Returning after 12 years, 1gb of RAM per tb of disk?

[allanonmage]

I've acquired some hardware, and downloaded the ISO and was about to start crimping fan wires when I decided to check RAM requirements for some reason. Woa, things have changes in the 12+ years since I last fiddled with TrueNAS (FreeNAS and NAS4Free at the time).

DDG seems to send me lots of requirement pages from many years ago (2011, 1026), but from what I can tell, the up to date recommendation for RAM is 1GB of RAM per 1TB of storage. That's a good rule, but does that ratio stay linear forever? Or does it top out at some point? The hardware recs page seems to be old, does this rule of thumb still apply?

For perspective, the last time I was playing with ZFS, I couldn't get it to use the 4GB I had in the system, even after lying to it that it had 8GB, so I repurposed that machine into a firewall, and just didn't have a NAS. Now I'm reading that it's memory hungry and will use all of the RAM. I guess things can change in 12 years, but that seems really strange.

I have 6x 14TB drives, and was thinking of a RAID6 style of setup where I could survive 2 drive failure. My intent is to build a home storage NAS to keep files accessible, and keep this machine to the task of storage. Usually 1 user, sometimes up to 4 max. It seems weird to me that all the NAS's nowadays have a full software suite to do all sorts of other things unrelated to storage, I would just spin up more hardware of VMs for different tasks. So this NAS will only do storage stuff, not other things. If I ever get around to Plex, that will be on a separate machine. Seeding Linux ISOs via torrents will also be on a seperate machine, though the storage might be on the NAS, I'm not sure. I could be flexible on this last point, if there isn't much overheard to that sort of thing.

As far as hardware, I have an AM4 motherboard with a Ryzen 5 CPU that has built in graphics, and an 8GB stick of RAM. The motherboard is a Gigabyte Aorus high end motherboard with built in Realtek 2.5GB network, 6 onboard SATA ports, and at some point I will use the SATA add-in card that I bought (it has 10 or 12 ports). It sounds like I need to add more RAM, but if I want to have a large ZFS array (~48TB usable), then it seems I will need a lot more RAM than I thought I would. The good thing is that the motherboard will support it. RAM isn't that big of a deal to updgrade, but I chose this CPU for it's built in graphics so I din't have to go find a discrete graphics card and waste a PICE slot. I see that encryption is available at the system/pool level, which sounds like an attractive feature, and the guide calls out a Xeon CPU for encryption. Except that's a 2012 era CPU listed and I literally donated such a system to Goodwill because it wasn't sellable in my area. I paid $120 for it 4 years ago and I think I paid a little too much. I suspect that the guide might be out of date for CPU recommendations then. I'm not opposed to a CPU upgrade, but I don't think there's a better CPU with built in graphics, so I'd have to re-engineer a few things. Video out is good for setting up systems and maintaining them without networking. pfSense offers a CLI via a serial port, and this motherboard has one, so I'd have to look into that if it's available.

 

[allanonmage]

I'll edit the poost when it goes live, but this is the case I'm using:

Rosewill RSV-L4500U 4U Server Chassis Rackmount Case | 15 3.5" HDD Bays | E-ATX Compatible | 6 Front 120mm Fans, 2 Rear 80mm Fans | USB 3.0, USB 2.0 | Front Panel Lock and Key | Silver/Black - Newegg.com

Buy Rosewill RSV-L4500U 4U Server Chassis Rackmount Case | 15 3.5" HDD Bays | E-ATX Compatible | 6 Front 120mm Fans, 2 Rear 80mm Fans | USB 3.0, USB 2.0 | Front Panel Lock and Key | Silver/Black with fast shipping and top-rated customer service. Once you know, you Newegg!

www.newegg.com

 

[Ericloewe]

That's a good rule, but does that ratio stay linear forever? Or does it top out at some point? The hardware recs page seems to be old, does this rule of thumb still apply?

It's a rule of thumb, mostly for smaller systems. It depends a lot on the specifics.

(FreeNAS and NAS4Free at the time).

FreeNAX 0.7, later called NAS4Free, now called XigmaNAS, has almost nothing to do with FreeNAS 8+/TrueNAS. In particular, FreeNAS 9.3 switched to only supporting ZFS for all storage. What seems to be happening is that you're comparing a legacy filesystem on a different codebase with ZFS, and yes, that would make for quite a big difference.

For perspective, the last time I was playing with ZFS, I couldn't get it to use the 4GB I had in the system, even after lying to it that it had 8GB, so I repurposed that machine into a firewall, and just didn't have a NAS. Now I'm reading that it's memory hungry and will use all of the RAM. I guess things can change in 12 years, but that seems really strange.

I won't speculate, as we're too far removed for it to be relevant, but I think it's safe to dismiss that previous experience you had.

It seems weird to me that all the NAS's nowadays have a full software suite to do all sorts of other things unrelated to storage,

It's not that I don't understand it at all, but I tend to agree with you.

The motherboard is a Gigabyte Aorus high end motherboard with built in Realtek 2.5GB network,

Realtek is quite the opposite of "high-end", and motherboard manufacturers had adopted that position before more recently jumping on the 2.5 GbE hype train, hanging off of Realtek for dear life. The motherboard market is in a miserable state these days.

I see that encryption is available at the system/pool level, which sounds like an attractive feature, and the guide calls out a Xeon CPU for encryption.

Be sure that you need it. Encryption means losing your data when things go wrong, and for many things that outweighs the benefits.
As for the CPU, the majority of modern CPUs support AES-NI. Xeon E5 v1s were early CPUs that supported AES-NI (not the earliest), and the Xeon E5 branding made it until the 2016 timeframe, so it's not quite as antiquated as strictly Xeon E5 v1. Even then, Xeon E5 v1/v2 are still okay and Xeon E5 v3/v4 can be pretty decent.

More broadly speaking, you should have a look at the Resources section, specifically the Hardware Recommendations Guide (yes, it's long in the tooth, but availability of the newer hardware is rough and prices astronomical) and the Introduction to ZFS.

 

[allanonmage]

In the desktop/gaming space, a Realtek 2.5GB NIC is superior to the troubled Intel NICs due to the Intel NICs causing crashes, system instability, etc. I had an ASUS motherboard for my main desktop with the Intel NIC and neither ASUS nor Intel are really doing anything to fix it, so I bought an addon card for it, and now it's a Minecraft server and 100% stable. You can't find much about it because of "reputation control" or whatever they call it (really it's censorship), but there's a few Reddit posts here and there, and I wanna say that LTT covered it at once point, but I may be mistaken.

Which really chafes since the only reason I opted for the Intel NIC motherboard was because of BSD's insistence that Intel NICs were the bar and everything else was trash.

The PDF in your link looks pretty good, so I'll finish reading it and then wee where I'm at.

 

[danb35]

It seems weird to me that all the NAS's nowadays have a full software suite to do all sorts of other things unrelated to storage

I think there are two major factors involved here:

• The assumption that for many users, the NAS will be the only server they have, and thus will want to run lots of other things on it; and
• The reality that many of those "things unrelated to storage" still interact with the storage, the most obvious example (to me, at least) being the media servers and the *arr suite. Sure, they can run on other systems (real or virtual), but there's some logic, IMO, to keeping those things local to the NAS. But of course there are lots of ways to skin that proverbial cat.

 

[Etorix]

An iGPU is essentially useless since, once set up, the NAS is administered remotely. Realtek NICs are known not to work reliably for TrueNAS, and SATA cards with port multipliers are a liability. Do check the hardware recommendations carefully.
As for RAM, the guidance is very rough (it is left undefined whether "storage" refers to the raw space or to total space, including parity, for instance…) and it does flatten out (at an equally undefined point), but you definitely need more than 8 GB and preferably more than the basic recommended 16 GB, though not necessarily up to 48-64 GB to strictly follow "1 GB pet TB". 32 GB should be fine—together with a LSI 9200/9300 HBA to replace the SATA card!

 

[NugentS]

I've acquired some hardware, and downloaded the ISO and was about to start crimping fan wires when I decided to check RAM requirements for some reason. Woa, things have changes in the 12+ years since I last fiddled with TrueNAS (FreeNAS and NAS4Free at the time).

DDG seems to send me lots of requirement pages from many years ago (2011, 1026), but from what I can tell, the up to date recommendation for RAM is 1GB of RAM per 1TB of storage. That's a good rule, but does that ratio stay linear forever? Or does it top out at some point? The hardware recs page seems to be old, does this rule of thumb still apply?

For perspective, the last time I was playing with ZFS, I couldn't get it to use the 4GB I had in the system, even after lying to it that it had 8GB, so I repurposed that machine into a firewall, and just didn't have a NAS. Now I'm reading that it's memory hungry and will use all of the RAM. I guess things can change in 12 years, but that seems really strange.

I have 6x 14TB drives, and was thinking of a RAID6 style of setup where I could survive 2 drive failure. My intent is to build a home storage NAS to keep files accessible, and keep this machine to the task of storage. Usually 1 user, sometimes up to 4 max. It seems weird to me that all the NAS's nowadays have a full software suite to do all sorts of other things unrelated to storage, I would just spin up more hardware of VMs for different tasks. So this NAS will only do storage stuff, not other things. If I ever get around to Plex, that will be on a separate machine. Seeding Linux ISOs via torrents will also be on a seperate machine, though the storage might be on the NAS, I'm not sure. I could be flexible on this last point, if there isn't much overheard to that sort of thing.

As far as hardware, I have an AM4 motherboard with a Ryzen 5 CPU that has built in graphics, and an 8GB stick of RAM. The motherboard is a Gigabyte Aorus high end motherboard with built in Realtek 2.5GB network, 6 onboard SATA ports, and at some point I will use the SATA add-in card that I bought (it has 10 or 12 ports). It sounds like I need to add more RAM, but if I want to have a large ZFS array (~48TB usable), then it seems I will need a lot more RAM than I thought I would. The good thing is that the motherboard will support it. RAM isn't that big of a deal to updgrade, but I chose this CPU for it's built in graphics so I din't have to go find a discrete graphics card and waste a PICE slot. I see that encryption is available at the system/pool level, which sounds like an attractive feature, and the guide calls out a Xeon CPU for encryption. Except that's a 2012 era CPU listed and I literally donated such a system to Goodwill because it wasn't sellable in my area. I paid $120 for it 4 years ago and I think I paid a little too much. I suspect that the guide might be out of date for CPU recommendations then. I'm not opposed to a CPU upgrade, but I don't think there's a better CPU with built in graphics, so I'd have to re-engineer a few things. Video out is good for setting up systems and maintaining them without networking. pfSense offers a CLI via a serial port, and this motherboard has one, so I'd have to look into that if it's available.

Lets pick apart your hardware (that you mention)
M/B - Gamer gear - not designed for 24*7 use - not to say it won't work. Contains lots of stuff you neither need or want - but you can mostly turn it off. 2.5Gb Realtek is shit. If you use Scale you milage will probably be better than Core. You can always add a proper NIC if your results are not good.

8GB RAM - is minimum for Core. Because of a RAM mangement issue in Linux I consider 16GB minimum for Scale. ZFS likes memory. A sensible RAM value for Scale I consider (YMMV) 32GB with a preference for 64. Core is 16 with a preference for 32. More is almost always better but if you are just serving files via SMB that should work well. ECC is better than non ECC if the board and CPU support it. For 6*14TB in Z2 - my memory suggestions should work well - even at the mimimum level of 8 (Core) and 16 (Scale).

CPU - it'll work. a NAS does not need much CPU - its all the addons (apps, VM's, Jails etc) that consume most CPU (and additional memory). Xeon is reccomended for just that reason - TN as a file server doesn't need that much. @Ericloewe has gone into the reasons - but TN does not need latest and greatest. Depends on your use case but take a look at the hardware specs of IX's smaller systems. Intel Atom which for just storage is easily enough.

Your SATA addin card - is this a PCIe X1 to lots of SATA ports "chinesium" special. In which case this is highly contra-indicated. It will likley work, for a bit and then not work at the cost of your data. An LSI HBA card is the way to expand SATA ports. There are some small non-name SATA addin cards with decent chipsets - but these tend to be PCIeX1 to at most 4 SATA ports (and hard to find). The reason(s):

1. The more ports you have the more likley they include a SATA Expander - this is a not good solution
2. PCIeX1 has very limited bandwidth - put many disks on one of these and you become bandwidth constrained. ZFS likes unrestricted access to disks and does not respond well to limitations
3. These are cheap for a reason - they tend to be crap

LSI HBA - for HDD's can be a 920x-8i (bit slow for many SSD's - you might want to use a 930x-8i) flashed with the latest IT firmware. Not a MegaRaid card although some of these can be lobotomized (I think). Not a RAID Card (Adaptec/3Ware etc). Look at "Art of Server" on ebay as a guide for whats out there. He is a bit more expensive than others - but he has a good rep and stands by his product. Cards come pre-flashed with correct firmware type (where relavent - he does sell other stuff). Warning LSI cards do need airflow - but thats a rackmount case - so there should be plenty of airflow

 

[Ericloewe]

In the desktop/gaming space, a Realtek 2.5GB NIC is superior to the troubled Intel NICs due to the Intel NICs causing crashes, system instability, etc.

2.5 GbE, I maintain, was always a low-effort, low-reward thing that only makes sense in very limited scenarios (the usual one quoted is high-end APs pushing >1 Gb/s). Since both Realtek (as they usually do) and Intel (through an oopsie) produced crap hardware, there is very, very little reason to even remotely consider 2.5 GbE on a serious machine (as opposed to narrow use-case embedded system).
The tl;dr is that if you want something faster than 1 Gb/s, go for 10 GbE or 25 GbE. It'll probably cost the same, anyway.

Which really chafes since the only reason I opted for the Intel NIC motherboard was because of BSD's insistence that Intel NICs were the bar and everything else was trash.

This is unequivocally true for FreeBSD and 1 GbE. For Linux, Broadcom has usable drivers (these days they've bumped up the price to gouge server OEMs, but there's plenty of junk cards left over from a decade of servers sold with dual- or quad-port Broadcom GbE NICs). For 10 GbE, Chelsio is a bit better overall, though Intel's hardware is said to be superior. So there is nuance to the statement, but it is true that it is hard to go wrong with Intel 1 GbE.

 

[NugentS]

The PDF in your link looks pretty good, so I'll finish reading it and then wee where I'm at.

With excitement?