jgreco
Resident Grinch
- Joined
- May 29, 2011
- Messages
- 18,680
Mod note: This thread is jgreco's original hardware recommendations guide, from back in the day. General advice still holds, but some specifics have changed.
The current Hardware Recommendations Guide can be found in the Resources section, at the following link:
https://forums.freenas.org/index.php?resources/hardware-recommendations-guide.12/
I keep seeing people come in asking about system builds and making what appear to be irrational choices.
The following is some guidance on how to select high performance hardware suitable for heavy-duty home or small office use. The goal isn't necessarily to pick the smallest, cheapest things that can possibly be used to get the job done, but rather how to throw a reasonable amount of money at the problem and get a solution that won't have to be totally replaced in a year because you underestimated everything. ZFS is piggy! But vendor-based NAS devices can be slow and are usually expensive. This is probably cheaper and faster. If you do want a vendor-based NAS device, see our friends at iXsystems, developers of FreeNAS who make the commercial TrueNAS product.
By the way, all these clicky links? Want more info? USE THEM. The intention is that you can use this to research stuff yourself...
ZFS is designed to protect your data, but cannot save you from poor hardware choices. ZFS works great with ECC (see cyberjock's sticky about why you really want ECC). ECC is not horribly expensive. ECC is well-supported with a Xeon CPU and a server-grade motherboard. ECC may not be well-supported on random prosumer grade boards and Pentium CPU's. If you are going to spend money on a system that's holding important data, get the right tool for the job. If your system is only dealing with backups, okay, fine, then go non-ECC if you really must.
Chassis:
For a smaller FreeNAS system, Fractal Design makes a bunch of products that I've never seen but many users rave about. As such, this isn't a suggestion so much as a recommendation that you check out the forum to see what people have said. It seems to me that the Node 304 could be ideal for a small 6-drive FreeNAS system. The Define R4 is attractive for up to 10 drives.
However, I'm simply not a fan of a chassis where you need to open it up to swap drives. The convenience of accessible drive trays is fairly large to me. That usually moves us up into rack-mount land, and most of these come with integrated boards.
ASUS makes the RS300-E7/PS4 which is an interesting 1U socket 1155 unit for around $550. Haven't played with one but heard it works well.
Supermicro has lots of options.
The SuperServer 5017C-URF is pricey at around $850, but offers redundant platinum-certified power supplies with the Intel C216 chipset. The SuperServer 5017C-MTF is probably the definitive prebuilt 4-drive, 1U small FreeNAS server based on C202, for around $525. The SC813MTQ-350CB chassis is available as a separate part if you prefer a different board. Note that Supermicro's chassis assume a Supermicro motherboard; ATX manufacturers have a variety of different placement strategies for their ports. Supermicro also offers a ton of 1U socket 2011 options if you need to go north of 32GB of RAM, redundant supplies, 10GbT, up to 768GB RAM, etc.
Supermicro falters a bit in the 2U-12 drive department. They don't actually have any single socket 1155 or 2011 as prebuilts, which would seem to be shortsighted as a NAS could reasonably need only a single CPU. But if you needed dual, then the 6027R-E1R12T or 6027-E1R12N would seem to be great choices. But you would probably be better off taking the CSE-826BE16-R920LPB chassis and integrating the board of your choice. Note that the "BE16" variant uses a backplane with an SFF-8087 for easy cabling and rapid assembly.
A lot of users have used Norco cases. These allow you to pick your own components a bit more easily, but by the time you end up swapping out the fan tray for the 120mm one, and you put in a decent power supply, and you deal with the headache of a bad backplane, pricewise you're within striking range of a Supermicro chassis, and the build quality of the Norco isn't as good.
Mainboards:
The cheap consumer boards and even many of the prosumer boards are ... well, "designed for Microsoft." Yes, your nifty prosumer board might sport USB3 and 8 SATA ports, but it also has a Realtek ethernet that really (makes you want to) scream, and capacitors that might have been bought in a back alley in Taiwan, and it wasn't truly designed to run 24/7. You do not want to overclock your system. That just adds stresses and increases the chance of failure. PCIe x16 is a waste (and might only support a video card) and you will have a hard time fitting an expansion HBA into a PCIe x1. Both are indicators to "look elsewhere."
So instead of that ECS Golden Z77H2-AX(1.0) LGA 1155 Intel Z77 for $319, or that ASUS Maximum VI Extreme for $400, both of which would admittedly make for an awesome PC, look at some server-grade boards. You're making a server. Why use a gaming enthusiast's hardware? We've been having great luck with Supermicro's X9S lineup of 1155 boards, but Tyan, HP, IBM, Dell, and others make usable hardware too. Unsure if it's a server board? Good server boards typically lack audio ports.
The Supermicro X9SCM ($154-$170) has 4 SATA2 and 2 SATA3 ports, dual Intel gigabit ethernets, a bunch of USB2 ports, and can be put in a nice 1U unit if you wish. It'll support ECC error correction and log ECC events.
The Supermicro X9SCA ($170) has both PCI and PCIe slots for more flexibility with any legacy stuff you might have.
The Supermicro boards have variations with IPMI, which means that you can access the console video and keyboard remotely, attach USB devices remotely, and basically never have to plug anything except power and network into the unit. They're also intended to run 24/7, and the Supermicro people who provide support know that a "server" isn't the waiter in a restaurant. Bonus: you get (at least) dual Intel server ethernets, instead of learning later that your Realtek is garbage and needing to fork out $35 for an Intel add-on desktop ethernet card. That effectively reduces the cost of your board by $35 (or even $70 if you count the second interface).
CPU's:
The Xeon E3-1230v2 is around $220. FOUR cores. Hyperthreading. 3.3GHz, 3.7GHz turbo for Samba zippiness. 8MB of cache. 69 watt TDP, and much lower consumption when idle. Up to 32GB ECC 1600MHz. AES-NI support for disk encryption fun. Even Intel VT-d in case you decide to play with virtualization.
Okay, so real low power? The Xeon E3-1220Lv2 is around $180. TWO cores, 2.3GHz with 3.5GHz turbo. 3MB of cache, 17 watt TDP. I suspect the 1230v2 idles at around the same level and may be a better choice due to its ability to bring lots of cycles to bear if needed, but the 1220Lv2 is an interesting option.
So, still too expensive? If cost is the all-controlling factor for your build, then at least pick a CPU with ECC support, because ZFS is designed assuming that the system isn't likely to randomly corrupt bits. I've heard good reports about the Intel Pentium G2020, around $60. Two cores, 2.9GHz, no turbo. 55 watt TDP. But you lose on AES-NI and other features that could be helpful. And I haven't actually tried one of these on a Supermicro board.
Memory:
Please don't guess at compatible memory for your system. Use a manufacturer's memory selection tool. This is your data that you're handling, after all. Do you really want to risk it?
Buy ECC. ZFS is designed with ECC in mind as part of the protection strategy.
Make sure you don't buy Registered memory if your system requires Unbuffered. Patrick Kennedy explains it all with pictures.
Don't buy low density and fill all your slots unless that is the only option available cost-wise. For example, don't buy 4GB sticks for socket 1155 systems with 4 slots. A 4GB KVR16E11/4 is $35. An 8GB KVR16E11/8I is $70. Given that pricing, there is NO REASON to buy 4GB modules. If you populate all your slots unnecessarily, then you lose out if you later decide you need more RAM. It is okay to buy two 8GB modules and leave two slots open.
ZFS loves memory. If you are shocked at the idea of a NAS needing 8GB of memory, get over it. Buy 16GB at a minimum. It's 2013. Memory is cheap. FreeNAS and ZFS requires a minimum of 8GB. You are not special. You are not the exception to that. You must outfit your box with at least 8GB. But seriously, you are strongly encouraged to go with 16 or more.
Power supply:
Don't buy a cheap power supply. It's your data, don't let a crummy power supply eat your system and take your data with it.
Do consider that a 24/7 supply that is less efficient costs a lot of money in the long run. Do yourself a favor and buy an 80Plus Gold rated supply, at least. You probably will never recover the cost differential of an 80Plus Platinum supply though.
Sizing a power supply is kind of a multidimensional problem. Remember that drive spin-up takes a LOT of current.
1) In terms of watts, a supply that is running at about 30-50% of its rated load is likely to be within its peak efficiency window, but is also likely to be relatively unstressed, leading to a longer life and reduced likelihood of premature failure.
2) You also need to estimate (by adding up the start currents for all the drives, usually around 2 amps each, and then estimating other 12V loads such as fans, CPU's, etc.) the 12V load, and verify that that will remain under the rated capacity of the power supply. You can CHEAT at this by hooking up the system without powering on the drives, letting it run memtest86, and checking power consumption. Take the watts, divide by 12, and that's a bad overestimate of the maximum 12V amps that the base system takes. Then you add the peak amps for all the drives, maybe add 10% more, check against the power supply rating, and there you have an easily derived pass-or-fail.
But you need BOTH these tests to pass and be reasonable, or you need to adjust your gameplan (mandatory staggered spin, etc). And for a NAS or other server with lots of 3.5" disks, this typically means selecting a power supply that runs at something closer to 30% of its rating, in my experience.
Additional SATA connectivity:
The IBM ServeRAID M1015, crossflashed to IT mode, is probably one of the best choices available if you need a bunch of extra SATA ports. It does take an extra ~12 watts however. Don't buy it new. They're available cheap on eBay all the time.
Random RAID controllers that are operating in RAID mode (showing virtual or logical devices to FreeNAS) are a very bad idea. If your controller costs more than a few hundred dollars, it may not be a good choice for FreeNAS.
That Adaptec RAID controller? Probably won't work. Neither will that SATA PCIe controller that you got in the 99c bin at the corner computer shop. Well, it might "work" but be cautious, it may like to drop bits. In general, ZFS works great if it has direct communication with a disk. It does not need to be fancy or expensive communication, but it does need to be reliable.
Do not obsess over SATA3. You do not need SATA3 ports, except to get the most out of a SSD. The average hard drive these days can read at about 150MBytes/sec. 3Gbit/s SATA2 works out to something like 375MBytes/sec.
Avoid the LSI SAS/SATA 3Gbps HBA/RAID controllers. They have a 2TB size limit. Ick.
Boot device:
Use a USB flash drive that you trust. I know the FreeNAS specs say 2GB. Most of the ones that claim to be 2GB are a bit less. Get three nice name-brand 4GB USB flash drives, not out of the 99c bin. SSD is totally awesome for a boot device, much faster, but eats a SATA port, is quite expensive (in comparison), and isn't necessarily any more reliable. Three USB flash drives means one for current use, one to install an upgrade, and one for backup/oops.
Hard drives:
Of course, you want to use cheap hard drives. But consider that you're building a system for hundreds of dollars. Add up the total cost of the system you propose with those "cheap" 2TB or 3TB drives, then divide by the number of usable TB you get. Then add up the total cost of a system built with 4TB drives, divide again. Shocked? The 4TB is often the less expensive choice per delivered TB, despite the drives being a bit of a price premium.
The current Hardware Recommendations Guide can be found in the Resources section, at the following link:
https://forums.freenas.org/index.php?resources/hardware-recommendations-guide.12/
I keep seeing people come in asking about system builds and making what appear to be irrational choices.
The following is some guidance on how to select high performance hardware suitable for heavy-duty home or small office use. The goal isn't necessarily to pick the smallest, cheapest things that can possibly be used to get the job done, but rather how to throw a reasonable amount of money at the problem and get a solution that won't have to be totally replaced in a year because you underestimated everything. ZFS is piggy! But vendor-based NAS devices can be slow and are usually expensive. This is probably cheaper and faster. If you do want a vendor-based NAS device, see our friends at iXsystems, developers of FreeNAS who make the commercial TrueNAS product.
By the way, all these clicky links? Want more info? USE THEM. The intention is that you can use this to research stuff yourself...
ZFS is designed to protect your data, but cannot save you from poor hardware choices. ZFS works great with ECC (see cyberjock's sticky about why you really want ECC). ECC is not horribly expensive. ECC is well-supported with a Xeon CPU and a server-grade motherboard. ECC may not be well-supported on random prosumer grade boards and Pentium CPU's. If you are going to spend money on a system that's holding important data, get the right tool for the job. If your system is only dealing with backups, okay, fine, then go non-ECC if you really must.
Chassis:
For a smaller FreeNAS system, Fractal Design makes a bunch of products that I've never seen but many users rave about. As such, this isn't a suggestion so much as a recommendation that you check out the forum to see what people have said. It seems to me that the Node 304 could be ideal for a small 6-drive FreeNAS system. The Define R4 is attractive for up to 10 drives.
However, I'm simply not a fan of a chassis where you need to open it up to swap drives. The convenience of accessible drive trays is fairly large to me. That usually moves us up into rack-mount land, and most of these come with integrated boards.
ASUS makes the RS300-E7/PS4 which is an interesting 1U socket 1155 unit for around $550. Haven't played with one but heard it works well.
Supermicro has lots of options.
The SuperServer 5017C-URF is pricey at around $850, but offers redundant platinum-certified power supplies with the Intel C216 chipset. The SuperServer 5017C-MTF is probably the definitive prebuilt 4-drive, 1U small FreeNAS server based on C202, for around $525. The SC813MTQ-350CB chassis is available as a separate part if you prefer a different board. Note that Supermicro's chassis assume a Supermicro motherboard; ATX manufacturers have a variety of different placement strategies for their ports. Supermicro also offers a ton of 1U socket 2011 options if you need to go north of 32GB of RAM, redundant supplies, 10GbT, up to 768GB RAM, etc.
Supermicro falters a bit in the 2U-12 drive department. They don't actually have any single socket 1155 or 2011 as prebuilts, which would seem to be shortsighted as a NAS could reasonably need only a single CPU. But if you needed dual, then the 6027R-E1R12T or 6027-E1R12N would seem to be great choices. But you would probably be better off taking the CSE-826BE16-R920LPB chassis and integrating the board of your choice. Note that the "BE16" variant uses a backplane with an SFF-8087 for easy cabling and rapid assembly.
A lot of users have used Norco cases. These allow you to pick your own components a bit more easily, but by the time you end up swapping out the fan tray for the 120mm one, and you put in a decent power supply, and you deal with the headache of a bad backplane, pricewise you're within striking range of a Supermicro chassis, and the build quality of the Norco isn't as good.
Mainboards:
The cheap consumer boards and even many of the prosumer boards are ... well, "designed for Microsoft." Yes, your nifty prosumer board might sport USB3 and 8 SATA ports, but it also has a Realtek ethernet that really (makes you want to) scream, and capacitors that might have been bought in a back alley in Taiwan, and it wasn't truly designed to run 24/7. You do not want to overclock your system. That just adds stresses and increases the chance of failure. PCIe x16 is a waste (and might only support a video card) and you will have a hard time fitting an expansion HBA into a PCIe x1. Both are indicators to "look elsewhere."
So instead of that ECS Golden Z77H2-AX(1.0) LGA 1155 Intel Z77 for $319, or that ASUS Maximum VI Extreme for $400, both of which would admittedly make for an awesome PC, look at some server-grade boards. You're making a server. Why use a gaming enthusiast's hardware? We've been having great luck with Supermicro's X9S lineup of 1155 boards, but Tyan, HP, IBM, Dell, and others make usable hardware too. Unsure if it's a server board? Good server boards typically lack audio ports.
The Supermicro X9SCM ($154-$170) has 4 SATA2 and 2 SATA3 ports, dual Intel gigabit ethernets, a bunch of USB2 ports, and can be put in a nice 1U unit if you wish. It'll support ECC error correction and log ECC events.
The Supermicro X9SCA ($170) has both PCI and PCIe slots for more flexibility with any legacy stuff you might have.
The Supermicro boards have variations with IPMI, which means that you can access the console video and keyboard remotely, attach USB devices remotely, and basically never have to plug anything except power and network into the unit. They're also intended to run 24/7, and the Supermicro people who provide support know that a "server" isn't the waiter in a restaurant. Bonus: you get (at least) dual Intel server ethernets, instead of learning later that your Realtek is garbage and needing to fork out $35 for an Intel add-on desktop ethernet card. That effectively reduces the cost of your board by $35 (or even $70 if you count the second interface).
CPU's:
The Xeon E3-1230v2 is around $220. FOUR cores. Hyperthreading. 3.3GHz, 3.7GHz turbo for Samba zippiness. 8MB of cache. 69 watt TDP, and much lower consumption when idle. Up to 32GB ECC 1600MHz. AES-NI support for disk encryption fun. Even Intel VT-d in case you decide to play with virtualization.
Okay, so real low power? The Xeon E3-1220Lv2 is around $180. TWO cores, 2.3GHz with 3.5GHz turbo. 3MB of cache, 17 watt TDP. I suspect the 1230v2 idles at around the same level and may be a better choice due to its ability to bring lots of cycles to bear if needed, but the 1220Lv2 is an interesting option.
So, still too expensive? If cost is the all-controlling factor for your build, then at least pick a CPU with ECC support, because ZFS is designed assuming that the system isn't likely to randomly corrupt bits. I've heard good reports about the Intel Pentium G2020, around $60. Two cores, 2.9GHz, no turbo. 55 watt TDP. But you lose on AES-NI and other features that could be helpful. And I haven't actually tried one of these on a Supermicro board.
Memory:
Please don't guess at compatible memory for your system. Use a manufacturer's memory selection tool. This is your data that you're handling, after all. Do you really want to risk it?
Buy ECC. ZFS is designed with ECC in mind as part of the protection strategy.
Make sure you don't buy Registered memory if your system requires Unbuffered. Patrick Kennedy explains it all with pictures.
Don't buy low density and fill all your slots unless that is the only option available cost-wise. For example, don't buy 4GB sticks for socket 1155 systems with 4 slots. A 4GB KVR16E11/4 is $35. An 8GB KVR16E11/8I is $70. Given that pricing, there is NO REASON to buy 4GB modules. If you populate all your slots unnecessarily, then you lose out if you later decide you need more RAM. It is okay to buy two 8GB modules and leave two slots open.
ZFS loves memory. If you are shocked at the idea of a NAS needing 8GB of memory, get over it. Buy 16GB at a minimum. It's 2013. Memory is cheap. FreeNAS and ZFS requires a minimum of 8GB. You are not special. You are not the exception to that. You must outfit your box with at least 8GB. But seriously, you are strongly encouraged to go with 16 or more.
Power supply:
Don't buy a cheap power supply. It's your data, don't let a crummy power supply eat your system and take your data with it.
Do consider that a 24/7 supply that is less efficient costs a lot of money in the long run. Do yourself a favor and buy an 80Plus Gold rated supply, at least. You probably will never recover the cost differential of an 80Plus Platinum supply though.
Sizing a power supply is kind of a multidimensional problem. Remember that drive spin-up takes a LOT of current.
1) In terms of watts, a supply that is running at about 30-50% of its rated load is likely to be within its peak efficiency window, but is also likely to be relatively unstressed, leading to a longer life and reduced likelihood of premature failure.
2) You also need to estimate (by adding up the start currents for all the drives, usually around 2 amps each, and then estimating other 12V loads such as fans, CPU's, etc.) the 12V load, and verify that that will remain under the rated capacity of the power supply. You can CHEAT at this by hooking up the system without powering on the drives, letting it run memtest86, and checking power consumption. Take the watts, divide by 12, and that's a bad overestimate of the maximum 12V amps that the base system takes. Then you add the peak amps for all the drives, maybe add 10% more, check against the power supply rating, and there you have an easily derived pass-or-fail.
But you need BOTH these tests to pass and be reasonable, or you need to adjust your gameplan (mandatory staggered spin, etc). And for a NAS or other server with lots of 3.5" disks, this typically means selecting a power supply that runs at something closer to 30% of its rating, in my experience.
Additional SATA connectivity:
The IBM ServeRAID M1015, crossflashed to IT mode, is probably one of the best choices available if you need a bunch of extra SATA ports. It does take an extra ~12 watts however. Don't buy it new. They're available cheap on eBay all the time.
Random RAID controllers that are operating in RAID mode (showing virtual or logical devices to FreeNAS) are a very bad idea. If your controller costs more than a few hundred dollars, it may not be a good choice for FreeNAS.
That Adaptec RAID controller? Probably won't work. Neither will that SATA PCIe controller that you got in the 99c bin at the corner computer shop. Well, it might "work" but be cautious, it may like to drop bits. In general, ZFS works great if it has direct communication with a disk. It does not need to be fancy or expensive communication, but it does need to be reliable.
Do not obsess over SATA3. You do not need SATA3 ports, except to get the most out of a SSD. The average hard drive these days can read at about 150MBytes/sec. 3Gbit/s SATA2 works out to something like 375MBytes/sec.
Avoid the LSI SAS/SATA 3Gbps HBA/RAID controllers. They have a 2TB size limit. Ick.
Boot device:
Use a USB flash drive that you trust. I know the FreeNAS specs say 2GB. Most of the ones that claim to be 2GB are a bit less. Get three nice name-brand 4GB USB flash drives, not out of the 99c bin. SSD is totally awesome for a boot device, much faster, but eats a SATA port, is quite expensive (in comparison), and isn't necessarily any more reliable. Three USB flash drives means one for current use, one to install an upgrade, and one for backup/oops.
Hard drives:
Of course, you want to use cheap hard drives. But consider that you're building a system for hundreds of dollars. Add up the total cost of the system you propose with those "cheap" 2TB or 3TB drives, then divide by the number of usable TB you get. Then add up the total cost of a system built with 4TB drives, divide again. Shocked? The 4TB is often the less expensive choice per delivered TB, despite the drives being a bit of a price premium.
Last edited by a moderator: