Looking for build critique and recommendations.

[xLogisticsx]

Hello FreeNAS community!

I am looking to build myself a server grade FreeNAS unit for home use and to tinker with some business/enterprise things on the side. I have plans to move in to the IT field in a few years and would like to play around with servers in the mean time.

I would like to post my build here for critique and recommendations. I already have two servers, a Dell PowerEdge 2950 Gen III running pfsense and an HP ProLiant DL380 G6 with Xenserver, to do most anything else I would want to mess with, so the FreeNAS server would likely be only for storage for home and VMs. I plan to have it store video media, personal use files, game/twitch streaming source video, backups for all devices, and surveillance footage in the future. Initially, I am going to start with a relatively low amount of storage in one vdev, then both add more vdevs to the zpool and expand older existing physical drives using autoexpand.

I think that covers the basic questions, so here what I have picked so far.

Chassis: Supermicro CSE-836TQ-R800B
Motherboard: Supermicro MBD-X10SRH-CLN4F-O
CPU: Intel Xeon E5-1620 v3 BX80644E51620V3
RAM: 1x Samsung 32G EEC LRDIMM MEM-DR432L-SL01-LR21
HDD: 4x WD Red 2TB drives (ZRAID2) WD20EFRX
Boot Drive: Supermicro 16GB DOM SSD-DM016-SMCMVN1
SLOG Drive: Supermicro 128GB DOM SSD-DM128-SMCMVN1 (not certain if I will add or not)
CPU cooler: Supermicro CPU cooler SNK-P0048AP4


Thanks for your time!

 

[Stux]

Hello FreeNAS community!

I am looking to build myself a server grade FreeNAS unit for home use and to tinker with some business/enterprise things on the side. I have plans to move in to the IT field in a few years and would like to play around with servers in the mean time.

I would like to post my build here for critique and recommendations. I already have two servers, a Dell PowerEdge 2950 Gen III running pfsense and an HP ProLiant DL380 G6 with Xenserver, to do most anything else I would want to mess with, so the FreeNAS server would likely be only for storage for home and VMs. I plan to have it store video media, personal use files, game/twitch streaming source video, backups for all devices, and surveillance footage in the future. Initially, I am going to start with a relatively low amount of storage in one vdev, then both add more vdevs to the zpool and expand older existing physical drives using autoexpand.

I think that covers the basic questions, so here what I have picked so far.

Chassis: Supermicro CSE-836TQ-R800B
Motherboard: Supermicro MBD-X10SRH-CLN4F-O
CPU: Intel Xeon E5-1620 v3 BX80644E51620V3
RAM: 1x Samsung 32G EEC LRDIMM MEM-DR432L-SL01-LR21
HDD: 4x WD Red 2TB drives (ZRAID2) WD20EFRX
Boot Drive: Supermicro 16GB DOM SSD-DM016-SMCMVN1
SLOG Drive: Supermicro 128GB DOM SSD-DM128-SMCMVN1 (not certain if I will add or not)
CPU cooler: Supermicro CPU cooler SNK-P0048AP4


Thanks for your time!

LRDIMM memory is not compatible with E5-16xx series. (Intel market segmentation). You need RDIMM (or a 2600 series CPU).

Supermicro DOMs make a terrible SLOG drive. They're not rated for writing like that and they don't have PLP. You want to look at an intel write optimized drive. Ultimate performance is from a PCIe NVMe drive, Ie the Intel P3600 (P3700 is most likely overkill), or P750 (underkill depending on write load). Or a SATA version (S3600 etc), if price is more important than performance.

For VM storage multiple mirrored vdevs are recommended, of course, you can ignore this if optimal performance is not important, and instead reliability/storage efficiency are. But with just 4 disks you might as well go with a pair of mirrors.

Worth checking out the E5-1650, as it's a great CPU, but overkill for just a storage box, but if you wanted to run ESXi + FreeNAS in this box, then 6 cores at up to 4ghz is fairly special.

Re: the chassis: supermicro says:

Direct attached HDD backplane (TQ version), multilane backplane (A version) and expander backplanes (E1C/E2C/E16/E26 versions) are available for any specific applications.

You don't want 'TQ'. 20 Sata cables is a mess. 'A' will reduce that to just 5 mini sas cables (4 lanes per cable). And the expander backplanes reduce that to 1 or 2!

 

[xLogisticsx]

LRDIMM memory is not compatible with E5-16xx series. (Intel market segmentation). You need RDIMM (or a 2600 series CPU).

Supermicro DOMs make a terrible SLOG drive. They're not rated for writing like that and they don't have PLP. You want to look at an intel write optimized drive. Ultimate performance is from a PCIe NVMe drive, Ie the Intel P3600 (P3700 is most likely overkill), or P750 (underkill depending on write load. Or a SATA version (S3600 etc), if price is more important than performance.

For VM storage multiple mirrored vdevs are recommended, of course, you can ignore thus if optimal performance is not important, and instead reliability/storage efficiency are. But with just 4 disks you might as well go with a pair of mirrors.

Worth checking out the E5-1650, as it's a great CPU, but overkill for just a storage box, but if you wanted to run ESXi + FreeNAS in this box, then 6 cores at up to 4ghz is fairly special.

Thanks for the tips. Totally didn't think to check for LRDIMM compatibility since I've never needed to with my personal builds. I'll likely just go with RDIMM since I will never need the 512GB of LRDIMM the MOBO can support. Though, I will look more in to my options overall.

For the SLOG drive, I don't even know if I will add it or not. While I do want to play around with as many things as I can, I also want to make sure that I can even utilize each thing I put in as well. That's why I strayed away from an L2ARC drive, since more RAM is better in most all cases. If I do add the SLOG drive though, I'll likely just get the 2x 2.5" drive expander for the chassis and put an SSD in there. Not sure if I want to go for an NVMe drive.

As far as the VM storage, it will really depend on the VM. The HP ProLiant already has 16 73GB 10k SAS drives in them that I have been thinking about replacing with SSDs for VMs that I want improved performance on. Other VMs, like the RTMP server, will run off of the ProLiant, but store their files on the FreeNAS.

 

[Stux]

32GB RDIMMs are the way to go with a 16xx series CPU. For a maximum eventual 256MB limit.

If you do use NFS/iSCSI for ESXi you will want a slog. Since all ESXi NFS writes are sync writes.

I did some basic tests here:
https://forums.freenas.org/index.php?threads/testing-the-benefits-of-slog-using-a-ram-disk.56561/

To a six drive Raidz2 with NFS, I was getting 5MB/s. With slog it's 100-400MB/s.

Of course, you could just disable sync.

You only really need mirrored slog if you absolutely can't lose data if a slog fails during a sudden shutdown event.

 

[farmerpling2]

My 2 cents...

• CPU: Intel Xeon E5-1620 v3 BX80644E51620V3 <-- See comment below
  
• RAM: <-- Get 2 or 4 16GB pieces to get best performance from memory
  
• HDD: 4x WD Red 2TB drives (ZRAID2) WD20EFRX <- Get 5 drives to get more usable area.
  
• Boot Drive: <-- Get >124GB SSD - cheap and easy to replace
  
• SLOG: <-- Unlikely you will need it. If you will be playing then do not get it. Just modify NFS to not be SYNC.

If you will be running many VM's get a CPU with more cores 6 or 8. Hyperthreading does not count, IMHO - +40% is not a core.

 

[Chris Moore]

Hello FreeNAS community!

I am looking to build myself a server grade FreeNAS unit for home use and to tinker with some business/enterprise things on the side. I have plans to move in to the IT field in a few years and would like to play around with servers in the mean time.

I would like to post my build here for critique and recommendations. I already have two servers, a Dell PowerEdge 2950 Gen III running pfsense and an HP ProLiant DL380 G6 with Xenserver, to do most anything else I would want to mess with, so the FreeNAS server would likely be only for storage for home and VMs. I plan to have it store video media, personal use files, game/twitch streaming source video, backups for all devices, and surveillance footage in the future. Initially, I am going to start with a relatively low amount of storage in one vdev, then both add more vdevs to the zpool and expand older existing physical drives using autoexpand.

I think that covers the basic questions, so here what I have picked so far.

Chassis: Supermicro CSE-836TQ-R800B
Motherboard: Supermicro MBD-X10SRH-CLN4F-O
CPU: Intel Xeon E5-1620 v3 BX80644E51620V3
RAM: 1x Samsung 32G EEC LRDIMM MEM-DR432L-SL01-LR21
HDD: 4x WD Red 2TB drives (ZRAID2) WD20EFRX
Boot Drive: Supermicro 16GB DOM SSD-DM016-SMCMVN1
SLOG Drive: Supermicro 128GB DOM SSD-DM128-SMCMVN1 (not certain if I will add or not)
CPU cooler: Supermicro CPU cooler SNK-P0048AP4

Thanks for your time!

Others have given you some good advice, Here are the things I think might be useful to you.

This would be a better chassis for you to use: http://www.ebay.com/itm/SuperMicro-...4xCaddy-SAS2-6Gbp-Expander-2xPSU/162597691498
It gives you more drive bays to work with and all of them can run from one SAS controller due to the integrated SAS expander in the backplane.
This would be an excelent match to the backplane that is in the chassis above: http://www.ebay.com/itm/New-IT-Mode...port-PCI-E-6Gb-s-Controller-Card/182444920218

I am not suggesting this system board, but it might not be bad if you put a couple of the compatible 6 core processors, it would give you 12 real cores and 24 with hyperthreading. A newer board can easily be swapped into this chassis and it would be more power efficient. Something to think about

In this chassis, you could use some 1GB drives in a pool or mirrors (two or three mirrors are much better due to IOPS) to run your VMs and you would still have room to run a pool at RAIDz2 for storage. The power supplies in this unit are also very good, but you might need to change the fans for quieter models.
I bought one like this and I plan to use it to rebuild one of my current systems when time permits.

On the hard drives, we have several systems where I work that have the WD Red Pro drives and I can't say they are any better than the Seagate drives I use at home. I definitely don't care for the diagnostic data they give. I feel like I am running with a blindfold on. The Seagate Barracuda drives I use at home give great diagnostic data. I have been running them at home for over 5 years and although I have had a few come up with bad sectors and need to be replaced, they are enough cheaper that you could buy spares and have money left over. Here is a link: https://www.newegg.com/Product/Product.aspx?Item=N82E16822178993
The WD Red drives at the same size are about $20 more per drive.

 

[xLogisticsx]

So I originally wanted to spend the money to build a server from scratch so I could have the "kid on Christmas" feel, but I've decided that 16 drive bays just won't cut it for me and any new Supermicro case with 24+ bays is more than I want to pay. So I am going to go the used route, like the two other servers I have, since those have not failed me.

With that said, all of the used ones I have found seem like major overkill for what I have planned for the server. All I want to use it for is storage. I don't have any plans to run any VMs on it. All of VMs are running off of another server and I plan to just have them utilize the FreeNAS server for things that I don't want to store on the VM's drive(s). All of them have dual CPUs and range from 4 cores to 8 cores each with varying clock speeds. If dual CPUs is all I can find, what amount of pCores and what clock speed would fit for my use case without being overkill?

 

[Ericloewe]

The more popular option is to buy an obsolete Supermicro system (say, X7), junk everything but the chassis (and maybe the fans, if they're in good shape), clean the thing, replace the backplane if needed and then add the desired components. I didn't mention PSUs, but you'd definitely want to replace them with modern ones if the thing is going to be outside of a datacenter, due to the obscene noise levels.

 

[Chris Moore]

This would be a better chassis for you to use: http://www.ebay.com/itm/SuperMicro-...4xCaddy-SAS2-6Gbp-Expander-2xPSU/162597691498
It gives you more drive bays to work with and all of them can run from one SAS controller due to the integrated SAS expander in the backplane.

The one I pointed you at above is an great option. You can always junk that system board and the backplane is great, power supplies should be too. I would buy two of those myself if I had the cash ...

 

[farmerpling2]

Others have given you some good advice, Here are the things I think might be useful to you.

This would be a better chassis for you to use: http://www.ebay.com/itm/SuperMicro-...4xCaddy-SAS2-6Gbp-Expander-2xPSU/162597691498
It gives you more drive bays to work with and all of them can run from one SAS controller due to the integrated SAS expander in the backplane.
This would be an excelent match to the backplane that is in the chassis above: http://www.ebay.com/itm/New-IT-Mode...port-PCI-E-6Gb-s-Controller-Card/182444920218
...

I think we should have anyone who wants to get a NAS system setup $10 to Chris and let him tell them what to get! ;)

His thoughts are well thought out and gets you a great bang for your buck!

I am impressed that he took the time to help someone and gather the data from ebay.