6 bay server

[lowerCamelCase]

Hello there!

I am building a FreeNAS machine right now. It will be used as data directory for 2 clients, shared data folder in 5 clients and as docker registry. In addition there will be a few clients saving some data from time to time, but those are pretty much irrelevant. The relevant data are a mixture of small files (<64kB) and medium files (<10mB), there are also a few multi gb files where I am mainly interested in fast reads. I want to build a raidz2 system.
All clients are Linux based and will propably use nfs. Generally speaking it is a home office environment with a bit more compute power than usual, but it will also serve to save a few private files, especially the picture database of my wife (in total about 2tb pictures in raw and jpg format).
I already own everything I need (not bought for this project but used for it now), but can go two ways from here.
Parts I plan to use:

• 2*128gb SSD (MDADM Raid 1 as boot device) (not preused)
• 6*4tb HGST Ultrastar 7k6000 plus 1 spare (nearly not preused)
• 2*40gbit qdr Mellanox Infiniband PCIe Cards (all relevant clients are connected via qdr inf) (preused)

Now I can go two(three) ways:

• dual Xeon X5650 with 12*8gb=96gb RAM (ddr3 reg ecc) (also possible to use 12*4gb=48gb) in combination with dell h310 raid controller (only 6 sata connectors on this generation of mainboards)
• dual Xeon E5-2637 v3 with 8*16gb=128gb RAM (ddr4 reg ecc) using on board sata connectors

For connectivity there is the qdr infiniband link to the most relevant systems, a gbit uplink to another backup layer (only used once a month), a gbit ethernet for general service, ipmi port and finally a wlan usb dongle (I might add my wifes smartphone to save some pictures from it via wifi, but this has very low priority).
I would actually prefer to use the older X5650 system, as I have other useful usage for the v3 system, not so much for the x5650. On the other hand I plan to use lz4 compression and I am not quite sure if the old Xeons can handle this task well enough.
What are your thoughts about this? Anything else I might have misdimensioned?
I have experience with bigger storage systems, contributed to beegfs, planed and used storage clusters in beegfs, ceph and lustre and have generally high knowledge about Linux systems. I have not used BSD in a while though. What I dont have experience with is small single noded storage systems as this one. I decided to go with FreeNAS as it seems to deliver good performance and I do not want to burn too much time on this project.

 

[Chris Moore]

Why do you feel like you need a dual socket system?

 

[lowerCamelCase]

For memory reasons mainly (using more modules per channel decreases the performance, in some workloads very significantly. Running in memory databeses I had 30% performance decrease going from 4 to 8 modules per socket on intel c612. Of course for the average usage its more like 5 to 10 percent decrease) . Also I already own these dual socket system and dont plan to invest into new mainboards of that older generations.

 

[Chris Moore]

I do not want to burn too much time on this project

There is a fair learning-curve to FreeNAS, because it is an appliance built on FreeBSD and some functionality has been intentionally restricted. That requires some time to figure out what you can and can't do and what can safely be worked around.
I would suggest going with FreeNAS version 11.1-U7 right now because they did a full UI redesign in 11.2 and most of the guides are still based around the old UI, so it might help you get up and running faster by using the previous UI and there are still a few bugs they are working to straighten out in the new UI. Here is some material to review to help familiarize you with ZFS, as ZFS is the ONLY file system used with FreeNAS including to mirror the boot pool (not MDADM) and to create the storage pool, so any SAS controller needs to be converted to HBA mode. That H310 (Dell PERC H310 I am guessing) can be crossflashed with the standard LSI IT mode firmware and we have a guide for how to do that on the forum if you need it.

The Xeon X5650 should be fine for FreeNAS. It has plenty of memory and FreeNAS is perfectly happy to use a SAS controller to talk to SATA drives, so no need to worry about the SATA ports. What model computer system is that?

2*40gbit qdr Mellanox Infiniband PCIe Cards (all relevant clients are connected via qdr inf) (preused)

I don't know if anyone has successfully used these with FreeNAS. You might have trouble with that..

Here are links to the reading material I mentioned:

Slideshow explaining VDev, zpool, ZIL and L2ARC
https://forums.freenas.org/index.ph...ning-vdev-zpool-zil-and-l2arc-for-noobs.7775/

Terminology and Abbreviations Primer
https://forums.freenas.org/index.php?threads/terminology-and-abbreviations-primer.28174/

Building, Burn-In, and Testing your FreeNAS system
https://forums.freenas.org/index.php?resources/building-burn-in-and-testing-your-freenas-system.38/

Github repository for FreeNAS scripts, including disk burnin
https://forums.freenas.org/index.ph...for-freenas-scripts-including-disk-burnin.28/

Detailed newcomers' guide to crossflashing LSI 9211 HBA and variants
https://forums.freenas.org/index.ph...o-crossflashing-lsi-9211-hba-and-variants.54/

Uncle Fester's Basic FreeNAS Configuration Guide
https://www.familybrown.org/dokuwiki/doku.php?id=fester:intro

 

[Chris Moore]

PS. Many folks that have a background in Linux or Unix start trying to change things from the command line because that is how they have done it elsewhere. It can lead to pain with FreeNAS because it rebuilds the system config at every boot, based on what is saved in the configuration database, so many changes made directly to config files will be lost when the system is rebooted. Changes need to be made through the GUI. The only real exception to that is changes made to the storage pool because those are saved in the pool.

If you have any questions, someone here will be happy to help.

 

[Chris Moore]

and finally a wlan usb dongle (I might add my wifes smartphone to save some pictures from it via wifi, but this has very low priority).

I forgot to mention, FreeNAS can't directly connect to WiFi. You will need to get a WiFi router or access point. That USB dongle isn't supposed to work. I won't say that it can't because someone would do it just to show me.

 

[rvassar]

PS. Many folks that have a background in Linux or Unix start trying to change things from the command line because that is how they have done it elsewhere. It can lead to pain with FreeNAS because it rebuilds the system config at every boot, based on what is saved in the configuration database, so many changes made directly to config files will be lost when the system is rebooted. Changes need to be made through the GUI. The only real exception to that is changes made to the storage pool because those are saved in the pool.

It actually goes a step further than this if you're a Unix native... I've found myself creating a directory in a pool via the CLI, rsyncing some data in, and adding a share, then being confused when I couldn't find the dataset in the GUI to take a snapshot. ;)

I suspect the X5650 will be OK, provided you don't expect full squish out of the Infiniband cards. The caveats are heat/power, older PCIe, and no Spectre/Meltdown microcode fixes, so play it safe on jails / VM's.

 

[lowerCamelCase]

Thank you for the big amount of response. I do know zfs on a user (not on a developer) level, so I am familiar with the terminology.
The information about the config storage however is a very usefull one! Thank you.
I did do some research and think I will be able to get the Mellanox card to work, but we will see. What I really hoped the gui will help me with is to easily get monitoring, as I dont want to set up icinga on this system.
This is a reason why I would prefer to go without the sas conzroller, for easier usage of smart, but its not really important either.
I was not aware that there is no wpa supplicant for bsd, but I did not check it, this really has low priority. I might aswell use an old router and another ethernet port.
Spectre/Meltdown are not an issue in this system, it wont be reachable by users without permission and wont run software where I need to take care. The docker repo is just working as repo, not to build or run images.
Getting qdr infiniband maxed out of cpurse is not a goal, it is not possible with the hdds. I just did not want the cpu to bottleneck the compression, as lz4 is propably not multithreaded the X5650 is not a strong contender.

 

[lowerCamelCase]

Flashing the controller was sth i was aware.
That even the boot device is not planed to be raided by mdadm suprises me.
Agin thank you for the input.

 

[Chris Moore]

for easier usage of smart

If you have the SAS controller flashed with the IT mode firmware and you are using it strictly as an HBA, it passes the SMART data over to the OS. I have (at work) hundreds of disks attached to FreeNAS systems by way of SAS controllers and SAS expander backplanes and it works fine.

I just did not want the cpu to bottleneck the compression, as lz4 is propably not multithreaded the X5650 is not a strong contender.

I did run FreeNAS on an older system of this vintage before and it should have no trouble with the lz4 compression. It isn't very processor intensive. I have a FreeNAS setup that is using gzip-9 and it gets higher ratio compression but it really punishes the CPU.

That even the boot device is not planed to be raided by mdadm suprises me.

The boot media is mirrored but it is done using ZFS. The mdadm software is not even installed in FreeNAS.

 

[anmnz]

I was not aware that there is no wpa supplicant for bsd

There is. It's specifically FreeNAS that doesn't do WiFi.

 

[Chris Moore]

Remember, FreeNAS is an appliance. The underlying operating system is FreeBSD, but it is highly customized and they make an effort to protect it from being reconfigured by the user. Configuration of the NAS is supposed to be done through the GUI, which makes the settings in the OS for you, not at the command line. There are a very few things that can or need to be done at the command line and most of those revolve around troubleshooting disk faults.
Just don't think it is the same as running FreeBSD.

 

[lowerCamelCase]

I just wanted to give some feedback after finishing my project, as I think this is part of a good behaviour.
First of all, thank you again for your help. I agree that I first had to fully accept FreeNAS beeing an appliance, not some addition to FreeBSD, with both up- and downsides.
I ended up using a machine somewhat in between the two mentioned earlier, utilizing a dual 2640v2 system with 128gb ram and LSI Broadcom 2308 SAS and a 12 bay expander back plane.
Installing FreeNAS was as straight forward as it can get and I had the system completly configurated within 10 minutes, including setting up raidz2 and nfs share via gbit ethernet.
I was pretty impressed by the GUI and how well it worked, also the Performance of the raidz2 with enabled lzo compression was even a lot better than expected, basically sustaining the maximum write speed the controller can put out, for big block sizes of course. Although I did not need it I liked the preconfigured docker images, like the gitlab one.
However setting up infiniband was a bit ruff. At the end I managed to get the Mellanox Card working (I had to compile the modules in a docker container first though). I also tried to get a Card by Qlogic to work in FreeNAS but this seemed to be impossible to me. Main issue here is that I do not trust the update stability after all the tweeking done by me, which sadly is just to much of a risk to availability and additional work for me.
Instead of changing the complete network from Infiniband to 10gbit Ethernet (which would have been quite some money to invest considering amount of machines and a lot of work too. In fact I would have needed to run the ethernet parallel as I need Infiniband for mpi stuff), I decided to go for a different system.
In the end I did set it up using debian, doing everything manually and add monitoring to an already running icinga2 instance on another server, which is now productive for two weeks and works quite well too, although there is a tendency for zol beeing slower than zfs on BSD.

If Infiniband is not part of the equation however, I would highly recommend FreeNAS. You did a great job there!

 

[Chris Moore]

If Infiniband is not part of the equation however, I would highly recommend FreeNAS. You did a great job there!

FreeNAS is not always the answer, but I am happy you were able to get something working.
Hopefully you will be able to get a FreeNAS system for home and continue learning about the features and functions that FeeNAS can do well.