The $1000 FreeNAS 2019 prosumer build. Will it FreeNAS? Will it TrueNAS?

[Constantin]

Great news.

When it comes to the # of drives to fit, consider the use case also. For example, I have a single-vdev Z3 array of 8 drives because I'm mostly focused on long-term, bit-rot-free storage, redundancy, etc. 3 disks provide parity, leaving 5 effectively for storage. Of those, I only intend to use 4, capacity-wise, to ensure that FreeNAS can operate efficiently (above 80% storage fill, data transfer performance plummets as FreeNAS starts to pack stuff into nooks and crannies).

My effective data capacity (4 drives) also matches the effective capacity on my backup arrays (RAID5 Oyen Digital Mobius 5 DAS) while using the same hard drives. That way, I only have to keep one spare on hand to drop in for a failure in either system.

If you want something more responsive re: IOPS, you'd want to consider a multi-VDEV system with more hard drives. (For example, a 12-drive pool consisting of two Z2 VDEVs with 6 drives each).

 

[Mr. Slumber]

Thanks for your post Constantin, that made me think and reconsider my plans :)

 

[Mr. Slumber]

HBA: LSI 9211-8i (for up to 8 HDDS/SSDs, already flashed to IT mode)

Can anybody please point me in the right direction speaking of high quality cables for HBAs?

I found a very good video explaing all the different HBA/SAS/Sata cables and so I thing for the LSI 9211-8i I need a "SFF-8087" type cable. Which are not of cheap quality? Thank you. :)

 

[Mr. Slumber]

I made a change: I was happy with the Rosewill case and for me it worked well but, well, I don't know, it had to be a little bit more perfect if you know what I mean.
So I did some research and switched over to Intertech 4U 4129-N. Yes it's a little bit more expensive compared to the Rosewill case but it feels more high-quality (near enterprise grade) and I like the three 120mm hot pluggable fans between the backplane and the mainboard. They provide a very good but quiet airflow.
It seems to be produced by Inter-tech but in the US it's sold by Norco as Norco RPC-450 4U but sadly to my knowledge without the 120mm fans.
More happy with this new case so let's see if it works out longterm.

 

[Mr. Slumber]

Good news everyone :)

After some issues with 11.2 releases on my test systems I decided to move on and updated this machine to 11.2-U3

So far, so good. I'm happy and until now haven't had any issues. (did a fresh install, no update!)

EDIT: 2019.10.14: Some problems with encrypted pool so back to 11.1-U7

 

[Mr. Slumber]

I also began populating the chassis with some more HDDs and had to find out that TLER was not enabled on any of these drives...

In fear of destroying my pool did some research and found this script: TLER script by spearfoot

I created a dataset "scripts" (this isn't shared and is owned by "root") uploaded the script to it and activated it under "Init/Shutdown Scripts" with the following settings: Type: script / Script: "put the path to the uploaded script here" / When: postinit

Works just great! :)

Highly recommended and thanks to @Spearfoot for this one!

 

[vitaprimo]

You know… I have only heard about people praising WD's Reds but I had a few of them that only gave me the weirdest problems I've had in storage so far. SMART wouldn't detect their problems, and even that extremely long full pass test on UnRAID systems I tried once would check out OK but then the drives intermittently error out. Most issues related to partitioning schemes/formats not so much as bad sectors. I still kept using these drives to see them break completely but they never do, they just pretend to work and F you up when the time is right...or not--whatever.

Of course I'm doing this on purpose trying to learn something out of it, I'm not trusting them with data but, even ZFS failed on 'em. WD is a really big player, so I'm beginning to think they're doing it like VPN "providers" (who really are just tunneled ISPs, not actual VPN solution providers) do and push for good reviews. On the other hand...from Seagate I've heard everything, yet I have very old mechanicals drives that should not be working today. Nothing valuable on them either, but sometimes old platter disks make good log archive collectors, and storing other things you might be hoarding anyway. :)

I'd personally go full server stuff, I have a 1U Cisco where it used to be my closet on the little hallway to the bedroom's bathroom so it's really-really close but I've gotten so used to it, I don't hear it anymore. The other day this week they worked on the grid and power was cut beyond the battery's capacity and things died. I went downstairs to cool off when it started to get hot without AC. Power was restored and I came back up to check if all the servers were online -- ADFS and Exchange most times have to be kickstarted manually. I felt my way to find the power buttons to the hypervisors and I though it was odd there was no loud jet blast when they POST, seconds later it was dead silent, it might as well be pitch black to match that silence. Server was already mid-power up and I cancelled it when I pushed on the button. I thought the forced air noise "murmur" was coming from the aircon unit in that cools the little room, but that thing can't really be heard on its own much less over the servers. There are more servers in the room but only the 1U makes this much noise. So maybe a 2U that has a 12-16, even two dozen 2.5" magazines on the front, and just casually load them up in solid state pairs as you go. You just got me in an eBay mood. :) Good luck, post pics of whatever you build/buy! :)

 

[Mr. Slumber]

(detailed view of the LSI 9211-8i, it's working just perfectly)

 

[Mr. Slumber]

What I really love about this build is that it's (thanks to Noctua fans) very very quiet compared to an enterprise server. You could definitely put it in quiet office space and it won't disturb you.
The three hot-swappable 120mm Noctua fans behind the backplane provide an excellent airflow as to be seen below:

 

[Mr. Slumber]

EDIT: 2019.10.14: Some problems with encrypted pool so back to 11.1-U7

Problems solved, did a fresh install of 11.2-U3 again and the machine is working really really well. Lovin' FreeNAS

 

[pro lamer]

Problems solved

What helped?

Sent from my phone

 

[Mr. Slumber]

What helped?

Did a lot of testing and this worked out for me: please read all the details here

 

[Mr. Slumber]

Just updated the machine to 11.2-U4. Reboot. Done. No problems. Very happy with this build it gets the job done :)

 

[Mr. Slumber]

Quick update to 11.2-U4.1. Reboot. Done. No problems.

 

[Mr. Slumber]

Just about two months later now and I thought it's a good time for a little review of this machine:

After sorting some trouble with the encrypted pools out (please see the posting above around April 6.th) and since I moved on to 11.2-U3 there were exactly 0 problems with the machine anymore. It's just great!

Here's why :)

• actually I have 62TB of storage HDDs installed (4x the shucked 8 TB WDs + 3x 10TB Seagate Exos Enterprise HDDs) (2 Pools, with 1 vdev each) which gives me a "practical usable storage capacity" (you can calculate this for yourself for your server here) of about 30TB. Plenty of space for my use case
  
  
• I really love the thought that all my pools/HDDs are completely encrypted so if one drive goes south I just send it back to Seagate/WD for replacement, no worries about the data on this drive
  
  
• although there are 8!!! fans installed in the machine thanks to design of the Noctua fans it's "flüsterleise" (nice german word for "silent like a whisper") and the cpu never gets "hotter" than 39°/40°C and the HDDs are between 29° - 35°C.
  
  
• speed wise it's about 110-115MB/s for my SMB shares most of the time over my 1GB Unifi network so I'm quite happy there also.
  
  
• stability: I hope this stays that way but for now I just had to reboot when I installed an update

TL;DR: all of my goals where achieved with this build and as said before I'm really happy with this machine.

Thanks again to everybody involved for your help and iXsystems for keeping up the good job! :)

 

[Mr. Slumber]

Quick update to 11.2-U5. Reboot. Done. No problems.

 

[pro lamer]

you can calculate this for yourself for your server here) of about 30TB

So both of the pools seem to be raidz1... The calculator gave roughly 30.5TiB when I used raidz1 settings.

How much data do you have in each pool already? How much time does it take to scrub? (I guess you hasn't had any opportunities for resilvering)...

Sent from my phone

 

[Mr. Slumber]

How much data do you have in each pool already?

just about 40% filled

How much time does it take to scrub?

Under "Tasks/Scrub Tasks" I set "Threshold days" to 21 so every 3 weeks both pools are scrubbed automatically. (Boot pool is being scrubbed every 7 days)
The tasks starts at 0.00 am and finishes around 4.00 pm so about 16hours for both pools in total. I'm always positively surprised because for me this seems kind of fast in comparison to my 2 year "old" storage server from a very popular brand. It was a racksize plus model but with lesser volume data the, how they call it, "data scrubbing" took nearly 2 days and the performance of the whole server was influenced heavily. So again one more thing that is just phantastic running FreeNAS :)

I guess you hasn't had any opportunities for resilvering

Happy to say: no resilvering yet hope it stays that way for a little while... ;)

 

[Mr. Slumber]

Quick update to 11.2-U6. Reboot. Done. Really enjoying the overall stability! It just works! :)

 

[Mr. Slumber]

Quick update to 11.2-U7. Reboot. Done. No problems. Very nice!