Advice for a newbie setting up ZFS

[SNE]

Hello. Recently I have inherited an older 45drives server (REV4) to manage with the following specs:
- 15 6TB WD RED NAS SATA drives
- 15 4TB Seagate SATA drives (sorry cannot recall model)
- CPU: i3 3220
- RAM: I will max it out at 32GB, DDR3 1333MHz, ECC
- Boot drive will be a 256GB SSD

The server has Windows server 2012 on it and I am going to replace this with FreeNAS. I have experimented very briefly with FreeNAS in the past. I have read about ZFS a bit. I plan on making the boot drive an SSD.

I am a little lost as to the best way to set up the drives I have with ZFS. How can I optimize the configuration? I will be setting up with RAIDZ, but I am not sure about how many vdevs to make, or how many disks to assign to each, how many pools to make, how many vdevs per pool - ??.

This will be primarily a file server with users that access via SMB and AFP with PC and Mac.

 

[sretalla]

ZFS perfection (alignment of blocks across disks) happens for RAIDZ2 at 6 disks. you have 30 disks, so you could consider giving up a little storage on one mixed VDEV of 3 WD and 3 seagate (perhaps in its own pool) and two other pools of 2x6disk VDEVS.

 

[SNE]

Thanks! Great advice!
I will have to read about the "ZFS perfection (alignment of blocks across disks) happens for RAIDZ2" - do you have a source?

Just curious - why did you mention RAIDZ2 rather than RAIDZ?

 

[danb35]

ZFS perfection (alignment of blocks across disks) happens for RAIDZ2 at 6 disks.

...except that with native ZFS compression this is no longer the case (see https://www.delphix.com/blog/delphi...or-how-i-learned-stop-worrying-and-love-raidz). Even when this was an issue, the rule was 2^n+p disks/vdev, so in RAIDZ2 you'd be looking at vdevs of 4, 6, or 10 disks. For this case, I'd probably recommend acquiring one more disk of each capacity, then setting up the pool with four vdevs of eight disks each in RAIDZ2.

why did you mention RAIDZ2 rather than RAIDZ?

RAIDZ isn't recommended with disks > 1 TB due to the risk of a read error on resilvering.

I'm also thinking that the max of 32 GB of RAM might be a little light for the amount of storage you're looking at, but it should be at least usable for the time being.

 

[SNE]

Thanks for the link and the information!
Unfortunately 32GB is the maximum RAM for that machine so can’t go above that.

This begs another few questions:
How do I know if there is not enough RAM?
What can I do if I cannot increase the RAM to optimize for the amount of RAM I have. I might be old-fashioned, but to me 32GB RAM is a lot.

 

[Chris Moore]

Disk controller details are missing here. Those old 45drives systems were using SATA port multiplier hardware if I recall correctly. Please share some more hardware info here.

 

[SNE]

Yes Chris, there are 3 cards in the machine (looks like PCI - I'm not at work now, so I cannot look again). Each card has 3 SATA cables going to a backplane. The backplane holds all the disks - up to 45 of course. So therefore, it looks like each SATA port on each card will take care of 5 drives.

Will that be relevant to my question concerning ZFS and the RAM limitation I have? viz, how do I optimize ZFS for the 32GB RAM, given the capacity I have described?

 

[Chris Moore]

Unfortunately 32GB is the maximum RAM for that machine so can’t go above that.

I bet the system board could be replaced.

 

[Chris Moore]

Will that be relevant to my question concerning ZFS and the RAM limitation I have?

The exact details here matter. SATA port multipliers are known to not work reliably with FreeNAS and ZFS. The source of these systems was a company called Backblaze and they were using Linux, if I recall correctly. They were never designed to be used with FreeBSD and ZFS.

 

[SNE]

I did call 45drives and had them pull up the old purchase and asked them if they would recommend FreeNAS, they said yes.

I bet the system board could be replaced.

Yes Chris I was thinking that is a possibilty. But the SATA multiplier cards are pretty old so I am not sure about that part of it - I really need to look at those cards - if they are PCI, then that's a bit more challenging to find an MB with 3 PCI slots that allows more than 32GB RAM... but maybe I can get away with 1 or 2 of the cards...hmm
Also I need the MB to support ECC RAM correct?

 

[Chris Moore]

I did call 45drives and had them pull up the old purchase and asked them if they would recommend FreeNAS, they said yes.

Most people are not aware of the limitations of SATA port multiplier hardware.

then that's a bit more challenging to find an MB with 3 PCI slots that allows more than 32GB RAM

If this is for your work, you might want to propose hardware that is better suited to the task as well as being newer. Is there any budget for this installation?

 

[SNE]

Most people are not aware of the limitations of SATA port multiplier hardware.


If this is for your work, you might want to propose hardware that is better suited to the task as well as being newer. Is there any budget for this installation?

It is at a State university so not much money around. I used some to max out the RAM -
I am usually expected to make do with what I have. I really want to use FreeNAS/ZFS for this both as a learning experience and because it appears to be a superior file system. The only other servers I have maintained for a long period have been Synology boxes.

 

[Chris Moore]

I am usually expected to make do with what I have. I really want to use FreeNAS/ZFS for this both as a learning experience and because it appears to be a superior file system.

ZFS is a superior file system. The original developer (Sun Microsystems, Inc.) spent years on R&D before it was open-sourced and others have taken that ball and run with it even after Oracle came along and closed the source again. It is fantastic.
The hardware you have is less than ideal, but you may have some success with it. Just don't put any data on the system before you do extensive validation testing. Here is a good setup guide:

https://www.familybrown.org/dokuwiki/doku.php?id=fester112:intro

 

[SNE]

Thanks for sticking with me Chris. That looks like a great resource - I will be sure to read.

One thing I have seen is that some people say you absolutely should not run ZFS without ECC RAM.
But in order for ECC RAM to work properly, the CPU has to support ECC RAM. I looked at the Intel Spec sheets for the i3 3220 and it does not appear to support ECC RAM. In fact to get a processor that will work in the socket on this MB and support ECC would be quite costly. What is your opinion about the ECC RAM issue?

 

[danb35]

One thing I have seen is that some people say you absolutely should not run ZFS without ECC RAM.

ECC RAM is wise any time you care about your data, regardless of OS or filesystem. Nothing about ZFS makes it any more important than it would be with any other filesystem, though.

But in order for ECC RAM to work properly, the CPU has to support ECC RAM.

Correct.

I looked at the Intel Spec sheets for the i3 3220 and it does not appear to support ECC RAM.

There's a bit of controversy around that, as that CPU (like all Socket 1155 i3s) was listed as supporting ECC, but isn't any more. Whether it actually does is anybody's guess at this point. But even lowly Celeron and Pentium CPUs in that socket are still listed:
https://ark.intel.com/content/www/u...ECCMemory=True&1_Filter-SocketsSupported=3636