RAID, vDev, pools, ZFS setup

[Isaac039]

If the title is a bit confusing it's because I'm a but confused on the storage specifics for setuping TrueNAS. I have a lot of years of experience in tech but RAID is one thing that I don't have much experience with.

I have a custom built PC which I'm looking at configuring with TrueNAS Scale. What I'm trying to accomplish is a TrueNAS build that will allow for some universal use with containers, VMs and NAS/storage for the containers, VMs and general file storage to be accessed from my Windows devices.

This is what I'm hoping to do and having some trouble with determining the best way to go about it. I have (4) 4TB hard drives and what I'm looking for is having 8TB, approximately, of storage with redundancy. I'd like to have the best performance possible when copying files and for the VMs/containers. I'm hoping I'd be able to lose one drive then be able to replace without any loss.

So basically I'm hoping for some help and guidance on how to approach doing the configs in TrueNAS to accomplish what I'm wanting to do with my setup.

Thank you in advance.

 

[Arwen]

Local VMs and any external VM storage want a ZFS pool made of Mirrors, to increase IOPS. ZFS pools made of 1 x RAID-Zx vDev have the IOPS, (I/O Operations Per Second), of the slowest device in the vDev. While multiple Mirrored vDevs have as many IOPS as their are Mirrored vDevs.

General purpose storage on the other hand, mostly wants more space per device usage, thus RAID-Zx.


Given just 4 x 4TB HDDs and a desire for 8TB of RAW storage, you have only a few choices;

• 2 x 2 disk Mirror vDevs, can loose 1 disk each, in each vDev before data loss. 8TB of RAW storage
• 4 disk RAID-Z1 vDev, can loose 1 disk but not generally recommended for disks larger that say 1TB or 2TB. Not recommended for VMs or external containers. 12TB RAW storage
• 4 disk RAID-Z2 vDev, can loose any 2 disks before data loss. Also not recommended for VMs or external containers. 8TB of RAW storage

Note that ZFS pools need to be kept at about 80% maximum full. Otherwise at a certain point ZFS slows down to use space efficiency over speed. Thus, the 8TB RAW storage ends up being less.


It is always helpful to read up on ZFS and TrueNAS. See the "Resources" section of the forum for intro guides and such. A link is at the top of every forum page for "Resources". I don't have a handy list of which Resources, (or sticky threads), that a new user should read. Perhaps someone else will make those recommendations.

 

[Isaac039]

Thank you very much for this info. I had done some research leading up to posting and was leaning towards doing a RAID-Z2 but your response has helped to confirm that this would indeed be a valid option given my current options with hard drives.

So a related question to all this. Would it be terribly hard to add an SSD to allow it to essentially act as the buffer and increase performance/IOPS?

 

[Arwen]

...
So a related question to all this. Would it be terribly hard to add an SSD to allow it to essentially act as the buffer and increase performance/IOPS?

Their is a read cache called ARC, Adaptive Replacement Cache, which resides in RAM. It basically will use normally un-used RAM for a read cache. In TrueNAS Core, based on FreeBSD, this ARC will use as much RAM as is not in use by programs or program data. ZFS was tightly integrated to the memory sub-system of FreeBSD, (similar to Solaris where ZFS was developed), thus making ARC work well.

A SSD or NVMe can be added or removed from a ZFS pool live to act as a Level 2 ARC, aka L2ARC. It acts as an extension to RAM, but does take some RAM to hold a directory of what is in the L2ARC. If the L2ARC device fails, for any reason, like just a block or the entire device, any read will go back to the data pool. Thus, a L2ARC is not considered critical.

The L2ARC should be not be considered unless you have more or less maxed out your normal ARC, aka RAM. So a server with only 8GB or 16GB of RAM is NOT a candidate for L2ARC. Generally, 64GBs of RAM is suggested before adding a L2ARC device.

The size of a L2ARC device should not exceed 5 or 10 times the size of RAM. Going too extreme on L2ARC size, like 2TB, is a bit silly. This is because too much RAM will be used to store the directory of what is in L2ARC, forcing ARC to be much smaller than it should be.


Their is something that helps data integrity for synchronous writes called a Separate intent Log, aka SLOG. This SLOG is NOT a write cache and only works for specific purposes, synchronous writes for NFS or iSCSI, nothing else. SLOGs can be added or removed from a ZFS pool live. The device requires Power Loss Prevention, PLP, and wants HIGH speed writes, and HIGH write endurance. SLOGs only come into play on a crash or power loss, no other time. It's main purpose is data integrity not write caching, which ZFS does not have an option for write caching outside of RAM.

In general, if you have to ask about a SLOG, you don't need one. It can always be added later if desired.

Note if you don't have a SLOG, and have NFS or iSCSI, the in-pool version of a transaction log does exist, called ZIL, (ZFS Intent Log). It just causes the data pool devices to be busier than with a SLOG.


Last, TrueNAS SCALE is based on Linux, and ZFS was not tightly integrated to the memory sub-system, (because, well, reasons), so the ARC usage is less than ideal. People are working to make it better, but it is not likely to be as good as TrueNAS Core, (based on FreeBSD), or Solaris, (the original source of ZFS), for at least a year.

 

[Isaac039]

Their is a read cache called ARC, Adaptive Replacement Cache, which resides in RAM. It basically will use normally un-used RAM for a read cache. In TrueNAS Core, based on FreeBSD, this ARC will use as much RAM as is not in use by programs or program data. ZFS was tightly integrated to the memory sub-system of FreeBSD, (similar to Solaris where ZFS was developed), thus making ARC work well.

A SSD or NVMe can be added or removed from a ZFS pool live to act as a Level 2 ARC, aka L2ARC. It acts as an extension to RAM, but does take some RAM to hold a directory of what is in the L2ARC. If the L2ARC device fails, for any reason, like just a block or the entire device, any read will go back to the data pool. Thus, a L2ARC is not considered critical.

The L2ARC should be not be considered unless you have more or less maxed out your normal ARC, aka RAM. So a server with only 8GB or 16GB of RAM is NOT a candidate for L2ARC. Generally, 64GBs of RAM is suggested before adding a L2ARC device.

The size of a L2ARC device should not exceed 5 or 10 times the size of RAM. Going too extreme on L2ARC size, like 2TB, is a bit silly. This is because too much RAM will be used to store the directory of what is in L2ARC, forcing ARC to be much smaller than it should be.


Their is something that helps data integrity for synchronous writes called a Separate intent Log, aka SLOG. This SLOG is NOT a write cache and only works for specific purposes, synchronous writes for NFS or iSCSI, nothing else. SLOGs can be added or removed from a ZFS pool live. The device requires Power Loss Prevention, PLP, and wants HIGH speed writes, and HIGH write endurance. SLOGs only come into play on a crash or power loss, no other time. It's main purpose is data integrity not write caching, which ZFS does not have an option for write caching outside of RAM.

In general, if you have to ask about a SLOG, you don't need one. It can always be added later if desired.

Note if you don't have a SLOG, and have NFS or iSCSI, the in-pool version of a transaction log does exist, called ZIL, (ZFS Intent Log). It just causes the data pool devices to be busier than with a SLOG.


Last, TrueNAS SCALE is based on Linux, and ZFS was not tightly integrated to the memory sub-system, (because, well, reasons), so the ARC usage is less than ideal. People are working to make it better, but it is not likely to be as good as TrueNAS Core, (based on FreeBSD), or Solaris, (the original source of ZFS), for at least a year.

This is some awesome detail so I greatly appreciate this extra insight on this subject and answering my questions. I finally carved out some time to get things assembled, installed and configured. So now I'm working on migrating data and setting up containers.

Thank you again for helping to answer my questions.

 

[Arwen]

Sure.

You might check out some of the Resources, which are articles on various subjects. See top of any forum page for a link to "Resources". Their are some general knowledge ones for ZFS, TrueNAS and networking. Plus, some detailed ones on specific less common subjects. We, the Forum Community started transferring "sticky" forum threads to Resources whence the forum software update included them. By now their are probably more than 100 Resource docs, (though some are not intended for newish users).