FreeNAS Disk Layout Suggestions Needed

chemikalguy

Cadet
Joined
Apr 2, 2020
Messages
5
I'm building a new FreeNAS box after having not had one for a number of years. I have acquired all of the hardware I'm going to use, with the exception of disks, maybe. The main hardware I'm going to be using is all a bit dated, but is still better than most of off-the-shelf home NAS units that are out there, unless I spend a ton of money. I have:

Quad Core Xeon E3-1225 (w/ integrated video support)
16GB of ECC RAM
Intel Server motherboard w/dual Intel NICs (w/ four SATA ports)
LSI 9211 SAS/SATA controller card

Anyway, I have a mini-itx case that holds 8 hotswap hard drives that would be connected to the SAS/SATA controller card. It also has a spot for SSD caching drives which I was going to run off of the motherboard port(s). I had originally planned on using an SSD or two for caching, and some spinning disks for data storage. I have a number of 2TB, 3TB and even some 4TB drives. This unit will likely replace one that has about 6TB of usable storage space in it.

I have a stack of 6 180GB SSDs, though, and wondered if its possible to do tiered storage in FreeNAS. In my daytime job, I work with enterprise storage, and it has the ability to tier storage so that data that isn't accessed very often will automatically get moved down to the slower drives from the faster SSDs. I'm not even sure it's worth using all those SSDs for a high performance tier, as this is for home storage, media serving, etc. These drives are fairly inexpensive now, and I can pick up two more for about $50-60 if I wanted a full 8-drive pool, which would give me roughly 1TB of high performance storage if I use a two drive protection level, but that doesn't address the rest of my storage needs to get me to at least an equivalent amount to my existing NAS. I do like the idea of tiered storage, but that might not be an option.

I'd like to know what people thing of the storage options, so I can decide how to proceed.
 

Inxsible

Guru
Joined
Aug 14, 2017
Messages
1,123
(w/ integrated video support)
Not useful for FreeNAS, but if you already have it, I wouldn't go change it.
I have a number of 2TB, 3TB and even some 4TB drives.
You will need to have the same size disks in a vdev to make use of all the available space. How many drives do you intend to use? If you have different sized disks, then you can choose to use mirror vdevs of same size disks. You haven't mentioned exactly how many drives you have, but you would need at least 4 to have a RAIDZ2 vdev.

You can create multiple pools -- one of SSDs and another with HDDs and use the SSDs etc. You can also use a hypervisor like ESXi or Proxmox and then use the SSD pool for OS storage and HDD pool for data storage and virtualize the FreeNAS appliance and other services that you want in your home. You haven't mentioned the make/model of your Intel motherboard, but you'd have to make sure the BIOS supports VT-d for passthrough of disks to FreeNAS, if you go the virtualization route.

Lots of possibilities, so without knowing your actual use case, it's hard to suggest. Have a look at my FreeNAS box which I use as a file server and Emby Media server -- I use a Pentium G3240 processor-- which is more than adequate for a file server and transcoding 1 stream.
 
Last edited:

chemikalguy

Cadet
Joined
Apr 2, 2020
Messages
5
So, I use it primarily for file storage and media serving (currently). A few years ago, I ripped all of my CDs to MP3 files (~600GB) and started ripping all my DVDs to mkv files. I only did a couple of DVDs, though, so I'd like to continue with doing that. I use Subsonic to stream my music, and would like to be able to stream video as well using Plex or something similar.

Since I have an off-the-shelf unit that has lots of plugins, I also use the built-in email aggregator (and associated iphone app), photo and video players, built-in antivirus, and system monitoring utilities.

My existing NAS is 90% full, and has a capacity of 5TB. I want at least that much space. I also do video creation with Adobe Premiere, so I have a couple of TB worth of video files. Most of my files aren't used all that often.

I seem to collect hard drives, so I have a large number of them. Something like 4 2TB, 5 3TB, 3 4TB, and 5 3TB SAS, plus the 6 180GB SSDs I mentioned. I suppose I could create a mirrored 'landing zone' for new files, and then manually move them to spindle-based storage for the long-term. I'm not a fan of the manual process, although I could probably write a script to move everything out once it is a month old or hasn't been accessed for a month.
 

Inxsible

Guru
Joined
Aug 14, 2017
Messages
1,123
My existing NAS is 90% full, and has a capacity of 5TB. I want at least that much space.
You would need much more than that. ZFS tanks in performance once the pool is more than 80% full. You have enough drives to be able to create a big pool with multiple vdevs, so I wouldn't worry too much.

Plex/Emby works well. You can also install Subsonic as well. All 3 have plugins already. A script for moving the file might work.

for media streaming -- do you intend to transcode? Do you intend to transcode multiple streams simultaneously? If so, you might need a CPU with enough grunt. If transcoding is not in your plans for at least 4-5 years, then a Pentium is more than sufficient -- as I can attest (see my signature for my build with a Pentium G3240).

You mentioned that you have a mini itx case. Do you intend to use that? How many drives can that fit? If it's too small, then you might need another case which can support your 17 drives. -- maybe a 4U rack mount case that can take 24 drives.
 

chemikalguy

Cadet
Joined
Apr 2, 2020
Messages
5
You would need much more than that. ZFS tanks in performance once the pool is more than 80% full. You have enough drives to be able to create a big pool with multiple vdevs, so I wouldn't worry too much.

Plex/Emby works well. You can also install Subsonic as well. All 3 have plugins already. A script for moving the file might work.

for media streaming -- do you intend to transcode? Do you intend to transcode multiple streams simultaneously? If so, you might need a CPU with enough grunt. If transcoding is not in your plans for at least 4-5 years, then a Pentium is more than sufficient -- as I can attest (see my signature for my build with a Pentium G3240).

You mentioned that you have a mini itx case. Do you intend to use that? How many drives can that fit? If it's too small, then you might need another case which can support your 17 drives. -- maybe a 4U rack mount case that can take 24 drives.

If I only did a RaidZ1 with the 3 4TB drives, that's still 8TB in the array, which would only be at ~60% capacity, and that's just one array. Since I have multiple drives each of several different sizes, I could create multiple arrays. For example, I could create these RAIDZ1s: 3x 4TB, 3x or 4x 3TB, and a 2TB mirror. Seems like there would be a performance hit for having multiple arrays, and it might make more sense to just make one larger one using 8x 3TB drives, which I have.

The case I have is a KT8+ NAS chassis like this guy here used in his build (https://imgur.com/gallery/LYBS5). It's a mini-ITX chassis, but it's by no means small. It has 8 3.5" hotswap drive bays in the front, so I could create multiple disk arrays using those. It also has a mount inside the top for a 2.5" SSD for caching. I 3D printed a mount so that I can stack 2 SSDs in there. I don't plan on using more than a total of 10 drives, unless I were to do an additional external storage array, but that's a whole different, down-the-road, project.

As far as transcoding, the pentium you have benchmarks somewhere in the 2000 range, while this xeon I have benchmarks somewhere in the 3500 range. I think if I do any transcoding, it should be fine, and that's if I don't yank the processor out of one of my other boxes and use that. It benchmarks in the 4600 range. Not sure how much transcoding I'll do, honestly. I really just need the extra processing power to handle all the other miscellaneous plugins that I'll have running.
 

Yorick

Wizard
Joined
Nov 4, 2018
Messages
1,912
Howdy,

give us the generation of your xeon (1225v?) and the model number of that Intel board, please?

ZFS and performance, and caching, and tiered storage. There is a good quick rundown here: https://www.reddit.com/r/zfs/comments/9duycb/zfs_tiered_storage/

In a nutshell: Read performance is handled by ARC (RAM cache); write (async) performance by having more vdevs. ZFS is built with a “write and don’t ever move”philosophy. It will never gain tiered storage.

I will make an assumption and say that you have GBit NICs on that board. Which means that a single vdev of spinning rust is enough to keep up with read and write, particularly if they are large files - video, backup, that kind of thing.

Which means you don’t need L2ARC and arguably don’t want it, because it takes away from ARC. With 16GiB of RAM, I'd go so far as to say you definitely do not want to add L2ARC (SSD cache) and instead increase ARC (RAM) if your cache hit rate is low.

ZFS requires a bit of a rethinking on how storage is laid out, and how caching works. You are on the right path with asking lots of questions.

Good ZFS primer video: https://youtu.be/x9A0dX2WqW8
 
Last edited:

chemikalguy

Cadet
Joined
Apr 2, 2020
Messages
5
Yorick,

The CPU is a Sandy Bridge-based Xeon (https://ark.intel.com/content/www/u...xeon-processor-e3-1225-6m-cache-3-10-ghz.html)

The board is an Intel Server board S1200KP (https://ark.intel.com/content/www/us/en/ark/products/60637/intel-server-board-s1200kp.html)

It has dual gigabit NICs onboard. It only supports 16GB of RAM, so I'm maxing it out.

Both are a bit dated, obviously, but performance should still be good on them. I just wanted to nail down areas where bottlenecks might occur, and figure out the best storage layout.

Thanks!

Scott
 

Yorick

Wizard
Joined
Nov 4, 2018
Messages
1,912
Oh, as in, literally 1225, no v anything! Fascinating :). That's a Gen 6 GPU, HD Graphics P3000 .

Your board is the C206 chipset, which does support integrated graphics. Which means you can, if so desired, enable hardware transcode for Plex here. In fact I kinda want you to, just as an experiment and to see how the visual quality of hardware transcode on the older GPU is. Consider it? :)

I don't see any bottlenecks for you. Your drives will saturate that GBit link, no trouble, and 16GiB of RAM is sufficient. I'd stay away from L2ARC unless zfs-stats -a shows you massive cache misses and you have performance issues, but I don't anticipate that happening. Come TrueNAS 12, you can, if you so desire, take a couple of your SSDs and designate them for metadata and small files in a fusion pool.
 

chemikalguy

Cadet
Joined
Apr 2, 2020
Messages
5
Your tone has that, "Wow! A Vic20!" edge to it. :)

I'll definitely experiment with settings to see what performance looks like. I've read that for Plex metadata, they recommend SSDs for performance reasons. Might be cool to try.

The other system I have sitting in my rack has a v2 of the CPU (https://ark.intel.com/content/www/u...n-processor-e3-1225-v2-8m-cache-3-20-ghz.html), which is a year newer, is an Ivy Bridge, has a bit more onboard cache, and actually consumes less power with a 25% performance boost. I might consider gutting the other system and using that CPU in this build. Lower power plus better performance seems like a win-win to me.
 

Inxsible

Guru
Joined
Aug 14, 2017
Messages
1,123
our tone has that, "Wow! A Vic20!" edge to it.
Don't worry about it. I have a SuperMicro X9SCL-F board with C202 chipset and I have a Xeon E3-1240 (yeah no v's) that I use as a VM server. I have used that as FreeNAS too for a short time while I was doing some maintenance on my main FreeNAS box.
 

Inxsible

Guru
Joined
Aug 14, 2017
Messages
1,123
Oh and a couple of posts back you gave examples of using RAIDZ1 for your pool --- yeah, don't !!

Use RAIDZ2. Resilvering can take a toll on the drives if one ever goes down and RAIDZ1 doesn't give you any protection when 1 drive is already degraded. This is especially true for larger sized drives.
 

pschatz100

Guru
Joined
Mar 30, 2014
Messages
1,184
If your use case is primarily file storage and serving media, and you are reading data more than writing data, then the speed of your hard drives will be a non-issue. In this case, tiered storage won't get you much benefit. When you play movies and music, the files transfer at their encoded bit-rates, which will be nowhere near your network speeds. Copying files will be limited by your network speed.

When I upgrade, I like to at least double my capacity. In your case, than would mean a goal of 12TB usable space. I also agree with the premise that you should have at least a RaidZ2 configuration. If it were me, and assuming that you do not want to purchase any additional drives, I would do the following:

1) Use one of the SSD's as a boot drive. If you feel more secure with mirrored boot drives, then use two of them.

2a) Use 6-8 of the 3TB drives in a RaidZ2 pool. That will give you nominal capacity of 12-18TB. I would check SMART info for all the drives and use the best ones. This will give you a nice upgrade from where you are now.
or
2b) Use 6x3TB drives in RaidZ2 for one 12 TB pool for general storage, plus another 2x3TB or 2x4TB mirrored pool for somewhat faster write performance - if you have a need for this.

**If you decide to put 8 drives in the case, then keep an eye on your cooling.

You did not say anything about the make and model of the drives. I would give preference to NAS drives versus consumer desktop drives. And keep in mind that 5400 RPM drives usually run cooler than 7200RPM drives.

If most of your 3TB drives are old, you might consider making a pool with the 4TB drives and the best 3TB drives. True, you won't use all the capacity of the 4TB drives - but that's a small price to pay for better reliability by using your best drives.

As for the other drives, I would get rid of any questionable ones and keep the rest for another project. I have some old 2TB drives in external enclosures that I use for backups.
 
Top