Building a 60 bay with FreeNAS

[clueluzz]

Hi,

I’m in the planning stages of building a NAS for editing videos across a number of video editors. I’m planning on using a HGST 60 Bay disk shelf and using a PC box to control that using a SAS HBA. All computers will be connected to the server using 10Gbe through a switch. As for the hardware side, I’m pretty much all set. However, I’m trying to wrap my head around the way FreeNAS handles the HDDs (pools and shares). Since I’m not going to populate the entire 60 bays right off, I’m wondering if anyone can share their thoughts and ideas how to tackle this.

I’m starting with 12x 6TB (set 1), 6x 12TB (set 2) and 2x 256GB as a cache. My objective is to use RAIDZ2 per 12 HDD.

With the above, do I go with the following:

Option A:
1. Do I create everything in one pool?
2. Later on, if I add another 6x 12TB, would it be able to expand set 1 without making this new set as a separate RAIDZ2.
3. If I continue to expand the pool until I populate all 60 bays, will it always be based on the sets of HDD before?

Option B:
3. Do I create separate pools for each set and continue to create separate pools each time I have a new set of drives?
4. Does the cache work globally for separate pools?

Please share your insights since I’m at the beginning of buiiding this. The last thing I need is to have a 144TB and need to rethink my breakdown as I populate the bays in the disk shelf.

 

[Chris Moore]

Slideshow explaining VDev, zpool, ZIL and L2ARC
https://www.ixsystems.com/community...ning-vdev-zpool-zil-and-l2arc-for-noobs.7775/

Terminology and Abbreviations Primer
https://www.ixsystems.com/community/threads/terminology-and-abbreviations-primer.28174/

 

[Chris Moore]

2x 256GB as a cache.

Why? What do you hope to accomplish with this?

Do I create everything in one pool?

It depends on your goals? Why are you saying that you want 12 drives in one "set" and 6 drives in another "set". Do you have some reason for these separations? What is the reason for the things you have proposed?

My objective is to use RAIDZ2 per 12 HDD

Why?

Do I create separate pools for each set and continue to create separate pools each time I have a new set of drives?

I would not.

Does the cache work globally for separate pools?

Cache is per pool.

Without knowing what it is you aim to accomplish, I would have to suggest that you use all the drives in one pool, six drives per vdev, all drives inside a single vdev need to be of the same capacity, but you can have ten vdevs in that chassis, all in the same pool. More vdevs generally equates to more speed. I have a system at work that has 60 drives in it, ten vdevs of six drives each, all in one pool, and it works great.

I just don't understand what you are thinking with the things you have suggested so far.

 

[clueluzz]

Hi Chris,

Thanks for replying my post so promptly. I read the slideshow that you provided and am starting to grasp of things. However, I’m sure I still have a lot to learn. My aim is to maximize usable storage with enough redundancy since we are editing multiple projects with lots of footage simultaneously.

So based on what I read and your summary (which actually is very close to what I want to achieve):
- The 2x 256GB is for L2ARC (which if I understand correctly is cache)
- The separation mentality was because of my base understanding of RAID. Since I’ve been using a 12 bay Synology, I’m applying the RAID6 mentality for that. However, after reading the slideshow and found that it is not recommended for more than 11 drives per vdev, I’ll be building 10HDD per vdev now since I’m aiming for RAIDZ2.
- I will be using one pool for all my vdevs so that the Cache SSD will handle all of it.

May I ask if you are using RAIDZ1 or RAIDZ2 for your ten vdevs? I’m guessing you’re using RAIDZ2 based on your slideshow :D

Looking forward to yours thoughts.

 

[Chris Moore]

May I ask if you are using RAIDZ1 or RAIDZ2 for your ten vdevs? I’m guessing you’re using RAIDZ2 based on your slideshow :D

Always RAIDz2. RAIDz1 is risky.

Why not to use RAID-5 or RAIDz1
https://www.zdnet.com/article/why-raid-5-stops-working-in-2009/

I have a server at work that is using 10 drives per vdev, and I wouldn't suggest that because it is slow. If you want better performance, 6 drives per vdev is the better option because it gives you more vdevs, which means better IOPS. I manage several servers at work with different characteristics.
We have one system with 8 vdevs of 8 x 4 TB drives (64 drives, using 16 bay SAS disk shelves), another with 6 vdevs of 10 x 6TB drives (60 x 6TB in a single chassi) and another of 10 vdevs of 6 drives (60 x 10TB drives) in a single chassis. Of all these, the fastest is the one with the most vdevs. The higher the vdev count, the faster the storage. It just depends on what your needs are, but I would stay away from the wide vdevs because they are relatively slow when compared against narrow vdevs.

 

[clueluzz]

Always RAIDz2. RAIDz1 is risky.

Why not to use RAID-5 or RAIDz1
https://www.zdnet.com/article/why-raid-5-stops-working-in-2009/

I’ve learnt my lesson not to ever use RAID5 anymore from a previous failure during re-silver years ago when I first got a RAID box.

The higher the vdev count, the faster the storage. It just depends on what your needs are, but I would stay away from the wide vdevs because they are relatively slow when compared against narrow vdevs.

Noted. Definitely need faster storage so I'm gonna go with your suggestion. Once again, thank you so much!

 

[AndroGen]

We have one system with 8 vdevs of 8 x 4 TB drives (64 drives, using 16 bay SAS disk shelves),
another with 6 vdevs of 10 x 6TB drives (60 x 6TB in a single chassi) and
another of 10 vdevs of 6 drives (60 x 10TB drives) in a single chassis.
Of all these, the fastest is the one with the most vdevs. The higher the vdev count, the faster the storage.

Jumping on this conversation
my build comes to the "final" assembly stage, and reading this post it has triggered some thoughts where your, Chris, input would be appreciated.
I am planning to have vdev with 10 disks 8 TB each, what would give about ~50 TB of the total available storage.
You refer to the big speed difference between vdev with 6 and 10 disks - how big this difference is?
Are we talking about 10% or 50+%?

 

[Chris Moore]

Jumping on this conversation

It is fairly simple. Each vdev (virtual device) in a pool behaves, in very rough terms, like a single disk. If you have all the disks in your pool in a single vdev, you get roughly the performance of a single mechanical drive. Take the same number of disks and make a pool with TWO vdevs and you get almost double the performance, like it were two disks in a stripe set. It is the reason that for high IOPS implementations, mirror vdevs are suggested. You can get more vdevs with less disks. For example with 12 disks, in mirror vdevs, you can have 6 vdevs in the pool, where if you were using RAIDz2, you could have two vdevs of 6 drives each. Same number of drives, but 6 vdevs is going to perform much better than two vdevs.

I went through this in my home NAS years (6 or 8, I don't remember exactly) ago. It was the third build I did and the first one where I put all the hardware together from scratch instead of using an old PC. I had initially built a RAIDz3 vdev with 12 drives but I found that this performed very poorly. It took days to copy the little amount of data that I had at the time into the pool and I couldn't even saturate the 1Gb network I had at the time. I struggled with different ways to optimize this system for about a year before I bit the bullet and reconfigured the same 12 drives into a pool with two vdevs of six drives each at RAIDz2 and roughly doubled my performance. For my purposes at home, it gives me adequate performance, but a single vdev was not cutting it. Back then I was using 1TB drives and I found that the speed increased with newer drives as I went to 2TB drives and then to 4TB drives because each successive generation of drives provided better performance to the pool, but still with the same underlying mathematical relationship.

 

[Chris Moore]

how big this difference is?
Are we talking about 10% or 50+%?

The amount of difference scales depending on the exact hardware in play, but the difference between one vdev and two vdevs is that two vdevs is about double the performance of one vdev. To put some numbers to that, with two vdevs I am getting about the performance of an SSD from my NAS over a 10Gb network.

For comparison, this is the speed of a SATA SSD:

This is the speed of my pool when I was using (correction) 2TB SATA hard drives:

When I added a NVMe SSD to that same pool as a SLOG device:

This was from several years ago and I am not still using that hardware.

 

[AndroGen]

The amount of difference scales depending on the exact hardware in play, but the difference between one vdev and two vdevs is that two vdevs is about double the performance of one vdev. To put some numbers to that, with two vdevs I am getting about the performance of an SSD from my NAS over a 10Gb network.
...
This was from several years ago and I am not still using that hardware.

Thanks
This would mean, the performance should be tested in multiple (more than initially planned) different configurations before going live.
This also shows me that using properly configured SLOG would add performance significantly - before that I was not sure whether I need separated SLOG, but now it sounds more like: I need to try it out (then it would be 2 mirrored NVMe M.2 SSDs mounted on PCIe card).
Current thinking: 10 of WD Red 8TB HDDs in one vdev in RAIDz2 + (still to be decided) SLOG.
Now it sounds like I really should consider SLOG to get the performance.

Just to give the prospective, main content: video and photos (storage, but processing on another machine via 10GB net) + some documents storage + few test VMs.
VMs might be moved to separated SSDs later - not decided yet.
The server is Intel Xeon 4110 (or Xeon D-2146NT) + 192GB (or 128) of RAM + 10GB network.

Another question, what would be the best tool for disks performance testing?
Should it be done from another PC connected to freeNAS via 10GB network?
Should be CrystalDiskMark or something else used for disk / freenas testing?
How big should NVMe M.2 SSDs be for SLOG for this vdev size, would 500GB be sufficient?

 

[Chris Moore]

This also shows me that using properly configured SLOG would add performance significantly - before that I was not sure whether I need separated SLOG, but now it sounds more like: I need to try it out (then it would be 2 mirrored NVMe M.2 SSDs mounted on PCIe card).

SLOG is mostly useful if you are doing synchronous write to the pool. In the test I did I had configured the share by way of iSCSI and had the pool set for sync always. It does make a difference in certain circumstances, but it is not always needed. It would completely not be used at all if you are not doing sync writes. NFS also uses sync write by default, but not SMB, so it is use dependent. If your use calls for SLOG, it is important that you choose wisely. Please look at this guidance:


The ZFS ZIL and SLOG Demystified
http://www.freenas.org/blog/zfs-zil-and-slog-demystified/

Some insights into SLOG/ZIL with ZFS on FreeNAS
https://www.ixsystems.com/community/threads/some-insights-into-slog-zil-with-zfs-on-freenas.13633/

Testing the benefits of SLOG using a RAM disk!
https://www.ixsystems.com/community/threads/testing-the-benefits-of-slog-using-a-ram-disk.56561/

SLOG benchmarking and finding the best SLOG
https://www.ixsystems.com/community/threads/slog-benchmarking-and-finding-the-best-slog.63521/

 

[Chris Moore]

Another question, what would be the best tool for disks performance testing?

This may not be the best, but it is the one I am familiar with:

solnet-array-test (for drive / array speed) non destructive test
https://www.ixsystems.com/community/resources/solnet-array-test.1/

There is also some information regarding performance testing at this site: https://calomel.org/zfs_raid_speed_capacity.html

I have not used it and am not familiar with that testing method, but it is a resource.

Should it be done from another PC connected to freeNAS via 10GB network?

Something to keep in mind, if you want the best performance from FreeNAS over the network, don't use jails or any of the things that create virtual network adapters for connecting VM / jail devices to the same interface that you are sharing your storage over. This slows the interface down significantly. Testing internal to FreeNAS can show you what your pool is capable of so you know if the limitation is a pool limit or a network limit. So it is important to test both.

Should be CrystalDiskMark or something else used for disk / freenas testing?

I only used CrystalDiskMark because I had an iSCSI disk mapped such that it looked to Windows to be a local hard disk drive. I wanted to see how it compared to a physical drive / SSD. After having used iSCS, I am not a fan. I would find another way if I had it to do again. For example, you can use NFS as a datastore to host the VMDKs for virtualization instead and it performs better from what I have seen.

 

[AndroGen]

Thanks, @Chris Moore, your input is very much appreciated.
I've already read linked above articles - nice ones.
I'll try both set-ups: with and without SLOG to see the impact. considering plans to have VMs I feel I would need it.
From these articles I have understood that SLOG might be not so big, below 100-200GB should be sufficient(the rest might be over-provisioned for SSD wear and tear).
Just found an interesting article related to that: https://www.thomas-krenn.com/en/wiki/SSD_Over-provisioning_using_hdparm

 

[clueluzz]

I’m having trouble locating the recommended HBA cards based on the compatible list (LSI / Broadcomm). Is the Microsemi Adaptec HBA 1100-8e compatible and good?

 

[Chris Moore]

Is the Microsemi Adaptec HBA 1100-8e compatible and good?

No. Not compatible.

Get one like this instead:
https://www.ebay.com/itm/LSI-9206-16e-6Gbps-SAS-HBA-P20-IT-mode-firmware-ZFS-FreeNAS-unRAID-Dell-TFJRW/163274936627
or
https://www.ebay.com/itm/LSI-9202-16e-6Gbps-SAS-HBA-low-profile-P20-IT-mode-firmware-ZFS-FreeNAS-unRAID/163239798646
or
https://www.ebay.com/itm/LSI-9201-16e-6Gbps-16-lane-external-SAS-HBA-P20-IT-Mode-ZFS-FreeNAS-unRAID-NoROM/162872615455
or
https://www.ebay.com/itm/LSI-9200-8e-6Gbps-8-lane-external-SAS-HBA-P20-IT-Mode-ZFS-FreeNAS-unRAID-NoROM/163534822734

There are so many options. Just keep looking.

 

[clueluzz]

How about LSI SAS 8300-e?

 

[AndroGen]

I’m having trouble locating the recommended HBA cards based on the compatible list (LSI / Broadcomm). Is the Microsemi Adaptec HBA 1100-8e compatible and good?

You might want to have a look on this article, where HBA topic is well covered, what is available, what models and so on
https://www.servethehome.com/buyers...s-freenas-nas-servers/top-picks-freenas-hbas/

 

[AndroGen]

No. Not compatible.

Get one like this instead:
...
There are so many options. Just keep looking.

If I look on e-bay (in Germany) - there are not too many local offers here, but a lot from China/Hong Kong/Taiwan marked as OEM something
What is the overall perception here in regards to such offers?

 

[Chris Moore]

What is the overall perception here in regards to such offers?

Often times used server gear is sent out for recycling and it ends up in those countries where it is dissembled and the components sold off. These are usually good, but sometimes you get something that is defective. Usually the firmware needs updating unless the vendor has already done that for you. I have bought a lot of used gear and only had a couple problems where something wasn't working.

How about LSI SAS 8300-e?

Something like this would be fine:
https://www.ebay.com/itm/LSI-SAS-93...-e3-0-Host-Bus-Adapter-US-seller/132716297980
You just need to ensure it has the latest IT mode firmware. I use the internal version of that card for one of my servers, like this:
https://www.ebay.com/itm/LSI-SAS300...-3-0-SATA-SAS-8-Port-SAS3-12Gb-s/292817215076
The thing is, you don't need 12Gb/s SAS for mechanical drives until you have over 80 drives (if I recall correctly) because the mechanical drives don't have enough bandwidth to saturate the interface. Usually you are just as well off with the less expensive 6Gb/s SAS controller because mechanical drives are just slow. Fast controllers are fine, but they really are not needed until you have a massive number of drives. The controller I have is supposed to be able to handle 1024 drives where the 6Gb SAS cards can "only" handle 512 drives or 256 drives depending on the card.

 

[Chris Moore]

I wish I could take photos at work but it is strictly against the rules. This is the kind of server I was talking about:
https://www.supermicro.com/products/system/4U/6049/SSG-6049P-E1CR60L.cfm

It has an internal "Broadcom 3008 AOC" that connects to the two SAS expander backplanes that all the drives plug into.
It is a fantastic unit and I have been able to fully saturate the 10Gb network when writing to it even when the writes were random small files. It is not quite as fast on reads, but our workload is a massive number of small files in a random workload. They call it a, "File GEO Database". If you were streaming a large sequential file, like a video, I am sure it would have no trouble at all.