SAS bandwidth question -- how many drives bottleneck?

lonelyzinc

Dabbler
Joined
Aug 8, 2019
Messages
35
This is my first storage build, and it's going to be a Supermicro system with a SAS3 backplane supporting 36 drives. I'm hoping to eventually use it in a production environment, and I have some questions about when the SAS HBA will bottleneck in performance testing.

My plan is to buy a bunch of used SAS2 1 TB drives to get a feel for the performance the ZFS system can produce.

Am I right to think that if each drive averages about 150 MB/s read/write speeds, and a SAS3 HBA provides 12 Gb/s (or 1500 MB/s) of bandwidth, that 12 disk drives (150 MB/s times 12 = 1800 MB/s) will be enough to fully test the I/O throughput?

My original thought was to get 24 drives, but wouldn't that be overkill? The numbers say that would be over 3600 MB/s of disk bandwidth, and through a SAS3 HBA it wouldn't ever all get used at once.
 

lonelyzinc

Dabbler
Joined
Aug 8, 2019
Messages
35
It's come to my attention from a colleague, that if I use a bunch of SAS2 drives, I will only get SAS2 speeds overall on the HBA.

So we can assume above, that I am using 1 TB SAS3 drives.
 

Jessep

Patron
Joined
Aug 19, 2018
Messages
379
You will likely be network limited before drive limited at that number of drives depending on a bunch of factors.
  • Pool layout (Number of vdevs)
  • Drive performance (SSD vs HDD vs rotational speed vs transfer rates, etc.)
  • Workload (iSCSI or other sync=always requirements (DB, VMs, NFS, etc.)
  • RAM amounts
  • CPU single thread performance (SMB is single threaded for instance)
Realistically you are at the tail wagging the dog level here. Start with your requirements and work from there.
 
Joined
May 10, 2017
Messages
838
SAS3 will give you around 4400MB/s max from a single link, and around 6000MB/s with dual link (PCIe will be the bottleneck here, assuming a PCIe 3.0 x8 HBA)

It's come to my attention from a colleague, that if I use a bunch of SAS2 drives, I will only get SAS2 speeds overall on the HBA.

With an LSI SAS3 HBA + expander (as used by Supermicro) you can get close to SAS3 speeds with SATA3/SAS2 devices because of the Databolt feature, I could get around 4000MB/s and 5500MB/s with single and dual link respectavely using SATA3 SSDs.
 

sretalla

Powered by Neutrality
Moderator
Joined
Jan 1, 2016
Messages
9,702
From the manufacturer's website:
LSI 9305-24i
Up to 1.5 million IOPs at 4 KB block size
Up to 6.5 GB/s sequential read throughput
 

lonelyzinc

Dabbler
Joined
Aug 8, 2019
Messages
35
To give it some more context, here's the server I'm looking at: link

At this stage I'm just going to buy 36 inexpensive (1TB SAS2?) drives and run disk speed tests. I'm not an expert on storage systems, I just know speed mainly through the NVMe drive on my workstation through copying files and disk speed tests. I want to bring this kinda speed to a storage pool

Will that backplane/HBA combo make the best use of all 36 bays in that system? Or do I need a more powerful backplane/HBA config? I noticed that there is an EL1 and EL2 version of those backplanes. Does that have anything to do with the dual-link that Johnnie Black mentioned?

Cheers guys!
 
Joined
May 10, 2017
Messages
838
Will that backplane/HBA combo make the best use of all 36 bays in that system?

Yes, and both have Databolt support which is good when using SAS2/SATA3 devices, use dual link for more bandwidth, though single link might be enough for 36 1TB disks.

I noticed that there is an EL1 and EL2 version of those backplanes. Does that have anything to do with the dual-link that Johnnie Black mentioned?

EL2 has dual expanders, but that's for failover/multipath, it won't have better performance.
 
Last edited:

lonelyzinc

Dabbler
Joined
Aug 8, 2019
Messages
35
Yes, and both have Databolt support which is good when using SAS2/SATA3 devices, use dual link for more bandwidth, though single link might be enough for 36 1TB disks.



EL2 has dual expanders, but that's for failover/multipath, it won't have better performance.
Thanks for the tips, have gone ahead and ordered!!
 

lonelyzinc

Dabbler
Joined
Aug 8, 2019
Messages
35
SAS3 will give you around 4400MB/s max from a single link, and around 6000MB/s with dual link (PCIe will be the bottleneck here, assuming a PCIe 3.0 x8 HBA)



With an LSI SAS3 HBA + expander (as used by Supermicro) you can get close to SAS3 speeds with SATA3/SAS2 devices because of the Databolt feature, I could get around 4000MB/s and 5500MB/s with single and dual link respectavely using SATA3 SSDs.
Now that the system is running, is there a simple way to see if the Databolt feature is working together on both fronts (HBA & SAS Drives)?
 
Joined
May 10, 2017
Messages
838
Now that the system is running, is there a simple way to see if the Databolt feature is working together on both fronts (HBA & SAS Drives)?

Databolt is just an expander + HBA feature, not drives, you'd need to run some benchmarks that test all drives simultaneously and check if you get more than SAS2 bandwidth, for example I ran some with Unraid (which makes it easier to test total bandwidth during a parity check) and was able to get around 4000MB/s with single link and 5500MB/s with dual link using SATA3 SSDs with an LSI SAS3 HBA + expander combo, considerebly above the max bandwith possible with SAS2/SATA3 devices without Datablot (which is around 2200MB/s and 4400MB/s respectively).
 
Top