Resource icon

Don't be afraid to be SAS-sy ... a primer on basic SAS and SATA

Status
Not open for further replies.

brando56894

Wizard
Joined
Feb 15, 2014
Messages
1,537
I'm going to start upgrading my NAS in the near future since I finally have a job again and I was wondering if it would be worth it to switch from SATA to SAS since I'm buying new HDDs anyway and my case already has a SAS backplane built in (I know I have to buy a controller PCI card). My current setup is as follows

Hardware
AsrockRack C2750D4I Intel Avoton
Silverstone CS-DS380B Case
2x8GB DDR3 1600 MHz ECC RAM (plan on upgrading to 4x16GB)

Pools
RAIDZ: 3x3TB WD Green and 1x3TB HGST NAS
Stripe: 2x1TB WD Red
Single: 1x750 GB 2.5" HDD (replacement for my nearly dead 128 GB Crucial M4 SSD, plan on replacing it with 2x32 or 64 GB Striped SSDs to help with wear leveling)


I'm going to replace the 4 3 TB drives with 5 or 6 8 TB drives and will probably convert it back over to RAIDZ2, but that's going to take a few months since it's going to cost a few grand haha. Before I go out and buy all these drives I'd like to get some opinions since I probably won't be doing an upgrade as large as this for years to come (it's going to take a while for me to fill up 30TB+ hahaha) so I want to make the right choices and not regret my decision down the line. This pool will mostly be for storage of HD multimedia (mostly 1080P at the moment, with some 720P. I'll be storing 4K content if I ever get a 4K TV), it's secondary functions will be for backups of anything I have on my PC that I want to save (not much, considering most of it is already on the NAS, a few hundred gigabytes) which includes a folder currently called "Safe Keeping" (the aforementioned few hundred GBs), and will probably include an iSCSI block device or two just for the hell of it. Data will be written to the pool on a daily basis by one or two specific programs, this won't be constant, but sporadically throughout the day for anywhere from 15 minutes to a few hours. Data will be read from the pool by the Kodi Media Centers throughout my apartment (one at a time, there are only two), along with Plex media server which is mostly used by my friends for remote streaming at random times.

The internal space in the case itself is very tight and makes cabling difficult, also there are currently 3 different controllers managing my 10 SATA ports:
Intel® C2750 for 2 x SATA3 6.0 Gb/s, 4 x SATA2 3.0 Gb/s
Marvell SE9172: 2 x SATA3 6.0 Gb/s,
Marvell SE9230: 4 x SATA3 6.0 Gb/s

I've read that people that have this board have hit bottlenecks when using all the ports, I already have 8 used and I'm probably reaching the limit. My thinking is that I could attach the 6x8 TB drive pool via SAS to an LSI PCI-E x8 card, use one SATA port for a SATA DOM (yea I know it's not necessary and it's overkill but so what, this whole build is overkill for my needs hahaha), connect my stripe (which will house my jails) to the Intel SATA 3 ports, then later on if and when it's necessary add an L2ARC and a ZIL to one of the Marvell controllers.
 

leoj3n

Dabbler
Joined
Jan 10, 2014
Messages
18
Here's why I decided to go with SAS-3 over SATA, today.

Quoting, roughly, from https://www.hgst.com/sites/default/files/resources/12GbpsSAS_KeyConsid_WP.pdf:
Unlike SATA, SAS works on a full-duplex basis and provides full bandwidth in both directions, a superior adapter that uses a superior command set. 12Gb/s SAS-3 addresses signal quality through “transmitter training.” SAS drives often have higher RPM rates and MTBF/AFR specs. SCSI has more reliability features than ATA to better protect data integrity. ATA does support queuing to a depth of 32, but SAS drives extend queue depth to 128 giving SAS drives more commands to choose from so they can execute them in the most efficient manner possible, yielding higher throughput. SATA does not currently have a future roadmap beyond 6Gb/s.

I am getting the amazing X10SRH-CLN4F thanks to a hat tip from this homelab build, because it comes with an integrated HBA: LSI 3008 SW controller w/ 8x SAS3 (12Gbps). So I don't need to buy an HBA to use SAS3. The board will also support my preference of E5-2648L v3 and DDR4 2133MHZ 1.2V LRDIMM.

I got lucky this morning and bought four "Seagate Enterprise Capacity 3.5 HDD 6TB 7200RPM 12Gb/s SAS 128 MB Cache" drives from Amazon for the same price as the SATA version @ $279.00. The price of the SAS version temporarily (randomly?) dropped to that of the SATA version while I was browsing, so I took that as a sign to pull the trigger! Maybe they were watching me. Anyways, the SAS3 version has since gone back up to their normal price of $289.00, but I saved $40.00, which makes up for the next obligatory purchase that had me scratching my head and asking around in IRC..

Apparently, to use these drives without a SAS backplane/chassis, I needed an 8087-to-8482 cable to connect the following adapters (EDIT: See UPDATE at bottom of this post):
Interesting documents about the SFF standard:
Amazon supplies two types of cable that can connect the 8087 HBA connector split into four "powered" 8482 drive connectors, solving my connection conundrum:
The quoted document from HGST also mentioned something called "Express Bay", that was an interesting read, but these cheap cables should work with the "superior command set", and drives that are significantly higher RPMs/cache/throughput than WD Reds (5400 RPM - 6 Gb/s - 64MB Cache @ $244.99) or significantly cheaper than WD Red Pro ($337.00) in a familiar 3.5-inch form factor that will fit my old ATX mid-tower.

I'm happy to get full-duplex and the superior command set of SAS3, for essentially the same price of SATA. Additionally, going SAS3 forced me to future-proof my RAM, CPU, and motherboard choices; I'm now ready for a future when SSDs compete with "spinning rust" in price, and I retain the ability to expand my RAM up to 256GB as DDR4 eventually takes over the memory market, and prices become comparable to DDR3.

UPDATE:

The LSI 3008 SW controller on the X10SRH-CLN4F is Mini SAS HD (SFF-8643), not Mini SAS (SFF-8087), so the cable above is NOT what is needed. It probably is what you need if you have a normal HBA that just plugs into PCI Express, but the Mini SAS HD connector looks very different:



So I ended up ordering another cable, from Newegg this time, which has been working well:

LSI LSI00412 0.6 meter Internal Cable SFF8643 to x4 SAS8482 w/power (mini SAS HD to SAS HDD)--Avago Technologies


I also bought 4 sata-to-molex connectors for connecting the molex to my PSU (because I could not find any 8643-to-8482 cable that uses sata power unfortunately):

C2G / Cables To Go 10149 15-Pin Serial ATA Male to LP4 Female Power Cable -6-Inch


There seems to be no issue with this setup so far, after I was able to resolve some trouble flashing the LSI 3008 SW controller to IT mode.
 
Last edited:
Joined
Nov 11, 2014
Messages
1,174
There is one thing I don't understand I did read the SAS-Y sticky to make sure I didn't missed something, but I can't find an answer to it:

Somebody mention 826TQ backplane which I have too, and the drive bays take SAS drives(sas drive connector and pins for the second path are there too) but on the back it has independent sata ports. How that will work then if I want to buy a SAS drives and used them with this backplane ? Let say I have LSI 9211-8i I want to use is that possible ?

With sata drives I just use the sata breakout cable(sff-8087 to 4 sata) from the LSI-9211-8i to the backplane , but with sas drives , how I am gonna be able to have connectivity to sas drive second path with this cable ?

P.S. I have to experience with sas drives , but interested to learn and try them.
 

jgreco

Resident Grinch
Joined
May 29, 2011
Messages
18,681
There's exceedingly little value to the redundant path in the context of FreeNAS in this context. Normally this is mostly an issue on an external chassis, where the opportunity for cables to be bumped, or a redundant NAS head unit to be used, are the issues at play.

Many SAS backplanes do not bother with the redundant connections, except to the extent that they would make sense in a larger topology. That's almost always SAS expander based. So what you'll find is that Supermicro doesn't bother to support the second channel on what's essentially designed to be a "SATA" backplane like the TQ, or, even, the "A" backplanes. You have to move into the realm of the B's, and specifically the ones with dual expander support like the BE26, to get access to the redundant connections.
 
Joined
Nov 11, 2014
Messages
1,174
There's exceedingly little value to the redundant path in the context of FreeNAS in this context. Normally this is mostly an issue on an external chassis, where the opportunity for cables to be bumped, or a redundant NAS head unit to be used, are the issues at play.

Many SAS backplanes do not bother with the redundant connections, except to the extent that they would make sense in a larger topology. That's almost always SAS expander based. So what you'll find is that Supermicro doesn't bother to support the second channel on what's essentially designed to be a "SATA" backplane like the TQ, or, even, the "A" backplanes. You have to move into the realm of the B's, and specifically the ones with dual expander support like the BE26, to get access to the redundant connections.

In this case I am hoping I can at least be able to use sas drives with "TQ", but not having the dual path ? So these pins on the connector for the second path don't go anywhere ?
 

jgreco

Resident Grinch
Joined
May 29, 2011
Messages
18,681
In this case I am hoping I can at least be able to use sas drives with "TQ", but not having the dual path ? So these pins on the connector for the second path don't go anywhere ?

Yes, of course you can. Very little of the Supermicro stuff is SATA-only. However, dual path isn't an option on the TQ. You'd need to move to the BE26 or BE2C, and retrofitting is not for the faint of heart. :smile:
 
Joined
Nov 11, 2014
Messages
1,174
I see. I am gonna have to think it over. Let me ask you this:
With SAS drives and two HBA (LSI9211-8i) is it possible to make freenas that will have redundancy if one of the HBA fails using the dual path SAS drives provide ?
 

jgreco

Resident Grinch
Joined
May 29, 2011
Messages
18,681
I see. I am gonna have to think it over. Let me ask you this:
With SAS drives and two HBA (LSI9211-8i) is it possible to make freenas that will have redundancy if one of the HBA fails using the dual path SAS drives provide ?

Yes, if done correctly.
 
Joined
Nov 11, 2014
Messages
1,174
Yes, if done correctly.

That is a great news!!!:)
What do I need then ?


P.S. For redundancy fan, this is such a wonderful music for my ears.:D
 
Last edited:
Joined
Nov 11, 2014
Messages
1,174
To do a bunch of homework, it sounds like.

Not really the answer I was hoping for but... :(
Perhaps is too complicated to explain, I just felt I am almost there but missing few pieces.
 
Joined
Nov 11, 2014
Messages
1,174
To do a bunch of homework, it sounds like.

Today the AC broke and spend half day working on fixing it, so I didn't have a time to do my homework. Can I borrow yours ?:smile:
 

ss4johnny

Explorer
Joined
Nov 15, 2013
Messages
55
I'm upgrading to a X9SRH-7F motherboard that has SAS ports. My chassis is a Supermicro 826TQ. I currently have 6 SATA drives and I plan to add 6 SAS drives in a separate VDEV when I switch to the new motherboard. I'm a little confused on what types of cables I need for the SAS drives.

The TQ has individual SAS ports. I see a lot of SAS cables for multi-lane and a lot of mini-SAS to internal SAS ones, like

http://www.amazon.com/dp/B018YHSACA/?tag=ozlp-20

But I don't think I need a mini-SAS cable. Any advice?
 

jgreco

Resident Grinch
Joined
May 29, 2011
Messages
18,681
I'm upgrading to a X9SRH-7F motherboard that has SAS ports. My chassis is a Supermicro 826TQ. I currently have 6 SATA drives and I plan to add 6 SAS drives in a separate VDEV when I switch to the new motherboard. I'm a little confused on what types of cables I need for the SAS drives.

The TQ has individual SAS ports. I see a lot of SAS cables for multi-lane and a lot of mini-SAS to internal SAS ones, like

http://www.amazon.com/dp/B018YHSACA/?tag=ozlp-20

But I don't think I need a mini-SAS cable. Any advice?

It's just straightforward SAS. I always kinda hated all the "mini-SAS" and other unusual terminology that gets created to make something that isn't that complicated, complicated. Just grab a bunch of SAS cables and you're good to go. Or SATA cables, which'll work too, except SATA cables are length limited.
 

ss4johnny

Explorer
Joined
Nov 15, 2013
Messages
55
It's just straightforward SAS. I always kinda hated all the "mini-SAS" and other unusual terminology that gets created to make something that isn't that complicated, complicated. Just grab a bunch of SAS cables and you're good to go. Or SATA cables, which'll work too, except SATA cables are length limited.

I just got everything installed and FreeNAS is picking them up just fine. Haven't made the new vdev yet as I'm getting a warning that:

"Firmware version 14 does not match driver version 20 for /dev/mps0. Please flash controller to P20 IT firmware."

I think I see a number of threads on how to fix it, but the explanations seemed a bit complex. Anyone know of a simple explanation of how to fix?
 

Mirfster

Doesn't know what he's talking about
Joined
Oct 2, 2015
Messages
3,215

ss4johnny

Explorer
Joined
Nov 15, 2013
Messages
55
Update the Firmware on your HBA to version 20. There are tons of threads on this with instructions and it can even be done from within FreeNas.

What's frustrating is that what is in the manual is terse and the forums have too much information spread out (including a lot of things that say "There are tons of threads on this with instructions"). The motherboard specifications say I have LSI 2308 and then when I go to the avago website there aren't any download options for that.

Edit: I eventually found
https://forums.freenas.org/index.ph...-with-lsi-2308-9-3-os-firmware-warning.27377/
which is helpful.
 
Last edited:

viniciusferrao

Contributor
Joined
Mar 30, 2013
Messages
192
I've a question about Multipath SAS.

I'm with a machine with a SuperMicro SAS Backplane with support for multipathing, but the cabling from the factory is a little weird. It came with the SAS cables from the HBA directly to the primary ports of the backplane, leaving the secondary ones empty.

The backplane has 8 ports in total, 4 in primary channel and 4 in the secondary channel. The manual shows two different configurations: one is done plugging the two ports of the HBA on the primary channel and a second one with two HBA's with ports on primary and secondary on the classical cross connection architecture.

Well, I was thinking in getting one port of my HBA on the primary connection and the other one on the secondary port. Just that. But what's the difference? Why it came with both cable only on the primary?

Here's the manual for the backplane:
https://www.supermicro.com/manuals/other/BPN-SAS3-846EL.pdf

My model is the EL2, with is the one with multipath.
 

ViciousXUSMC

Dabbler
Joined
May 12, 2014
Messages
49
Thanks for the write up, wish I had found it 2 or 3 weeks ago :) but still nice to re-affirm all the info I found the hard way.
 

ioquatix

Dabbler
Joined
May 9, 2017
Messages
48
If I have a BE1C backplane can I connect the two 2x SFF-8643 from the motherboard to the backplane? Does this increase throughput? Or is this not possible?
 
Status
Not open for further replies.
Top