Hello world! Since I'm pretty sure this is my first post I'll start by saying thanks for all the help I've gotten from this forum over the years... this is certainly one of my, if not my, favorite open source community. Ya'll are just incredible! I've built countless FreeNAS boxes and have had lots of questions as I've educated myself. I would like to believe I'm the world's best googler but sadly, too often I can't find the answer I'm looking for even though I know it's been asked and answered before. I'll start to write the beginning of a new thread here to ask my questions and as I go through the process of researching my post to ensure I'm asking intelligent questions, I find the answer I was looking for. Every. Single. Time. And honestly, this time is no exception but f-it, I'm posting this today because a) I'd actually like to participate in a discussion around here and b) maybe (just maybe) I need some human interaction with being stuck at home so much lately. XD
So, here's what I'm looking to accomplish... I want to build a 24-drive (or greater) system and pull out all the stops such that the only bottleneck are the drives themselves. I'm not concerned about price at all; I can afford COTS solutions but would prefer to use FreeNAS for <reasons>. The drives I'm being given for this are 600GB 15K SAS3 drives (I don't have the specific part/model number yet.) I've never built a system with a SAS3 backplane so that's what I expect this conversation to primarily revolve around. Since these are spinning disks, my assumption is that SAS3 just doesn't matter... but given that they actually make and sell these things (for a considerable premium) maybe I'm not appreciating some aspect of SCSI communication. My understanding is that even the fastest SSDs can't come close to 12Gbps, let alone spinning disks. But does SAS3 allow for considerably higher IOPS as compared to SAS2? Even with spinning disks? I want to engineer this such that if in the future these drives get swapped out for SSDs, or I built additional systems using SSDs from the outset, I will be able to take full advantage of the drives so don't get too stuck on the fact that I'm using spinning disks today.
I built a system (that I made several copies of) around 2013 which used the Supermicro 846XA-R1K23B chassis and mated the six SFF-8087 connectors on the backplane to three LSI 9211-8i RAID controllers. I didn't want to use SAS expanders because of the oversubscription it entails. But now that I think about it, that PCIE bus was oversubscribed which I probably didn't even account for at the time, beyond ensuring each 9211 got a full 8 lanes.
Looking at the 846 chassis it doesn't appear they have a SAS3-passthrough capable chassis... the 846XA-R1K23B only claims support for 'SAS 6Gbps' and the other SAS3 options all use an expander. But the passthrough backplane I understood as being entirely passive; I thought it was literally just an electrical interconnect so unless SAS3 uses previously unused pins, or the physical connecter differs in some way that I'm unaware of, I'm confused as to why the passthrough backplane wouldn't support SAS3. I'm guessing the answer is that given the drive speeds and the speed of the SAS bus, using an expander is just not an issue but I'd love to see some math around that if somebody has done it.
In regards to the 846 chassis, it takes 3.5 inch drives by default. I've previously purchased drive cages for this that support 2.5 inch disks but now I'm not seeing that accessory listed on SuperMicro's site... did they discontinue that? I liked this because I could use either drive size in any slot. If that's gone I'm gonna have to look at something other than the 846 because the current requirement is for 2.5 inch drives.
In terms of HBAs, I loved my 9211s and am thinking the upgrade there is the 9300-8i because I want to stick with LSI/Broadcom given my previous experience, the wide user base, and very good support in FreeNAS. But, at 8-lanes, each card will only get ~8Gbps. With 8 drives per card, and assuming they're high-performance SSDs, we already have a chokepoint, right? Are there any x16 HBAs you would recommend? Also, PCIE 4 is becoming a thing... is this worth looking at?
I went Intel/Xeon in the previous build I mentioned but I'm leaning toward AMD Epyc this time around. Thoughts on that? 128 lanes of PCIE 4, yes please. But if I use HBAs that are only PCIE 3, how does that affect the overall PCIE bus? I haven't looked but are there MBs out there with EIGHT x16 PCIE slots?!?!?
I've purposely not stated the use case thusfar as these tend to move around a lot in my world and my design approach is really to over-engineer the devil out of it so that it can be used wherever it's needed. The most common use case is supporting a vSphere infrastructure, via either NFS or iSCSI with a heavy preference toward NFS for simplicity. But this has the potential to be used as a backup target, a file server, temporary storage for migrations, and 'other duties as assigned.'
I think that's enough to start the discussion. Since this is my first post I also just want to give a shout out to the late great cyberjock, who's posts have been priceless to me over the years. Yes, I know he got into a lot of people's feels and I haven't read every one of his posts, so maybe he got WAY out of line somewhere. But I've always had a soft spot in my heart for the curmudgeonly SME type. I will concede that the vibe here is a bit more positive these days which I think is great. I'll share some related advice that has helped some of my coworkers deal with other coworkers: take solace in the fact that your earnings potential in this field is much higher than theirs. (If that's important to you... if you're a tree-huggin hippie like me who has different success metrics you're likely more tolerant of these people anyway... but there's other silver linings for you too.) Contrary to popular belief, highly skilled engineers are not in short supply these days... highly-skilled engineers who can effectively communicate with business leaders and decision-makers and not piss off their peers? Well, you can write your own check. Just my 2 pennies on that.
And one more thank you to all. If you made it this far in my post, you're what makes this place what it is. Cheers
So, here's what I'm looking to accomplish... I want to build a 24-drive (or greater) system and pull out all the stops such that the only bottleneck are the drives themselves. I'm not concerned about price at all; I can afford COTS solutions but would prefer to use FreeNAS for <reasons>. The drives I'm being given for this are 600GB 15K SAS3 drives (I don't have the specific part/model number yet.) I've never built a system with a SAS3 backplane so that's what I expect this conversation to primarily revolve around. Since these are spinning disks, my assumption is that SAS3 just doesn't matter... but given that they actually make and sell these things (for a considerable premium) maybe I'm not appreciating some aspect of SCSI communication. My understanding is that even the fastest SSDs can't come close to 12Gbps, let alone spinning disks. But does SAS3 allow for considerably higher IOPS as compared to SAS2? Even with spinning disks? I want to engineer this such that if in the future these drives get swapped out for SSDs, or I built additional systems using SSDs from the outset, I will be able to take full advantage of the drives so don't get too stuck on the fact that I'm using spinning disks today.
I built a system (that I made several copies of) around 2013 which used the Supermicro 846XA-R1K23B chassis and mated the six SFF-8087 connectors on the backplane to three LSI 9211-8i RAID controllers. I didn't want to use SAS expanders because of the oversubscription it entails. But now that I think about it, that PCIE bus was oversubscribed which I probably didn't even account for at the time, beyond ensuring each 9211 got a full 8 lanes.
Looking at the 846 chassis it doesn't appear they have a SAS3-passthrough capable chassis... the 846XA-R1K23B only claims support for 'SAS 6Gbps' and the other SAS3 options all use an expander. But the passthrough backplane I understood as being entirely passive; I thought it was literally just an electrical interconnect so unless SAS3 uses previously unused pins, or the physical connecter differs in some way that I'm unaware of, I'm confused as to why the passthrough backplane wouldn't support SAS3. I'm guessing the answer is that given the drive speeds and the speed of the SAS bus, using an expander is just not an issue but I'd love to see some math around that if somebody has done it.
In regards to the 846 chassis, it takes 3.5 inch drives by default. I've previously purchased drive cages for this that support 2.5 inch disks but now I'm not seeing that accessory listed on SuperMicro's site... did they discontinue that? I liked this because I could use either drive size in any slot. If that's gone I'm gonna have to look at something other than the 846 because the current requirement is for 2.5 inch drives.
In terms of HBAs, I loved my 9211s and am thinking the upgrade there is the 9300-8i because I want to stick with LSI/Broadcom given my previous experience, the wide user base, and very good support in FreeNAS. But, at 8-lanes, each card will only get ~8Gbps. With 8 drives per card, and assuming they're high-performance SSDs, we already have a chokepoint, right? Are there any x16 HBAs you would recommend? Also, PCIE 4 is becoming a thing... is this worth looking at?
I went Intel/Xeon in the previous build I mentioned but I'm leaning toward AMD Epyc this time around. Thoughts on that? 128 lanes of PCIE 4, yes please. But if I use HBAs that are only PCIE 3, how does that affect the overall PCIE bus? I haven't looked but are there MBs out there with EIGHT x16 PCIE slots?!?!?
I've purposely not stated the use case thusfar as these tend to move around a lot in my world and my design approach is really to over-engineer the devil out of it so that it can be used wherever it's needed. The most common use case is supporting a vSphere infrastructure, via either NFS or iSCSI with a heavy preference toward NFS for simplicity. But this has the potential to be used as a backup target, a file server, temporary storage for migrations, and 'other duties as assigned.'
I think that's enough to start the discussion. Since this is my first post I also just want to give a shout out to the late great cyberjock, who's posts have been priceless to me over the years. Yes, I know he got into a lot of people's feels and I haven't read every one of his posts, so maybe he got WAY out of line somewhere. But I've always had a soft spot in my heart for the curmudgeonly SME type. I will concede that the vibe here is a bit more positive these days which I think is great. I'll share some related advice that has helped some of my coworkers deal with other coworkers: take solace in the fact that your earnings potential in this field is much higher than theirs. (If that's important to you... if you're a tree-huggin hippie like me who has different success metrics you're likely more tolerant of these people anyway... but there's other silver linings for you too.) Contrary to popular belief, highly skilled engineers are not in short supply these days... highly-skilled engineers who can effectively communicate with business leaders and decision-makers and not piss off their peers? Well, you can write your own check. Just my 2 pennies on that.
And one more thank you to all. If you made it this far in my post, you're what makes this place what it is. Cheers
