My new 48 bay build.

[Chris Moore]

Could the faster pool scrub in the new chassis be due to less vibration ?
There is a video showing the vibration caused by just shouting at a set of discs can cause slower disc accesses...
https://www.youtube.com/watch?v=tDacjrSCeq4

The pool was in a rack-mount chassis before, but this chassis is much more weighty, so I suppose there could be less vibration. I am curious about it. One thing that changed from the old chassis to the new is, I was running all 16 drives direct attached to two SAS controllers before where now there is just one SAS controller connected to the SAS Expander backplanes that the dives connect to. Because I am nowhere near the theoretical speed limit (based on drive count) I didn't expect to see any change, certainly not an improvement.

 

[LTCM]

It's pretty cool. They come on their own little mini-crate.

I was a little surprised which how flexible the drive caddies are. I'm sure it's not a problem, but compared to the OS drive caddies the main ones are much less rigid.

I haven't read the manual yet, does it give you any advice on the order to install drives? I'm guessing it's best to fill an entire row at a time? I don't even have a system to install in it yet as my current box is a proprietary server and it wont fit in this case.

 

[nightshade00013]

Yeah the crate is all cardboard, it's interesting but got tossed the same day, LOL.

The caddies are just plastic aligners from what I can tell. They do kinda click in place with a little friction but no latches, once the lid is on though they won't go anywhere and gravity will hold them down as well..

Not sure on the drives but I would try and spread the load between each backplane. When I get started transplanting I am going to label the drives with Pool, vDev, and Serial so I can keep track of everything.

 

[Chris Moore]

I haven't read the manual yet, does it give you any advice on the order to install drives? I'm guessing it's best to fill an entire row at a time? I don't even have a system to install in it yet as my current box is a proprietary server and it won't fit in this case.

The drive slots on each backplane are numbered from the end that connects to the SAS controller and if you are looking at it from that end, the numbers go left to right across, but it doesn't really matter because FreeNAS enumerates the drives in the order that they come ready. That means they are not numbered in the OS in the order that they are in the backplane. Because of that, I put the last four digits of the serial number on the bracket that holds the drive and it makes it much easier to locate a failed drive.

 

[Chris Moore]

I admit when I am wrong. Apparently, the backplane slots are numbered from the front of the chassis with the higher numbers at the end toward the system board. Here is what I noticed in my log:

Code:

> ses0: da7,pass7: Element descriptor: 'Slot 017'
> ses0: da7,pass7: SAS Device Slot Element: 1 Phys at Slot 17
> ses0:  phy 0: SATA device
> ses0: da8,pass8: Element descriptor: 'Slot 018'
> ses0: da8,pass8: SAS Device Slot Element: 1 Phys at Slot 18
> ses0:  phy 0: SATA device
> ses0: da9,pass9: Element descriptor: 'Slot 019'
> ses0: da9,pass9: SAS Device Slot Element: 1 Phys at Slot 19
> ses0:  phy 0: SATA device
> ses0: da10,pass10: Element descriptor: 'Slot 020'
> ses0: da10,pass10: SAS Device Slot Element: 1 Phys at Slot 20
> ses0:  phy 0: SATA device
> ses0: da2,pass2: Element descriptor: 'Slot 021'
> ses0: da2,pass2: SAS Device Slot Element: 1 Phys at Slot 21
> ses0:  phy 0: SATA device
> ses0: da11,pass11: Element descriptor: 'Slot 022'
> ses0: da11,pass11: SAS Device Slot Element: 1 Phys at Slot 22
> ses0:  phy 0: SATA device
> ses0: da1,pass1: Element descriptor: 'Slot 023'
> ses0: da1,pass1: SAS Device Slot Element: 1 Phys at Slot 23

I have not looked yet to ensure that the physical drive locations correspond properly to these 'Slot' numbers, but I will report what I find when I have time to work on it this weekend. It might be helpful in locating a drive when there are more drives in the system.

 

[diskdiddler]

I don’t know if this has been asked, but why not just use much larger hard drives, than so many? Surely the heat and noise is a concern? As well as the power usage, physical space taken up?

 

[Chris Moore]

I don’t know if this has been asked, but why not just use much larger hard drives, than so many? Surely the heat and noise is a concern? As well as the power usage, physical space taken up?

Because large hard drives only give you mass storage capacity, they don't give you massive speed. I want to have more spindles (drives) so I can have more IOPS. My main storage pool is RAID-z2 and that works great for things like movies and family photos, but I want to create a second pool that is made up as a stripe of mirrors and use it with iSCSI to connect my ESXi system where I run virtual machines. It is a work in progress because I don't have the cash to run out and buy it all at once.
If I had piles of cash, I would just buy a bunch of SSDs and make it really fast, but I will probably end up with a bunch of 1TB HDD. I don't need a lot of storage, it just needs to be fast and that means many disks.

PS. My storage pool is less than 40% full, so I am good on capacity, and when I need more capacity in the main pool, I will go to larger drives instead of adding more drives. I had actually thought of getting 8TB drives and migrating the main pool to fewer drives to save power.

 

[LTCM]

I don’t know if this has been asked, but why not just use much larger hard drives, than so many? Surely the heat and noise is a concern? As well as the power usage, physical space taken up?

I can only speak for myself, but here are my reasons.

It's a high quality case & PSU that will last me a long time.
It's more expensive than a traditional tower case, but not obscenely so. A decent quality tower + PSU would cost what, $175-200?
For me, heat and noise aren't a huge problem. My server sits in my semi-finished/unfinished basement with temps from 55 (winter) to 63 (summer). I run FreeNAS on a Dell 1950 right now, which makes more noise than most jet engines and hasn't been an issue so far.
Having what I consider unlimited disk slots allows for me to procure disks in a wide range of capacities. Meaning I can shop for deals and size the number of drives accordingly; something you cannot do if you are limited by slots constraints. I find that smaller disks are can be more economical at times.
The ability to use the backplane off of a single HBA is a major plus for me.
It comes with rails, which is worth at least a few bucks if you need them.
Cause the chicks dig it (aka Vanity)

In the end, there is no right answer. Everyone has their reasons.

Edit: Chris must have submitted his reply while I was composing mine.

 

[Chris Moore]

Cause the chicks dig it (aka Vanity)

LOL...

My wife said the new server was quieter than the old one. I spend time and effort making the old one quiet and this one was quieter with no work on my part.

 

[LTCM]

LOL...

My wife said the new server was quieter than the old one. I spend time and effort making the old one quiet and this one was quieter with no work on my part.

The layout seems to be incredible for airflow - few cables and no metal drive cages like you'd find in a desktop. Just six fans moving air between the drives.

I'll admit, I took apart some parts of the case. It looks to me that the power supplies have heat sinks which transfer heat to the PSU frames. Which then make contact with the case. From what I can see, they are basically using the case itself as a heat sink to minimize the speed of those tiny little fans. Pretty smart if you ask me.

 

[LTCM]

@Chirs Moore

If you have time, can you answer one more question for me? I was looking at the spec sheet for the Dell H310 and it states the card only supports up to 32 drives in non-RAID. Is that going to cause a problem if I purchase one for this case? I have no plans for anywhere near that many, but could there be an issue given the two backplanes have total spots great than 32?

 

[diskdiddler]

Because large hard drives only give you mass storage capacity, they don't give you massive speed. I want to have more spindles (drives) so I can have more IOPS. My main storage pool is RAID-z2 and that works great for things like movies and family photos, but I want to create a second pool that is made up as a stripe of mirrors and use it with iSCSI to connect my ESXi system where I run virtual machines. It is a work in progress because I don't have the cash to run out and buy it all at once.
If I had piles of cash, I would just buy a bunch of SSDs and make it really fast, but I will probably end up with a bunch of 1TB HDD. I don't need a lot of storage, it just needs to be fast and that means many disks.

PS. My storage pool is less than 40% full, so I am good on capacity, and when I need more capacity in the main pool, I will go to larger drives instead of adding more drives. I had actually thought of getting 8TB drives and migrating the main pool to fewer drives to save power.

You can afford 48 disks so it seems you might have been a good person to actual experiment with maybe some SSHDs or SSDs or something?

It would't been interesting to see the performance of 8x SSHDs or something like that.

I guess different needs for diff people, I got the impression a big CPU, a lot of ram and anything over 12 disks is already going to perform very well, but maybe you have some big big performance needs :)

 

[Chris Moore]

If you have time, can you answer one more question for me?

Any time. Just ask.

Dell H310 and it states the card only supports up to 32 drives in non-RAID. Is that going to cause a problem if I purchase one for this case?

That is what I am using and the limitation is because of the Dell firmware. Once it is flashed with the LSI firmware, that limitation is gone and it will support (if I recall correctly) 128 drives.

EDIT-
I can admit when I am wrong. I just did this sas2ircu 0 DISPLAY

Code:

  Controller type						 : SAS2008
  BIOS version							: 0.00.00.00
  Firmware version						: 20.00.07.00
  Channel description					 : 1 Serial Attached SCSI
  Initiator ID							: 0
  Maximum physical devices				: 255
  Concurrent commands supported		   : 3432
  Slot									: Unknown
  Segment								 : 0
  Bus									 : 1
  Device								  : 0
  Function								: 0
  RAID Support							: No

So, it says 255 is the max number of devices. This includes the SAS controller itself, each of the backplanes and all the connected drives.

 

[Chris Moore]

You can afford 48 disks

I probably won't fill it all the way up and not all at once.

It would't been interesting to see the performance of 8x SSHDs or something like that.

I can buy a lot of HDDs before I spend as much as 8 SSDs. I will probably pickup some used enterprise SAS drives for the iSCSI pool.

 

[nightshade00013]

Even for me right now I have a single 7 drive RaidZ3 pool. It will take a while before I fill 48 drive bays and I will probably end up going to a 12 drive vDev in RaidZ3 with two vDev's with this case at some point. The case I still have everything in would only fit 15 drives and with the SAS and SATA connections I could do 14 drives total without an expander or another SAS controller hence the 7 drive vDev.

The beauty of having a case that will support 48 drives is that it will support any number of drives up to 48, plus a couple more inside the other section. That is hard to beat even if you only use a few. Plus the power redundancy for keeping the system up as long as possible. As I said in a previous post I will keep this case for the rest of my life.

 

[bollar]

The hardest part of connecting the SAS cables was the fact that all the hard drive trays need to come out first. Then the trays that hold the backplanes can be unscrewed (thumbscrews on the front of the chassis) and slide forward a few inches (without disconnecting any of the already connected cables) to allow fingers to get into where the connectors are.

I'm new to backplanes and wonder if there's an advantage to running each backplane on separate cards? I have a Supermicro X9DRD-7LNF4 with dual onboard LSI 2308 SAS controllers. I also have a couple of LSI 9207-8i cards, so my questions are:

1. Is there an advantage on moving the backplane connection to one of the cards?
2. Is there an advantage on running each backplane on a different card?

My guess is I can run it all off the motherboard, but if your experience differs, I can try something different.

Thanks for your advice.

 

[Chris Moore]

I'm new to backplanes and wonder if there's an advantage to running each backplane on separate cards? I have a Supermicro X9DRD-7LNF4 with dual onboard LSI 2308 SAS controllers. I also have a couple of LSI 9207-8i cards, so my questions are:

1. Is there an advantage on moving the backplane connection to one of the cards?
2. Is there an advantage on running each backplane on a different card?

My guess is I can run it all off the motherboard, but if your experience differs, I can try something different.

Thanks for your advice.

My understanding is that if you dual link to a single backplane from one card, it increases the potential bandwidth between the system and the dives, but with spinning disks you won't run into the need for that due to the relatively slow transfer from the disks themselves. If you were doing SSDs, that is a different story. If you use one link from a single card to each backplane, it should be adequate. If you use a second card and do another single link to each backpland, that gives you dual path, for redundancy (if a SAS card fails) but it does not increase bandwidth.

 

[bollar]

My understanding is that if you dual link to a single backplane from one card, it increases the potential bandwidth between the system and the dives, but with spinning disks you won't run into the need for that due to the relatively slow transfer from the disks themselves. If you were doing SSDs, that is a different story. If you use one link from a single card to each backplane, it should be adequate. If you use a second card and do another single link to each backpland, that gives you dual path, for redundancy (if a SAS card fails) but it does not increase bandwidth.

Hm. Thanks!

So I guess I could double link from each of the cards and single link from the motherboard (and watch the whole thing go up in smoke).

Then start replaceibg with SSDs at some point after I win lotto.

 

[Chris Moore]

So I guess I could double link from each of the cards and single link from the motherboard

The backplanes (the ones I have dealt with) only support two "Incoming" connections from a SAS controller. The third connector is always an "Outgoing" connector that is intended to allow for a cascade connection to another SAS expander backplane. I do not profess to be familiar with all the options that may exist, this is just what I have worked with. Each of the 24 port backplanes in this server have a port labeled "In", one labeled "In/Out" and one labeled "Out". I am only using a single link from one SAS controller to the "In" port on each backplane and the performance is as good or better than it was when every drive was directly connected to a SAS controller in the old chassis.

 

[bollar]

Fry's no longer carries SAS cables (who knew?) so I have to wait until Monday to get the mini-SAS SFF-8087 cables. I entertained myself by finding a lifetime's worth of old drives and loading them into the caddies.