BUILD SuperMicro X10SRL-F + 3 846 Chassis + 72 Disks

[SweetAndLow]

Wait a second... :D What do you mean with "excessive standoffs"?

I'm a bit worried right now, because I remember mounting a lot of mainboards with these 1cm tall mounting / standoff things... :D I thought the purpose of them is to establish a tiny distance between the solder joints on the backside of the mainboard and the chassis?

Sometimes standoffs come pre installed in strange places for other motherboard standards. These can interfere with motherboard that are of a different form factor. This causes shorts and can ruin your motherboard.

 

[Scampicfx]

ah okay, that's clear! wow, what a nice forum... 2 replies within seconds... thanks! :D

 

[Stux]

Oh. Well thar's yer problem. I noticed that this was a spectacu-fail when I tried it several years back. SAS wideporting to 8 lanes ought to be possible, but it doesn't seem to work by default. On a 846BE26, with a single SFF8087 to either backplane connector, everything was peachy and would see almost full throughput to all 12 drives (half empty). When connecting the second one, it would fall apart and performance would be 10-50% of a single SFF8087. Since it was for an archival pool, I really didn't care and never bothered to dig into it further. I seem to recall there's an LSI utility that lets you chat with the SAS expander for configuration management purposes. I was guessing that perhaps it needed some special tweaking to make x8 wideport work.

Remove one of the SFF8088's and it'll start to fly. Should be safe to remove hot. Also there's a standing offer that if you actually figure out what needs to happen to make this work, you can get to tell me in what way I was an idiot. ;-) I'm curious, just not curious enough to do the work.

The backplane in each chassis is dual linked to a dedicated LSI controller, so I have 48Gbps bandwidth to each chassis.

I believe LSI 2008 is PCIe Gen 2, and 8 lanes. 500MB/s per lane, total of 32gbps.

You won't get full bandwidth out of it I don't think, and I imagine it might actually not enjoy running at 100% utilization

 

[maglin]

Speaking of SuperMicro LEDs, I really wish my red LEDs would light up when there's a fault detected with a given drive. My understanding is that this is implemented in TrueNAS for SuperMicro chassis. I wonder what the odds are that feature will trickle down into FreeNAS?

I'm pretty sure that is controller by the SAS controller. I believe they end up being OEM specific on a lot of occasions. I only see my red lights flash on initial power up. After that is all blue flashes with disk activity.

But having a drive kick out a SMART error and then the corresponding red light come on would be just amazing for finding that said drive.

 

[SMnasMAN]

awesome thread!! exactly what ive been searching for (but kept google/forum searching for the wrong disk counts and was finding little related).

is there anyway to have the images start showing up again? (ie in OP's post on the 1st page, no images show up). i assume a free image hosting service was used and that has expired maybe? (or is it something on my end?).
thanks

(btw im just starting my build sm x9 48+ disk build later this week:
https://forums.freenas.org/index.ph...4u-disk-shelf-2u-disk-shelf-48-disks-s.71624/)

EDIT- i now see that the images were hosted on what looks like a totally dead site (or hopefully just down temporarily), ie:
http://www.cstone.net/~dk/cosmos-04.JPG
if OP has the picts, i can host them if interested (even at exact same urls if he can point dns to my server).

EDIT2- dang, almost! but wayback machine did not archive the actual images, only the directory listing:
https://web.archive.org/web/20161115010616/http://www.cstone.net/~dk/

 

[danb35]

in OP's post on the 1st page, no images show up

...which is why we recommend that images be uploaded directly to the forum rather than externally hosted.

 

[Ericloewe]

It would seem to be a host which the OP has more control over than your typical imgur or whatever. And it seems to be broken.