BUILD SuperMicro X10SRL-F + 3 846 Chassis + 72 Disks

[DataKeeper]

With a base freenas install in my system with 18 4TB WD Reds I'm sitting at 216w idle. It's peek was 288w.

 

[pclausen]

I wonder how much the motherboard itself draws. 75w? If so, that's about 8 watts per drive. If the board draws 100w, it would be about 6.4w per drive. Regardless, that way way better than my old X7 room heater. I can't wait to shut that down! Once I get my 4th chassis from work, I'll be able to transfer 20 1TB drives over to external chassis #1, run badblocks on the drives, and then build VDevs 3 and 4 and begin data migration from that X7 based chassis and finally shut it down for good.

 

[TXAG26]

The X10SRH-CLN4F with E5-2618L v3, 4x 16GB DDR4 RDIMMS, 1x 120mm fan, and 2x USB 3.0 flash drives (no HDDs/SSDs) draws running from a 450w Seasonic Gold PSU (consumption measured with a Kill-a-Watt):
58w @ bios screen
87w running Memtest86 (1 cpu core active)
118w running Memtest86 (8 cpu cores active)

 

[pclausen]

Thanks for that detail! I have a digital current clamp and a special power cord with hot, neutral and ground exposed so that I can clamp on hot (or neutral) to get current reading. I'll see how those readings compare to the IPMI values. Amazing how much more efficient X10 is compared to X7.

 

[pclausen]

Have a couple of updates to report. I got in 36 new disk trays complete with the plastic inserts for proper air flow for bays with actual drives. Paid $48 with shipping, which works out to like $1.33 a pop. Seemed like too great a deal to pass up. :)

I also swapped my IBM 1015 for an actual LSI SAS9211-8i. I like it better because the SFF-8087 connectors point towards the backplane, so I was able to use my shorter SFF-8087 cables.

Also installed both of my SAS9200-8E HBAs:

And all 3 are playing nicely together it seems:

Couple of shots of my 72 bays stacked together on the floor for some additional testing and burn-in, before I begin actually standing up the production pool.

 

[DataKeeper]

And the lights dim when it's all turned on! Lol

 

[TXAG26]

That is some serious hardware! I'd imagine if it is ever fully loaded you'd need to feed it from multiple AC circuits (pull in another dedicated AC circuit).

Standard AC receptacles are typically fed by 12ga wiring and are attached to 15 or 20 amp breakers, which gives you about 1,800 to 2,400w of real power from the wall (assuming nothing else like lights/other equipment/computers are also feeding from that same circuit).

 

[pclausen]

Standard AC receptacles are typically fed by 12ga wiring and are attached to 15 or 20 amp breakers, which gives you about 1,800 to 2,400w of real power from the wall (assuming nothing else like lights/other equipment/computers are also feeding from that same circuit).

Got that covered in spades. :D

I got 3 20A quad receptacles (each feed from its own 20A breaker) AND dual 30A outlets (each feed from its own 30A breaker). The 30A outlets each feed a 3KW UPS.

One of my 3KW UPS is down at the moment due to a spectacular failure, complete with smoke and sparks. I plan to retire him. All 3 of these UPS are very old, and the batteries in the 1400 Watt one are gone. So I will definitely be doing some much needed maintenance in the power department before deploying my FreeNAS.

Also, I have 400A service feeding a pair of 200A panels. Both of those panels are feed via an Asco 300 automatic transfer switch:

Power routing before Asco 300 deployments:

After:

Also running an Asco Power Manager:

Power Manager has Ethernet module to monitor power and email alarms of any issues related to mains power.

Oh, and the whole house generator that comes online should the main power fail, is a Military grade 4 cylinder diesel. It's a little on the loud side, but it is extremely reliable and built like a tank. Literally. :D

 

[DataKeeper]

You sir.. Have a problem :D

I'll be running 3 new 20A circuits soon to supply power to the server room, separate from the lights. Likely only need 2 but might as well run the third for any future needs. The wife has been on me for the last few years about a whole house generator as I'm usually out of the country whenever she loose power. I said that it didn't bother me. ;) Then ducked the right hook. :D Just about every year the power goes out for a day or two and I've always been away lol. The rest of the year it's fine. Likely go with a NG setup as we have that piped into the house and have a conversion kit for regular fuel if need be.

 

[DrKK]

You have fucking 10 or 12 AWG Romex just sitting out in the open in userspace without conduits sir?

Make sure your insurance company doesn't find out. Amazon.com sells the National Electric Code.

 

[DataKeeper]

10 year old house here with unfinished basement. All 12 AWG Romeo in the basement from the main panel to the furnace, water heater, lights, sump, 10 double outlets are openly run. They are neatly run and secured, not hanging down loose. The panel wiring itself is impeccable. Had the house fully inspected 4 years ago and it passed.

And yes.. Insurance Co got the fully report and photos.

 

[TXAG26]

10 year old house here with unfinished basement. All 12 AWG Romeo in the basement from the main panel to the furnace, water heater, lights, sump, 10 double outlets are openly run. They are neatly run and secured, not hanging down loose. The panel wiring itself is impeccable. Had the house fully inspected 4 years ago and it passed.

And yes.. Insurance Co got the fully report and photos.

I guarantee you that open #4 from your genset to the house wouldn't pass any type of inspection and probably violates half a dozen building/electrical codes. Just an FYI...

 

[pclausen]

You have fucking 10 or 12 AWG Romex just sitting out in the open in userspace without conduits sir?

Make sure your insurance company doesn't find out. Amazon.com sells the National Electric Code.

Lol. It's not like that anymore. That generator pic was from when I first tested the transfer switch running off generator power.

These pics are from when I load tested that unit. :D

 

[pclausen]

I guarantee you that open #4 from your genset to the house wouldn't pass any type of inspection and probably violates half a dozen building/electrical codes. Just an FYI...

Not sure I follow. Are you referring to the grounding? Everything is grounded at the service disconnect. The bond within the 200A panels between neutral and ground bars are lifted to prevent a ground loop. The generator is NOT individually grounded, as per code in my area.

 

[Bidule0hm]

I think he talks about the unprotected cables on the floor on this picture: http://www.cstone.net/~dk/xferswitch-11-6-03.JPG ;)

 

[TXAG26]

Not sure I follow. Are you referring to the grounding? Everything is grounded at the transfer switch. The bond within the 200A panels between neutral and ground bars are lifted to prevent a ground loop. The generator is NOT individually grounded, as per code in my area.

I was referring to the 4 AWG connecting from the house to the genset (you just replied that was a temporary pic, so its all good if that's all in conduit).
Awesome setup btw, most businesses don't even have that type of storage infrastructure! Be sure to let know what type of amperage that setup pulls! I hope you're on an electrical co-op in VA as that would be quite the pricey setup to feed in most parts of the country!

 

[pclausen]

Ah ok. Got it. Yes everything is in conduit at this point. Rigid until the last foot or so where it connects to the generator, that part is flexible conduit, per code.

Also, I mistyped/spoke before. All grounding meet at the service disconnect, not the transfer switch. Here's the layout:

By default, service panels have a bond between Neutral and Ground. You always want to remove this when you introduce a transfer switch upstream of the service panels. If you don't, you have 2 paths to ground, which you don't want.

I apologize if I upset any of you guys, that was not my intent. I should have mentioned that my current generator hookup does not look like the the picture I posted.

 

[pclausen]

I rebuilt surplus military gensets as a hobby/side business and make a few bucks in the process, sometimes. Here's a pic of me unloading that infamous MEP-003A generator in the above pics. I got 2 MEP-802As on the same trailer load.

I'm an IT geek by trade, but it sure is nice to get a break and play with completely (well almost) unrelated stuff sometimes!

 

[TXAG26]

The X10SRH-CLN4F with E5-2618L v3, 4x 16GB DDR4 RDIMMS, 1x 120mm fan, and 2x USB 3.0 flash drives (no HDDs/SSDs) draws running from a 450w Seasonic Gold PSU (consumption measured with a Kill-a-Watt):
58w @ bios screen
87w running Memtest86 (1 cpu core active)
118w running Memtest86 (8 cpu cores active)

So I added 1x SSD, 14x 2TB HDD's (5900/5400 rpm), as well as 5x 120mm fans (0.16amp, 12v) and am up to:
144w FreeNAS (idle).
190w FreeNAS (when doing large network file transfer/backup, RaidZ2, LZ4 enabled).

 

[pclausen]

Wow, that's pretty sweet. Can't wait to shut down my space heater X7 server!

Been crazy busy with other stuff, but finally had a chance to clean up my racks, shuffle things around, and install the new APC 2200 UPS and get the rest of the gear mounted in the racks.

I still have some stuff I haven't put back in, but I'll have plenty of room to grow. :D

Rear view:

Waiting on my 1U Supermicro chassis that will be running pfsense with a pair of front facing RJ45s. It will mount up under the Netgear switch. I'm finally getting fiber out here in the sticks where I live. Have a spot picked out where the ONT is going to be installed. Can't wait to ditch my wireless service that get's maybe 8 Mbps down on a good day off peak. :(

Still waiting for the 4th 846 chassis to be released by accounting at work so that I can finally get my last 2 vdevs (each 10 x 1TB) online and start copying data from my X7 based windows server. If they keep dragging their feet, I might just have to invest and some 4TB Reds and move on. I was really hoping to avoid that expense right now.