BUILD SuperMicro X10SRL-F + 3 846 Chassis + 72 Disks

[DataKeeper]

When doing the badblocks tests use the 'screen' program. Ssh in, start screen and work from there. If you haven't used it before google it and enjoy. :) It's an amazing little program that allows you to detach and logout, or reconnect if connection is lost, and reattach to the job/jobs.

 

[Apollo]

Thanks. So basically just create a new single disk vdev in a new zpool, create a datastore within in, and copy the content from my existing datastore into it?

That pretty much it.
What you need is to create a recursive snapshot of your entire pool.
then you can replicate the pool to your newly added single disk by running the following:

This place the destination drive into read-only mode. Replication will still be able to be done, but when you need to access the stored data, it will prevent the state of the dataset to be modified which could prove problematic latter on when incremental replication need to take place. This is not really an issue in your case.

Code:

sudo zfs set readonly=on YYY

Code:

zfs send -vvR XXX@manual-20150502  | zfs receive -vvF YYY

If you create a dataset on the YYY pool called "my_backup_dataset" then the command can be as follow:

Code:

zfs send -vvR XXX@manual-20150502  | zfs receive -vvF YYY/my_backup_dataset

You only need to run this command once, with XXX being the name of your pool you want to replicate and YYY the name of the pool where the replication must be stored. You will have many snapshots of the different datasets but they will be copied as well as the command is using the recursive -R option.
The pool receiving the data must be large enough to account for all your data.
That should do it.

To restore the entire pool in its former glory, you need to run the same command and changing the snapshot source and destination:

Code:

zfs send -vvR YYY/my_backup_dataset@manual-20150502  | zfs receive -vvF XXX

 

[pclausen]

I've been playing around with screen and it's pretty cool. I forced dropped my ssh session and was able to come back and see my top session still running. So I'm set to run my badblocks tests.

Apollo, appreciate the detailed response on how to backup and restore my pool. I'll add that to my ever growing list of reference material. For right now, I ended up just blowing away my pool without backing up anything, since I just had some test data (maybe 1 TB worth of media), and 1 jail that will be pretty easy to re-configure. Besides, I want to change things around a little bit on my 2nd try.

I did find that my Hitachi 2TB Deskstars don't support the conveyance SMART test. A bit odd I think since my much older Seagate ES.2 1TB do support it.

Turns out wired.com shipped that power controller board with "Direct Signature Required", so they are going to re-attempt delivery today so I have to work from home today. I'll survive that I think. :)

 

[pclausen]

My CB3 power controller just showed up along with the mounting bracket.

I was kinda expecting there to be an I/O plate and that the mounting bracket would have allowed it to mount in place of a motherboard so that the RJ45 jack was going through the I/O plate.

From the looks of things, I could have probably saved the $20 on the bracket and just drilled and tapped 4 holes in the bottom on the chassis and installed 4 standoffs.

Oh well, at least I only ordered one of these to start. I got impatient waiting on the 846 chassis from work, so I ordered another one from eBay (same vendor DataKeeper got his from). So I guess I'll end up with 96 hot swap bays by the time it's all done. :D

 

[marbus90]

we're closing in to 5x24bay heh. ;)

 

[pclausen]

Lol, the thought crossed my mind! :D

Decided to play around with the power controller while waiting for the Chassis. Hacked it into this 2U HTPC I have:

Works great, just like a real X10 mobo.

And I did set the jumper to SAS2 as opposed to the SAS3 default:

Once I get it in the proper chassis, it looks like it might be worthwhile to get some I2C cables to go between the power controller and SAS2 backplane to monitor temps and power.

 

[pclausen]

Mail guy just dropped off some more goodies.

LSI SAS9200-8e for connecting the first external chassis via dual-link for 48Gbps.

I could have used my 1015s for this, but it would be messy routing cables between the internal connectors, through a slot on the main chassis, then in through another slot in the external chassis, and then finally to the backplane. This is how I rolled in the past, but maintenance was a pain.

So the external chassis will each get one of these dual-link SFF-8088 to SFF-8087 brackets:

And finally, to connect the all the chassis together, I picked up a 10 pack of these 1 meter SFF-8088 to SFF-8088 cables for real cheap on eBay:

I'm toying with the idea of getting a bunch of used 1TB Green drives (about $20 a pop) and filling up all my bays with those instead of going out and buying expensive 4TB units. Here's one example of a lot of 20 for $390:

http://www.ebay.com/itm/Lot-of-20-W...422?pt=LH_DefaultDomain_0&hash=item567210e976

EDIT: Wow, the price really went up. They were $390 Monday. I'm sure they will roll back around at this price point again.

I'll need to calculate how many years it will take to break even on power consumption using 1TB drives vs 4TB drives. :)

 

[marbus90]

tbh, I'd rather pick these: http://www.ebay.com/itm/171531159397 ;)

Still they seem $/GB wise close to new 3TB Reds. then again... Reds are aimed at smaller NAS systems and show bigger failure rates than Seagate non-NAS drives in Storagepods.

Maybe SAS drives are acceptable as well. The market isn't as big, better chances for a deal.

 

[LongHair4277]

I have a question regarding power for all these external chassis. What will happen to the vdev or zpool if for some reason one of the chassis (24 drives) loses power and is no longer visible to the controller? I am a noob when it comes to FreeNas and ZFS. I know RAID5/6 and for it, that wouldn't be a good thing at all. Thanks Dean

 

[Ericloewe]

I have a question regarding power for all these external chassis. What will happen to the vdev or zpool if for some reason one of the chassis (24 drives) loses power and is no longer visible to the controller? I am a noob when it comes to FreeNas and ZFS. I know RAID5/6 and for it, that wouldn't be a good thing at all. Thanks Dean

If the whole pool goes down, data should be intact. I imagine FreeNAS might not enjoy what's going on and make life a tad harder than it needed to be, strictly speaking.

SLOG inside the host system adds another layer of complexity to the scenario that might cause trouble.

 

[pclausen]

Got my "new" 846 chassis today. I must say I'm very impressed with the condition. Appears to be fully reconditioned. Fans look completely dirt free, as does the power supplies and main chassis. Got the power board installed. Turns out the bracket I got adapts the the JB3 to the hole spacing of the JB1/2.

I'll need some 4-pin fan extension cables (don't feel like soldering extensions onto the existing ones like I did the single rear fan for the main chassis). I'm lazy. :)

Or, I get a I2C cable to see if I can actually control the fan speed for the midplane fans if connected to the SAS2 expander. It does have 4 pin fan connectors, so in theory, that should be possible.

On the Rj45 Ethernet connector, do they make brackets with a female RJ45 on the bracket with a pigtail make RF45 that I can connect to the power board? That would be the cleanest solutions for connecting my network to the controller.

 

[Bidule0hm]

I have a question since I saw this power board: if you connect the 24 pins ATX connector to this board then what do you connect to the MB to power it?

As for the RJ45: if you know how to use a soldering iron then do your own with an old ethernet PCI card and some ethernet cable. A less DIY (and probably cleaner) solution would be to find a female/female coupler and a regular patch cable as I don't think a female/male cable exists.

Edit: ok, actually I was wrong, they do exist :D There's even some with threaded inserts like this one for example ;)

 

[Apollo]

I have a question since I saw this power board: if you connect the 24 pins ATX connector to this board then what do you connect to the MB to power it?

I don't think this box is intended at host a MB, OP mentioned he would connect across boxes with SFF8088 extenders.
Such a big baox for such a tiny board. Quite a waste of space in my opinion, as the drives are going to be quite cramped.

 

[Bidule0hm]

Oh ok, I need some sleep :P

Yeah, we need to find something to put in this empty space. I'd say you can put your coffee to keep it warm... :D or... more drives; you never have too much drives...

 

[DataKeeper]

I'm thinking there's room for about 36 extra drives inside that thing with the power controller mounted vertical next to the PSUs :D

 

[marbus90]

I'm thinking there's room for about 36 extra drives inside that thing with the power controller mounted vertical next to the PSUs :D

Supermicro sells 44/45bay JBODs from factory, to be found under the model# 847. 24 in the front, 20-21 in the back with the known hotswap trays. Looks the same from the front as the 846 series, so if you can hide the invoice, the wife wouldn't notice a difference.

 

[DataKeeper]

Yes.. I'm familiar with that model and very happy I decided to pass on it! :D For anyone looking.... Here's the end of your storage problems for $629.00

Also.. Please! Don't encourage this lol I've got 2 Dell U2515H screens, the A1SRi-2758f main-board, 16GB ECC ram, an Intel S3500 SSD & a 2nd 6TB WD RED.

 

[marbus90]

I don't like that exact model since you'd need either expander backplanes (one from the 846, one from the 826) or 5x LSI 9211/9207 HBAs to connect all these SFF-8087 ports. Since it's low-profile only, you'll be stuck with 8port variants - the 9201-16i couldn't be used (and you'd still need 2x 9201-16i + 1x 9211-8i or 9211-4i4e).

A single HBA would be sufficient with Expander backplanes, as you can run 2x SFF-8087 to the front expander and from there 1x SFF-8087 to the rear backplane. As traffic is theoretically split 33/66% between the backplanes, that works out acceptably.

Still, it's not the JBOD SKU which comes with said 45 bays (the space for the mobo/expansion card is used for HDDs as well), expanders and 4x SFF-8088 connectors ;)

 

[pclausen]

Nah, the 847 is old school, the new 847D is where it's at. 90 3.5" in 4U. Each bay holds 2 disks, so 45 x 2 = 90 disks. Also has triple 1280W power supplies.

http://www.supermicro.com/products/chassis/4U/847/SC847DE16-R1K28LP.cfm

BiduleOhm, thanks for that last link. That's exactly what I need. I just need to find a bracket to screw the female RJ45 into and I'll be all set.

 

[pclausen]

Got my 4pin fan extension cables and installed the dual SFF-8088 to SFF-8087 bracket and neatened up the wiring a little in the 1st external chassis.

Power consumption according to IPMI is 36 watt after power on with no drives installed. The fans are running at very low rpm. Then after about 60 seconds they speed up for about 20 seconds, then settle in to the "Optimal" speed as set via IPMI and the power consumption is a steady 44 watts.

I slid in a single Barracuda ES.2 1TB drive and consumption went up to 58W, so 14 watts it seemed. I then slid in 5 more (3 1TB ES.2, a 7200.12 1TB and a 7200.12 750GB, and power climbed to 104 watts. So an average of 10 watts per drive. So fully loaded with these old drives, I should pull about 284 watts, but actual might be a little higher as I suspect the fans will pick up some speed to deal with the heat those drives will generate.

I wonder what the actual consumption is of a modern 4TB red in comparison. 5 watts?