BUILD SuperMicro X10SRL-F + 3 846 Chassis + 72 Disks

[Darren Myers]

Well great, now my gf is gonna REALLY complain when i come home with a 42U rack filled with 4U supermicro cases because *SOMEONE* just haddddd to go and make something so awesome :P

 

[pclausen]

Lol! Yeah, my wife is sort of "given me the eye" about all this. And that's putting it nicely. :D

Anyway, I picked up the 4th chassis! :D

The good:
It was free!
24 2TB Hard Drives
Supermicro X8STE Mobo w/ Xeon L5506 & 3 x 2 GB ECC RAM
Newer chassis with 4 pin fans

The Bad:
Areca 1880i RAID controller (bad for FreeBSD, not bad for eBay)

The Ugly:
SAS1 backplane expander
900W Screamer power supplies

Fortunately I had a spare SAS2 backplane that I swapped in. SAS2 on top, SAS1 on bottom. The 3 pin fan connectors are a dead give away to spot the SAS1 backplanes. EDIT: the forum decided to flip the below pic upside down for some reason. Strange...

Disks were an interesting mix of Hitachi Deskstars, WD Greens and Blank (only one unfortunately) and a pair of blacks that were used for boot.

On the greens, I did the wdidle3 trick:

Also updated the firmware on all drives.

96 hot swap bays of glory!

Rear view. Still have some work to do. And of course I still need to copy all my data over from the Windows chassis.

View of drives from FreeNAS:

I hate how FreeBSD "randomize" the da## assignments, but I guess I'll learn to live with it.

But I did manage to put all 10 1TB drives into their own vdev and then have 3 vdevs of 10 2TB drives each. They have a mix of 7200 and 5200 rpm drives, but hey, what can you do.

Now it's time to run smart tests and badblocks on the new drives.

 

[marbus90]

I think I'm a little lost now. Can you create an excel/google sheet which bay has which HDD? As in manufacturer, model, size and in which array it currently is? ;)

 

[DataKeeper]

Isn't "The Look" great at times lol

I'll agree, I dislike how FreeBSD changes the device name at times. Is their an actual reason it does this? Coming from 20years with Linux and other Unix OSs I don't understand why it should change. To me it's so wrong!

 

[pclausen]

The thing that gets me is that if I go into the LSI BIOS on boot, there is a clear rigid relationship between the SAS expander, bay and disk within it.

It would be nice if FreeBSD would interrogate that and assign da## numbers accordingly. My drives are physically installed as follows:

The bays in green are currently populated. Pink are drives that are part of my current Windows drive pool that will be going away once the data had been copied over to FreeNAS. Chassis 3 is not powered up yet since there are no drives in it yet.

I'll take some pictures of the LSI BIOS screen to verify the above and associate serial numbers and full model numbers for each slot.

Chassis 4 is the X7 based one running windows w/ stablebit DrivePool.

A friend of my has a similar setup (2 SuperMicro 846 chassis) with a mix of 2 and 4TB drives. Have about 70% overlap on media. He has agreed that we should sync our collections. This way we will each have a full backup of what the other person has. We are in the processing of mailing a 5TB USB3 drive back and fourth until we are synched up.

He runs Windows 7 with Areca RAID6 controller where I will be running FreeNAS. I won't be able to convince him to switch to FreeNAS. I'm thinking Bacula could be an option for us to do 2-way syncs once our datasets get close to being the same. Is there anything else I should consider that might be easier to stand up than Bacula to keep about 60TB of data synched up between 2 remote locations?

 

[marbus90]

btsync probably.

 

[pclausen]

I thought btsync was only between my server and the cloud, not server to server? If I can do server to server without having to pay a monthly fee, that would be great. Will definitely have to look into that!

 

[DataKeeper]

Ive been chatting to a friend in Canada about the same thing. He'd be using RH or Debian Linux. Not for everything but syncing our most important files between each other. Likely no more than 4-6TB in my case.

 

[jgreco]

Isn't "The Look" great at times lol

I'll agree, I dislike how FreeBSD changes the device name at times. Is their an actual reason it does this? Coming from 20years with Linux and other Unix OSs I don't understand why it should change. To me it's so wrong!

Because the process for scanning the bus is asynchronous. Unless you've wired down the drives, it assigns them in order they're detected. These days, with gpt labels, this really isn't an issue, but in the old days, yeah, I remember wiring down 72 drives... heh.

 

[pclausen]

How do you "wire down" drives?

Anyway, I went into the LSI BIOS and as it turns out, some of my drives were not in the bays I thought they were. After given it some thought, I decided to retire my 10 1TB drives since I really don't need them to complete the initial transfer. So with those out of the way, and rearranging the rest in neat order, here's what the BIOS is showing:

Chassis 1

Chassis 2

It took FreeNAS a long time to boot, it was wondering what happened to the 10 1TB drives that made up a vdev in the zpool. It was probably also really worried about the 2TB drives that had been swapped between vdevs as well. :D

It finally came up and the disk assignments almost match the physical in the BIOS. I also went ahead and keyed in descriptions for each drive that matched what the LSI BIOS had.

One of the Seagate drives slipped into da24 and shifted the rest down by one. Not too bad.

Ran short SMART and conveyance (which the Hitachi don't support) and am in the process of running the long SMART test on all the drives now. I know I had initially tested some of them, but with all the moving disks around and what not, I figured I'd hit them all again. This will also give me an opportunity to see if any of the SMART values have increased since the original test. Of course with all the 1TB Seagates out of the picture, there should be very few SMART errors left in the fold.

Hopefully I'll get a chance to kick off badblocks on all 30 drives before heading out in the morning. And then, at long last, I can stand up the first 3 production vdevs in my pool and start copying some data for real.

 

[pclausen]

Badblocks kicked off on the 30 2TB drives:

Its pretty obvious which chassis is #1 and #2. :D

I moved the 10 1TB drives to the 3rd chassis down from the top. You can see them not pushed in all the way in bays 15-24. I also peeled off all those bay # stickers from the trays.

Drive temps holding in the low to mid 30's for the Hitachi in chassis #1 and in the upper 20's for the newer drives in chassis #2.

Does that 2nd number at the end indicate the max temp the drive has been exposed to? If so, I got a couple that have seen upper 50's...

 

[jgreco]

How do you "wire down" drives?

Kernel config. You wire scbus's to specific controllers, then drives to the scbusses. This totally sucks because it's wiring in specific hardware knowledge.

Code:

#
# Configuration for DELL PowerEdge 4250 with 39160's
#
#XXDELLXX#device                scbus0  at ahc6 # SCSI bus (required)
#XXDELLXX#device                scbus1  at ahc0 # SCSI bus (required)
#XXDELLXX#device                scbus2  at ahc1 # SCSI bus (required)
#XXDELLXX#device                scbus3  at ahc2 # SCSI bus (required)
#XXDELLXX#device                scbus4  at ahc3 # SCSI bus (required)
#XXDELLXX#device                scbus5  at ahc4 # SCSI bus (required)
#XXDELLXX#device                scbus6  at ahc5 # SCSI bus (required)
#XXDELLXX#device                scbus7  at ahc8 # SCSI bus (required)
#XXDELLXX#device                scbus8  at ahc9 # SCSI bus (required)
#XXDELLXX#device                scbus9  at ahc7 # SCSI bus (required)
#
# Configuration for standard AHC systems
#
#XXAHCXX#device         scbus0  at ahc0 # SCSI bus (required)
#XXAHCXX#device         scbus1  at ahc1 # SCSI bus (required)
#XXAHCXX#device         scbus2  at ahc2 # SCSI bus (required)
#XXAHCXX#device         scbus3  at ahc3 # SCSI bus (required)
#XXAHCXX#device         scbus4  at ahc4 # SCSI bus (required)
#XXAHCXX#device         scbus5  at ahc5 # SCSI bus (required)
#XXAHCXX#device         scbus6  at ahc6 # SCSI bus (required)
#XXAHCXX#device         scbus7  at ahc7 # SCSI bus (required)
#XXAHCXX#device         scbus8  at ahc8 # SCSI bus (required)
#XXAHCXX#device         scbus9  at ahc9 # SCSI bus (required)
#
# Configuration for standard AHD systems
#
#XXAHDXX#device         scbus0  at ahd0 # SCSI bus (required)
#XXAHDXX#device         scbus1  at ahd1 # SCSI bus (required)
#XXAHDXX#device         scbus2  at ahd2 # SCSI bus (required)
#XXAHDXX#device         scbus3  at ahd3 # SCSI bus (required)
#XXAHDXX#device         scbus4  at ahd4 # SCSI bus (required)
#XXAHDXX#device         scbus5  at ahd5 # SCSI bus (required)
#XXAHDXX#device         scbus6  at ahd6 # SCSI bus (required)
#XXAHDXX#device         scbus7  at ahd7 # SCSI bus (required)
#XXAHDXX#device         scbus8  at ahd8 # SCSI bus (required)
#XXAHDXX#device         scbus9  at ahd9 # SCSI bus (required)
#
# Configuration for standard NCR systems (boot on AHC)
#
#XXNCRXX#device         scbus0  at ahc0 # SCSI bus (required)
#XXNCRXX#device         scbus1  at ncr0 # SCSI bus (required)
#XXNCRXX#device         scbus2  at ncr1 # SCSI bus (required)
#XXNCRXX#device         scbus3  at ncr2 # SCSI bus (required)
#XXNCRXX#device         scbus4  at ncr3 # SCSI bus (required)
#XXNCRXX#device         scbus5  at ncr4 # SCSI bus (required)
#XXNCRXX#device         scbus6  at ncr5 # SCSI bus (required)
#XXNCRXX#device         scbus7  at ncr6 # SCSI bus (required)
#XXNCRXX#device         scbus8  at ncr7 # SCSI bus (required)
#XXNCRXX#device         scbus9  at ncr8 # SCSI bus (required)

device          da0  at scbus0 target  0 unit 0
device          da1  at scbus0 target  1 unit 0
device          da2  at scbus0 target  2 unit 0
device          da3  at scbus0 target  3 unit 0
device          da4  at scbus0 target  4 unit 0
device          da5  at scbus0 target  5 unit 0
device          da6  at scbus0 target  6 unit 0
device          da8  at scbus0 target  8 unit 0
device          da9  at scbus0 target  9 unit 0

device          da10 at scbus1 target  0 unit 0
device          da11 at scbus1 target  1 unit 0
device          da12 at scbus1 target  2 unit 0
device          da13 at scbus1 target  3 unit 0
device          da14 at scbus1 target  4 unit 0
device          da15 at scbus1 target  5 unit 0
device          da16 at scbus1 target  6 unit 0
device          da18 at scbus1 target  8 unit 0
device          da19 at scbus1 target  9 unit 0

device          da20 at scbus2 target  0 unit 0
device          da21 at scbus2 target  1 unit 0
device          da22 at scbus2 target  2 unit 0
device          da23 at scbus2 target  3 unit 0
device          da24 at scbus2 target  4 unit 0
device          da25 at scbus2 target  5 unit 0
device          da26 at scbus2 target  6 unit 0
device          da28 at scbus2 target  8 unit 0
device          da29 at scbus2 target  9 unit 0

device          da30 at scbus3 target  0 unit 0
device          da31 at scbus3 target  1 unit 0
device          da32 at scbus3 target  2 unit 0
device          da33 at scbus3 target  3 unit 0
device          da34 at scbus3 target  4 unit 0
device          da35 at scbus3 target  5 unit 0
device          da36 at scbus3 target  6 unit 0
device          da38 at scbus3 target  8 unit 0
device          da39 at scbus3 target  9 unit 0

device          da40 at scbus4 target  0 unit 0
device          da41 at scbus4 target  1 unit 0
device          da42 at scbus4 target  2 unit 0
device          da43 at scbus4 target  3 unit 0
device          da44 at scbus4 target  4 unit 0
device          da45 at scbus4 target  5 unit 0
device          da46 at scbus4 target  6 unit 0
device          da48 at scbus4 target  8 unit 0
device          da49 at scbus4 target  9 unit 0

device          da50 at scbus5 target  0 unit 0
device          da51 at scbus5 target  1 unit 0
device          da52 at scbus5 target  2 unit 0
device          da53 at scbus5 target  3 unit 0
device          da54 at scbus5 target  4 unit 0
device          da55 at scbus5 target  5 unit 0
device          da56 at scbus5 target  6 unit 0
device          da58 at scbus5 target  8 unit 0
device          da59 at scbus5 target  9 unit 0

device          da60 at scbus6 target  0 unit 0
device          da61 at scbus6 target  1 unit 0
device          da62 at scbus6 target  2 unit 0
device          da63 at scbus6 target  3 unit 0
device          da64 at scbus6 target  4 unit 0
device          da65 at scbus6 target  5 unit 0
device          da66 at scbus6 target  6 unit 0
device          da68 at scbus6 target  8 unit 0
device          da69 at scbus6 target  9 unit 0

device          da70 at scbus7 target  0 unit 0
device          da71 at scbus7 target  1 unit 0
device          da72 at scbus7 target  2 unit 0
device          da73 at scbus7 target  3 unit 0
device          da74 at scbus7 target  4 unit 0
device          da75 at scbus7 target  5 unit 0
device          da76 at scbus7 target  6 unit 0
device          da78 at scbus7 target  8 unit 0
device          da79 at scbus7 target  9 unit 0

device          da80 at scbus8 target  0 unit 0
device          da81 at scbus8 target  1 unit 0
device          da82 at scbus8 target  2 unit 0
device          da83 at scbus8 target  3 unit 0
device          da84 at scbus8 target  4 unit 0
device          da85 at scbus8 target  5 unit 0
device          da86 at scbus8 target  6 unit 0
device          da88 at scbus8 target  8 unit 0
device          da89 at scbus8 target  9 unit 0

device          da90 at scbus9 target  0 unit 0
device          da91 at scbus9 target  1 unit 0
device          da92 at scbus9 target  2 unit 0
device          da93 at scbus9 target  3 unit 0
device          da94 at scbus9 target  4 unit 0
device          da95 at scbus9 target  5 unit 0
device          da96 at scbus9 target  6 unit 0
device          da98 at scbus9 target  8 unit 0
device          da99 at scbus9 target  9 unit 0

As you can see despite SCSI supporting 15 devices, we only set up for 10, because the shelves we used only supported 10 (full height 3.5"), and because it made the numbering work out such that "daXZ" identified X as shelf and Z as drive.

But the problem is that depending on the types of controller deployed, you might need to futz around. For example, the internal controller on the Dell 4250 would show up in an odd spot in the probe sequence, so the scbus assignment wiring for that was a little funky.

Things are much better now since there are better ways to identify drives in the modern era. If you were to take two of the drives in the system above and swap them, this would be a real problem because the ccd config and fstab would need to be modified in order to configure things correctly. Now, with modern GPT, the drives can be assigned labels and it doesn't matter where a drive shows up - as long as it is attached and labeled, it is all good.

Don't get me wrong - I cut my teeth on 4BSD and SysV style systems so I was always real comfortable with things like "daX" naming, but over the years I've come to appreciate that it can go train wreck very quickly when you fail to anticipate something happening. For example, in the above kernel config, the primary reason we did this wasn't actually to get pretty device numbering (what this thread seems to be about), but rather so that when da23 failed, and the system rebooted, all the higher number drives would be reduced by one, and the system would catastrophically fail because the ccd concatenations that added them into filesystems were adding the wrong devices into the wrong places. The pretty device names were just a happy side effect.

 

[pclausen]

Yeah, that doesn't sound like fun at all!

So with FreeNAS 9.3, when I assign disks to VDevs, even though I'm doing it by picking daX numbers, behind the scenes some other distinct ID, such as the serial number is used?

So if I was to swap a couple of drives around, the DAx values would change, but FreeNAS would be smart enough to still properly associate disks with the correct VDevs based on serial number or some other marker?

 

[jgreco]

As long as you're doing it through the GUI, yes. We used to have problems with people trying to manipulate ZFS from the CLI using "posts they found on the intarwebz" and that has some significant potential for risk. If you pull a "zpool status" you'll note the components are identified by GPT ID.

Code:

          raidz3-0                                      ONLINE       0     0     0
            gptid/73b10d5c-28b6-11e3-a1b2-005056b349b2  ONLINE       0     0     0
            gptid/746b1991-28b6-11e3-a1b2-005056b349b2  ONLINE       0     0     0
            gptid/752b8b61-28b6-11e3-a1b2-005056b349b2  ONLINE       0     0     0
            gptid/75e6a66d-28b6-11e3-a1b2-005056b349b2  ONLINE       0     0     0
            gptid/7b94555a-28b6-11e3-a1b2-005056b349b2  ONLINE       0     0     0
            gptid/77528cf6-28b6-11e3-a1b2-005056b349b2  ONLINE       0     0     0
            gptid/7807e3f6-28b6-11e3-a1b2-005056b349b2  ONLINE       0     0     0
            gptid/78c1f204-28b6-11e3-a1b2-005056b349b2  ONLINE       0     0     0
            gptid/797ee920-28b6-11e3-a1b2-005056b349b2  ONLINE       0     0     0
            gptid/7a1a0ccf-28b6-11e3-a1b2-005056b349b2  ONLINE       0     0     0
            gptid/7ac74f21-28b6-11e3-a1b2-005056b349b2  ONLINE       0     0     0

Those devices will populate into /dev/gptid/ using those identifiers regardless of where you plug them into your FreeNAS box ... SAS ports, SAS expanders, SATA ports, even USB.

 

[Bidule0hm]

Does that 2nd number at the end indicate the max temp the drive has been exposed to? If so, I got a couple that have seen upper 50's...

Sadly, yes.

So if I was to swap a couple of drives around, the DAx values would change, but FreeNAS would be smart enough to still properly associate disks with the correct VDevs based on serial number or some other marker?

Yes, no problem with swapping drives around.

Also you might find my drive identification script useful, see the useful scripts link in my signature ;)

 

[pclausen]

Glad to hear FreeNAS is quite robust when when it comes to moving drives around.

That script link you your sig is awesome! I'll definitely be setting some/most of those up along the way.

3 of my 30 2TB drives completed the badblocks test earlier today after about 30 hours or so. The remaining 27 are still chucking along after close to 42 hours now.

The 3 that completed already (2 Seagate ST2000VN000 5,400 RPM NAS and 1 WD Black) have 2 things in common:

1. They talk to the SAS expander backplane at 6Gbps
2. They were all in the 2nd chassis that only has 6 drives in it

I do have a one more 6Gbps SATA drive in bay 24 on chassis 1 that is still running. (Hitachi HDS72302).

I never really paid much attention to the 3Gbps vs 6Gbps SATA speed in the past since both are higher than what any 7,200 RPM drive can actually read or write at, but maybe there are other advantages to having a 6Gbps SATA link than raw read/write performance?

 

[DataKeeper]

Because the process for scanning the bus is asynchronous. Unless you've wired down the drives, it assigns them in order they're detected. These days, with gpt labels, this really isn't an issue, but in the old days, yeah, I remember wiring down 72 drives... heh.

I remember having someone explain the wiring down drives to me to which I replied.. no, No & NO! :D My last dealings with BSD was with 4.0 and upgrading to 4.1 (which had no rescue diskette option btw) and after that was able to switch back to Debian.

 

[DataKeeper]

...3 of my 30 2TB drives completed the badblocks test earlier today after about 30 hours or so. The remaining 27 are still chucking along after close to 42 hours now...

hehe All my 4TB drives ran iirc roughly 72 hours and ended within minutes of each other. The 6TB drives took about 96 hours.

ohh and yes.. Bidule0hm's scripts are great :)

 

[jgreco]

Glad to hear FreeNAS is quite robust when when it comes to moving drives around.

[...]

I never really paid much attention to the 3Gbps vs 6Gbps SATA speed in the past since both are higher than what any 7,200 RPM drive can actually read or write at, but maybe there are other advantages to having a 6Gbps SATA link than raw read/write performance?

Not really. Slightly lower latency, but that's down in the noise.

 

[jgreco]

I remember having someone explain the wiring down drives to me to which I replied.. no, No & NO! :D My last dealings with BSD was with 4.0 and upgrading to 4.1 (which had no rescue diskette option btw) and after that was able to switch back to Debian.

Well, yeah, that was 35 years ago and you had to bootstrap 4.1 BSD from tape.

Maybe you meant FreeBSD 4.1, but that doesn't make any sense, because FreeBSD 4.1 definitely had a fixit floppy. ftp://ftp.sol.net/pub/FreeBSD/releases/i386/4.1-RELEASE/floppies