BUILD SuperMicro X10SRL-F + 3 846 Chassis + 72 Disks

[pclausen]

That's what I thought, which is why I'm perplexed why my 6Gbps SATA drives (5,400 RPM at that) are completing their badblocks runs way ahead of the 3Gbps SATA drives, which are all 7,200 RPM.

The single 6Gbps SATA drive in the full chassis completed sometime during the night. All the remaining 26 drives, all 3Gbps SATA, are still chucking along. Currently testing with pattern 0x00 at 75%, so there's still a reading and comparing cycle to complete.

So for whatever reason, the drives with 6Gbps SATA interfaces are somehow completing badblocks much faster than drives with 3Gbps interfaces... Almost twice as fast in fact.

 

[jgreco]

Well, wait. My interpretation of what you posted was that you had some drives in a chassis with only a few other drives and they completed faster. The other drives that were in a full chassis completed slower. That makes sense to me, because at 24Gbps you only have about 1Gbps (100-125MBytes/sec in practice) available for each drive, but modern disks can manage about 150-200MBytes/sec. The real question is what sort of throughput are you ACTUALLY getting to each disk in the fully loaded chassis, because there's not really a guarantee that you can get a full 24Gbps out of a SFF8087.

 

[pclausen]

The backplane in each chassis is dual linked to a dedicated LSI controller, so I have 48Gbps bandwidth to each chassis.

Chassis #1 contains 23 3Gbps drives and one 6Gbps drive. The 6Gbps drive in this chassis finished sometime last night, where the other 23 are still chucking along.

Chassis #2 contains 3 3Gbps drives and 3 6Gbps drives. The 6Gbps drives all finished yesterday mid day.

I agree that I'm unlikely to get the full 24Gbps out of each SFF8087 link, but when dual linked, I would not think there would be a problem for the HBA to get whatever bandwidth it could use.

Just took another look at the completed percentages, and they all over the place:

da0 2%
da1 2%
da2 7%
da3 7%
da4 8%
da5 8%
da6 3%
da7 10%
da8 4%
da9 6%
da10 16%
da11 5%
da12 2%
da13 4%
da14 15%
da15 7%
da16 15%
da17 5%
da18 5%
da19 9%
da20 4%
da21 6%
da22 7%
da23 done (6 Gbps)
da24 done (6 Gbps)
da25 done (6 Gbps)
da26 85%
da27 72%
da28 60%
da29 done (6 Gbps)

da24 - 29 are in chassis #2.

da0 -22 are all identical Hitachi 7200 RPM Deskstar drives. Not sure why there's a variance. None of them are showing any badblocks errors. Maybe I'll find a bunch of smart errors on the slow drives once the tests complete and that's why they are slower? Will be interesting to look at the results once this is all done.

 

[jgreco]

The backplane in each chassis is dual linked to a dedicated LSI controller, so I have 48Gbps bandwidth to each chassis.

Oh. Well thar's yer problem. I noticed that this was a spectacu-fail when I tried it several years back. SAS wideporting to 8 lanes ought to be possible, but it doesn't seem to work by default. On a 846BE26, with a single SFF8087 to either backplane connector, everything was peachy and would see almost full throughput to all 12 drives (half empty). When connecting the second one, it would fall apart and performance would be 10-50% of a single SFF8087. Since it was for an archival pool, I really didn't care and never bothered to dig into it further. I seem to recall there's an LSI utility that lets you chat with the SAS expander for configuration management purposes. I was guessing that perhaps it needed some special tweaking to make x8 wideport work.

Remove one of the SFF8088's and it'll start to fly. Should be safe to remove hot. Also there's a standing offer that if you actually figure out what needs to happen to make this work, you can get to tell me in what way I was an idiot. ;-) I'm curious, just not curious enough to do the work.

 

[pclausen]

Interesting. Unfortunately, the external chassis linked via a pair of SFF-8088's is the one with only 6 drives in it.

However, a few weeks back, I did play with the LSI utility and was able to access the backplane and check the firmware version (they were all very old). I found a command line utility (xflash I think was the name) and I was able to update the firmware on all my backplanes to the latest version.

I'll play around with LSI gui utility some more to see if there's anything there about dual linking.

If I do drop back to just s single link to each backplane, I wonder if there would be a benefit to still dedicate a LSI controller to each, or if I should just run 2 backplanes of a single controller?

 

[pclausen]

I found a SM tool that sees my backplane through the HBA, but it does little more than tell me the firmware version running on it.

I'll dig around the LSI site some more to see what I can find.

 

[Ericloewe]

I'm sure LSI has more comprehensive tools, since the firmware is upgradeable.

 

[pclausen]

Right and I did upgrade the firmware on all my SAS2 backplanes to the latest version. The latest version came out in June 2013 and provided the following bug fixes / improvements:

1. FW 55.14.11.0 set number of power supplies to zero in Server configuration, and set to two in JBOD configuration.
The SES-2 diagnostic page length will therefore be variable. It is found that some application software cannot
support this, so FW 55.14.18.0 set number of power supplies to two in all circumstances while power supplies status
will be shown as "not applicable" in Server configuration.

2. FW 55.14.11.0 allow primary and secondary expanders to monitor all SES elements at same time for HA application.
However, it has a FW bug that only backplanes with certain SAS address could support this. FW 55.14.18.0 get this
fixed.

3. Fix incorrect cross-reference link in few SES-2 diagnostic pages.

4. Fix compatibility issue with Adaptec HBA / RAID controllers.

I was running ancient firmware prior to upgrading.

This link has some low level stuff for communicating with the expander, but it might be a little too deep for me.

http://sg.danny.cz/sg/smp_utils.html

From what I can tell, LSI does not appear to have any tools available to the public to download for their expanders from what I have found so far.

I could have sworn I read somewhere that the Supermicro LSI expanders in the 846's supported dual-linking and that you could plug into any 2 of the 3 available SFF-8087 connectors.

After badblocks completes, will it be ok to power down my system and pull the 2nd SFF-8087 cable running to each backplane, and then power back up and run the long SMART test again before looking at the SMART results?

 

[Darren Myers]

@pclausen it was me and @DataKeeper who have 2 SAS connections, and i actually asked some of the members of [H]ardforums as i found a very old thread about the very dual linking as i wanted to know for sure, the answer i got is found here http://hardforum.com/showthread.php?t=1856746

 

[pclausen]

Thanks Darren. I guess the jury is still out on whether is actually increases performance or not? I went ahead and pulled out the 2nd cable to both the internal chassis as well as the external one. I'll do some testing this way, add them back, and run the same tests again.

I finally got about 60% done copying my data over to my FreeNAS server. I'm at 67% capacity on the pool now and need to add another 10 x 2TB VDev soon. I decided to take a break here and check things out a little before continuing. So here's a storage view:

And using biduleohm's excellent SMART script, I got a nice summary of my SMART status:

What does the question mark next to da10 and da16 indicate?

Decided to kick off a scrub of the volume. Here's what it looked like after 10 minutes:

Not sure what that message about some devices being configured to use non-native block sized is all about? Is this because these older drives are 512 and FreeNAS defaults to 4k blocks? Should I worry about this?

Scrub is running at about 1G/s. Not sure if that is good or bad given my hardware?

CPU util and load both seem nice and low.

All 30 disks looks like these first 2

Assuming the scrub completes without errors, is there any other tests I should perform before I start deleting data I have already copied from the source server to free up more disks to create my 4th 10 x 2TB VDev?

 

[Bidule0hm]

What does the question mark next to da10 and da16 indicate?

It's the warning symbol (can't find better than that, I already use the exclamation mark as the critical symbol). It's here to point out that a drive has maybe a problem and that you should check the values. The exclamation mark is here to tell you "this drive has a problem, you should do something now!". You can change these symbols to use any character you want and you can change the thresholds too (see at the top of the script), the details are in the description of the script in the scripts' thread if you want more infos ;)

Is this because these older drives are 512 and FreeNAS defaults to 4k blocks? Should I worry about this?

Yes. No.

Scrub is running at about 1G/s. Not sure if that is good or bad given my hardware?

I'd say it's pretty good, didn't see many systems that can reach the GB/s. Maybe you should have more given the number of drives you have, wait for an answer from a more experienced member.

Assuming the scrub completes without errors, is there any other tests I should perform before I start deleting data I have already copied from the source server to free up more disks to create my 4th 10 x 2TB VDev?

If you've already tested the system and the drives then it should be ok ;)

 

[pclausen]

Thanks BiduleOhm. I should have actually looked at the script before asking. Duh!

So I'm getting 1 GB/s with 3 10 disk VDevs. I take it that means I'm getting about 333 MB/s per VDev? I'll find out soon enough. This morning I was able to free up 10 2TB drives from the Windows server and expanded my V1 pool with those. Copying data over now from my final set of 10 4TB drives. Once that completes, I'll run another scrub to see if I get 1.33 GB/s. Than add the fifth and final (for now) VDev using the 4TB drives and see if my scrubs run and 1.66 GB/s. Then I'll add in the 2nd SFF-8087 / SFF-8088 cables to each chassis and test again.

With 50 drives and 120TB, I wonder if I should consider upgrading from 32Gig to 64Gig of ECC ram?

 

[jgreco]

Wow, yes, at 32GB you're going to be somewhat stressed. Depends on various factors. If performance suddenly seems to fall off a cliff, adding memory is very likely to be the "fix."

 

[pclausen]

Thanks. I'll keep an eye on performance as I continue to expand.

Once I have all the data copied over and the last VDev added, my plan is to add another datastore to the pool, and move all the data over to it and then delete the original source datastore. That will force the data being spread evenly across all 5 VDevs, correct?

 

[jgreco]

Should!

 

[mjws00]

Nice test of 32GB limits. Not much REAL data on 120TB with that amount. That's double cj's stretch. If things work nicely on a light media style load, that would be a neat data point, also where/when/if it falls off the cliff.

Cool.

 

[pclausen]

Copying ~50TB from datastore media1 to media will be a good test. I'll be sure to start small until I get a feel for how it goes.

I went ahead and created my permanent "media" datastore and set it up exactly the same as the temporary "media1"as far as permissions, etc.

I also created an identical CIFS share, so now have the following:
Path ----------------Name
/mnt/v1/media----media
/mnt/v1/media1---media1

All is well and I can access both from my windows machines.

I have a single plugin installed (Emby) and I have storage associated with it as follows:

Jail -------- Source ------------ Destination
emby_1---/mnt/v1/media1 -- /media

So can I just go ahead and add this additional storage to the existing jail without any issues:

Jail -------- Source ------------ Destination
emby_1---/mnt/v1/media --- /media

Will the jail merge everything from both sources into the single /media destination?

This is only temporary until I delete the /media1 stuff.

I just want to make sure I don't break something at the 11th hour as I'm completing this migration!

 

[pclausen]

Btw, when I expanded the pool by adding the 4th VDev, the CIFS share dropped out temporarily while the Samba service restarted. Had me really concerned for about 30 seconds until it all came back! :D

 

[pclausen]

Snaps of main stats working through the current copy operation with 40 2 TB drives.

 

[Darren Myers]

I found a SM tool that sees my backplane through the HBA, but it does little more than tell me the firmware version running on it.

I'll dig around the LSI site some more to see what I can find.

Where is this utility found, and how did you flash/update your backplane...its one thing i havent done yet.