Recommendations for a proper rebuild

[techiem2]

I've been running my old 7 system using a standard self-built rack machine with a 4 disk zraid, but I'm starting to think of expanding and would like to do it properly, so I'm thinking of a new server with Freenas 8.
What I'm thinking is:
Basic rack server of some sort with lots of ram (of course) to be the core FreeNAS host.
24 Bay SATA/SAS Rack storage box to connect to it (The NORCO DS-24E on newegg looks interesting to me) for the current disks and for adding new sets of disks.
A Controller card in the FreeNAS system to connect the storage box to (I saw the LSI LSI00192 and LSI00188 and on newegg and they looks interesting, as well as the LSI00276).

I'm looking for recommendations as I haven't dealt with external SAS/Sata boxes before, just an older 4 bay eSATA/USB box that I used in a previous fileserver build.
As I understand it, FreeNAS primarily needs RAM rather than CPU power.

So I guess I'm basically wondering what the recommendations would be for a basic rack server and controller card that could be used together successfully with FreeNAS 8 to hook up to such a box.
I like that Norco since it uses a single connection from it to the host server, but am open to suggestions.

At this point I don't really have any kind of solid budget decided, but I'm kind of thinking $2500 would be a reasonable target for a basic server + storage box + controller.

My plan would be to get it up and running then move my current drives into it to import the ZFS pool (iirc I setup each drive with a single gpt partition and then used those to build the ZFS array).

Along with that, as I understand it a pool can be expanded by adding in similar zraid sets or adding more drives to the existing zraid? I.e. I have a 4 disk zraid, so I could add 4 more disks and expand the zraid, or build a new zraid off them and add them to the existing pool?

Thanks for any input/ideas/clarifications.

Mark II

 

[cyberjock]

You will not be able to import the drives with the GPT partitions into the ZFS pool and keep the data. Simply put, the drive will be "formatted" when you add them to the pool. If you have enough hard drives you could create a zpool, copy the data to the zpool so your remaining hard drives are blank, then add the remaining disks to the zpool.

 

[Joshua Parker Ruehlig]

You can add additional raidz to an existing pool, I recommend each raidz is identical though, in number of disk and size of disk.
Wow $2500 on a FreeNAS box, I wish I could make something like that, that case looks amazing!

 

[techiem2]

You will not be able to import the drives with the GPT partitions into the ZFS pool and keep the data. Simply put, the drive will be "formatted" when you add them to the pool. If you have enough hard drives you could create a zpool, copy the data to the zpool so your remaining hard drives are blank, then add the remaining disks to the zpool.

So FreeNAS 8 requires the drives to be unpartitioned and won't import the old zpool made on gpt partitioned disks?
I was kinda afraid that might be the case...that makes things a bit more complicated (6TB is a fair bit of data to find somewhere to temporarily store while you rebuild your array.....).

 

[techiem2]

You can add additional raidz to an existing pool, I recommend each raidz is identical though, in number of disk and size of disk.
Wow $2500 on a FreeNAS box, I wish I could make something like that, that case looks amazing!

Ok that's what I thought. Mine would be identical in number of disks, but wouldn't the same size raidz (new raidz would likely be 3TB disks instead of 2TB disks like the current raidz in the pool).

 

[cyberjock]

You can add additional raidz to an existing pool, I recommend each raidz is identical though, in number of disk and size of disk.
Wow $2500 on a FreeNAS box, I wish I could make something like that, that case looks amazing!

Thanks for correcting me. I could have sworn the OP had said that he was using Windows Server 2008. In which case he wouldn't have a zpool but just GPT partitions formatted NTFS. Maybe I had read it in another thread. /shrug

 

[techiem2]

Thanks for correcting me. I could have sworn the OP had said that he was using Windows Server 2008. In which case he wouldn't have a zpool but just GPT partitions formatted NTFS. Maybe I had read it in another thread. /shrug

Right, just a standard FreeNAS 7 setup. 4 disks with a single GPT partition created on them and then setup as a raidz in the zpool (I think I read here somewhere that settings up a GPT parttion rather than passing the raw disk was the proper way to do things...but I could be wrong). Does that mean I CAN import the existing pool into FreeNAS 8?
My current plan is to get a controller card and the drive bay for my existing box, then eventually replace the box itself with something sleeker.

 

[Joshua Parker Ruehlig]

I think you can import your old pool. gpt/mbr doesn't matter. my disk were gpt before they went into my freenas machine and have stayed gpt. If you do import your old pool I suggest you run 'zpool upgrade' to upgrade the zpool version to 15.

 

[techiem2]

I think you can import your old pool. gpt/mbr doesn't matter. my disk were gpt before they went into my freenas machine and have stayed gpt. If you do import your old pool I suggest you run 'zpool upgrade' to upgrade the zpool version to 15.

Ok I'll probably give that a shot and see.
I'm planning to switch from running it on an old quantum bigfoot to a cf card in a sata adapter like I'm doing with my build over at the church/school anyway, so I'll get that and install 8 to it and see if it sees the existing pool.

 

[techiem2]

Ok. The rebuild is going well so far. I got the CF card and adapter and installed Freenas8 to it in KVM while sitting at work with my laptop one night. :P
When I switched the main box over I had to force import the pool and then reboot. Once I did that the auto import in the gui worked to add the pool to it.
Everything seems to be back up and running now and configured how I want.
One thing I noticed is that apparently user home dirs have to be configured to be inside your pool or else they disappear on reboot (I tried /mnt/home/blah and that went poof on reboot).
I also added a second NIC (Intel Pro-1000 GT) to it (in addition to the onboard NIC) to get the server on the main lan subnet as well as the server subnet to get my main lan traffic going direct instead of through the router/firewall/etc. box (most of my transfers to/from it come from the main lan subnet anyway) and this made a HUGE difference in my transfer speeds (I was rarely breaking 20MB/s when going through the router, but now my transfers are starting around 20MB/s and tpyically up to 40-50MB/s on large transfers).
I've ordered the SAS controller card (the LSI 9200-8e) to add in. I'll add the drive bay when I can afford it and then work on adding sets of 4 disks as I can afford a set.

NOTE TO ANYONE WANTING THE LSI 9200-8E:
This card was recently discontinued (March 22 according to the CDW product page).
I was going to put off buying it for a month or two until I found that out (due to newegg popping in and out of stock over a couple days and then their status on it changing to discontinued).
Newegg is now out of stock and it seems the price is starting to go up at the places that still have some.
If you want one, buy it quickly (buy.com itself had some as of when I ordered this weekend, but the other places they list as selling them may or may not - Tiger Direct was the default seller but apparently did not actually have any - "The quanity you want isn't available..." - it looked like a seller or two on Amazon had a few left as well).

 

[Joshua Parker Ruehlig]

glad everything's going well. any reason you choose CF over usb? (maybe CF over IDE is better then usb2.0? never though about that..)

Also got a Intel Pro-1000 GT for $10 shipped from China on ebay, the specific chipset has a bug in freebsd 7.2+ drivers. I ended up just forking out another $20 for a Intel PWLA8490MT PRO/1000 MT and I hope I wont experience the same driver hiccups.

 

[techiem2]

I'm running CF over SATA.
Mainly because 1. 8GB CF cards are cheap and 2. I like it mostly internal and hooked up as a normal disk rather than fooling around with USB (making sure you are booting usb first and making sure things are setup in a way the flash drive won't get bumped....)

hmm...Maybe it's an MT not a GT...not sure. But it's working great so far (and I've dumped a fair amount of data across the critter so far lol). :P

 

[techiem2]

I upgraded the server to 16GB RAM since it was having memory issues as the array got near full (and I have no swap due to the install on a CF card, need to toss an old drive in at some point to use for swap just in case).
I finally got the Norco DS-24E enclosure and it is nice!
I moved the 4 array disks from the fileserver into the enclosure, plugged it into the LSI 9200-8e card, and booted it up. In the process I found that one of the disks was misbehaving, so I ordered a replacement for that and pulled the disk out (more on that later). Freenas booted up and all was well, except for the degraded array due to the missing disk. When I got the replacement disk I put it in the enclosure and the disk detected messages popped up on the console. At this point I did the zpool replace to replace the removed disk with the new one, and then zpool detach to remove the old disk when it was done resilvering (apparently this is supposed to be done automatically but doesn't always for some reason).
So the server is happily running now using the DS-24E on the LSI 9200-8e.
My current plan is to add a set of 4 3TB disks to the enclosure to add another 9TB to the array (hopefully this year).
After that I will need to replace the server itself since the RAM is maxed and I'll need to increase that before adding more storage.
I'm thinking a nice 1U with 64GB RAM would work well, since a full enclosure of 3TB disks would give me 54TB of storage, and that would take me quite a while to acquire all the disks for and then to fill up, I hope (and after that I'd need another enclosure soooooo).

So that's where the server is at for now.
Now about that disk....

The particular disk (a Hitachi 2TB) has always been a bit clicky, but always seemed to work fine, so I didn't think much of it.
When I moved it to the enclosure and started testing, I found that the LSI sometimes would and sometimes wouldn't see the disk when the card initialized. When it DID see the disk, Freenas would get stuck during boot, apparently trying to do some initialization on it. I think Freenas also got stuck when the controller didn't see it at initialization and was somehow trying to look at that slot anyway. As soon as I pulled the disk out of the enclosure everything booted fine.
I've been testing the disk a bit and have found that it seems that when the disk is idle, it clicks, but when it is in active use, it is quiet.
I'm currently running a full scrub and then badblocks run on the disk to make sure the media itself is good before I repurpose it or try to get it warranty replaced.
My only thought is that there must be something flaky with the controller or firmware on the disk that standard SATA controllers (or USB SATA adapters) are fine with but that the higher end controllers in the enclosure and on the LSI card don't like (as well as possibly something with the motor due to the clicking when idle).