Upgrading capacity.

[albrandwood]

I have a business critical FreeNAS 9.2.1.7 setup ...

Dell R710 48Gb ram + LSI 12 bay SAS enclosure with 7x 2TB Raid-Z3 + SSD for Zil + SSD for L2ARC. we are getting close to 80% capacity, so I want to increase the capacity by migrating the existing pool to a new 11x 2TB Raid-Z3 using a second SAS enclosure.

(a) can I add external SAS + drives enclosures without shutting down FreeNAS
(b) am I best migrating the data using snapshots or Replication (and if so, could anyone give me the correct CLI commands?)
(c) I'd like to recycle the two SSD for the new Raid-Z3 (The SSD are internal to the R710, as they are SATA no SAS). I'm guessing that they would need to be removed from the existing zpool, and added to the new zpool? or wait till the old zpool is removed/destroyed, then re-add them to the new zpool?
(d) when finished, can I remove the existing drives from system without shutting down FreeNAS? (I plan on using the existing enclosure for a couple of 15krpm Raid-10 pools).

Thanks

@shley

 

[cyberjock]

A. You should be able to. Depends on if the hardware supports hot-swap and hot-plugging. If you aren't 100% sure, I would do it powered down just to be safe.
B. Replication is probably the best. The command depends on lots of factors, so you'll have to figure out what is appropriate for your situation.
C. You do have to remove them from the existing pool and add them to the new pool.
D. Depends on if your hardware supports hot-swap and hot-plugging. If you aren't 100% sure I would do it powered down just to be safe.

Any reason you don't just add another vdev of 7 disks? That seems like the best way to expand your zpool.

 

[albrandwood]

A. The hardware supports hot-swap/hot-plugging. I just hate having to shutdown the VMWare servers (iSCSI). But I'll probably play it safe anyway.
B. OK - Guess I'll start reading up on replication.
C. So I should remove them using zpool CLI. (I'm guessing that should remove any metadata etc).
D. See A.

As adding another vdev of 7 disks? That sounds the easiest, but ...
1) I'm not comfortable having zpools spread across multiple enclosures (although my brain says its not an issue).
2) The existing drives 2TB units are coming up on 5 years of continuous use, and I don't really want to push their reliability :)
3) Adding another vdev of 7 disks would be wasting 3 drives bays. I think Z3 will have enough redundancy for 11 drives.
4) I'd like to move my 4x 15krpm raid10 for vmware iSCSI from inside the R710 to the original enclosure, and probably add more 15krpm to it later in the year.
(I found out the hardware that SATA adapters don't work well in SAS hardware when used with ZFS, which means if I want SSD they have to be internal to the R710)

I guess my only real concern is how to do the replication... (and only so I don't have to go through the effort of rebuilding all the ZFS Datasets, Jails and Samba Shares.)

 

[Ericloewe]

1) I'm not comfortable having zpools spread across multiple enclosures (although my brain says its not an issue).

Not an issue, if hardware behaves normally. If it doesn't, you do have a 2*p chance of having a problem, instead of p. Extra points of failure and so on (nothing unusual).

2) The existing drives 2TB units are coming up on 5 years of continuous use, and I don't really want to push their reliability :)

If it's better than possible downtime for your business, sure.

(I found out the hardware that SATA adapters don't work well in SAS hardware when used with ZFS, which means if I want SSD they have to be internal to the R710)

What do you mean, exactly?

I guess my only real concern is how to do the replication... (and only so I don't have to go through the effort of rebuilding all the ZFS Datasets, Jails and Samba Shares.)

Some people have replicated to a pool that, in the end, was named the same as the original one. I don't know the exact process, but it should allow you to:

1. Replicate from A to B
2. Shut down and remove A, leaving B, which is renamed to A (or was named A all along, I don't know)
3. Boot and everything should just work

 

[albrandwood]

What do you mean, exactly?

SATA drives in multi-pathed SAS enclosures (such as the ones I'm using) require adapters to handle the multi-pathing. The problem is that the SAS/SATA interposer devices hide the drive from ZFS, which is bad enough, but they also seem to mistranslates errors ... The end result is when a SATA drive signals "please wait whilst I empty my buffer" ZFS sees it as "OH F**K DRIVE NOT RESPONDING ... END OF THE WORLD IS UPON US!!!", and promptly kicks the drive from the pool. As such, my only way to use SATA SSD is to put them inside the R710 which allows native SATA devices (as well as SAS devices).

 

[Ericloewe]

SATA drives in multi-pathed SAS enclosures (such as the ones I'm using) require adapters to handle the multi-pathing. The problem is that the SAS/SATA interposer devices hide the drive from ZFS, which is bad enough, but they also seem to mistranslates errors ... The end result is when a SATA drive signals "please wait whilst I empty my buffer" ZFS sees it as "OH F**K DRIVE NOT RESPONDING ... END OF THE WORLD IS UPON US!!!", and promptly kicks the drive from the pool. As such, my only way to use SATA SSD is to put them inside the R710 which allows native SATA devices (as well as SAS devices).

Ah, I was wondering if you were talking about interposers. Yeah, those things are painful.

I'm not familiar with complicated SAS layouts (particularly the edge cases), so I can't offer alternative solutions (other than a separate, simple HBA+ external chassis with expander solution).

 

[sfcredfox]

Just out of curiosity, what controller is running those internal drives?

I'm only familiar with HP gear, so all I've ever heard from Dell people is RAID controllers for internal drives. Did you swap it out with an aftermarket HBA or is it am OEM HBA?