For a small server (where disk costs aren't a big issue) I find mirrors much more flexible than RAIDZ. For a large server (where disk costs dominate), I can't afford 100% redundancy and have to use RAIDZ[123]. I'm going to ramble a bit about how I see the tradeoffs over the life of a server, in hopes that helps some people fill in their mental maps with more detail.
I find that cheap servers are limited by drive bay space (especially if you want hot-swap) plus motherboard controller provision (6 controllers on the two motherboards I've used for ZFS stuff). (One of my servers has a second controller card, so it can handle 14 drives.) (I haven't yet learned about backplane extenders and various ways of getting more drives off of advanced controllers, and should some day; but so far paying for 6 current-tech drives is pushing my limits).
Anyway, let me explain the flexibility using mirrors gives me, for a small household server (currently running 3 mirror vdevs plus a hot spare disk, 3.63T; this is the same server, but different disks, that I started with in 2006, and it's a Solaris not FreeNAS box).
When you plan your zpool, if you intend to keep it running a long time (as opposed to replacing it and copying the data over somewhere else), some advance planning helps a lot.
The basic limitation that constrains this is that you can NOT ever take a vdev out of a zpool. Once you've created a vdev in your zpool, you're stuck with that vdev forever.
You can add a new vdev at any time. However, once you've done so, you're stuck with it forever.
Mirrors are very flexible; you can attach an additional drive to a mirror at any time, and that new drive will get the data copied to it. When the copy completes, you then have a three-way mirror. You can also
detach a drive from a mirror at any time. This reduces your redundancy, so you don't want to do it casually, but it's possible.
With a mirror vdev, you can increase its size
without ever reducing the redundancy below the starting value. If you start out with a two-way mirror, here's how you increase it later:
- Physically install a new, larger, drive.
- Attach that drive to the mirror vdev, and wait for it to "resilver". You now have a three-way mirror.
- Detach one of the original, small drives from the vdev. You now have a two-way mirror again.
- Physically remove the detached original drive.
- Physically install a second new, larger, drive.
- Attach that drive to the mirror vdev, and wait for it to resilver. You now have a three-way mirror.
- Detach the one remaining original, small, drive. You now have a two-way mirror again. And, if the auto-expand property is on, it's bigger (the size of the new big drives).
- Physically remove the detached original drive.
Note that, to do this, you need to be able to add one additional drive to your server temporarily. So you can't do this if you have filled it 100%.
RAIDZ[123] vdevs will also auto-expand if you replace all their disks (one at a time, waiting for them to resilver) with larger disks. The difference, though, is that you can't temporarily increase the redundancy on a RAIDZ vdev. You can replace one disk with a bigger disk, wait for it to resilver, and then repeat -- but during that resilver (and a resilver can take a LONG time with 4TB disk drives!) the redundancy is down by one. And, of course, the load of the resilver is
in addition to any other load, so the remaining disks are being worked harder than usual, and hence are more likely than usual to fail, just when the redundancy is reduced. So it's considerably more risky.
I've built one mirror-based zpool and one RAIDZ-based zpool, and I think each was the right choice for the budget and storage needs I build them for. I hope my rambling on at length has helped somebody understand the constraints and tradeoffs in these choices over the lifespan of their server! And that those of you who find it uninteresting have stopped reading before now!
(As an example of the ridiculous extremes mirrors can be pushed to, somebody at Sun once built a 48-way mirror on one of their X4500 "Thumper" fileserver boxes. Worked fine.)