New to ZFS - Quick Query

[Mrrodge]

Hi there,

New to all this and am trying to get my head around IOPS (I work in IT but am new to SAN/virtualization). I can see this post being the first of many!

I have a network topology in my office that isn't virtualized, so I'm looking to move away from the server's inbuilt RAID10 array to a SAN with Fibre Channel connection.

I'm not entirely sure how we're going to implement it yet, so at present I'm on a learning quest with some old desktop hardware I've thrown together and I'm playing with FreeNAS (don't worry, this will never be put on production, we'll probably by a TrueNAS or HP or something but I want to get a full understanding of the workings of these things).

Current hardware as follows:

AMD FX 4100 Quad-Core
16GB DDR3
2x 120GB SSDs
1x 640GB HDD
2x 2TB HDDs
2 x 1TB HDDs

At present, I have the array set up with one SSD for ZIL, the other for L2ARC, 640GB for hot spare and the remaining drives mirrored and striped in a RAID10-style setup.

Couple of queries with my current setup:

• With the 640GB as a hot spare, in the event of a failure of one of the 2TB drives, and that drive was, say, 750GB full, what happens? I was surprised FreeNAS allowed me to add a hot spare that was smaller than my biggest drive.
• I've read that should the ZIL SSD die, the entire pool is lost and I should mirror it. Is this right?
• The SSDs are standard and have no battery/supercapacitor for cache flushing.
• So, in theory, if I added the extra SSD for ZIL and the whole system had a redundant power supply and external UPS, it'd be pretty much power failure proof?

That's pretty much it on the test rig I have now, but I also have some more production-specific questions for the future, based on multiple VMs running server roles such as Exchange, Active Directory, SQL Server, File shares etc. Obviously for this we'd use a FC connection to the SAN and IOPS will need to be maximised.

• Production setup requires maximum IOPS. Aside from the SAS/SATA and drive RPM debate, how can I maximise IOPS? I gather that if I got a couple of Z2 vdevs and striped them in a pool, I'd increase performance but only to the sum of the slowest drive in each vdev at best, but doesn't the same apply with striped mirrors? I mean for example if I have a volume of 2 vdevs each with 2 drives at 4TB, total storage would be the sum of the slowest performing drive in each vdev, just like with RAIDZ2, Right?
• So to put the above into practice, say I have 12 4TB drives. I can configure RAIDZ2 in 2 vdevs giving me approx. 22TB storage and up to four possible simultaneous drive failures with no data loss. Or, I can have 4 vdevs containing 3 mirrored drives, all striped. For this I'd have 16TB storage and much, much faster rebuild and performance, with the same/better redundancy?

Any feedback would be great!

Thanks.

 

[hugovsky]

Hi and welcome. To better answer your questions, take a look at the links in my sig.

If you need IOPS, mirrors is the way to go.

 

[danb35]

how can I maximise IOPS?

Striped mirrors.

 

[BigDave]

At present, I have the array set up with one SSD for ZIL, the other for L2ARC, 640GB for hot spare and the remaining drives mirrored and striped in a RAID10-style setup.

I second @hugovsky's listed recommendations of the information in his signature.

Your current test rig will not perform well with the small amount of RAM (16GB) and the use if the L2ARC
will slow down rather than help performance. This can and will probably change once you buy production
hardware...

 

[danb35]

I've read that should the ZIL SSD die, the entire pool is lost and I should mirror it. Is this right?

It's been some years since this was the case.

 

[Ericloewe]

I gather that if I got a couple of Z2 vdevs and striped them in a pool, I'd increase performance but only to the sum of the slowest drive in each vdev at best, but doesn't the same apply with striped mirrors?

Mostly, but you have more mirror vdevs than RAIDZ2 vdevs for the same amount of drives.

Or, I can have 4 vdevs containing 3 mirrored drives, all striped. For this I'd have 16TB storage and much, much faster rebuild

Well, faster rebuilds - for now. With the improvements to resilver/scrub performance that are being worked on, we might end up with the funny situation that RAIDZ scrubs faster due to higher throughput (the work in question focuses on minimizing the amount of IOPS eaten up while reading everything on the pool).

...and performance

Yeah, IOPS-wise, definitely.

You can also size ARC/L2ARC to minimze reads from the disks, which would allow for more IOPS to be used for writing.

 

[Mrrodge]

Thanks for the replies - some valuable information.

I knew system requirements were high for ZFS, but didn't realise they were just so high, which makes me wonder if, for our situation, SAN is worth having at all.

Let's say, for arguments sake that I have an LSI RAID controller running 5 drives in hardware RAID 5. The controller cost circa £300. If I built a FreeNAS system for £300 (excluding drives) and ran it with the same 5 drives, in a RAIDZ1 configuration, how would its performance compare? Think via fibre channel connection, Fibre channel HA's at nil cost to rule out network bandwidth.

I know there are other benefits such as scalability, hardware independence etc but our priority will essentially be performance for VMs.

Also I have a couple more queries:

• If I expand a pool when the remaining capacity gets low and I'm taking advantage of mirrors, how do I re-balance the pool? Won't all writes just go to the new vdev and bottleneck peformance?
• Does anyone know a way of having indicator lights for failed drives? We'd like to have this functionality.
• Also, how does the 'spare' functionality work? Is it the same as a hot spare on a normal RAID controller? i.e. the spare is automatically used as a replacement for a failed drive? If so, can a hot inserted drive be automatically allocated as a new spare?

Thanks!

 

[danb35]

If I expand a pool when the remaining capacity gets low and I'm taking advantage of mirrors, how do I re-balance the pool?

You don't.

Won't all writes just go to the new vdev and bottleneck peformance?

It will balance itself out in due course.

Does anyone know a way of having indicator lights for failed drives?

It's very hardware-dependent, so I'm not aware of a good general solution. I ran across a script that looks like it could give the foundation for something like this, but it's going to need some work to get going on FreeNAS.

Also, how does the 'spare' functionality work? Is it the same as a hot spare on a normal RAID controller? i.e. the spare is automatically used as a replacement for a failed drive?

Pretty much.

If so, can a hot inserted drive be automatically allocated as a new spare?

Insert a drive, and the system automatically picks it up and adds it to the pool as a spare? No, and I don't think you'd really want that--otherwise, when you decided to expand your pool, your drives would be allocated as spares before you had time to add them to the pool as a new vdev.

 

[Mrrodge]

You don't.

It will balance itself out in due course.

How, if much of the data isn't written to again? And doesn't performance take a hit?

You don't.

Insert a drive, and the system automatically picks it up and adds it to the pool as a spare? No, and I don't think you'd really want that--otherwise, when you decided to expand your pool, your drives would be allocated as spares before you had time to add them to the pool as a new vdev.

Fair point, thanks.

 

[Stux]

Let's say, for arguments sake that I have an LSI RAID controller running 5 drives in hardware RAID 5. The controller cost circa £300. If I built a FreeNAS system for £300 (excluding drives) and ran it with the same 5 drives, in a RAIDZ1 configuration, how would its performance compare?

I would be surprised if it was significantly worse. Hardware RAID controllers run each of the drives in lock-step, essentially limiting IOPs to the performance of a single drive.... just like ZFS.

In fact, with ZFS ARC and Transaction Groups, it should be better. ZFS will batch transactions to turn random write loads into sequential writes, the RAID card probably can't do that as well.

BUT ZFS is a CoW filesystem, and in some pathologicial work-loads, that could make a difference.