ZFS and SSD cache size (log (zil) and L2ARC)

[RamGuy]

How large SSD sizes are required in order to have successful SSD caching of both the log / zil and L2ARC on my setup running 7x 2TB Western Digital RE4 hard drives in either RAIDZ (10TB) or RAIDZ2 (8TB) with 16GB (4x 4GB) DDR3 1333MHz ECC unbuffered.

I guess the log / zil doesn't require much space? Perhaps even a Intel SSD 311 / 313 series 20GB SLC will be sufficient? In regards of the L2ARC matters gets a bit more complicated. For a 8 or 10TB ZFS array will a Intel 520 series 60GB, 120GB, 180GB or 240GB be enough?

 

[louisk]

The ZIL doesn't require much space, 8G would be sufficient. I used a 32G SSD because it was cheap.

L2ARC is up to you. How much do you want to cache? How much data do you typically access that you need to be faster than coming off the spindles? This will tell you what size SSD to get for the cache.

 

[RamGuy]

Then the Intel 311 / 313 series SSD's should be a perfect fit for ZIL caching, considering the benefit of SLC and the fact that 20 / 24GB should be plenty of storage. For L2ARC I guess the more the merrier, so perhaps around 120 to 180GB but which series of SSD's are more suitable for caching?

 

[b1ghen]

I have been looking into the 311 and now the 313 for ZIL for a while, please report how it works out for you if you go that route. I think it would be the perfect match for cheap SLC ZIL.

 

[bfishernc]

Sorry to revive an old thread... but researching adding SSD for ZIL - looking for feedback from anyone who's done it...

- Did you see a performance improvement? (yes, I realize everyone's situation is different... just looking for broad patterns here)
- Did you use MLC or SLC SSD?
- Did you mirror your ZIL?
- Did you add SSD for L2ARC? See an improvement?

I see lots of people asking questions, but not many people posting follow up if they did it and if it helped or not... Thanks!

 

[Brosif_My_Nif]

I am also looking at adding an Intel 313 20GB ZIL as well as an Intel 520 120 or 180GB L2ARC. How did they work out for you?

 

[cyberjock]

To answer the questions of if you saw a performance improvement or not, it's much more important that the way in which the zpool is used is exactly the same as what you use it for. In one setting you could see a phenomenal increase in performance. In another setting with the exact same hardware the difference in performance could be minimal.

It's much more important that you read up on ZIL and L2ARC and understand how they work and what they cache and determine if that function works for you. If you simply say you want it because its a "read" cache you do NOT understand how it works.

 

[Brosif_My_Nif]

noobsauce80,

I understand how it works and I think I could benefit from adding a couple SSD's for ZIL and L2ARC.

Here is a real world example of what I use my FreeNAS box for. I run a small wedding photography business. Me and my wife store, edit and work with hundreds of RAW photo files (Roughly 60MB Each - D800 Camera Bodies) from the NAS. Bandwidth isn't an issue as I have LACP set up on a managed switch. I think it would help to increase the responsiveness while editing off the NAS. Do you think my situation would benefit from the addition of a couple SSD's for ZIL and L2ARC?

I have to agree with bfishernc. I have not seen many real world examples or people following up after they've added their ZIL or L2ARC SSD's.

Anways, thanks for taking the time to respond.

 

[cyberjock]

That's a tough call. The files aren't several GB. 60-100MB pictures would mean a second or 2 to send or receive the whole file. If you want to try a ZIL, I'd recommend waiting for 8.3(There's a bug in ZFS v15 that can cause a loss of the entire pool). The L2ARC is easy enough to try now if desired.

I imagine you probably open a picture, then edit it, maybe save it several times at various stages of editing. If that is your "expected" work load, I'd expect that a ZIL may increase the responsiveness of your saves. But for reads probably not much to help with an L2ARC.

 

[Brosif_My_Nif]

Thanks noobsauce80. The workflow you described is exactly what I do with the images and why I think it would be beneficial to add an SSD for the ZIL.

I upgraded my pools to v28, and found an old 60GB Vertex to test on L2ARC until I receive the 20GB Intel 313 ZIL I am about to pull the trigger on. It seemed to speed up the large folders (150 - 200GB of RAW Photo Files) as it read it off the spinning disks and I was navigating through them. Not by much, but a little. Could be because this old Vertex is pretty slow compared to what's on the market now.

I am just going to get the Intel 20GB for the ZIL for now. I'll report back and let you guys know if it helps with the quick saves while I'm editing.

 

[Brosif_My_Nif]

Sorry to revive an old thread... but researching adding SSD for ZIL - looking for feedback from anyone who's done it...

- Did you see a performance improvement? (yes, I realize everyone's situation is different... just looking for broad patterns here)
- Did you use MLC or SLC SSD?
- Did you mirror your ZIL?
- Did you add SSD for L2ARC? See an improvement?

I see lots of people asking questions, but not many people posting follow up if they did it and if it helped or not... Thanks!

1. When I get my ZIL, I will post back and let you know if I see a speed improvement.

2. SLC SSD for ZIL because it's constantly writing to it. They last much longer and remember, you don't need much space. I am going to purchase an Intel 313 20GB SSD for my ZIL and a 520 120GB SSD for my L2ARC a little bit later.

3. Mirror your ZIL if you're on ZFS v15, but from what I have read, v28 does not need to be mirrored because if you disconnect the ZIL on v28, you will not sustain data loss from the changes they have made. I tried it myself on a test pool I made just a few minutes ago and yanked out my Sata Cable from an old SSD and did not sustain any data loss on the test pool.

4. I noticed a small improvement when I added an old SSD (Vertex 60GB) for the L2ARC, but nothing substantial.

 

[bfishernc]

I purchased the SSDs but have not implemented them. I was waiting for the release of the next version of FreeNAS to eliminate any chance of corruption with the ZIL. Once we get the new version, I will try.

My use case is slightly different... virtual machines in a home lab. I believe a SSD ZIL will help significantly as it is many small writes (once VM is running and stable). Not as certain about the L2ARC but have a SSD to try it. Since I'm not using the ZIL yet, I don't have any VMs running (on my FreeNAS) so no reason to test the SSD L2Arc. Will test both once next release is available.

 

[TScott]

In regards to some of the earlier questions about the size of SSD to use for L2ARC, I would caution against going too large. I'm quite new to ZFS as well, but from what I've read your money is always best spent on memory (ARC) before an SSD for L2ARC (as memory is noteably faster). Along with that, its vital not to oversize your L2ARC, because system memory (ARC) is used by ZFS to manage L2ARC. The larger the L2ARC pool, the more of the main, much faster, system memory is eaten. A few links I used for reference:

http://www.mail-archive.com/zfs-discuss@opensolaris.org/msg34674.html
http://www.nexentastor.org/boards/2/topics/4633
http://mirror-admin.blogspot.com/2011/12/how-l2arc-works.html
http://mail.opensolaris.org/pipermail/zfs-discuss/2011-May/048185.html

Something to consider for ZIL, some of the large SSDs (in the same product line) have faster write speeds. So even while the space might not be used, the additional bandwidth may be of benefit.

 

[Brosif_My_Nif]

In regards to some of the earlier questions about the size of SSD to use for L2ARC, I would caution against going too large. I'm quite new to ZFS as well, but from what I've read your money is always best spent on memory (ARC) before an SSD for L2ARC (as memory is noteably faster). Along with that, its vital not to oversize your L2ARC, because system memory (ARC) is used by ZFS to manage L2ARC. The larger the L2ARC pool, the more of the main, much faster, system memory is eaten. A few links I used for reference:

http://www.mail-archive.com/zfs-discuss@opensolaris.org/msg34674.html
http://www.nexentastor.org/boards/2/topics/4633
http://mirror-admin.blogspot.com/2011/12/how-l2arc-works.html
http://mail.opensolaris.org/pipermail/zfs-discuss/2011-May/048185.html

Something to consider for ZIL, some of the large SSDs (in the same product line) have faster write speeds. So even while the space might not be used, the additional bandwidth may be of benefit.

I completely agree with everything you state in the first paragraph. Very good information and great links!


The only thing I want to comment on is the use of MLC vs. SLC. Here's a simple video comparing the two:

http://searchsolidstatestorage.techtarget.com/video/SLC-vs-MLC-explained

The faster SSD's all seem to be based on MLC NAND. Although faster and cheaper to produce, it does not have the write endurance that SLC offers. Unless TRIM is introduced down the road for wear leveling, I would stick with SLC NAND for the use of a ZIL because of the large amount of write cycles it offers.

Or... Buy the cheaper MLC for a ZIL and replace it every few years until you kill it with write cycles if you're on v28 if you want the cheaper, faster SSD, but that's not something I would recommend.

 

[TScott]

I completely agree with everything you state in the first paragraph. Very good information and great links!


The only thing I want to comment on is the use of MLC vs. SLC. Here's a simple video comparing the two:

http://searchsolidstatestorage.techtarget.com/video/SLC-vs-MLC-explained

The faster SSD's all seem to be based on MLC NAND. Although faster and cheaper to produce, it does not have the write endurance that SLC offers. Unless TRIM is introduced down the road for wear leveling, I would stick with SLC NAND for the use of a ZIL because of the large amount of write cycles it offers.

Or... Buy the cheaper MLC for a ZIL and replace it every few years until you kill it with write cycles if you're on v28 if you want the cheaper, faster SSD, but that's not something I would recommend.

Excellent point about MLC vs SLC for use in ZIL. Nice link as well, I've been meaning to find an updated article in regards to MLC vs SLC.

Although I was speaking more in general terms, the speed difference within the same line of SSDs does appear to be more prevalent in the larger MLC drives (I had the Samsung 830 Series MLC SSD line in mind, in particular). But, even the Intel 313 series shows this to a certain (small) extent. For example:

Intel 313 20GB
Sequential Read 220 MB/s
Sequential Write 100 MB/s
Random Read (8GB Span) 36000 IOPS
Random Write (8GB Span) 3300 IOPS
Latency - Read 72 µs
Latency - Write 90 µs

Intel 313 24GB
Sequential Read 160 MB/s
Sequential Write 115 MB/s
Random Read (8GB Span) 33000 IOPS
Random Write (8GB Span) 4000 IOPS
Latency - Read 72 µs
Latency - Write 90 µs

SLC vs MLC for heavy write usage aside, what I'd really like to know is how much of a difference the added write bandwidth/IOps makes when it comes to the ZIL drive and the pools performance (generally speaking, not just with the example drives above)? Given the very reasonable price of small, reliable, SLC drives like the Intel ($99 at NewEgg) would the added performance of some of the faster MLC drives even be a factor (or faster SLCs, for that matter)? I'm sure it will depend on the workload and the configuration, but it would interesting to know at what point it might be worth it to get a faster drive for ZIL. I think I'll do some digging to see if I can find any relevant comparisons.

 

[mattlach]

To answer the questions of if you saw a performance improvement or not, it's much more important that the way in which the zpool is used is exactly the same as what you use it for. In one setting you could see a phenomenal increase in performance. In another setting with the exact same hardware the difference in performance could be minimal.

It's much more important that you read up on ZIL and L2ARC and understand how they work and what they cache and determine if that function works for you. If you simply say you want it because its a "read" cache you do NOT understand how it works.

I'm trying to understand as much as I can, but the intricacies of file system operations sometimes escape me.

My main interest in L2ARC and ZIL is not necessarily raw write or read performance, but rather in helping in situations where there are multiple simultaneous reads/writes by different processes/users.

I have noted a SIGNIFICANT slowdown on my system - for instance - if there is a continuous write in process (like from a backup process writing to the array) if then a user tries to read from it at the same time.

In situations like these I'll get rather high system loads (of approximately 6) with close to 0 CPU load, so it must be binding up on I/O speed.

My theory is that L2ARC will help a lot in this case if whatever the user is trying to read is in the cache, as then the continuous write will not be interrupted.

While I am struggling with the differences between synchronous and asynchronous writes, I'm also hoping a ZIL will help me overcome the relatively slow write speeds of RAIDz2. I don't have any significant databases running off of the array, mostly multiple users storing and requesting files, at the same time as backups may be taking place.

Proposed System (not yet built):
- AMD FX-8120 32GB RAM running VMWare ESXi 5.1
- FreeNAS running as Guest with 2 cores and 16GB RAM assigned
- 3x3TB WD Green drives in RAIDz2
- Potentially (based on this discussion) 20GB Intel 313 ZIL drive and 128GB Larger faster MLC drive for L2ARC

System load is mostly from 4 end users in the house, but we are tasking users.

At any given time, the following could be happening simultaneously:
- 4 users opening, saving and copying files
- Automated backups from clients to NAS (7 computers in house, 3 PC desktops, 3 PC laptops, 1 Apple MacBook Pro)
- Apple MBP running time machine backup to NAS
- rtorrent running and saving data on server (up to 85mbit downstream network speeds)
- NAS being backed up to offsite cloud backup (up to 35Mbit upstream network speeds)

So as mentioned above, I definitely think L2ARC will help me, but will ZIL? Not as certain. For me my focus is on using a combination of RAM/SSD to compensate for the relatively poor simultaneous read/write performance of my WD Green drives.

I Appreciate any feedback

 

[duomenox]

mattlach,

I don't think your bottleneck is memory caching... you are running 3 slow drives in RZ2, so for every single operation you are calculating double parity. I think you would be better off expanding your RZ2 pool using at least 2 additional drives. By adding the drives, you will increase the number of spindles in the array and allow for more IOPS.

From the list of potential simultaneous operations, I bet you are running into an IOPS bottleneck. Adding a L2ARC cache may help for a very small amount of time, but when the cache fills up you'll be back where you started.

Just my opinion, I'm not a ZFS guru, but I've seen IOPS bottlenecks in SMB VMware environments and adding spindles was a tremendous help.

If you cannot add spindles, consider buying faster drives and dropping down to RZ1, Attach a large USB Raid 1 enclosure and automate a backup schedule to ensure data integrity (if needed).

 

[mattlach]

I should have clarified.

My current setup is not FreeNAS based.

I'm using an external array via USB, and since it was through VMWare ESXi USB is pretty slow.

I was just looking forward to my new RAIDz setup, and thinking about ways to avoid the problem.

I have done some tests with my 4x3TB WD Greens in RAIDz2 and without any caching (ZIL or L2ARC) I'm getting about 200MB/s reads and 160MB/s writes which I am considering adding SSD caching to further help avoid the type of showdowns I've seen...

 

[joeschmuck]

Interesting thread. Right now I'm looking into if it's possible to take a single SSD, say 250GB, and create two partitions for it, 8GB for a ZIL and the remaining for an L2ARC. Based on this thread I see no reason to use an entire SSD for a ZIL. I'm only doing this to play around and see what kind of performance gains I could obtain, if any for my use. So I'm in search of commands to join the cache and zil manually since the GUI only accepts the entire drive. Time to continue my search.

EDIT: Found the commands I needed, easy search on Google: (pool is called pool, SSD is da5)

zpool add pool log da5p1
zpool add pool cache da5p2

 

[bollar]

Interesting thread. Right now I'm looking into if it's possible to take a single SSD, say 250GB, and create two partitions for it, 8GB for a ZIL and the remaining for an L2ARC.

It will be interesting to see what you find. I would worry about the system slowing down because of the the limited bandwidth available to the SSD via SATA -- if the SSD had a max throughput of 500MB, for example, how much of that would be used on the ZIL, and what's left over for the L2ARC. My hunch would be that the system would be overall slower than if you just allocated the entire drive to L2ARC.