FreeNAS Mini XL L2ARC

[turboaaa]

I have a FreeNAS Mini XL used as a non-critical archive server. It is currently holding a copy of a filesystem with many many little files. Everytime a replication job is run it takes a good while for file contents to be compared. Though this is not to be a permanent solution, it will be THE solution for at least another year.

The system currently has the full 32GB RAM and a FreeNAS Mini Read Cache and ZIL Drives. From my understanding, and limited testing, it is better to have more RAM than L2ARC since the L2ARC DB is held in RAM. But one area I am weak in is implications of having L2ARC that is very large on a system with very low RAM such as this. (Again, because normally I would use more RAM)

The files I want to keep replicated is less than 2TB. If I was to install a 2TB SSD to hold this data in L2ARC, it seems it would still be faster with the improved random I/O of the SSD over the mechanical disks and 32GB ARC.

 

[turboaaa]

Looks like the MiniXL will take 64GB of RAM as well. Found a good price on ebay (Too good to be true? Possibly :-s ), if the RAM works out I plan on grabbing a 2TB L2ARC disk and see what happens. Worst case scenario the system becomes unstable and I put back the 120GB drive and upgrade my workstation to a 2TB SSD.

If anyone has experience doing this, let me know your results.

 

[jgreco]

I'd be cautious about going too big with the L2ARC. We usually suggest a 5:1 ratio of L2ARC:ARC, and then only starting at 64GB, but lots of improvements in the L2ARC subsystem suggest you might be able to do better.. I don't think there'd be any problems with a 120-256GB SSD on 32GB RAM, and definitely I'd experiment with 512GB SSD on 64GB RAM. That's my feeling at least.

Replication does not compare the contents of files. It does involve tracking changes to the contents, and therefore what you're seeing is a combination of metadata accesses and updated file contents. This would translate to lots of seeks. ZFS has a good chance of minimizing a lot of that if you can get the hot bits into ARC and then into L2ARC. If the files are mostly static, I'd suggest focusing on getting a smaller L2ARC device to work well.

 

[turboaaa]

Replication does not compare the contents of files. It does involve tracking changes to the contents, and therefore what you're seeing is a combination of metadata accesses and updated file contents. This would translate to lots of seeks. ZFS has a good chance of minimizing a lot of that if you can get the hot bits into ARC and then into L2ARC. If the files are mostly static, I'd suggest focusing on getting a smaller L2ARC device to work well.

I am a little confused by your statement. It still sounds like it would be best to have those seeks performed in L2ARC on a system that is limited by RAM (assuming no other workloads are present on the system). Though I will definitely defer to your advise in the ARC to L2ARC ratio. I did forget to mention dedup is enabled on the volume and that takes up a chunk of RAM as well.

When you say "getting a smaller :2ARC devic to work well" what would you recommend I do to optimize existing L2ARC?

Two updates
1. I received the 64GB of RAM. Let's hope it works as expected when I install first thing this week.
2. I found an issue with the network that is slowing down the replication process. Now that I know the problem is exaggerated by the network, I will leave L2ARC alone.

Thank you for your insights. It is appreciated.

 

[turboaaa]

Crud, no wonder the RAM was cheap. So I found out the MiniXL does not like RDIMM modules. Thankfully they cost me next to nothing, a real upgrade will be shy of $400. After a couple projects finish I will look into this again.

After doing some digging I did find that I have the auto tuner enabled. This seemed like a good idea since it would let FreeNAS decide what is best for the resources available, but there have been some reports that disabling this helps. Any input?