L2ARC impact on rsync performance for largely dormant data

Constantin

Vampire Pig
Joined
May 19, 2017
Messages
1,827
I've been doing some more testing and the evidence suggests that using a L2ARC with a metadata only flag has a potential substantial positive impact on rsync performance over AFP. For largely dormant data, the combination of L2ARC dedicated to metadata seems to benefit rsync performance a lot.

For example, I ran rsync on a 1.15TB iTunes folder with 30k+ music files, assorted images, iOS apps, etc. and the rsync completed in 5 minutes over a 10 GBe connection. 1.5GB of newly-added songs was also copied to an external RAID5 array. Previously, this task would take 40+ minutes.

Granted, my new server has a more powerful CPU, but I'd wager that the spinning storage is the bottleneck here, esp. for small files like all the iOS stuff still stored in my iTunes folder. I also find browsing the various directories to be much quicker now. It's almost like DAS.

Thus, folk with similar use cases may want to consider experimentally adding a L2ARC with the metadata only flag even if their FreeNAS doesn't have 64GB of RAM, etc. However, to fill the L2ARC with metadata, you will have to run the rsync a couple of times to fill it - each rsync should decrease in time as the L2ARC fills with metadata.
 

Mr. Slumber

Contributor
Joined
Mar 10, 2019
Messages
182
Great news, thank you.

may want to consider experimentally adding a L2ARC with the metadata only flag

Really would like to try this but I'm not sure how to do this. Got a Samsung EVO 850 Pro SSD laying around and my new FreeNAS box got "only" 16GB ECC ram. Thanks for pointing me in the right direction :)
 

Constantin

Vampire Pig
Joined
May 19, 2017
Messages
1,827
For something as small as 16GB, keep an eye on the regular ARC memory. Cyber jock recommends at least 64GB of memory before enabling a L2ARc.

L2ARC consumes memory that otherwise would have been available to ARC. If your pool isn’t huge, this may not be a problem. Also, best suggested practices is a ~5:1 ratio of L2Arc vs. arc. So you may want to partition that SSD to be <75GB as a starting point.

You can add a L2ARc in the current ui by opening storage, selecting your pool and then selecting extend. At the bottom, you’ll see various tabs like cache, log, etc. select cache, then the select the SSD from the list above (check box) confirm adding it to the cache.

The metadata flag has to be set via the CLI, IIRC.
 
Last edited:

Mr. Slumber

Contributor
Joined
Mar 10, 2019
Messages
182

Constantin

Vampire Pig
Joined
May 19, 2017
Messages
1,827
It likely depends on part on how you use the L2ARC. How much space for metadata is actually needed? That in turn might limit the arc you’d gobble up with L2ARC indices.

For my use case, L2ARc may make sense even at 32GB of RAM. But... that’s because much of my data is pretty static, which means that a L2ARC with metadata only flags set will quickly confirm to rsync that the files are good and let it move on.
 

Mr. Slumber

Contributor
Joined
Mar 10, 2019
Messages
182
L2ARc may make sense even at 32GB of RAM. But... that’s because much of my data is pretty static

Ok I think I get this now. So the data has to be relatively static for that to work out?! e.g. backups of VMs which change only once a day
 

Constantin

Vampire Pig
Joined
May 19, 2017
Messages
1,827
More testing is required.... L2ARC likely benefits rsync most with static data because it allows rsync to skip over unchanged content. Similarly, I really like how responsive the system has become while browsing (currently with AFP, though SMB allegedly similarly benefits).

However, if the content is constantly changing, then rsync will have to transfer all that content to the backup medium anyway, so more bandwidth, time, etc. will be lost to that task. The time needed to browse and compare directories will be less relevant, ditto the metadata stored in the L2ARC for the few files that have not changed...

Apparently, you can enable L2ARC selectively (not just on a pool by pool basis) though that is not a GUI-enabled feature. Filling the L2ARC takes time (see the settings) and hence browsing your pool may become more responsive in some sections (that are used a lot) than others.
 

Mr. Slumber

Contributor
Joined
Mar 10, 2019
Messages
182

Constantin

Vampire Pig
Joined
May 19, 2017
Messages
1,827
One more thing - apparently, L2ARC is not persistent - so your NAS gets to rebuild it every time it reboots.
 

Constantin

Vampire Pig
Joined
May 19, 2017
Messages
1,827
I thought I'd run a few tests to give you all some data to chew on. It seems to illustrate nicely how the eventual filling of the L2ARC with metadata contributes to much faster rsync performance. The following datasets were rsynced with a DAS array over a 10GBe connection in a FreeNAS rig that was otherwise unchanged (no new RAM, etc.). Some minor data additions may have happened between sets, reflecting the usual realities of a backup server.

iTunes folder (1.36TB of music files, iOS backups):
Run 1: 1 hour, 37 minutes​
Run 2: 33 minutes​
Run 3: 4 minutes, 41 seconds​
Run 4: 5 minutes, 4 seconds​

Pictures folder (1.74TB of family pics)
Run 1: 3 hours, 31 minutes​
Run 2: 2 hours, 20 minutes​
Run 3: 50 minutes, 43 seconds​
Run 4: 42 minutes, 37 seconds​

Time Machine folder (1.16TB of sparse bundles)
Run 1: 48 minutes​
Run 2: 28 minutes, 44 seconds​
Run 3: 28 minutes, 47 seconds​
Run 4: 3 minutes, 29 seconds​
Your setup may not experience the same benefit if your backup program is smarter than mine (Carbon Copy Cloner) and hence keeps track of what was copied previously. CCC is a great program re: copying, verifying, etc. but it's also conservative by checking each file individually instead of building backup indices to compare directory data against. L2ARC benefits seem to accrue until the 3rd-4th run, after that, execution speed seems relatively constant.

The above was over 10GBe network connections. The same L2ARC benefit seems to be true for "slower" network assets repeatedly working with rsync also though other factors conspire to lessen the benefit. The following iTunes backup (same folder as above) is from the NAS to my "burner" 2.5" HDD (this is how I keep Sonos out of the NAS). The HDD is slow, attached via USB 2.0 to a Apple Airport Extreme Gen 6, with a 1GBe wired connection.
Run 1: 3 hours, 41 minutes​
Run 2: 1 hour, 46 minutes​
Run 3: 1 hour, 48 minutes​
Note: Run 1 above had benefitted from at least one previous rsync of the Music library (NAS to DAS). Thus, the first run would likely have taken significantly longer and runs 2 and up would not be so similar.

All in all, for my limited use case (backup file server with largely dormant data), L2ARC with metadata = only seems to have a significant benefit. Too bad that L2ARC has yet to become persistent between reboots!
 
Last edited:

Mr. Slumber

Contributor
Joined
Mar 10, 2019
Messages
182
Wow, thanks, very interesting! :)
 

Constantin

Vampire Pig
Joined
May 19, 2017
Messages
1,827
One of the things I want to explore is just how full the L2ARC set of disks actually is now that they are "filled" with metadata. I want to believe it's not the 900GB of SSD I've thrown at the L2ARC thus far. But it could be... Anyhow, a 10x improvement in rsync performance makes me very happy.
 

Constantin

Vampire Pig
Joined
May 19, 2017
Messages
1,827
Used the following command to get stats on memory, ARC, and L2ARC: /usr/local/www/freenasUI/tools/arc_summary.py

Here is the output. I wonder if a 22% hit ratio for the L2ARC is considered OK when L2ARC is metadata=only? Also, am I interpreting the L2ARC report correctly in that the L2ARC is currently using about 250 GB?

First, memory:
Code:
System Memory:

        0.09%   114.53  MiB Active,     77.77%  96.93   GiB Inact
        19.56%  24.37   GiB Wired,      0.00%   0       Bytes Cache
        2.08%   2.59    GiB Free,       0.50%   641.04  MiB Gap

        Real Installed:                         128.00  GiB
        Real Available:                 99.89%  127.86  GiB
        Real Managed:                   97.47%  124.63  GiB

        Logical Total:                          128.00  GiB
        Logical Used:                   22.25%  28.48   GiB
        Logical Free:                   77.75%  99.52   GiB

Kernel Memory:                                  1.62    GiB
        Data:                           97.46%  1.58    GiB
        Text:                           2.54%   42.29   MiB

Kernel Memory Map:                              124.63  GiB
        Size:                           4.97%   6.20    GiB
        Free:                           95.03%  118.43  GiB


Now ARC:
Code:
ARC Summary: (HEALTHY)
        Storage pool Version:                   5000
        Filesystem Version:                     5
        Memory Throttle Count:                  0

ARC Misc:
        Deleted:                                94.05m
        Mutex Misses:                           208.78m
        Evict Skips:                            208.78m

ARC Size:                               26.86%  15.44   GiB
        Target Size: (Adaptive)         26.89%  15.45   GiB
        Min Size (Hard Limit):          26.89%  15.45   GiB
        Max Size (High Water):          3:1     57.47   GiB

ARC Size Breakdown:
        Recently Used Cache Size:       64.45%  9.96    GiB
        Frequently Used Cache Size:     35.55%  5.49    GiB

ARC Hash Breakdown:
        Elements Max:                           12.02m
        Elements Current:               97.89%  11.76m
        Collisions:                             77.60m
        Chain Max:                              9
        Chains:                                 2.62m

ARC Total accesses:                                     3.63b
        Cache Hit Ratio:                94.67%  3.44b
        Cache Miss Ratio:               5.33%   193.86m
        Actual Hit Ratio:               92.87%  3.38b

        Data Demand Efficiency:         87.46%  191.88m
        Data Prefetch Efficiency:       1.92%   65.71m

        CACHE HITS BY CACHE LIST:
          Anonymously Used:             0.78%   26.93m
          Most Recently Used:           3.26%   112.28m
          Most Frequently Used:         94.83%  3.26b
          Most Recently Used Ghost:     0.97%   33.28m
          Most Frequently Used Ghost:   0.15%   5.24m

        CACHE HITS BY DATA TYPE:
          Demand Data:                  4.88%   167.82m
          Prefetch Data:                0.04%   1.26m
          Demand Metadata:              92.33%  3.18b
          Prefetch Metadata:            2.75%   94.80m

        CACHE MISSES BY DATA TYPE:
          Demand Data:                  12.41%  24.06m
          Prefetch Data:                33.24%  64.45m
          Demand Metadata:              37.55%  72.79m
          Prefetch Metadata:            16.80%  32.56m


... followed by the L2ARC output:
Code:
L2 ARC Summary: (HEALTHY)
        Passed Headroom:                        1.02m
        Tried Lock Failures:                    1.64m
        IO In Progress:                         439
        Low Memory Aborts:                      333.99k
        Free on Write:                          11.84k
        Writes While Full:                      6.58k
        R/W Clashes:                            0
        Bad Checksums:                          0
        IO Errors:                              0
        SPA Mismatch:                           406.84m

L2 ARC Size: (Adaptive)                         248.42  GiB
        Compressed:                     97.35%  241.84  GiB
        Header Size:                    0.36%   908.05  MiB

L2 ARC Breakdown:                               193.85m
        Hit Ratio:                      22.59%  43.79m
        Miss Ratio:                     77.41%  150.06m
        Feeds:                                  541.12k

L2 ARC Buffer:
        Bytes Scanned:                          64.59   TiB
        Buffer Iterations:                      541.12k
        List Iterations:                        2.16m
        NULL List Iterations:                   55.73k

L2 ARC Writes:
        Writes Sent:                    100.00% 235.19k

DMU Prefetch Efficiency:                        643.64m
        Hit Ratio:                      13.92%  89.60m
        Miss Ratio:                     86.08%  554.04m


and last but not least, the Auto-tunables:
Code:
ZFS Tunable (sysctl):
        kern.maxusers                           8519
        vm.kmem_size                            133819486208
        vm.kmem_size_scale                      1
        vm.kmem_size_min                        0
        vm.kmem_size_max                        1319413950874
        vfs.zfs.vol.immediate_write_sz          32768
        vfs.zfs.vol.unmap_sync_enabled          0
        vfs.zfs.vol.unmap_enabled               1
        vfs.zfs.vol.recursive                   0
        vfs.zfs.vol.mode                        2
        vfs.zfs.sync_pass_rewrite               2
        vfs.zfs.sync_pass_dont_compress         5
        vfs.zfs.sync_pass_deferred_free         2
        vfs.zfs.zio.dva_throttle_enabled        1
        vfs.zfs.zio.exclude_metadata            0
        vfs.zfs.zio.use_uma                     1
        vfs.zfs.zil_slog_bulk                   786432
        vfs.zfs.cache_flush_disable             0
        vfs.zfs.zil_replay_disable              0
        vfs.zfs.version.zpl                     5
        vfs.zfs.version.spa                     5000
        vfs.zfs.version.acl                     1
        vfs.zfs.version.ioctl                   7
        vfs.zfs.debug                           0
        vfs.zfs.super_owner                     0
        vfs.zfs.immediate_write_sz              32768
        vfs.zfs.standard_sm_blksz               131072
        vfs.zfs.dtl_sm_blksz                    4096
        vfs.zfs.min_auto_ashift                 12
        vfs.zfs.max_auto_ashift                 13
        vfs.zfs.vdev.queue_depth_pct            1000
        vfs.zfs.vdev.write_gap_limit            4096
        vfs.zfs.vdev.read_gap_limit             32768
        vfs.zfs.vdev.aggregation_limit_non_rotating131072
        vfs.zfs.vdev.aggregation_limit          1048576
        vfs.zfs.vdev.trim_max_active            64
        vfs.zfs.vdev.trim_min_active            1
        vfs.zfs.vdev.scrub_max_active           2
        vfs.zfs.vdev.scrub_min_active           1
        vfs.zfs.vdev.async_write_max_active     10
        vfs.zfs.vdev.async_write_min_active     1
        vfs.zfs.vdev.async_read_max_active      3
        vfs.zfs.vdev.async_read_min_active      1
        vfs.zfs.vdev.sync_write_max_active      10
        vfs.zfs.vdev.sync_write_min_active      10
        vfs.zfs.vdev.sync_read_max_active       10
        vfs.zfs.vdev.sync_read_min_active       10
        vfs.zfs.vdev.max_active                 1000
        vfs.zfs.vdev.async_write_active_max_dirty_percent60
        vfs.zfs.vdev.async_write_active_min_dirty_percent30
        vfs.zfs.vdev.mirror.non_rotating_seek_inc1
        vfs.zfs.vdev.mirror.non_rotating_inc    0
        vfs.zfs.vdev.mirror.rotating_seek_offset1048576
        vfs.zfs.vdev.mirror.rotating_seek_inc   5
        vfs.zfs.vdev.mirror.rotating_inc        0
        vfs.zfs.vdev.trim_on_init               1
        vfs.zfs.vdev.bio_delete_disable         0
        vfs.zfs.vdev.bio_flush_disable          0
        vfs.zfs.vdev.cache.bshift               16
        vfs.zfs.vdev.cache.size                 0
        vfs.zfs.vdev.cache.max                  16384
        vfs.zfs.vdev.default_ms_shift           29
        vfs.zfs.vdev.min_ms_count               16
        vfs.zfs.vdev.max_ms_count               200
        vfs.zfs.vdev.trim_max_pending           10000
        vfs.zfs.txg.timeout                     5
        vfs.zfs.trim.enabled                    1
        vfs.zfs.trim.max_interval               1
        vfs.zfs.trim.timeout                    30
        vfs.zfs.trim.txg_delay                  32
        vfs.zfs.spa_min_slop                    134217728
        vfs.zfs.spa_slop_shift                  5
        vfs.zfs.spa_asize_inflation             24
        vfs.zfs.deadman_enabled                 1
        vfs.zfs.deadman_checktime_ms            5000
        vfs.zfs.deadman_synctime_ms             1000000
        vfs.zfs.debug_flags                     0
        vfs.zfs.debugflags                      0
        vfs.zfs.recover                         0
        vfs.zfs.spa_load_verify_data            1
        vfs.zfs.spa_load_verify_metadata        1
        vfs.zfs.spa_load_verify_maxinflight     10000
        vfs.zfs.max_missing_tvds_scan           0
        vfs.zfs.max_missing_tvds_cachefile      2
        vfs.zfs.max_missing_tvds                0
        vfs.zfs.spa_load_print_vdev_tree        0
        vfs.zfs.ccw_retry_interval              300
        vfs.zfs.check_hostid                    1
        vfs.zfs.mg_fragmentation_threshold      85
        vfs.zfs.mg_noalloc_threshold            0
        vfs.zfs.condense_pct                    200
        vfs.zfs.metaslab_sm_blksz               4096
        vfs.zfs.metaslab.bias_enabled           1
        vfs.zfs.metaslab.lba_weighting_enabled  1
        vfs.zfs.metaslab.fragmentation_factor_enabled1
        vfs.zfs.metaslab.preload_enabled        1
        vfs.zfs.metaslab.preload_limit          3
        vfs.zfs.metaslab.unload_delay           8
        vfs.zfs.metaslab.load_pct               50
        vfs.zfs.metaslab.min_alloc_size         33554432
        vfs.zfs.metaslab.df_free_pct            4
        vfs.zfs.metaslab.df_alloc_threshold     131072
        vfs.zfs.metaslab.debug_unload           0
        vfs.zfs.metaslab.debug_load             0
        vfs.zfs.metaslab.fragmentation_threshold70
        vfs.zfs.metaslab.force_ganging          16777217
        vfs.zfs.free_bpobj_enabled              1
        vfs.zfs.free_max_blocks                 18446744073709551615
        vfs.zfs.zfs_scan_checkpoint_interval    7200
        vfs.zfs.zfs_scan_legacy                 0
        vfs.zfs.no_scrub_prefetch               0
        vfs.zfs.no_scrub_io                     0
        vfs.zfs.resilver_min_time_ms            3000
        vfs.zfs.free_min_time_ms                1000
        vfs.zfs.scan_min_time_ms                1000
        vfs.zfs.scan_idle                       50
        vfs.zfs.scrub_delay                     4
        vfs.zfs.resilver_delay                  2
        vfs.zfs.top_maxinflight                 32
        vfs.zfs.delay_scale                     500000
        vfs.zfs.delay_min_dirty_percent         60
        vfs.zfs.dirty_data_sync                 67108864
        vfs.zfs.dirty_data_max_percent          10
        vfs.zfs.dirty_data_max_max              4294967296
        vfs.zfs.dirty_data_max                  4294967296
        vfs.zfs.max_recordsize                  1048576
        vfs.zfs.default_ibs                     15
        vfs.zfs.default_bs                      9
        vfs.zfs.zfetch.array_rd_sz              1048576
        vfs.zfs.zfetch.max_idistance            67108864
        vfs.zfs.zfetch.max_distance             33554432
        vfs.zfs.zfetch.min_sec_reap             2
        vfs.zfs.zfetch.max_streams              8
        vfs.zfs.prefetch_disable                0
        vfs.zfs.send_holes_without_birth_time   1
        vfs.zfs.mdcomp_disable                  0
        vfs.zfs.per_txg_dirty_frees_percent     30
        vfs.zfs.nopwrite_enabled                1
        vfs.zfs.dedup.prefetch                  1
        vfs.zfs.dbuf_cache_lowater_pct          10
        vfs.zfs.dbuf_cache_hiwater_pct          10
        vfs.zfs.dbuf_cache_shift                5
        vfs.zfs.dbuf_cache_max_bytes            4148304512
        vfs.zfs.arc_min_prescient_prefetch_ms   6
        vfs.zfs.arc_min_prefetch_ms             1
        vfs.zfs.l2c_only_size                   0
        vfs.zfs.mfu_ghost_data_esize            1984747520
        vfs.zfs.mfu_ghost_metadata_esize        4070629888
        vfs.zfs.mfu_ghost_size                  6055377408
        vfs.zfs.mfu_data_esize                  1314999296
        vfs.zfs.mfu_metadata_esize              2694144
        vfs.zfs.mfu_size                        3774576640
        vfs.zfs.mru_ghost_data_esize            6311910400
        vfs.zfs.mru_ghost_metadata_esize        3471030272
        vfs.zfs.mru_ghost_size                  9782940672
        vfs.zfs.mru_data_esize                  43289088
        vfs.zfs.mru_metadata_esize              1953792
        vfs.zfs.mru_size                        6746905088
        vfs.zfs.anon_data_esize                 0
        vfs.zfs.anon_metadata_esize             0
        vfs.zfs.anon_size                       288768
        vfs.zfs.l2arc_norw                      0
        vfs.zfs.l2arc_feed_again                1
        vfs.zfs.l2arc_noprefetch                0
        vfs.zfs.l2arc_feed_min_ms               200
        vfs.zfs.l2arc_feed_secs                 1
        vfs.zfs.l2arc_headroom                  2
        vfs.zfs.l2arc_write_boost               40000000
        vfs.zfs.l2arc_write_max                 10000000
        vfs.zfs.arc_meta_limit                  15426000000
        vfs.zfs.arc_free_target                 696014
        vfs.zfs.arc_kmem_cache_reap_retry_ms    1000
        vfs.zfs.compressed_arc_enabled          1
        vfs.zfs.arc_grow_retry                  60
        vfs.zfs.arc_shrink_shift                7
        vfs.zfs.arc_average_blocksize           8192
        vfs.zfs.arc_no_grow_shift               5
        vfs.zfs.arc_min                         16593218048
        vfs.zfs.arc_max                         61704000000
        vfs.zfs.abd_chunk_size                  4096
 

Chris Moore

Hall of Famer
Joined
May 2, 2015
Messages
10,080
One more thing - apparently, L2ARC is not persistent - so your NAS gets to rebuild it every time it reboots.
Absolutely true. It is flushed and needs to be rebuilt based on usage. I think there was some talk of changing that functionality so it could be saved and restored but I don't know that anything will ever come of that.
 

Constantin

Vampire Pig
Joined
May 19, 2017
Messages
1,827
Not trying to ressurrect an old thread but just a quick note that L2ARCs can be made persistent as of TrueNAS 12. That way, their contents are not flushed every time the machine reboots. If you need something slightly faster (especially for small files), then very carefully consider a special VDEV (sVDEV) instead. Here is a link to a thread comparing L2ARC to sVDEV.
 

Stux

MVP
Joined
Jun 2, 2016
Messages
4,358
Interesting thread.

I had a similar problem with CCC style backup performance via rsync just taking too long to scan... Eventually it got to the point where the systems were just scanning the whole time.

I ended up re-architecting things so that backups are done by ZFS replication from one ZFS server to another (and offsite too), meanwhile, changes to OSX requiring SMB3, and TimeMacine moving to APFS helped make the time machine backups faster and that co-incided with upgrading everything to 10gbps, which led to re-architecting the pool to mirrors from RaidZ2..

Which pretty much made the problem go away.

BUT if I still had that problem I know I'd look into this solution :)
 

Constantin

Vampire Pig
Joined
May 19, 2017
Messages
1,827
I found that rsync performance really cratered when it had to deal with literally millions of small files such as all the lproj files that OSX is peppered with. Putting old system backups into disk images based on sparse bundles solved that problem by presenting bands to rsync instead of lots of small files. Works just fine too when I have to mount them.

Ditto document backups, with the added bonus of encryption.

If you use disk images with sparse bundles, only the bands that change have to be updated when something is altered. Otherwise, they present themselves as large contiguous files to rsync, which speeds rsync execution dramatically. This works very well for my use case (quasi-WORM) but may not be ideal for other use cases.
 
Top