shpokas
Dabbler
- Joined
- Oct 3, 2014
- Messages
- 32
Hi guys,
I have a server with 12 x 1 TB SSD disks on LSI SAS3008 storage controller in 6 mirrored VDEVs.
Server has 192 GB of memory
It hosts about 30 virtual machines on zvols over iscsi through 10 GbE to two ESXi servers.
I can add two more NVMe's as a mirrored set to TrueNAS server.
For the setup above and performance stats below - the question is - add them as SLOG, L2ARC or special vdev for metadata?
Some zfs statistics of the TrueNAS server below.
I have a server with 12 x 1 TB SSD disks on LSI SAS3008 storage controller in 6 mirrored VDEVs.
Server has 192 GB of memory
It hosts about 30 virtual machines on zvols over iscsi through 10 GbE to two ESXi servers.
I can add two more NVMe's as a mirrored set to TrueNAS server.
For the setup above and performance stats below - the question is - add them as SLOG, L2ARC or special vdev for metadata?
Some zfs statistics of the TrueNAS server below.
[root@tnas ~]# uptime
11:02AM up 92 days, 11:18, 2 users, load averages: 3.58, 1.44, 0.82# zpool iostat -v POOL 5
capacity operations bandwidth
pool alloc free read write read write
---------------------------------------------- ----- ----- ----- ----- ----- -----
POOL 1.38T 4.05T 0 1.38K 1.58K 42.1M
mirror-0 236G 692G 0 212 0 6.62M
gptid/10f184d5-965d-11ed-802b-a0369ff33974 - - 0 104 0 3.31M
gptid/111e5308-965d-11ed-802b-a0369ff33974 - - 0 107 0 3.31M
mirror-1 235G 693G 0 252 0 6.18M
gptid/113319ed-965d-11ed-802b-a0369ff33974 - - 0 124 0 3.09M
gptid/110052e0-965d-11ed-802b-a0369ff33974 - - 0 127 0 3.09M
mirror-2 237G 691G 0 270 806 8.75M
gptid/10fa7c66-965d-11ed-802b-a0369ff33974 - - 0 135 0 4.37M
gptid/1108d3f3-965d-11ed-802b-a0369ff33974 - - 0 135 806 4.37M
mirror-3 235G 693G 0 163 0 4.48M
gptid/10c327dc-965d-11ed-802b-a0369ff33974 - - 0 81 0 2.24M
gptid/1128d106-965d-11ed-802b-a0369ff33974 - - 0 82 0 2.24M
mirror-4 237G 691G 0 275 0 8.01M
gptid/1103a197-965d-11ed-802b-a0369ff33974 - - 0 138 0 4.00M
gptid/10c88186-965d-11ed-802b-a0369ff33974 - - 0 136 0 4.00M
mirror-5 237G 691G 0 243 806 8.07M
gptid/10eb3468-965d-11ed-802b-a0369ff33974 - - 0 121 806 4.03M
gptid/11307a07-965d-11ed-802b-a0369ff33974 - - 0 122 0 4.03M
---------------------------------------------- ----- ----- ----- ----- ----- -----
------------------------------------------------------------------------
ZFS Subsystem Report Thu Mar 28 10:36:24 2024
FreeBSD 13.1-RELEASE-p9 zpl version 5
Machine: tnas (amd64) spa version 5000
ARC status: HEALTHY
Memory throttle count: 0
ARC size (current): 79.8 % 137.8 GiB
Target size (adaptive): 79.9 % 138.0 GiB
Min size (hard limit): 3.5 % 6.0 GiB
Max size (high water): 28:1 172.7 GiB
Most Frequently Used (MFU) cache size: 76.1 % 102.2 GiB
Most Recently Used (MRU) cache size: 23.9 % 32.2 GiB
Metadata cache size (hard limit): 75.0 % 129.5 GiB
Metadata cache size (current): 3.9 % 5.0 GiB
Dnode cache size (hard limit): 10.0 % 13.0 GiB
Dnode cache size (current): 2.3 % 308.3 MiB
ARC hash breakdown:
Elements max: 17.8M
Elements current: 70.8 % 12.6M
Collisions: 929.6M
Chain max: 10
Chains: 1.8M
ARC misc:
Deleted: 74.2M
Mutex misses: 231.1k
Eviction skips: 48.3k
Eviction skips due to L2 writes: 0
L2 cached evictions: 0 Bytes
L2 eligible evictions: 2.7 TiB
L2 eligible MFU evictions: 92.3 % 2.5 TiB
L2 eligible MRU evictions: 7.7 % 215.4 GiB
L2 ineligible evictions: 40.7 GiB
ARC total accesses (hits + misses): 48.7G
Cache hit ratio: 99.9 % 48.7G
Cache miss ratio: 0.1 % 34.9M
Actual hit ratio (MFU + MRU hits): 99.9 % 48.7G
Data demand efficiency: 99.5 % 4.7G
Data prefetch efficiency: 88.1 % 75.7M
Cache hits by cache type:
Most frequently used (MFU): 94.5 % 46.0G
Most recently used (MRU): 5.4 % 2.6G
Most frequently used (MFU) ghost: < 0.1 % 10.5M
Most recently used (MRU) ghost: < 0.1 % 2.1M
Anonymously used: < 0.1 % 8.7M
Cache hits by data type:
Demand data: 9.6 % 4.7G
Prefetch data: 0.1 % 66.6M
Demand metadata: 90.2 % 43.9G
Prefetch metadata: < 0.1 % 8.8M
Cache misses by data type:
Demand data: 62.6 % 21.8M
Prefetch data: 25.9 % 9.0M
Demand metadata: 3.9 % 1.4M
Prefetch metadata: 7.5 % 2.6M
DMU prefetch efficiency: 1.9G
Hit ratio: 5.1 % 95.4M
Miss ratio: 94.9 % 1.8G
L2ARC not detected, skipping section
Tunables:
abd_scatter_enabled 1
abd_scatter_min_size 4097
allow_redacted_dataset_mount 0
anon_data_esize 0
anon_metadata_esize 0
anon_size 43586048
arc.average_blocksize 8192
arc.dnode_limit 0
arc.dnode_limit_percent 10
arc.dnode_reduce_percent 10
arc.evict_batch_limit 10
arc.eviction_pct 200
arc.grow_retry 0
arc.lotsfree_percent 10
arc.max 185454000000
arc.meta_adjust_restarts 4096
arc.meta_limit 0
arc.meta_limit_percent 75
arc.meta_min 0
arc.meta_prune 10000
arc.meta_strategy 1
arc.min 0
arc.min_prefetch_ms 0
arc.min_prescient_prefetch_ms 0
arc.p_dampener_disable 1
arc.p_min_shift 0
arc.pc_percent 0
arc.prune_task_threads 1
arc.shrink_shift 0
arc.sys_free 0
arc_free_target 1044346
arc_max 185454000000
arc_min 0
arc_no_grow_shift 5
async_block_max_blocks 18446744073709551615
autoimport_disable 1
btree_verify_intensity 0
ccw_retry_interval 300
checksum_events_per_second 20
commit_timeout_pct 5
compressed_arc_enabled 1
condense.indirect_commit_entry_delay_ms 0
condense.indirect_obsolete_pct 25
condense.indirect_vdevs_enable 1
condense.max_obsolete_bytes 1073741824
condense.min_mapping_bytes 131072
condense_pct 200
crypt_sessions 0
dbgmsg_enable 1
dbgmsg_maxsize 4194304
dbuf.cache_shift 5
dbuf.metadata_cache_max_bytes 18446744073709551615
dbuf.metadata_cache_shift 6
dbuf_cache.hiwater_pct 10
dbuf_cache.lowater_pct 10
dbuf_cache.max_bytes 18446744073709551615
dbuf_state_index 0
ddt_data_is_special 1
deadman.checktime_ms 60000
deadman.enabled 1
deadman.failmode wait
deadman.synctime_ms 600000
deadman.ziotime_ms 300000
debug 0
debugflags 0
dedup.prefetch 0
default_bs 9
default_ibs 15
delay_min_dirty_percent 60
delay_scale 500000
dirty_data_max 4294967296
dirty_data_max_max 4294967296
dirty_data_max_max_percent 25
dirty_data_max_percent 10
dirty_data_sync_percent 20
disable_ivset_guid_check 0
dmu_object_alloc_chunk_shift 7
dmu_offset_next_sync 1
dmu_prefetch_max 134217728
dtl_sm_blksz 4096
embedded_slog_min_ms 64
flags 0
fletcher_4_impl [fastest] scalar superscalar superscalar4 sse2 ssse3
free_bpobj_enabled 1
free_leak_on_eio 0
free_min_time_ms 1000
history_output_max 1048576
immediate_write_sz 32768
initialize_chunk_size 1048576
initialize_value 16045690984833335022
keep_log_spacemaps_at_export 0
l2arc.exclude_special 0
l2arc.feed_again 1
l2arc.feed_min_ms 200
l2arc.feed_secs 1
l2arc.headroom 2
l2arc.headroom_boost 200
l2arc.meta_percent 33
l2arc.mfuonly 0
l2arc.noprefetch 0
l2arc.norw 0
l2arc.rebuild_blocks_min_l2size 1073741824
l2arc.rebuild_enabled 0
l2arc.trim_ahead 0
l2arc.write_boost 40000000
l2arc.write_max 10000000
l2arc_feed_again 1
l2arc_feed_min_ms 200
l2arc_feed_secs 1
l2arc_headroom 2
l2arc_noprefetch 0
l2arc_norw 0
l2arc_write_boost 40000000
l2arc_write_max 10000000
l2c_only_size 0
livelist.condense.new_alloc 0
livelist.condense.sync_cancel 0
livelist.condense.sync_pause 0
livelist.condense.zthr_cancel 0
livelist.condense.zthr_pause 0
livelist.max_entries 500000
livelist.min_percent_shared 75
lua.max_instrlimit 100000000
lua.max_memlimit 104857600
max_async_dedup_frees 100000
max_auto_ashift 14
max_dataset_nesting 50
max_log_walking 5
max_logsm_summary_length 10
max_missing_tvds 0
max_missing_tvds_cachefile 2
max_missing_tvds_scan 0
max_nvlist_src_size 0
max_recordsize 1048576
metaslab.aliquot 1048576
metaslab.bias_enabled 1
metaslab.debug_load 0
metaslab.debug_unload 0
metaslab.df_alloc_threshold 131072
metaslab.df_free_pct 4
metaslab.df_max_search 16777216
metaslab.df_use_largest_segment 0
metaslab.find_max_tries 100
metaslab.force_ganging 16777217
metaslab.fragmentation_factor_enabled 1
metaslab.fragmentation_threshold 70
metaslab.lba_weighting_enabled 1
metaslab.load_pct 50
metaslab.max_size_cache_sec 3600
metaslab.mem_limit 25
metaslab.preload_enabled 1
metaslab.preload_limit 10
metaslab.segment_weight_enabled 1
metaslab.sm_blksz_no_log 16384
metaslab.sm_blksz_with_log 131072
metaslab.switch_threshold 2
metaslab.try_hard_before_gang 0
metaslab.unload_delay 32
metaslab.unload_delay_ms 600000
mfu_data_esize 107134380032
mfu_ghost_data_esize 66160526336
mfu_ghost_metadata_esize 10336311808
mfu_ghost_size 76496838144
mfu_metadata_esize 268150784
mfu_size 109762782720
mg.fragmentation_threshold 95
mg.noalloc_threshold 0
min_auto_ashift 9
min_metaslabs_to_flush 1
mru_data_esize 29634497536
mru_ghost_data_esize 69762585088
mru_ghost_metadata_esize 1614550528
mru_ghost_size 71377135616
mru_metadata_esize 670851584
mru_size 34543990272
multihost.fail_intervals 10
multihost.history 0
multihost.import_intervals 20
multihost.interval 1000
multilist_num_sublists 0
no_scrub_io 0
no_scrub_prefetch 0
nocacheflush 0
nopwrite_enabled 1
obsolete_min_time_ms 500
pd_bytes_max 52428800
per_txg_dirty_frees_percent 30
prefetch.array_rd_sz 1048576
prefetch.disable 0
prefetch.max_distance 67108864
prefetch.max_idistance 67108864
prefetch.max_sec_reap 2
prefetch.max_streams 8
prefetch.min_distance 4194304
prefetch.min_sec_reap 1
read_history 0
read_history_hits 0
rebuild_max_segment 1048576
rebuild_scrub_enabled 1
rebuild_vdev_limit 67108864
reconstruct.indirect_combinations_max 4096
recover 0
recv.queue_ff 20
recv.queue_length 16777216
recv.write_batch_size 1048576
removal_suspend_progress 0
remove_max_segment 16777216
resilver_disable_defer 0
resilver_min_time_ms 3000
scan_blkstats 0
scan_checkpoint_intval 7200
scan_fill_weight 3
scan_ignore_errors 0
scan_issue_strategy 0
scan_legacy 0
scan_max_ext_gap 2097152
scan_mem_lim_fact 20
scan_mem_lim_soft_fact 20
scan_report_txgs 0
scan_strict_mem_lim 0
scan_suspend_progress 0
scan_vdev_limit 16777216
scrub_min_time_ms 1000
send.corrupt_data 0
send.no_prefetch_queue_ff 20
send.no_prefetch_queue_length 1048576
send.override_estimate_recordsize 0
send.queue_ff 20
send.queue_length 16777216
send.unmodified_spill_blocks 1
send_holes_without_birth_time 1
slow_io_events_per_second 20
spa.asize_inflation 24
spa.discard_memory_limit 16777216
spa.load_print_vdev_tree 0
spa.load_verify_data 1
spa.load_verify_metadata 1
spa.load_verify_shift 4
spa.slop_shift 5
space_map_ibs 14
special_class_metadata_reserve_pct 25
standard_sm_blksz 131072
super_owner 0
sync_pass_deferred_free 2
sync_pass_dont_compress 8
sync_pass_rewrite 2
sync_taskq_batch_pct 75
top_maxinflight 1000
traverse_indirect_prefetch_limit 32
trim.extent_bytes_max 134217728
trim.extent_bytes_min 32768
trim.metaslab_skip 0
trim.queue_limit 10
trim.txg_batch 32
txg.history 100
txg.timeout 5
unflushed_log_block_max 131072
unflushed_log_block_min 1000
unflushed_log_block_pct 400
unflushed_log_txg_max 1000
unflushed_max_mem_amt 1073741824
unflushed_max_mem_ppm 1000
user_indirect_is_special 1
validate_skip 0
vdev.aggregate_trim 0
vdev.aggregation_limit 1048576
vdev.aggregation_limit_non_rotating 131072
vdev.async_read_max_active 3
vdev.async_read_min_active 1
vdev.async_write_active_max_dirty_percent 60
vdev.async_write_active_min_dirty_percent 30
vdev.async_write_max_active 5
vdev.async_write_min_active 1
vdev.bio_delete_disable 0
vdev.bio_flush_disable 0
vdev.cache_bshift 16
vdev.cache_max 16384
vdev.cache_size 0
vdev.def_queue_depth 32
vdev.default_ms_count 200
vdev.default_ms_shift 29
vdev.file.logical_ashift 9
vdev.file.physical_ashift 9
vdev.initializing_max_active 1
vdev.initializing_min_active 1
vdev.max_active 1000
vdev.max_auto_ashift 14
vdev.min_auto_ashift 9
vdev.min_ms_count 16
vdev.mirror.non_rotating_inc 0
vdev.mirror.non_rotating_seek_inc 1
vdev.mirror.rotating_inc 0
vdev.mirror.rotating_seek_inc 5
vdev.mirror.rotating_seek_offset 1048576
vdev.ms_count_limit 131072
vdev.nia_credit 5
vdev.nia_delay 5
vdev.queue_depth_pct 1000
vdev.read_gap_limit 32768
vdev.rebuild_max_active 3
vdev.rebuild_min_active 1
vdev.removal_ignore_errors 0
vdev.removal_max_active 2
vdev.removal_max_span 32768
vdev.removal_min_active 1
vdev.removal_suspend_progress 0
vdev.remove_max_segment 16777216
vdev.scrub_max_active 3
vdev.scrub_min_active 1
vdev.sync_read_max_active 10
vdev.sync_read_min_active 10
vdev.sync_write_max_active 10
vdev.sync_write_min_active 10
vdev.trim_max_active 2
vdev.trim_min_active 1
vdev.validate_skip 0
vdev.write_gap_limit 4096
version.acl 1
version.ioctl 15
version.module v2023120100-zfs_f4871096b
version.spa 5000
version.zpl 5
vnops.read_chunk_size 1048576
vol.mode 2
vol.recursive 0
vol.unmap_enabled 1
wrlog_data_max 8589934592
xattr_compat 1
zap_iterate_prefetch 1
zevent.len_max 512
zevent.retain_expire_secs 900
zevent.retain_max 2000
zfetch.max_distance 67108864
zfetch.max_idistance 67108864
zil.clean_taskq_maxalloc 1048576
zil.clean_taskq_minalloc 1024
zil.clean_taskq_nthr_pct 100
zil.maxblocksize 131072
zil.min_commit_timeout 5000
zil.nocacheflush 0
zil.replay_disable 0
zil.slog_bulk 786432
zio.deadman_log_all 0
zio.dva_throttle_enabled 1
zio.exclude_metadata 0
zio.requeue_io_start_cut_in_line 1
zio.slow_io_ms 30000
zio.taskq_batch_pct 80
zio.taskq_batch_tpq 0
zio.use_uma 1
VDEV cache disabled, skipping section
ZIL committed transactions: 1.6G
Commit requests: 1.6G
Flushes to stable storage: 1.4G
Transactions to SLOG storage pool: 0 Bytes 0
Transactions to non-SLOG storage pool: 10.9 TiB 951.4M
