I've recently moved to Truenas 13.0-U1 and noticed that some of my grafana I/O graphs are no longer working. Not totally unexpected, however in trying to fix them, I can't seem to find separate read/write IO metrics in any of the measurements that are being stored in influxdb. Truenas is clearly recording them since I can see separate read/write graphs in Truenas's native Reporting sections, but I only get net I/O, not separate iops for read or write.
Code:
> show measurements name: measurements name ---- arcstat_ratio_arc-hits arcstat_ratio_arc-l2_hits arcstat_ratio_arc-l2_misses arcstat_ratio_arc-misses arcstat_ratio_data-demand_data_hits arcstat_ratio_data-demand_data_misses arcstat_ratio_data-prefetch_data_hits arcstat_ratio_data-prefetch_data_misses arcstat_ratio_metadata-demand_metadata_hits arcstat_ratio_metadata-demand_metadata_misses arcstat_ratio_metadata-prefetch_metadata_hits arcstat_ratio_metadata-prefetch_metadata_misses arcstat_ratio_mu-mfu_ghost_hits arcstat_ratio_mu-mfu_hits arcstat_ratio_mu-mru_ghost_hits arcstat_ratio_mu-mru_hits cache_eviction-cached cache_eviction-eligible cache_eviction-ineligible cache_operation-allocated cache_operation-deleted cache_ratio-arc cache_result-demand_data-hit cache_result-demand_data-miss cache_result-demand_metadata-hit cache_result-demand_metadata-miss cache_result-mfu-hit cache_result-mfu_ghost-hit cache_result-mru-hit cache_result-mru_ghost-hit cache_result-prefetch_data-hit cache_result-prefetch_data-miss cache_result-prefetch_metadata-hit cache_result-prefetch_metadata-miss cache_size-L2 cache_size-anon_size cache_size-arc cache_size-bonus_size cache_size-c cache_size-c_max cache_size-c_min cache_size-dbuf_size cache_size-dnode_size cache_size-hdr_size cache_size-metadata_size cache_size-mfu_ghost_size cache_size-mfu_size cache_size-mru_ghost_size cache_size-mru_size cache_size-other_size cache_size-p counter-journal-bytes counter-journal-rotates cpu-idle cpu-interrupt cpu-nice cpu-system cpu-user df_complex-free df_complex-reserved df_complex-used disk_io_time disk_octets disk_octets-0-0 disk_ops disk_ops-0-0 disk_time disk_time-0-0 gauge-tree_depth gauge-tree_nodes gauge_arcstats_raw-l2_asize gauge_arcstats_raw-l2_hdr_size gauge_arcstats_raw-l2_size gauge_arcstats_raw_arcmeta-arc_meta_limit gauge_arcstats_raw_arcmeta-arc_meta_max gauge_arcstats_raw_arcmeta-arc_meta_min gauge_arcstats_raw_arcmeta-arc_meta_used gauge_arcstats_raw_counts-allocated gauge_arcstats_raw_counts-deleted gauge_arcstats_raw_counts-mutex_miss gauge_arcstats_raw_counts-recycle_miss gauge_arcstats_raw_counts-stolen gauge_arcstats_raw_cp-c gauge_arcstats_raw_cp-c_max gauge_arcstats_raw_cp-c_min gauge_arcstats_raw_cp-p gauge_arcstats_raw_demand-demand_data_hits gauge_arcstats_raw_demand-demand_data_misses gauge_arcstats_raw_demand-demand_metadata_hits gauge_arcstats_raw_demand-demand_metadata_misses gauge_arcstats_raw_duplicate-duplicate_buffers gauge_arcstats_raw_duplicate-duplicate_buffers_size gauge_arcstats_raw_duplicate-duplicate_reads gauge_arcstats_raw_evict-evict_l2_cached gauge_arcstats_raw_evict-evict_l2_eligible gauge_arcstats_raw_evict-evict_l2_ineligible gauge_arcstats_raw_evict-evict_skip gauge_arcstats_raw_hash-hash_chain_max gauge_arcstats_raw_hash-hash_chains gauge_arcstats_raw_hash-hash_collisions gauge_arcstats_raw_hash-hash_elements gauge_arcstats_raw_hash-hash_elements_max gauge_arcstats_raw_hits_misses-hits gauge_arcstats_raw_hits_misses-misses gauge_arcstats_raw_l2-l2_cksum_bad gauge_arcstats_raw_l2-l2_feeds gauge_arcstats_raw_l2-l2_hits gauge_arcstats_raw_l2-l2_io_error gauge_arcstats_raw_l2-l2_misses gauge_arcstats_raw_l2-l2_rw_clash gauge_arcstats_raw_l2_compress-l2_compress_failures gauge_arcstats_raw_l2_compress-l2_compress_successes gauge_arcstats_raw_l2_compress-l2_compress_zeros gauge_arcstats_raw_l2_free-l2_cdata_free_on_write gauge_arcstats_raw_l2_free-l2_free_on_write gauge_arcstats_raw_l2abort-l2_abort_lowmem gauge_arcstats_raw_l2bytes-l2_read_bytes gauge_arcstats_raw_l2bytes-l2_write_bytes gauge_arcstats_raw_l2evict-l2_evict_lock_retry gauge_arcstats_raw_l2evict-l2_evict_reading gauge_arcstats_raw_l2write-l2_write_buffer_bytes_scanned gauge_arcstats_raw_l2write-l2_write_buffer_iter gauge_arcstats_raw_l2write-l2_write_buffer_list_iter gauge_arcstats_raw_l2write-l2_write_buffer_list_null_iter gauge_arcstats_raw_l2write-l2_write_full gauge_arcstats_raw_l2write-l2_write_in_l2 gauge_arcstats_raw_l2write-l2_write_io_in_progress gauge_arcstats_raw_l2write-l2_write_not_cacheable gauge_arcstats_raw_l2write-l2_write_passed_headroom gauge_arcstats_raw_l2write-l2_write_pios gauge_arcstats_raw_l2write-l2_write_spa_mismatch gauge_arcstats_raw_l2write-l2_write_trylock_fail gauge_arcstats_raw_l2writes-l2_writes_done gauge_arcstats_raw_l2writes-l2_writes_error gauge_arcstats_raw_l2writes-l2_writes_hdr_miss gauge_arcstats_raw_l2writes-l2_writes_sent gauge_arcstats_raw_memcount-memory_throttle_count gauge_arcstats_raw_mru-mfu_ghost_hits gauge_arcstats_raw_mru-mfu_hits gauge_arcstats_raw_mru-mru_ghost_hits gauge_arcstats_raw_mru-mru_hits gauge_arcstats_raw_prefetch-prefetch_data_hits gauge_arcstats_raw_prefetch-prefetch_data_misses gauge_arcstats_raw_prefetch-prefetch_metadata_hits gauge_arcstats_raw_prefetch-prefetch_metadata_misses gauge_arcstats_raw_size-data_size gauge_arcstats_raw_size-hdr_size gauge_arcstats_raw_size-other_size gauge_arcstats_raw_size-size geom_busy_percent-ada0 geom_busy_percent-ada1 geom_busy_percent-ada2 geom_busy_percent-ada3 geom_busy_percent-ada4 geom_busy_percent-ada5 geom_busy_percent-ada6 geom_busy_percent-da0 geom_busy_percent-da1 geom_bw-ada0 geom_bw-ada1 geom_bw-ada2 geom_bw-ada3 geom_bw-ada4 geom_bw-ada5 geom_bw-ada6 geom_bw-da0 geom_bw-da1 geom_latency-ada0 geom_latency-ada1 geom_latency-ada2 geom_latency-ada3 geom_latency-ada4 geom_latency-ada5 geom_latency-ada6 geom_latency-da0 geom_latency-da1 geom_ops-ada0 geom_ops-ada1 geom_ops-ada2 geom_ops-ada3 geom_ops-ada4 geom_ops-ada5 geom_ops-ada6 geom_ops-da0 geom_ops-da1 geom_ops_rwd-ada0 geom_ops_rwd-ada1 geom_ops_rwd-ada2 geom_ops_rwd-ada3 geom_ops_rwd-ada4 geom_ops_rwd-ada5 geom_ops_rwd-ada6 geom_ops_rwd-da0 geom_ops_rwd-da1 geom_queue-ada0 geom_queue-ada1 geom_queue-ada2 geom_queue-ada3 geom_queue-ada4 geom_queue-ada5 geom_queue-ada6 geom_queue-da0 geom_queue-da1 hash_collisions if_errors if_octets if_packets io_octets-L2 load memory-active memory-cache memory-free memory-inactive memory-laundry memory-wired memory_throttle_count mutex_operations-miss nfsstat-access nfsstat-commit nfsstat-create nfsstat-fsinfo nfsstat-fsstat nfsstat-getattr nfsstat-link nfsstat-lookup nfsstat-mkdir nfsstat-mknod nfsstat-pathconf nfsstat-read nfsstat-read_bytes nfsstat-readdir nfsstat-readirplus nfsstat-readlink nfsstat-remove nfsstat-rename nfsstat-rmdir nfsstat-setattr nfsstat-symlink nfsstat-write nfsstat-write_bytes operations-receive-flush operations-receive-update operations-write-data_sets operations-write-updates pending_operations ps_state-blocked ps_state-idle ps_state-running ps_state-sleeping ps_state-stopped ps_state-wait ps_state-zombies queue_length swap-free swap-used temperature uptime
