I just completed a upgrade of my SIG, main FN system, from 11.2-U5 ->13.0u3.1) (is a physical system, TrueNas only - ie TN is not virtualized, and within TN i do NOT run ANY jails nor VMs).
I always felt i was getting reduced performance out of my flash based pools (most specifically - my 1.6t nvme Mirror , and my 3x vDev of 6x z2 ssd Pool), but my HDD Pools seemed to perform "as expected". All are pools used for NFS (to 3x vmware hosts) and all use a optane p900 as slog (i partitioned it into 16gb slices).
(NOTE: i have still NOT upgraded my pools' zfs feature flags yet, incase issues pop up, although there have been 0 issues thus far)
Since the 11.2->13.x upgrade, performance has increased quite a bit!:
The main question of this thread relates to DELETE IOPS (in regards to Read / Write / Delete - IOPS).
Were these constant delete IOPS due to TRIM being enabled under 11.2 ? and as i understand it, under 13.0 , temporarily (SMR), trim is disabled on all pools.
Or was this caused by something else perhaps? (see images below)
(should i consider using cron to weekly run: zpool trim ssdpoolXYZ )
zpool status -t
zpool get autotrim
I keep very detailed stats on this system, mainly through TN->graphite->grafana , and some custom scripts i run as cron tasks (also going to graphite or splunk).
overall, im seeing a huge improvement in perf. since upgrading to 13, and a BIG, decrease in the load to this SLOG only optane (anecdotally - as i always have "gstat -s -p" running in a ssh session on my desktop, and keep an eye on it) -
( btw, the upgrade went flawlessly! i did manual updates via .tar uploads , following the path described in other threads/reddit posts of: (starting at 11.2-u5) - 11.2-U5 -> 11.2 U8 > 11.3u5 > 12.0u8.1 > 13.0u3.1).
(+ i have yet to update the pools feature flags yet!)
In the images below, these are from one ssd from the ssd3x6x480GBz2 (the stats on the others drives in this pool look the exact same, as expected).
also including a snap of stats from Truenas 13 web gui (to confirm / show that the stats match my external stats/graphite/grafana)
View attachment 63128
View attachment 63129
View attachment 63130
View attachment 63132
I always felt i was getting reduced performance out of my flash based pools (most specifically - my 1.6t nvme Mirror , and my 3x vDev of 6x z2 ssd Pool), but my HDD Pools seemed to perform "as expected". All are pools used for NFS (to 3x vmware hosts) and all use a optane p900 as slog (i partitioned it into 16gb slices).
(NOTE: i have still NOT upgraded my pools' zfs feature flags yet, incase issues pop up, although there have been 0 issues thus far)
Since the 11.2->13.x upgrade, performance has increased quite a bit!:
example #1- being the LOWER guest-vm-observed disk latency via VMware Stats, of my NFS shares
example #2- much less IOwait in vmware guests (i also keep track of this, and have seen a large decrease / improvement)
The main question of this thread relates to DELETE IOPS (in regards to Read / Write / Delete - IOPS).
Were these constant delete IOPS due to TRIM being enabled under 11.2 ? and as i understand it, under 13.0 , temporarily (SMR), trim is disabled on all pools.
Or was this caused by something else perhaps? (see images below)
(should i consider using cron to weekly run: zpool trim ssdpoolXYZ )
zpool status -t
zpool get autotrim
Code:
[root@freenas ~/grafanaSENDscripts]# zpool get autotrim
NAME PROPERTY VALUE SOURCE
freenas-boot autotrim off default
he5x8TBz1 autotrim off default
he8x8TBz2 autotrim off default
nvmeMIRROR autotrim off default
red4tbENCz1 autotrim off default
ssd3x6x480GBz2 autotrim off default
ssd4x1600gZ1 autotrim off default
sun2x7TBrd0 autotrim off default
sun7x8TBz1 autotrim off default
pool: ssd3x6x480GBz2
state: ONLINE
status: Some supported and requested features are not enabled on the pool.
The pool can still be used, but some features are unavailable.
action: Enable all features using 'zpool upgrade'. Once this is done,
the pool may no longer be accessible by software that does not support
the features. See zpool-features(7) for details.
scan: scrub repaired 0B in 00:46:38 with 0 errors on Sun Jan 8 19:46:41 2023
config:
NAME STATE READ WRITE CKSUM
ssd3x6x480GBz2 ONLINE 0 0 0
raidz2-0 ONLINE 0 0 0
gptid/b238bbd8-d82e-11e9-bae4-00259084f1c8 ONLINE 0 0 0 (untrimmed)
gptid/b2e8a1de-d82e-11e9-bae4-00259084f1c8 ONLINE 0 0 0 (untrimmed)
gptid/b382351d-d82e-11e9-bae4-00259084f1c8 ONLINE 0 0 0 (untrimmed)
gptid/b421e71c-d82e-11e9-bae4-00259084f1c8 ONLINE 0 0 0 (untrimmed)
gptid/b4d79d37-d82e-11e9-bae4-00259084f1c8 ONLINE 0 0 0 (untrimmed)
gptid/b580104e-d82e-11e9-bae4-00259084f1c8 ONLINE 0 0 0 (untrimmed)
raidz2-1 ONLINE 0 0 0
gptid/b670b4d0-d82e-11e9-bae4-00259084f1c8 ONLINE 0 0 0 (untrimmed)
gptid/b714597f-d82e-11e9-bae4-00259084f1c8 ONLINE 0 0 0 (untrimmed)
gptid/b7d8bca7-d82e-11e9-bae4-00259084f1c8 ONLINE 0 0 0 (untrimmed)
gptid/b88979e7-d82e-11e9-bae4-00259084f1c8 ONLINE 0 0 0 (untrimmed)
gptid/b94e1216-d82e-11e9-bae4-00259084f1c8 ONLINE 0 0 0 (untrimmed)
gptid/b9f0c93b-d82e-11e9-bae4-00259084f1c8 ONLINE 0 0 0 (untrimmed)
raidz2-2 ONLINE 0 0 0
gptid/bb0fa8d3-d82e-11e9-bae4-00259084f1c8 ONLINE 0 0 0 (untrimmed)
gptid/bbb3f9cf-d82e-11e9-bae4-00259084f1c8 ONLINE 0 0 0 (untrimmed)
gptid/bc74e749-d82e-11e9-bae4-00259084f1c8 ONLINE 0 0 0 (untrimmed)
gptid/bd35b67b-d82e-11e9-bae4-00259084f1c8 ONLINE 0 0 0 (untrimmed)
gptid/bde00cf9-d82e-11e9-bae4-00259084f1c8 ONLINE 0 0 0 (untrimmed)
gptid/be9f1fbc-d82e-11e9-bae4-00259084f1c8 ONLINE 0 0 0 (untrimmed)
logs
gptid/f166ac78-c7bc-11e9-975e-00259084f1c8 ONLINE 0 0 0 (untrimmed)
smart from one of the ssds above:
root@freenas:~ # smart 13
smartctl 7.2 2021-09-14 r5236 [FreeBSD 13.1-RELEASE-p2 amd64] (local build)
Copyright (C) 2002-20, Bruce Allen, Christian Franke, www.smartmontools.org
=== START OF INFORMATION SECTION ===
Vendor: HGST
Product: HSCP0480S5xnNMRI
Revision: 3P02
Compliance: SPC-4
User Capacity: 480,999,997,440 bytes [480 GB]
Logical block size: 512 bytes
Physical block size: 4096 bytes
LU is resource provisioned, LBPRZ=1
Rotation Rate: Solid State Device
Form Factor: 2.5 inches
Logical Unit id: 0x5000cca04e0c69e4
Serial number: 0QV6UMUA
Device type: disk
Transport protocol: SAS (SPL-3)
Local Time is: Tue Jan 31 09:37:03 2023 CST
SMART support is: Available - device has SMART capability.
SMART support is: Enabled
Temperature Warning: Enabled
=== START OF READ SMART DATA SECTION ===
SMART Health Status: OK
Percentage used endurance indicator: 8%
Current Drive Temperature: 33 C
Drive Trip Temperature: 70 C
Accumulated power on time, hours:minutes 55900:42
Manufactured in week 04 of year 2015
Specified cycle count over device lifetime: 0
Accumulated start-stop cycles: 0
Specified load-unload count over device lifetime: 0
Accumulated load-unload cycles: 0
Vendor (Seagate Cache) information
Blocks sent to initiator = 157920815440986112
Error counter log:
Errors Corrected by Total Correction Gigabytes Total
ECC rereads/ errors algorithm processed uncorrected
fast | delayed rewrites corrected invocations [10^9 bytes] errors
read: 0 0 0 0 0 116421.109 0
write: 0 0 0 0 0 202844.592 0
verify: 0 0 0 0 0 56865.379 0
Non-medium error count: 0
SMART Self-test log
Num Test Status segment LifeTime LBA_first_err [SK ASC ASQ]
Description number (hours)
# 1 Background long Completed - 26272 - [- - -]
Long (extended) Self-test duration: 3468 seconds [57.8 minutes]
I keep very detailed stats on this system, mainly through TN->graphite->grafana , and some custom scripts i run as cron tasks (also going to graphite or splunk).
overall, im seeing a huge improvement in perf. since upgrading to 13, and a BIG, decrease in the load to this SLOG only optane (anecdotally - as i always have "gstat -s -p" running in a ssh session on my desktop, and keep an eye on it) -
( btw, the upgrade went flawlessly! i did manual updates via .tar uploads , following the path described in other threads/reddit posts of: (starting at 11.2-u5) - 11.2-U5 -> 11.2 U8 > 11.3u5 > 12.0u8.1 > 13.0u3.1).
(+ i have yet to update the pools feature flags yet!)
In the images below, these are from one ssd from the ssd3x6x480GBz2 (the stats on the others drives in this pool look the exact same, as expected).
also including a snap of stats from Truenas 13 web gui (to confirm / show that the stats match my external stats/graphite/grafana)
View attachment 63128
View attachment 63129
View attachment 63130
View attachment 63132
