Slow write speeds ~10-20MB/s

icepicknz · Feb 11, 2023

Hi Everyone.

Been looking through the forums and trying to find the answer, have tried a number of things and have done benchmark tests from other peoples troubleshooting.
I noticed the issue recently when I copied my Plex media from one folder to another, it took nearly a day, so I decided to investigate.

Vol1 is used for Jails/VM - very low usage, just nginx, Pritunl, Plex / System Dataset Pool
Disk da8 & da9 make up this volume and are SEAGATE ST4000NM0023 10k rpm (See test below)

Code:

NAME                                            STATE     READ WRITE CKSUM
Vol1                                            ONLINE       0     0     0
  mirror-0                                      ONLINE       0     0     0
    gptid/64ebe6e0-bd0e-11eb-ac17-b82a72cfcabd  ONLINE       0     0     0
    gptid/6538a3ce-bd0e-11eb-ac17-b82a72cfcabd  ONLINE       0     0     0

Vol2 is used for media only, so written to very seldom, mostly read by Plex.
The remainder of the disks are SEAGATE ST4000NM0023, 7200 RPM (see test below)

Code:

NAME                                            STATE     READ WRITE CKSUM
Vol2                                            ONLINE       0     0     0
  raidz1-0                                      ONLINE       0     0     0
    gptid/1c2f74d9-bd0f-11eb-ac17-b82a72cfcabd  ONLINE       0     0     0
    gptid/1d830a1b-bd0f-11eb-ac17-b82a72cfcabd  ONLINE       0     0     0
    gptid/1e07214a-bd0f-11eb-ac17-b82a72cfcabd  ONLINE       0     0     0
    gptid/1f835716-bd0f-11eb-ac17-b82a72cfcabd  ONLINE       0     0     0
    gptid/1f470494-bd0f-11eb-ac17-b82a72cfcabd  ONLINE       0     0     0
    gptid/1fea528c-bd0f-11eb-ac17-b82a72cfcabd  ONLINE       0     0     0
    gptid/20b39770-bd0f-11eb-ac17-b82a72cfcabd  ONLINE       0     0     0
    gptid/214276cf-bd0f-11eb-ac17-b82a72cfcabd  ONLINE       0     0     0
    gptid/21337812-bd0f-11eb-ac17-b82a72cfcabd  ONLINE       0     0     0
    gptid/21af5b63-bd0f-11eb-ac17-b82a72cfcabd  ONLINE       0     0     0
    gptid/21be580a-bd0f-11eb-ac17-b82a72cfcabd  ONLINE       0     0     0
    gptid/21cf6056-bd0f-11eb-ac17-b82a72cfcabd  ONLINE       0     0     0

Code:

NAME        STATE     READ WRITE CKSUM
boot-pool   ONLINE       0     0     0
  da14p2    ONLINE       0     0     0

One of the two VM/Jail disks / System Dataset Pool

Code:

[root@TGA-Truenas ~]# diskinfo -tv /dev/da8
/dev/da8
    512             # sectorsize
    300000000000    # mediasize in bytes (279G)
    585937500       # mediasize in sectors
    0               # stripesize
    0               # stripeoffset
    36472           # Cylinders according to firmware.
    255             # Heads according to firmware.
    63              # Sectors according to firmware.
    SEAGATE ST300MM0006    # Disk descr.
    S0K2F6GC        # Disk ident.
    mps0            # Attachment
    No              # TRIM/UNMAP support
    10000           # Rotation rate in RPM
    Not_Zoned       # Zone Mode

Seek times:
    Full stroke:      250 iter in   2.764277 sec =   11.057 msec
    Half stroke:      250 iter in   1.852089 sec =    7.408 msec
    Quarter stroke:   500 iter in   3.090188 sec =    6.180 msec
    Short forward:    400 iter in   1.258590 sec =    3.146 msec
    Short backward:   400 iter in   1.198859 sec =    2.997 msec
    Seq outer:       2048 iter in   0.099816 sec =    0.049 msec
    Seq inner:       2048 iter in   0.256881 sec =    0.125 msec

Transfer rates:
    outside:       102400 kbytes in   0.625300 sec =   163761 kbytes/sec
    middle:        102400 kbytes in   0.598765 sec =   171019 kbytes/sec
    inside:        102400 kbytes in   0.871804 sec =   117458 kbytes/sec

One of the remaining 12 disks making up Vol2 (these all produce similar results)

[root@TGA-Truenas ~]# diskinfo -tv /dev/da0
/dev/da0
512 # sectorsize
4000787030016 # mediasize in bytes (3.6T)
7814037168 # mediasize in sectors
0 # stripesize
0 # stripeoffset
486401 # Cylinders according to firmware.
255 # Heads according to firmware.
63 # Sectors according to firmware.
SEAGATE ST4000NM0023 # Disk descr.
Z1Z3GTRQ # Disk ident.
mps0 # Attachment
No # TRIM/UNMAP support
7200 # Rotation rate in RPM
Not_Zoned # Zone Mode

Seek times:
Full stroke: 250 iter in 4.395745 sec = 17.583 msec
Half stroke: 250 iter in 3.565977 sec = 14.264 msec
Quarter stroke: 500 iter in 4.313871 sec = 8.628 msec
Short forward: 400 iter in 2.099903 sec = 5.250 msec
Short backward: 400 iter in 2.121230 sec = 5.303 msec
Seq outer: 2048 iter in 0.145272 sec = 0.071 msec
Seq inner: 2048 iter in 0.098212 sec = 0.048 msec

Transfer rates:
outside: 102400 kbytes in 0.535380 sec = 191266 kbytes/sec
middle: 102400 kbytes in 0.714354 sec = 143346 kbytes/sec
inside: 102400 kbytes in 1.154738 sec = 88678 kbytes/sec

Running fio on Vol2 says its going to take 13minutes and jumps between KiB/s and up to 80MiB/s (very seldom higher number)

[root@TGA-Truenas ~]# sync ; fio --randrepeat=1 --direct=1 --gtod_reduce=1 --numjobs=10 --name=test --filename=/mnt/Vol2/test --bs=4k --iodepth=64 --size=2G --readwrite=randwrite --ramp_time=4 --group_reporting
test: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=64
...
fio-3.28
Starting 10 processes
test: Laying out IO file (1 file / 2048MiB)
test: Laying out IO file (1 file / 2048MiB)
test: Laying out IO file (1 file / 2048MiB)
test: Laying out IO file (1 file / 2048MiB)
test: Laying out IO file (1 file / 2048MiB)
test: Laying out IO file (1 file / 2048MiB)
test: Laying out IO file (1 file / 2048MiB)
test: Laying out IO file (1 file / 2048MiB)
test: Laying out IO file (1 file / 2048MiB)
test: Laying out IO file (1 file / 2048MiB)
Jobs: 10 (f=10): [w(10)][6.1%][w=11.8MiB/s][w=3025 IOPS][eta 13m:06s]

Run status group 0 (all jobs):

CANCELLED WITH CTRL C

WRITE: bw=25.8MiB/s (27.1MB/s), 25.8MiB/s-25.8MiB/s (27.1MB/s-27.1MB/s), io=3517MiB (3688MB), run=136279-136279msec

Pretty much seeing the same result with Vol1 tests

Run status group 0 (all jobs):
WRITE: bw=23.5MiB/s (24.7MB/s), 23.5MiB/s-23.5MiB/s (24.7MB/s-24.7MB/s), io=623MiB (654MB), run=26508-26508msec

Any help or direction would be greatly appreciated.
I have purchased another server which will be here in about a week so I can start from scratch, but I'll need to move all the data over.
Finding the cause to this issue will allow me to start fresh and in the correct way I hope.

P.S. I have ordered NVMe drives for log/cache, will run RAID2 with multiple vdevs for the pool

icepicknz · Feb 11, 2023

Just thought of a logical quick test and getting 48-50MB/s if copying between drives; still very slow in my mind?

[root@TGA-Truenas ~]# dd if=/mnt/Vol2/Media/Movies/test.mkv of=/mnt/Vol1/test
5974082+1 records in
5974082+1 records out
3058730246 bytes transferred in 63.234389 secs (48371310 bytes/sec)

[root@TGA-Truenas ~]# dd if=/mnt/Vol1/test of=/mnt/Vol2/test
5974082+1 records in
5974082+1 records out
3058730246 bytes transferred in 60.466885 secs (50585213 bytes/sec)

Samuel Tai · Feb 12, 2023

Your Vol2 VDEV is way too wide. All your IO is eaten up with calculating checksums, and figuring out where to deposit checksums. A RAIDZ* VDEV should be 6-8 wide, tops.

awasb · Feb 12, 2023

While I agree that wider VDEVs won't shine on benchmarks, the "deficit" shown above can't be explained by that alone IMHO.

Compare: 10x 2.5" WD RED laptop grade spinners ... 5.400rpm ... at APM128 ... with all the PowerManagement there is (standby excluded) ... LPM ... ALPM ... encryption ... compression ...

Code:

root@truenas ~ #  sync ; fio --randrepeat=1 --direct=1 --gtod_reduce=1 --numjobs=10 --name=test --filename=/mnt/data/test --bs=4k --iodepth=64 --size=2G --readwrite=randwrite --ramp_time=4 --group_reporting [0]
test: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=64
...
fio-3.28
Starting 10 processes
Jobs: 8 (f=8): [w(2),E(1),w(6),E(1)][87.0%][w=736MiB/s][w=188k IOPS][eta 00m:06s]
test: (groupid=0, jobs=10): err= 0: pid=81170: Sun Feb 12 14:54:12 2023
  write: IOPS=133k, BW=518MiB/s (543MB/s)(17.9GiB/35455msec); 0 zone resets
   bw (  KiB/s): min=56497, max=1503551, per=99.24%, avg=526492.32, stdev=46935.38, samples=695
   iops        : min=14120, max=375882, avg=131619.94, stdev=11733.85, samples=695
  cpu          : usr=3.88%, sys=30.84%, ctx=1196071, majf=0, minf=1
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,4702441,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=64

Run status group 0 (all jobs):
  WRITE: bw=518MiB/s (543MB/s), 518MiB/s-518MiB/s (543MB/s-543MB/s), io=17.9GiB (19.3GB), run=35455-35455msec

@icepicknz Please post Your exact machine specs. And after that the output of zfs get all Vol2, please.

icepicknz · Feb 12, 2023

Thanks @awasb

I have added to signature, but encase it doesn't come up:

### Server 1 - TrueNas Core ###
- Dell PowerVault NX3200
- CPU: Intel(R) Xeon(R) CPU E5-2630 v2 @ 2.60GHz (2600.11-MHz K8-class CPU) 24 core
- 32Gb Ram
- Vol1 - Mirror - da8/da9 SEAGATE ST300MM0006 10k RPM - (System Dataset Pool)
- Vol2 - RAIDZ1 - 12 disks SEAGATE ST4000NM0023 7200 RPM - Media

awasb · Feb 12, 2023

What controllers are attached? What mode is set? Did You flash anything?

And again: please post the output of the above given command zfs get all Vol2

icepicknz · Feb 12, 2023

Not entirely sure of the controller as I set it up back in May 2021, but I do recall googling for it and flashing as per a recommendation on this forum. Wish I knew what it was but its set to pass through the disks.

Have also disabled sync on both volumes 1 & 2 and speeds are the same.

So sorry, forgot to attach the ZFS all

root@TGA-Truenas[~]# zfs get all Vol2
NAME PROPERTY VALUE SOURCE
Vol2 type filesystem -
Vol2 creation Tue May 25 16:10 2021 -
Vol2 used 13.1T -
Vol2 available 25.6T -
Vol2 referenced 2.85G -
Vol2 compressratio 1.00x -
Vol2 mounted yes -
Vol2 quota none default
Vol2 reservation none default
Vol2 recordsize 128K default
Vol2 mountpoint /mnt/Vol2 default
Vol2 sharenfs off default
Vol2 checksum on default
Vol2 compression off local
Vol2 atime on default
Vol2 devices on default
Vol2 exec on default
Vol2 setuid on default
Vol2 readonly off default
Vol2 jailed off default
Vol2 snapdir hidden default
Vol2 aclmode passthrough local
Vol2 aclinherit passthrough local
Vol2 createtxg 1 -
Vol2 canmount on default
Vol2 xattr on default
Vol2 copies 1 local
Vol2 version 5 -
Vol2 utf8only off -
Vol2 normalization none -
Vol2 casesensitivity sensitive -
Vol2 vscan off default
Vol2 nbmand off default
Vol2 sharesmb off default
Vol2 refquota none default
Vol2 refreservation none default
Vol2 guid 15713293758960918662 -
Vol2 primarycache all default
Vol2 secondarycache all default
Vol2 usedbysnapshots 0B -
Vol2 usedbydataset 2.85G -
Vol2 usedbychildren 13.1T -
Vol2 usedbyrefreservation 0B -
Vol2 logbias latency default
Vol2 objsetid 54 -
Vol2 dedup off default
Vol2 mlslabel none default
Vol2 sync disabled local
Vol2 dnodesize legacy default
Vol2 refcompressratio 1.00x -
Vol2 written 2.85G -
Vol2 logicalused 13.1T -
Vol2 logicalreferenced 2.85G -
Vol2 volmode default default
Vol2 filesystem_limit none default
Vol2 snapshot_limit none default
Vol2 filesystem_count none default
Vol2 snapshot_count none default
Vol2 snapdev hidden default
Vol2 acltype nfsv4 default
Vol2 context none default
Vol2 fscontext none default
Vol2 defcontext none default
Vol2 rootcontext none default
Vol2 relatime off default
Vol2 redundant_metadata all default
Vol2 overlay on default
Vol2 encryption off default
Vol2 keylocation none default
Vol2 keyformat none default
Vol2 pbkdf2iters 0 default
Vol2 special_small_blocks 0 local
Vol2 org.freebsd.ioc:active no local
root@TGA-Truenas[~]#

icepicknz · Feb 12, 2023

Just found this useful command

root@TGA-Truenas[~]# sas2flash -listall
LSI Corporation SAS2 Flash Utility
Version 16.00.00.00 (2013.03.01)
Copyright (c) 2008-2013 LSI Corporation. All rights reserved
Adapter Selected is a LSI SAS: SAS2308_2(D1)
Num Ctlr FW Ver NVDATA x86-BIOS PCI Addr
----------------------------------------------------------------------------
0 SAS2308_2(D1) 20.00.07.00 14.01.00.06 No Image 00:02:00:00
Finished Processing Commands Successfully.
Exiting SAS2Flash.

awasb · Feb 12, 2023

Do you rely on atime? If not, please do a zfs set atime=off Vol2.

Please check again (after disabling) with zfs get all Vol2 | grep atime.

awasb · Feb 12, 2023

icepicknz said:
Just found this useful command

Please be warned concerning the following question/„advice“: I‘m by far no LSI flashing expert. (Did it only two times.)

Do you skip the LSI uefi/bios? I think it‘s a bad idea. If the controller will want to complain about something, without BIOS you will never find out AFAIK.

Nonetheless: Please add sas2flash -c 0 -list.

icepicknz · Feb 12, 2023

I thanks you for taking the time looking into this with me.

atime disabled as suggested:

[root@TGA-Truenas ~]# zfs get all Vol2 | grep atime
Vol2 atime off local
Vol2 relatime off default
[root@TGA-Truenas ~]# zfs get all Vol1 | grep atime
Vol1 atime off local
Vol1 relatime off default

Transfer speeds copying from disk to disk the same:

[root@TGA-Truenas ~]# dd if=/mnt/Vol2/Media/Movies/Saw\ III\ \(2006\)/Saw\ III.mkv of=/mnt/Vol1/test
5974082+1 records in
5974082+1 records out
3058730246 bytes transferred in 56.407609 secs (54225491 bytes/sec)
[root@TGA-Truenas ~]# dd if=/mnt/Vol1/test of=/mnt/Vol2/test
5974082+1 records in
5974082+1 records out
3058730246 bytes transferred in 55.561532 secs (55051222 bytes/sec)

FIO result to Vol2: Cancelled after about a minute as said 12m remaining:

[root@TGA-Truenas ~]# sync ; fio --randrepeat=1 --direct=1 --gtod_reduce=1 --numjobs=10 --name=test --filename=/mnt/Vol2/test --bs=4k --iodepth=64 --size=2G --readwrite=randwrite --ramp_time=4 --group_reporting
test: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=64
...
fio-3.28
Starting 10 processes
^Cbs: 10 (f=10): [w(10)][4.6%][w=12.0MiB/s][w=3066 IOPS][eta 14m:03s]
fio: terminating on signal 2
test: (groupid=0, jobs=10): err= 0: pid=51743: Mon Feb 13 10:13:05 2023
write: IOPS=6060, BW=23.7MiB/s (24.8MB/s)(870MiB/36767msec); 0 zone resets
bw ( KiB/s): min= 8900, max=132153, per=100.00%, avg=24312.89, stdev=2732.34, samples=730
iops : min= 2220, max=33037, avg=6075.68, stdev=683.09, samples=730
cpu : usr=0.54%, sys=2.97%, ctx=222337, majf=0, minf=1
IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=0,222825,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=64

Run status group 0 (all jobs):
WRITE: bw=23.7MiB/s (24.8MB/s), 23.7MiB/s-23.7MiB/s (24.8MB/s-24.8MB/s), io=870MiB (913MB), run=36767-36767msec

Output as requested: sas2flash -c 0 -list

[root@TGA-Truenas ~]# sas2flash -c 0 -list
LSI Corporation SAS2 Flash Utility
Version 16.00.00.00 (2013.03.01)
Copyright (c) 2008-2013 LSI Corporation. All rights reserved
Adapter Selected is a LSI SAS: SAS2308_2(D1)
Controller Number : 0
Controller : SAS2308_2(D1)
PCI Address : 00:02:00:00
SAS Address : 500605b-1-2345-6777
NVDATA Version (Default) : 14.01.00.06
NVDATA Version (Persistent) : 14.01.00.06
Firmware Product ID : 0x2214 (IT)
Firmware Version : 20.00.07.00
NVDATA Vendor : LSI
NVDATA Product ID : SAS9207-8i
BIOS Version : N/A
UEFI BSD Version : N/A
FCODE Version : N/A
Board Name : SAS9207-8i
Board Assembly : N/A
Board Tracer Number : N/A
Finished Processing Commands Successfully.
Exiting SAS2Flash.

awasb · Feb 12, 2023

Could you please check gstat -p while copying locally? Are there any dubious activities/load metrics on certain disks?

Is smartd running?

icepicknz · Feb 12, 2023

Yes Smartd is running

[root@TGA-Truenas ~]# ps -auxwww | grep smartd
root 2821 0.0 0.0 17712 1592 - Is Thu12 0:01.64 /usr/local/sbin/smartd -n -i 1800 -c /usr/local/etc/smartd.conf -p /var/run/smartd-daemon.pid

gstat shows hardly any activity on
Vol2, a couple percent here and there then 0.
Vol1 hops around from 0, in the teens and every now and then into 50-90% for a second but more idle than anything.

gstat below while copying, jumps from 2 to 43 to 90% then drops down, to 5, 74, 99, then drops again 3, 60, 90
So it ramps up, then dies off, then ramps up and dies off.

L(q) ops/s r/s kBps ms/r w/s kBps ms/w %busy Name
0 12 2 16 4.6 8 32 19.2 8.1| da0
0 14 2 24 6.9 10 40 15.2 10.7| da1
0 13 2 24 5.1 9 36 18.7 9.6| da2
0 12 2 24 0.2 8 32 21.3 8.1| da3
0 13 2 24 5.1 9 36 16.2 9.6| da4
0 12 2 24 0.2 8 32 18.3 7.8| da5
0 11 1 12 0.2 8 40 21.4 8.1| da6
0 12 2 24 10.9 8 36 21.3 10.1| da7

L(q) ops/s r/s kBps ms/r w/s kBps ms/w %busy Name
1 168 19 225 0.7 149 12617 9.5 74.8| da0
1 154 11 130 0.1 143 12594 9.4 72.6| da1
1 168 19 225 0.9 149 12594 9.4 74.3| da2
1 162 20 229 1.5 142 12625 10.0 76.2| da3
1 154 8 95 0.1 146 12605 9.4 73.8| da4
3 166 20 154 1.8 146 12617 9.7 75.5| da5
1 164 19 225 0.2 145 12625 9.2 72.3| da6
1 165 20 229 1.6 145 12625 9.4 75.3| da7

L(q) ops/s r/s kBps ms/r w/s kBps ms/w %busy Name
1 223 25 454 114.4 198 17504 9.4 99.6| da0
1 212 19 295 71.4 193 17488 10.0 99.2| da1
1 217 24 478 74.7 193 17293 9.5 99.3| da2
1 217 24 478 72.8 193 17488 9.7 99.1| da3
1 214 21 322 75.0 193 17524 9.6 99.8| da4
5 212 22 414 78.6 190 16867 9.6 98.6| da5
2 210 18 370 79.8 192 17293 9.5 98.2| da6
1 220 26 502 91.3 194 17500 10.0 98.7| da7

icepicknz · Feb 12, 2023

Not sure if this will help, I notice a 2G slice on da0, da1, da3 & da6
These are all on Vol2 but not on all the disks it would seem; is this a ZFS parity thing or swap?

[root@TGA-Truenas ~]# glabel list | grep -B 1 2.0G
1. Name: gptid/2056f31e-bd0f-11eb-ac17-b82a72cfcabd
Mediasize: 2147483648 (2.0G)
--
1. Name: da6p1
Mediasize: 2147483648 (2.0G)
--
1. Name: gptid/1f67d0c0-bd0f-11eb-ac17-b82a72cfcabd
Mediasize: 2147483648 (2.0G)
--
1. Name: da0p1
Mediasize: 2147483648 (2.0G)
--
1. Name: gptid/1bfaf8f2-bd0f-11eb-ac17-b82a72cfcabd
Mediasize: 2147483648 (2.0G)
--
1. Name: da3p1
Mediasize: 2147483648 (2.0G)
--
1. Name: gptid/1f0bf740-bd0f-11eb-ac17-b82a72cfcabd
Mediasize: 2147483648 (2.0G)
--
1. Name: da1p1
Mediasize: 2147483648 (2.0G)

[root@TGA-Truenas ~]# swapinfo
Device 1K-blocks Used Avail Capacity
/dev/mirror/swap0.eli 2097152 245800 1851352 12%
/dev/mirror/swap1.eli 2097152 242424 1854728 12%
/dev/mirror/swap2.eli 2097152 241632 1855520 12%
/dev/mirror/swap3.eli 2097152 246252 1850900 12%
/dev/mirror/swap4.eli 2097152 248052 1849100 12%
Total 10485760 1224160 9261600 12%

awasb · Feb 12, 2023

Right. It's the swap. What version of TrueNAS Core are you running?

Concerning the general problem:

Did you remount the pool after disabling atime? If not, please disconnect/export and reconnect/import. Just for sure. Please do some zpool list and zpool status after that and post the output.

If that doesn't help and no other oddities show up, you will

have to rely on your backup (in fact, you do it already, since raidz1 with 12 disks is no redundancy in case something has to be rebuild) and
have to take Samuel's advice and try to recreate the pool with a different layout.

Even though I still doubt, it's the width of the VDEV (alone), that's bringing things down to a crawl.

You wrote

I noticed the issue recently when I copied my Plex media from one folder to another

Was it better before? Did you test it? Or could it be that it was always this bad and you never noticed?

icepicknz · Feb 12, 2023

Ok. We have a cyclone going on here at the moment, but I may be getting the new server tomorrow, in hopes I do so I can start fresh I ask the following recommendation.

1./ I’ll boot off mirrored nvme 128g, is there a way and should I put swap on it?
2./ what should I put the System Dataset Pool onto, should these also be nvme 512g or 1tb mirrored or a pair of 1.8gb Sas 10k disks?

The system arriving has the following resources.
-512g ram
-21 x SAS 1.8gb 10k disks
-2 x 128g NVMe
-2 x 512g NVMe
-4 x 1tb NVMe

How best would you use the above resources?

I was planning on a pool for
1./ high availability nfs mounted proxmox VM’s’s (maybe 1.8Tb raidz2/stripe) add log 512gb
2./ pool for dev vm’s and media that’s not the end of the world if I lost it so raidz1 or 2. Also add log 512g

awasb · Feb 13, 2023

Ad 1./ With 512GB RAM swap is no issue. (Usually.) For sure it won't hurt, no matter where it's created. Just let TrueNAS do its "thing".
Ad 2./ It doesn't matter IMHO. (I'd prefer the most redundant.)

Concerning resource planning:

It all depends on the use case (per pool). But one thing's for sure: Get decent (S)LOG devices (maybe mirrored) with low latency and a huge write endurance for serving VMs via nfs. In most cases You don't need more than 16GB per pool (after "overprovisioning").

For all the rest ...

ZFS Storage Pool Layout

This amazing document, created by iXsystems in February 2022 as a "White Paper", cleanly explains how to qualify pool performance touching briefly on how ZFS stores data and presents the advantages, performance and disadvantages of each pool...

www.truenas.com

Resource - The path to success for block storage

ZFS does two different things very well. One is storage of large sequentially-written files, such as archives, logs, or data files, where the file does not have the middle bits modified after creation. This is optimal for RAIDZ. It's what most...

www.truenas.com

Slideshow explaining VDev, zpool, ZIL and L2ARC for noobs!

Slideshow explaining VDev, zpool, ZIL and L2ARC and other newbie mistakes! I've put together a Powerpoint presentation(and PDF) that gives some useful info for newbies to FreeNAS. I decided to create this slideshow because in the last 5 months I've been on this forum I've seen a lot of people...

www.truenas.com

Some insights into SLOG/ZIL with ZFS on FreeNAS

What is the ZIL? POSIX provides a facility for the system or an application to make sure that data requested to be written is actually committed to stable storage: a synchronous write request. Upon completion of a sync write request, the underlying filesystem is supposed to guarantee that a...

www.truenas.com

icepicknz · Feb 13, 2023

Thanks you for these resources, many of which I have seen on YouTube which is great; only wish I had done the research 2 years ago properly, but the new system I intend to do it right.

With the following resources, I plan to do the following
-512g ram
-21 x SAS 1.8gb 10k disks
-2 x 128g NVMe
-2 x 512g NVMe
-4 x 1tb NVMe

boot-pool - mirror (boot disk and swap)
- 512g NVMe
- 512g NVMe

Vol1 (Virtual Machines)
-udev1 mirror
--- 1TB NVMe
--- 1TB NVMe
-udev2 mirror
--- 1TB NVMe
--- 1TB NVMe

Vol2 (File Storage for Media) 4 x RAIDz2
-vdev1 RAIDz2
--- 1.8g 10k sas
--- 1.8g 10k sas
--- 1.8g 10k sas
--- 1.8g 10k sas
--- 1.8g 10k sas
-vdev2 RAIDz2
--- 1.8g 10k sas
--- 1.8g 10k sas
--- 1.8g 10k sas
--- 1.8g 10k sas
--- 1.8g 10k sas
-vdev3 RAIDz2
--- 1.8g 10k sas
--- 1.8g 10k sas
--- 1.8g 10k sas
--- 1.8g 10k sas
--- 1.8g 10k sas
-vdev4 RAIDz2
--- 1.8g 10k sas
--- 1.8g 10k sas
--- 1.8g 10k sas
--- 1.8g 10k sas
--- 1.8g 10k sas
-log vdev mirror
--- 128g NVMe
--- 128g NVMe
-1 hot spare left 1.8g 10k sas

Does this look ok?

awasb · Feb 13, 2023

Looks good enough to test it. I'm looking forward to see the first benchmarks.

Again: Those (s)log NVMEs need really, really good write endurance, PLP and low latency.

icepicknz · Feb 13, 2023

Sadly will be about 3 weeks, waiting for the NVMe disks and PCIe for them to arrive in 3 weeks.
Also waiting on a 10g switch (2 weeks) for testing the proxmox hypervisors over NFS

Will report back once I have a test locally at least

Important Announcement for the TrueNAS Community.

Slow write speeds ~10-20MB/s

Dabbler

Dabbler

Never underestimate your own stupidity

Patron

Dabbler

Patron

Dabbler

Dabbler

Patron

Patron

Dabbler

Patron

Dabbler

Dabbler

Patron

Dabbler

Patron

Dabbler

Patron

Dabbler

Similar threads