Hi all,
I have a problem that I am not able to solve and hope someone could help me :) I am running the following server with TrueNAS-13.0-U3.1 installed :
But the weekly scrub tests and long SMART tests are all OK. Furthermore I did a FIO test in the "good" state after a reboot and the "bad" state after a couple of days. Performance is the same:
Result of testing single 1MiB random write process:
fio --name=random-write --ioengine=posixaio --rw=randwrite --bs=1m --size=16g --numjobs=1 --iodepth=1 --runtime=60 --time_based --end_fsync=1
result: (after server reboot with good network speed ~110mb/s)
run status group 0 (all jobs): WRITE: bw=345MiB/s (361MB/s), 345MiB/s-345Mib/s (361MB/s-361Mb/s), io=22.7 GiB (24.4GB), run=67540-67540msec
result: (after couple of days with bad network speed ~1mb/s)
run status group 0 (all jobs): WRITE: bw=375MiB/s (393MB/s), 375MiB/s-375MiB/s (393MB/s-393MB/s), io=24,4 GiB (26.2GB), run=66563-66563msec
So i'm personally ruling out my pool. And speed seems in my opinion what you would expect from such drives.
As a second trouble shooting step i'm focusing now on my network connection.
Running iperf3 from a linux box on the same network gives me the following results:
After reboot (looks good):
iperf3 -c 1192.168.10.11 -bidir
Connecting to host 192.168.10.11, port 5201
[ 5] local 192.168.10.158 port 43080 connected to 192.168.10.11 port 5201
[ ID] Interval Transfer Bitrate Retr Cwnd
[ 5] 0.00-1.00 sec 114 MBytes 959 Mbits/sec 0 757 KBytes
[ 5] 1.00-2.00 sec 111 MBytes 933 Mbits/sec 0 757 KBytes
[ 5] 2.00-3.00 sec 111 MBytes 933 Mbits/sec 0 757 KBytes
[ 5] 3.00-4.00 sec 111 MBytes 933 Mbits/sec 0 757 KBytes
[ 5] 4.00-5.00 sec 111 MBytes 933 Mbits/sec 0 757 KBytes
[ 5] 5.00-6.00 sec 111 MBytes 933 Mbits/sec 0 757 KBytes
[ 5] 6.00-7.00 sec 111 MBytes 933 Mbits/sec 0 757 KBytes
[ 5] 7.00-8.00 sec 111 MBytes 933 Mbits/sec 0 757 KBytes
[ 5] 8.00-9.00 sec 111 MBytes 933 Mbits/sec 0 757 KBytes
[ 5] 9.00-10.00 sec 111 MBytes 933 Mbits/sec 0 757 KBytes
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bitrate Retr
[ 5] 0.00-10.00 sec 1.09 GBytes 936 Mbits/sec 0 sender
[ 5] 0.00-10.01 sec 1.09 GBytes 933 Mbits/sec receiver
Than after a couple of days speed drops by a factor 20:
Connecting to host 192.168.10.11, port 5201
[ 5] local 192.168.10.158 port 54886 connected to 192.168.10.11 port 5201
[ ID] Interval Transfer Bitrate Retr Cwnd
[ 5] 0.00-1.00 sec 6.15 MBytes 51.6 Mbits/sec 1464 4.24 KBytes
[ 5] 1.00-2.00 sec 6.02 MBytes 50.5 Mbits/sec 1440 1.41 KBytes
[ 5] 2.00-3.00 sec 6.32 MBytes 53.0 Mbits/sec 1510 4.24 KBytes
[ 5] 3.00-4.00 sec 5.68 MBytes 47.7 Mbits/sec 1284 1.41 KBytes
[ 5] 4.00-5.00 sec 6.20 MBytes 52.0 Mbits/sec 1504 2.83 KBytes
[ 5] 5.00-6.00 sec 6.41 MBytes 53.8 Mbits/sec 1488 2.83 KBytes
[ 5] 6.00-7.00 sec 6.26 MBytes 52.5 Mbits/sec 1508 2.83 KBytes
[ 5] 7.00-8.00 sec 6.32 MBytes 53.0 Mbits/sec 1516 2.83 KBytes
[ 5] 8.00-9.00 sec 5.86 MBytes 49.2 Mbits/sec 1401 2.83 KBytes
[ 5] 9.00-10.00 sec 6.11 MBytes 51.2 Mbits/sec 1457 2.83 KBytes
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bitrate Retr
[ 5] 0.00-10.00 sec 61.3 MBytes 51.4 Mbits/sec 14572 sender
[ 5] 0.00-10.00 sec 61.2 MBytes 51.3 Mbits/sec receiver
My attention is on the high Retr number? That does not seem right. Any further steps I could take?
I'm using the onboard intel NIC. Changing cable or ports on the switch did not solve it as well.
Thanks in advance for the help. Much appriciated.
regards,
Whizzle
I have a problem that I am not able to solve and hope someone could help me :) I am running the following server with TrueNAS-13.0-U3.1 installed :
- Motherboard make and model: fujitsu d3644-b
- CPU make and model: Pentium G5600
- RAM quantity: 32gb ECC
- Hard drives, quantity, model numbers, and RAID configuration, including boot drives: boot drive is SSD mirror, ZFS pool: 6x 4tb WD RED (EFRX model) the CMR type
- Hard disk controllers:
- Network cards: onboard
But the weekly scrub tests and long SMART tests are all OK. Furthermore I did a FIO test in the "good" state after a reboot and the "bad" state after a couple of days. Performance is the same:
Result of testing single 1MiB random write process:
fio --name=random-write --ioengine=posixaio --rw=randwrite --bs=1m --size=16g --numjobs=1 --iodepth=1 --runtime=60 --time_based --end_fsync=1
result: (after server reboot with good network speed ~110mb/s)
run status group 0 (all jobs): WRITE: bw=345MiB/s (361MB/s), 345MiB/s-345Mib/s (361MB/s-361Mb/s), io=22.7 GiB (24.4GB), run=67540-67540msec
result: (after couple of days with bad network speed ~1mb/s)
run status group 0 (all jobs): WRITE: bw=375MiB/s (393MB/s), 375MiB/s-375MiB/s (393MB/s-393MB/s), io=24,4 GiB (26.2GB), run=66563-66563msec
So i'm personally ruling out my pool. And speed seems in my opinion what you would expect from such drives.
As a second trouble shooting step i'm focusing now on my network connection.
Running iperf3 from a linux box on the same network gives me the following results:
After reboot (looks good):
iperf3 -c 1192.168.10.11 -bidir
Connecting to host 192.168.10.11, port 5201
[ 5] local 192.168.10.158 port 43080 connected to 192.168.10.11 port 5201
[ ID] Interval Transfer Bitrate Retr Cwnd
[ 5] 0.00-1.00 sec 114 MBytes 959 Mbits/sec 0 757 KBytes
[ 5] 1.00-2.00 sec 111 MBytes 933 Mbits/sec 0 757 KBytes
[ 5] 2.00-3.00 sec 111 MBytes 933 Mbits/sec 0 757 KBytes
[ 5] 3.00-4.00 sec 111 MBytes 933 Mbits/sec 0 757 KBytes
[ 5] 4.00-5.00 sec 111 MBytes 933 Mbits/sec 0 757 KBytes
[ 5] 5.00-6.00 sec 111 MBytes 933 Mbits/sec 0 757 KBytes
[ 5] 6.00-7.00 sec 111 MBytes 933 Mbits/sec 0 757 KBytes
[ 5] 7.00-8.00 sec 111 MBytes 933 Mbits/sec 0 757 KBytes
[ 5] 8.00-9.00 sec 111 MBytes 933 Mbits/sec 0 757 KBytes
[ 5] 9.00-10.00 sec 111 MBytes 933 Mbits/sec 0 757 KBytes
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bitrate Retr
[ 5] 0.00-10.00 sec 1.09 GBytes 936 Mbits/sec 0 sender
[ 5] 0.00-10.01 sec 1.09 GBytes 933 Mbits/sec receiver
Than after a couple of days speed drops by a factor 20:
Connecting to host 192.168.10.11, port 5201
[ 5] local 192.168.10.158 port 54886 connected to 192.168.10.11 port 5201
[ ID] Interval Transfer Bitrate Retr Cwnd
[ 5] 0.00-1.00 sec 6.15 MBytes 51.6 Mbits/sec 1464 4.24 KBytes
[ 5] 1.00-2.00 sec 6.02 MBytes 50.5 Mbits/sec 1440 1.41 KBytes
[ 5] 2.00-3.00 sec 6.32 MBytes 53.0 Mbits/sec 1510 4.24 KBytes
[ 5] 3.00-4.00 sec 5.68 MBytes 47.7 Mbits/sec 1284 1.41 KBytes
[ 5] 4.00-5.00 sec 6.20 MBytes 52.0 Mbits/sec 1504 2.83 KBytes
[ 5] 5.00-6.00 sec 6.41 MBytes 53.8 Mbits/sec 1488 2.83 KBytes
[ 5] 6.00-7.00 sec 6.26 MBytes 52.5 Mbits/sec 1508 2.83 KBytes
[ 5] 7.00-8.00 sec 6.32 MBytes 53.0 Mbits/sec 1516 2.83 KBytes
[ 5] 8.00-9.00 sec 5.86 MBytes 49.2 Mbits/sec 1401 2.83 KBytes
[ 5] 9.00-10.00 sec 6.11 MBytes 51.2 Mbits/sec 1457 2.83 KBytes
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bitrate Retr
[ 5] 0.00-10.00 sec 61.3 MBytes 51.4 Mbits/sec 14572 sender
[ 5] 0.00-10.00 sec 61.2 MBytes 51.3 Mbits/sec receiver
My attention is on the high Retr number? That does not seem right. Any further steps I could take?
I'm using the onboard intel NIC. Changing cable or ports on the switch did not solve it as well.
Thanks in advance for the help. Much appriciated.
regards,
Whizzle