Hi there!
This is my first time posting here so please excuse any newbie errors. I've recently built a couple of TrueNAS systems to replace a Synology enclosure I had been using before. I've built one high performance system as my main NAS and a secondary one as a backup. Specs for the main unit as follows:
CPU: 2x E5-2660 v3 @ 2.60GHz (10 cores each)
RAM: 128GB ECC
Motherboard: Supermicro X10DRU-i+
Network: Mellanox CX314A ConnectX-3 Pro 40GbE
HBA: Supermicro AOC-S3008L-L8E / Supermicro SAS3 backplane
Pool: 12x HGST SAS3 8TB (RAID Z2) w/ 512GB MVMe cache
My primary workstation is a trash can Mac Pro with a Thunderbolt 10GbE adapter (Sonnet SFP+). I've been pulling my hair out for the last few months trying to diagnose inconsistent transfer speeds. I've got two areas of concern:
First issue - inconsistent client transfer speeds
File transfer speeds from my main workstation to the primary NAS have been all over the place. They started fairly strong with SMB transfers in the 600MB/sec range but now I'm lucky to receive more than 150MB/sec. With the system specs, even the 600MB/sec speed seems slow but I'll leave that aside for a moment.
For ages I thought it was an issue with jumbo frames. I connect to the NAS via 2 switches. The primary switch in the closet is a HPE 5700 and my workstation connects to that through a newer Aruba 10GbE switch. At one point, I was unable to connect to TrueNAS from my workstation through the Aruba switch with jumbo frames enabled (it literally just stopped working one day). I addressed that with HPE support who replaced the switch. To my surprise, I still wasn't able to connect with jumbo frames enabled to the primary NAS but I was able to connect to the backup which has the same NIC. Even with the second switch taken out of the equation. Go figure. After plenty of research around headaches caused by jumbo frames, I decided to disable them on my workstation. Jumbo frames seem to work fine with the ESXi hosts that have iSCSI datastores on the TrueNAS units.
Now I'm being plagued by slow transfer speeds from my workstation to both NAS units. Iperf tests from my workstation to both NAS units as follows:
Main unit
Connecting to host 10.10.209.10, port 5201
[ 4] local 10.10.202.233 port 56604 connected to 10.10.209.10 port 5201
[ ID] Interval Transfer Bandwidth
[ 4] 0.00-1.00 sec 203 MBytes 1.71 Gbits/sec
[ 4] 1.00-2.00 sec 211 MBytes 1.77 Gbits/sec
[ 4] 2.00-3.00 sec 198 MBytes 1.66 Gbits/sec
[ 4] 3.00-4.00 sec 190 MBytes 1.59 Gbits/sec
[ 4] 4.00-5.00 sec 197 MBytes 1.66 Gbits/sec
[ 4] 5.00-6.00 sec 188 MBytes 1.58 Gbits/sec
[ 4] 6.00-7.00 sec 204 MBytes 1.71 Gbits/sec
[ 4] 7.00-8.00 sec 198 MBytes 1.66 Gbits/sec
[ 4] 8.00-9.00 sec 186 MBytes 1.56 Gbits/sec
[ 4] 9.00-10.00 sec 181 MBytes 1.52 Gbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bandwidth
[ 4] 0.00-10.00 sec 1.91 GBytes 1.64 Gbits/sec sender
[ 4] 0.00-10.00 sec 1.91 GBytes 1.64 Gbits/sec receiver
Backup unit
Connecting to host 10.10.209.20, port 5201
[ 4] local 10.10.202.233 port 56632 connected to 10.10.209.20 port 5201
[ ID] Interval Transfer Bandwidth
[ 4] 0.00-1.00 sec 163 MBytes 1.36 Gbits/sec
[ 4] 1.00-2.00 sec 171 MBytes 1.44 Gbits/sec
[ 4] 2.00-3.00 sec 175 MBytes 1.47 Gbits/sec
[ 4] 3.00-4.00 sec 169 MBytes 1.42 Gbits/sec
[ 4] 4.00-5.00 sec 169 MBytes 1.42 Gbits/sec
[ 4] 5.00-6.00 sec 175 MBytes 1.46 Gbits/sec
[ 4] 6.00-7.00 sec 169 MBytes 1.42 Gbits/sec
[ 4] 7.00-8.00 sec 173 MBytes 1.45 Gbits/sec
[ 4] 8.00-9.00 sec 164 MBytes 1.38 Gbits/sec
[ 4] 9.00-10.00 sec 165 MBytes 1.39 Gbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bandwidth
[ 4] 0.00-10.00 sec 1.65 GBytes 1.42 Gbits/sec sender
[ 4] 0.00-10.00 sec 1.65 GBytes 1.42 Gbits/sec receiver
Thinking it could be a switch issue, I also ran a test from my workstation to an ESXi VM connected to the same main switch as the TrueNAS units (via a Mellanox 10GbE NIC). That displays results that are certainly better:
[ 4] local 10.10.202.233 port 56686 connected to 10.10.201.22 port 5201
[ ID] Interval Transfer Bandwidth
[ 4] 0.00-1.00 sec 241 MBytes 2.02 Gbits/sec
[ 4] 1.00-2.00 sec 246 MBytes 2.06 Gbits/sec
[ 4] 2.00-3.00 sec 412 MBytes 3.45 Gbits/sec
[ 4] 3.00-4.00 sec 396 MBytes 3.32 Gbits/sec
[ 4] 4.00-5.00 sec 392 MBytes 3.29 Gbits/sec
[ 4] 5.00-6.00 sec 381 MBytes 3.20 Gbits/sec
[ 4] 6.00-7.00 sec 329 MBytes 2.76 Gbits/sec
[ 4] 7.00-8.00 sec 659 MBytes 5.54 Gbits/sec
[ 4] 8.00-9.00 sec 722 MBytes 6.05 Gbits/sec
[ 4] 9.00-10.00 sec 752 MBytes 6.31 Gbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bandwidth
[ 4] 0.00-10.00 sec 4.42 GBytes 3.80 Gbits/sec sender
[ 4] 0.00-10.00 sec 4.42 GBytes 3.80 Gbits/sec receiver
For completeness, here are test results from that Ubuntu VM to the main NAS. Results are exactly as expected:
[ 5] local 10.10.201.22 port 46562 connected to 10.10.209.10 port 5201
[ ID] Interval Transfer Bitrate Retr Cwnd
[ 5] 0.00-1.00 sec 958 MBytes 8.03 Gbits/sec 0 1.33 MBytes
[ 5] 1.00-2.00 sec 1.08 GBytes 9.23 Gbits/sec 0 1.33 MBytes
[ 5] 2.00-3.00 sec 1.07 GBytes 9.19 Gbits/sec 0 1.33 MBytes
[ 5] 3.00-4.00 sec 1.07 GBytes 9.21 Gbits/sec 0 1.33 MBytes
[ 5] 4.00-5.00 sec 1.09 GBytes 9.36 Gbits/sec 0 1.33 MBytes
[ 5] 5.00-6.00 sec 1.07 GBytes 9.23 Gbits/sec 0 1.33 MBytes
[ 5] 6.00-7.00 sec 1.07 GBytes 9.18 Gbits/sec 0 1.33 MBytes
[ 5] 7.00-8.00 sec 1.08 GBytes 9.23 Gbits/sec 0 1.33 MBytes
[ 5] 8.00-9.00 sec 1.07 GBytes 9.20 Gbits/sec 0 1.33 MBytes
[ 5] 9.00-10.00 sec 1.06 GBytes 9.12 Gbits/sec 0 1.34 MBytes
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bitrate Retr
[ 5] 0.00-10.00 sec 10.6 GBytes 9.10 Gbits/sec 0 sender
[ 5] 0.00-10.00 sec 10.6 GBytes 9.10 Gbits/sec receiver
It's pretty clear to me from those results that something is going on between my workstation and both TrueNAS units but I'm at a loss as to what could be causing the issues. Any suggestions would be greatly appreciated!
Second issue - inconsistent transfer speeds between main and backup TrueNAS units
When I first built the backup unit and ran a replication task from one to the other, that completed at around 700MiB/sec. The 40GbE interface is admittedly overkill for that, but that was the free port I had on the switch so that's what I used! Assuming the limiting factor here was just the write speed of 6x 8TB SATA disks.
Now for the weirdness. I added 2 more 8TB disks to that unit. It's backup data so I just wiped and re-created the volume (RAID Z1 - same as before but with the two additional disks). When I ran replication task to re-copy the data, it ran along no greater than 300MiB/sec. If anything, I would have expected that to increase a little given the extra two disks.
Again, for completeness, here are the iperf results from the main to backup NAS. Nowhere close to the full 40Gb but still more than sufficient:
Connecting to host 10.10.209.20, port 5201
[ 5] local 10.10.209.10 port 30217 connected to 10.10.209.20 port 5201
[ ID] Interval Transfer Bitrate Retr Cwnd
[ 5] 0.00-1.00 sec 1.76 GBytes 15.1 Gbits/sec 120 121 KBytes
[ 5] 1.00-2.00 sec 1.50 GBytes 12.9 Gbits/sec 63 91.2 KBytes
[ 5] 2.00-3.00 sec 1.54 GBytes 13.2 Gbits/sec 88 113 KBytes
[ 5] 3.00-4.00 sec 1.56 GBytes 13.4 Gbits/sec 88 134 KBytes
[ 5] 4.00-5.00 sec 1.51 GBytes 12.9 Gbits/sec 82 98.3 KBytes
[ 5] 5.00-6.00 sec 1.51 GBytes 13.0 Gbits/sec 102 133 KBytes
[ 5] 6.00-7.00 sec 1.55 GBytes 13.4 Gbits/sec 84 91.2 KBytes
[ 5] 7.00-8.00 sec 1.52 GBytes 13.0 Gbits/sec 112 133 KBytes
Any thoughts what could be causing this nearly 50% speed reduction? Apart from adding the two disks in the second unit, nothing else changed with the hardware config. (That's a Xeon E31230 @ 3.20GHz w/ 32GB ECC memory.)
Many thanks in advance for any assistance!
All the best
This is my first time posting here so please excuse any newbie errors. I've recently built a couple of TrueNAS systems to replace a Synology enclosure I had been using before. I've built one high performance system as my main NAS and a secondary one as a backup. Specs for the main unit as follows:
CPU: 2x E5-2660 v3 @ 2.60GHz (10 cores each)
RAM: 128GB ECC
Motherboard: Supermicro X10DRU-i+
Network: Mellanox CX314A ConnectX-3 Pro 40GbE
HBA: Supermicro AOC-S3008L-L8E / Supermicro SAS3 backplane
Pool: 12x HGST SAS3 8TB (RAID Z2) w/ 512GB MVMe cache
My primary workstation is a trash can Mac Pro with a Thunderbolt 10GbE adapter (Sonnet SFP+). I've been pulling my hair out for the last few months trying to diagnose inconsistent transfer speeds. I've got two areas of concern:
First issue - inconsistent client transfer speeds
File transfer speeds from my main workstation to the primary NAS have been all over the place. They started fairly strong with SMB transfers in the 600MB/sec range but now I'm lucky to receive more than 150MB/sec. With the system specs, even the 600MB/sec speed seems slow but I'll leave that aside for a moment.
For ages I thought it was an issue with jumbo frames. I connect to the NAS via 2 switches. The primary switch in the closet is a HPE 5700 and my workstation connects to that through a newer Aruba 10GbE switch. At one point, I was unable to connect to TrueNAS from my workstation through the Aruba switch with jumbo frames enabled (it literally just stopped working one day). I addressed that with HPE support who replaced the switch. To my surprise, I still wasn't able to connect with jumbo frames enabled to the primary NAS but I was able to connect to the backup which has the same NIC. Even with the second switch taken out of the equation. Go figure. After plenty of research around headaches caused by jumbo frames, I decided to disable them on my workstation. Jumbo frames seem to work fine with the ESXi hosts that have iSCSI datastores on the TrueNAS units.
Now I'm being plagued by slow transfer speeds from my workstation to both NAS units. Iperf tests from my workstation to both NAS units as follows:
Main unit
Connecting to host 10.10.209.10, port 5201
[ 4] local 10.10.202.233 port 56604 connected to 10.10.209.10 port 5201
[ ID] Interval Transfer Bandwidth
[ 4] 0.00-1.00 sec 203 MBytes 1.71 Gbits/sec
[ 4] 1.00-2.00 sec 211 MBytes 1.77 Gbits/sec
[ 4] 2.00-3.00 sec 198 MBytes 1.66 Gbits/sec
[ 4] 3.00-4.00 sec 190 MBytes 1.59 Gbits/sec
[ 4] 4.00-5.00 sec 197 MBytes 1.66 Gbits/sec
[ 4] 5.00-6.00 sec 188 MBytes 1.58 Gbits/sec
[ 4] 6.00-7.00 sec 204 MBytes 1.71 Gbits/sec
[ 4] 7.00-8.00 sec 198 MBytes 1.66 Gbits/sec
[ 4] 8.00-9.00 sec 186 MBytes 1.56 Gbits/sec
[ 4] 9.00-10.00 sec 181 MBytes 1.52 Gbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bandwidth
[ 4] 0.00-10.00 sec 1.91 GBytes 1.64 Gbits/sec sender
[ 4] 0.00-10.00 sec 1.91 GBytes 1.64 Gbits/sec receiver
Backup unit
Connecting to host 10.10.209.20, port 5201
[ 4] local 10.10.202.233 port 56632 connected to 10.10.209.20 port 5201
[ ID] Interval Transfer Bandwidth
[ 4] 0.00-1.00 sec 163 MBytes 1.36 Gbits/sec
[ 4] 1.00-2.00 sec 171 MBytes 1.44 Gbits/sec
[ 4] 2.00-3.00 sec 175 MBytes 1.47 Gbits/sec
[ 4] 3.00-4.00 sec 169 MBytes 1.42 Gbits/sec
[ 4] 4.00-5.00 sec 169 MBytes 1.42 Gbits/sec
[ 4] 5.00-6.00 sec 175 MBytes 1.46 Gbits/sec
[ 4] 6.00-7.00 sec 169 MBytes 1.42 Gbits/sec
[ 4] 7.00-8.00 sec 173 MBytes 1.45 Gbits/sec
[ 4] 8.00-9.00 sec 164 MBytes 1.38 Gbits/sec
[ 4] 9.00-10.00 sec 165 MBytes 1.39 Gbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bandwidth
[ 4] 0.00-10.00 sec 1.65 GBytes 1.42 Gbits/sec sender
[ 4] 0.00-10.00 sec 1.65 GBytes 1.42 Gbits/sec receiver
Thinking it could be a switch issue, I also ran a test from my workstation to an ESXi VM connected to the same main switch as the TrueNAS units (via a Mellanox 10GbE NIC). That displays results that are certainly better:
[ 4] local 10.10.202.233 port 56686 connected to 10.10.201.22 port 5201
[ ID] Interval Transfer Bandwidth
[ 4] 0.00-1.00 sec 241 MBytes 2.02 Gbits/sec
[ 4] 1.00-2.00 sec 246 MBytes 2.06 Gbits/sec
[ 4] 2.00-3.00 sec 412 MBytes 3.45 Gbits/sec
[ 4] 3.00-4.00 sec 396 MBytes 3.32 Gbits/sec
[ 4] 4.00-5.00 sec 392 MBytes 3.29 Gbits/sec
[ 4] 5.00-6.00 sec 381 MBytes 3.20 Gbits/sec
[ 4] 6.00-7.00 sec 329 MBytes 2.76 Gbits/sec
[ 4] 7.00-8.00 sec 659 MBytes 5.54 Gbits/sec
[ 4] 8.00-9.00 sec 722 MBytes 6.05 Gbits/sec
[ 4] 9.00-10.00 sec 752 MBytes 6.31 Gbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bandwidth
[ 4] 0.00-10.00 sec 4.42 GBytes 3.80 Gbits/sec sender
[ 4] 0.00-10.00 sec 4.42 GBytes 3.80 Gbits/sec receiver
For completeness, here are test results from that Ubuntu VM to the main NAS. Results are exactly as expected:
[ 5] local 10.10.201.22 port 46562 connected to 10.10.209.10 port 5201
[ ID] Interval Transfer Bitrate Retr Cwnd
[ 5] 0.00-1.00 sec 958 MBytes 8.03 Gbits/sec 0 1.33 MBytes
[ 5] 1.00-2.00 sec 1.08 GBytes 9.23 Gbits/sec 0 1.33 MBytes
[ 5] 2.00-3.00 sec 1.07 GBytes 9.19 Gbits/sec 0 1.33 MBytes
[ 5] 3.00-4.00 sec 1.07 GBytes 9.21 Gbits/sec 0 1.33 MBytes
[ 5] 4.00-5.00 sec 1.09 GBytes 9.36 Gbits/sec 0 1.33 MBytes
[ 5] 5.00-6.00 sec 1.07 GBytes 9.23 Gbits/sec 0 1.33 MBytes
[ 5] 6.00-7.00 sec 1.07 GBytes 9.18 Gbits/sec 0 1.33 MBytes
[ 5] 7.00-8.00 sec 1.08 GBytes 9.23 Gbits/sec 0 1.33 MBytes
[ 5] 8.00-9.00 sec 1.07 GBytes 9.20 Gbits/sec 0 1.33 MBytes
[ 5] 9.00-10.00 sec 1.06 GBytes 9.12 Gbits/sec 0 1.34 MBytes
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bitrate Retr
[ 5] 0.00-10.00 sec 10.6 GBytes 9.10 Gbits/sec 0 sender
[ 5] 0.00-10.00 sec 10.6 GBytes 9.10 Gbits/sec receiver
It's pretty clear to me from those results that something is going on between my workstation and both TrueNAS units but I'm at a loss as to what could be causing the issues. Any suggestions would be greatly appreciated!
Second issue - inconsistent transfer speeds between main and backup TrueNAS units
When I first built the backup unit and ran a replication task from one to the other, that completed at around 700MiB/sec. The 40GbE interface is admittedly overkill for that, but that was the free port I had on the switch so that's what I used! Assuming the limiting factor here was just the write speed of 6x 8TB SATA disks.
Now for the weirdness. I added 2 more 8TB disks to that unit. It's backup data so I just wiped and re-created the volume (RAID Z1 - same as before but with the two additional disks). When I ran replication task to re-copy the data, it ran along no greater than 300MiB/sec. If anything, I would have expected that to increase a little given the extra two disks.
Again, for completeness, here are the iperf results from the main to backup NAS. Nowhere close to the full 40Gb but still more than sufficient:
Connecting to host 10.10.209.20, port 5201
[ 5] local 10.10.209.10 port 30217 connected to 10.10.209.20 port 5201
[ ID] Interval Transfer Bitrate Retr Cwnd
[ 5] 0.00-1.00 sec 1.76 GBytes 15.1 Gbits/sec 120 121 KBytes
[ 5] 1.00-2.00 sec 1.50 GBytes 12.9 Gbits/sec 63 91.2 KBytes
[ 5] 2.00-3.00 sec 1.54 GBytes 13.2 Gbits/sec 88 113 KBytes
[ 5] 3.00-4.00 sec 1.56 GBytes 13.4 Gbits/sec 88 134 KBytes
[ 5] 4.00-5.00 sec 1.51 GBytes 12.9 Gbits/sec 82 98.3 KBytes
[ 5] 5.00-6.00 sec 1.51 GBytes 13.0 Gbits/sec 102 133 KBytes
[ 5] 6.00-7.00 sec 1.55 GBytes 13.4 Gbits/sec 84 91.2 KBytes
[ 5] 7.00-8.00 sec 1.52 GBytes 13.0 Gbits/sec 112 133 KBytes
Any thoughts what could be causing this nearly 50% speed reduction? Apart from adding the two disks in the second unit, nothing else changed with the hardware config. (That's a Xeon E31230 @ 3.20GHz w/ 32GB ECC memory.)
Many thanks in advance for any assistance!
All the best
