Leopard Blanc
Cadet
- Joined
- Sep 18, 2019
- Messages
- 9
Hi, dear FreeNAS users !
I'm struggling with file transfer between two FreeNAS boxes, ServerA and serverB over NFS.
I started testing NFS after noticing that a snapshot replication was running at ~60 MB/s (on 100Gb network).
I will test other share possibilities after I'm done with those tests.
I'm looking for help identifying the bottleneck.
Machines
ServerA:
Hostname ServerA
Build FreeNAS-11.1-U6
Platform Intel(R) Xeon(R) Gold 6128 CPU @ 3.40GHz
Memory 195196MB
System Time Fri, 4 Oct 2019 18:22:30 +0200
Uptime 6:22PM up 290 days, 3:26, 5 users
Load Average 1.83, 2.31, 2.45
network: double 100 Gb in bond
Storage: 32x NVMe SSDs
no SLOG/L2ARC
ServerB:
Hostname ServerB
Build FreeNAS-11.2-U6
Platform Intel(R) Xeon(R) CPU E5-2670 v2 @ 2.50GHz
Memory 65465MB
System Time Fri, 4 Oct 2019 18:23:13 +0200
Uptime 6:23PM up 2 days, 2:02, 5 users
Load Average 2.29, 2.01, 1.74
network: 04:00.0 Ethernet controller: Mellanox Technologies MT27800 Family [ConnectX-5] (100 Gb)
Storage: pool of 11 x 10TB drives in RaidZ (not good,I know, but let's forget it for the sake of the current problem)
no SLOG/L2ARC
iperf (throughput > 3 GB/s)
IOzone Locally on serverB (writing > 2 GB/s)
IOzone on serverA mounted NFS export from serverB (writing < 200 MB/s)
iozone -ac -i 0 -s 128g -y 64k -q 1024k -Rb output.ods
Temporary conclusion
If I can run any kind of test or give some info about the setup, please let me know.
Thank for any help, idea, testing tool etc...
All the best,
Leo
I'm struggling with file transfer between two FreeNAS boxes, ServerA and serverB over NFS.
I started testing NFS after noticing that a snapshot replication was running at ~60 MB/s (on 100Gb network).
I will test other share possibilities after I'm done with those tests.
I'm looking for help identifying the bottleneck.
Machines
ServerA:
Hostname ServerA
Build FreeNAS-11.1-U6
Platform Intel(R) Xeon(R) Gold 6128 CPU @ 3.40GHz
Memory 195196MB
System Time Fri, 4 Oct 2019 18:22:30 +0200
Uptime 6:22PM up 290 days, 3:26, 5 users
Load Average 1.83, 2.31, 2.45
network: double 100 Gb in bond
Storage: 32x NVMe SSDs
no SLOG/L2ARC
ServerB:
Hostname ServerB
Build FreeNAS-11.2-U6
Platform Intel(R) Xeon(R) CPU E5-2670 v2 @ 2.50GHz
Memory 65465MB
System Time Fri, 4 Oct 2019 18:23:13 +0200
Uptime 6:23PM up 2 days, 2:02, 5 users
Load Average 2.29, 2.01, 1.74
network: 04:00.0 Ethernet controller: Mellanox Technologies MT27800 Family [ConnectX-5] (100 Gb)
Storage: pool of 11 x 10TB drives in RaidZ (not good,I know, but let's forget it for the sake of the current problem)
no SLOG/L2ARC
iperf (throughput > 3 GB/s)
Code:
root@serverA:~ # iperf -c serverB -P 8 ------------------------------------------------------------ Client connecting to serverB, TCP port 5001 TCP window size: 64.2 KByte (default) ------------------------------------------------------------ [ 9] local serverA port 54407 connected with serverB port 5001 [ 5] local serverA port 54406 connected with serverB port 5001 [ 7] local serverA port 54404 connected with serverB port 5001 [ 10] local serverA port 54408 connected with serverB port 5001 [ 6] local serverA port 54405 connected with serverB port 5001 [ 4] local serverA port 54403 connected with serverB port 5001 [ 3] local serverA port 54401 connected with serverB port 5001 [ 8] local serverA port 54402 connected with serverB port 5001 [ ID] Interval Transfer Bandwidth [ 9] 0.0-10.0 sec 3.46 GBytes 2.98 Gbits/sec [ 5] 0.0-10.0 sec 2.46 GBytes 2.12 Gbits/sec [ 7] 0.0-10.0 sec 6.27 GBytes 5.39 Gbits/sec [ 10] 0.0-10.0 sec 4.32 GBytes 3.71 Gbits/sec [ 6] 0.0-10.0 sec 6.21 GBytes 5.33 Gbits/sec [ 4] 0.0-10.0 sec 5.55 GBytes 4.76 Gbits/sec [ 3] 0.0-10.0 sec 6.10 GBytes 5.24 Gbits/sec [ 8] 0.0-10.0 sec 4.09 GBytes 3.51 Gbits/sec [SUM] 0.0-10.0 sec 38.5 GBytes 33.0 Gbits/sec
IOzone Locally on serverB (writing > 2 GB/s)
Code:
iozone -a -i 0 -s 128g -y 64k -q 1024k -Rb output.ods The top row is records sizes, the left column is file sizes Writer Report 64 128 256 512 1024 134217728 1860641 2533219 2661507 2804997 2899522 134217.728 1860.641 2533.219 2661.507 2804.997 2899.522 Re-writer Report 64 128 256 512 1024 134217728 2088055 2594037 2656033 2853643 2799819
IOzone on serverA mounted NFS export from serverB (writing < 200 MB/s)
iozone -ac -i 0 -s 128g -y 64k -q 1024k -Rb output.ods
Temporary conclusion
- Network is fast
- Local writing is fast
- Writing over NFS is 100 times slower than saturation (CPUs are far from being saturated)
If I can run any kind of test or give some info about the setup, please let me know.
Thank for any help, idea, testing tool etc...
All the best,
Leo
Last edited: