goosesensor
Dabbler
- Joined
- Jan 7, 2019
- Messages
- 10
I've recently put together a new FreeNAS box and invested in some 10G infrastructure equipent for my home office. Now that the dust has settled and the NAS has been up and running for a while, I'm turing my attention to performance tuning. Depending on client OS/protocol (see below) I get around 450MB/sec read/write transfers via SFP+. I'd like to improve this number. Here are the details of the setup:
FreeNAS 11.2
ASRock C236 WSI
Xeon E3-1245
16GB ECC Memory
8x Pioneer 1TB SSDs connected via SATA
Mellanox Connectx2 SFP+ NIC, MTU 9000
Pool is RAIDZ2 across all drives, have tired lz4 and uncompressed datasets.
The Workstation PC (main client) is a beefy Intel machine with NVMe storage and a 10Gtek Intel chipset SFP+ NIC.
The NAS and PC are connected through a MikroTik CRS305-1G-4S+in, NAS via DAC and PC via OM3. I have tested with the NAS & PC connected directly via OM3 with no difference in performance.
For a given platform I see the best performance using its native file sharing protocol.
Arch Linux: ~ 470MB/sec up&down via NFS with "-rw,soft,intr,rsize=131072,wsize=131072,async", NIC MTU 9000
Windows: ~ 470MB/sec up&down via SMB, default NIC MTU
macOS: ~ 470MB/sec up&down via AFP, NIC MTU 9000
To test NAS disk perfoamnce, I made an uncompressed dataset and used dd for reading and writing about 20GB to/from /dev/zero. I see about 1,400MB/sec read and 1,000MB/sec write:
To test TCP performance, I used iperf yielding a full 10Gbps between NAS and PC:
iperf PC -> NAS:
iperf NAS -> PC:
iperf NAS loopback:
iperf PC loopback:
During sustained large file transfers between NAS and Workstation via NFS I see a NAS CPU usage of about 30% and memory usage usually hovers around about 80%. The workstation doesn't seem to blink.
So where is the bottleneck happening? Here's what I think I've ruled out:
- NAS resource constraints. CPU and memory are happy. Disks are fast.
- Workstation resource constraints. CPU and memory are happy. NVMe should be able to keep up, and testing with a ramdisk yields no changes.
- Network resource connstrains. The SFP+ switch seems happy, as bypassing it yields no changes.
This makes me think that NFS is improperly configured on the host, client, or both.
I've done some Googling and tinkering and most common suggestions I come across are:
1. Increase MTU [done]
2. Ensure full duplex [done]
3. For NFS (my main use) increate rsize and wsize [done]
4. For NFS use async. [not done]
For #4 -- async NFS -- I can't see to get this working. I've tried adding it to the server exports but it just gets overriten on restart. I've also tried setting vfs.nfsd.async=1 with sysctl but it too disappears after reboot. 'nfsstat -m' on the workstation never reports 'async' for and shares.
I'm out of ideas... help!
FreeNAS 11.2
ASRock C236 WSI
Xeon E3-1245
16GB ECC Memory
8x Pioneer 1TB SSDs connected via SATA
Mellanox Connectx2 SFP+ NIC, MTU 9000
Pool is RAIDZ2 across all drives, have tired lz4 and uncompressed datasets.
The Workstation PC (main client) is a beefy Intel machine with NVMe storage and a 10Gtek Intel chipset SFP+ NIC.
The NAS and PC are connected through a MikroTik CRS305-1G-4S+in, NAS via DAC and PC via OM3. I have tested with the NAS & PC connected directly via OM3 with no difference in performance.
For a given platform I see the best performance using its native file sharing protocol.
Arch Linux: ~ 470MB/sec up&down via NFS with "-rw,soft,intr,rsize=131072,wsize=131072,async", NIC MTU 9000
Windows: ~ 470MB/sec up&down via SMB, default NIC MTU
macOS: ~ 470MB/sec up&down via AFP, NIC MTU 9000
To test NAS disk perfoamnce, I made an uncompressed dataset and used dd for reading and writing about 20GB to/from /dev/zero. I see about 1,400MB/sec read and 1,000MB/sec write:
Code:
% dd if=/dev/zero of=testfile bs=2048k count=10000 10000+0 records in 10000+0 records out 20971520000 bytes transferred in 19.359261 secs (1083280995 bytes/sec)
Code:
% dd if=testfile of=/dev/zero bs=2048k count=10000 10000+0 records in 10000+0 records out 20971520000 bytes transferred in 14.378746 secs (1458508227 bytes/sec)
To test TCP performance, I used iperf yielding a full 10Gbps between NAS and PC:
iperf PC -> NAS:
Code:
% iperf -c 10.0.1.2 ------------------------------------------------------------ Client connecting to 10.0.1.2, TCP port 5001 TCP window size: 845 KByte (default) ------------------------------------------------------------ [ 3] local 10.0.1.4 port 50812 connected with 10.0.1.2 port 5001 [ ID] Interval Transfer Bandwidth [ 3] 0.0-10.0 sec 11.4 GBytes 9.80 Gbits/sec
iperf NAS -> PC:
Code:
% iperf -c 10.0.1.4 ------------------------------------------------------------ Client connecting to 10.0.1.4, TCP port 5001 TCP window size: 35.0 KByte (default) ------------------------------------------------------------ [ 3] local 10.0.1.2 port 22862 connected with 10.0.1.4 port 5001 [ ID] Interval Transfer Bandwidth [ 3] 0.0-10.0 sec 11.4 GBytes 9.75 Gbits/sec
iperf NAS loopback:
Code:
iperf -c 10.0.1.2 ------------------------------------------------------------ Client connecting to 10.0.1.2, TCP port 5001 TCP window size: 47.9 KByte (default) ------------------------------------------------------------ [ 3] local 10.0.1.2 port 37612 connected with 10.0.1.2 port 5001 [ ID] Interval Transfer Bandwidth [ 3] 0.0-10.0 sec 71.1 GBytes 61.1 Gbits/sec
iperf PC loopback:
Code:
% iperf -c 10.0.1.4 ------------------------------------------------------------ Client connecting to 10.0.1.4, TCP port 5001 TCP window size: 2.50 MByte (default) ------------------------------------------------------------ [ 3] local 10.0.1.4 port 35818 connected with 10.0.1.4 port 5001 [ ID] Interval Transfer Bandwidth [ 3] 0.0-10.0 sec 98.8 GBytes 84.9 Gbits/sec
During sustained large file transfers between NAS and Workstation via NFS I see a NAS CPU usage of about 30% and memory usage usually hovers around about 80%. The workstation doesn't seem to blink.
So where is the bottleneck happening? Here's what I think I've ruled out:
- NAS resource constraints. CPU and memory are happy. Disks are fast.
- Workstation resource constraints. CPU and memory are happy. NVMe should be able to keep up, and testing with a ramdisk yields no changes.
- Network resource connstrains. The SFP+ switch seems happy, as bypassing it yields no changes.
This makes me think that NFS is improperly configured on the host, client, or both.
I've done some Googling and tinkering and most common suggestions I come across are:
1. Increase MTU [done]
2. Ensure full duplex [done]
3. For NFS (my main use) increate rsize and wsize [done]
4. For NFS use async. [not done]
For #4 -- async NFS -- I can't see to get this working. I've tried adding it to the server exports but it just gets overriten on restart. I've also tried setting vfs.nfsd.async=1 with sysctl but it too disappears after reboot. 'nfsstat -m' on the workstation never reports 'async' for and shares.
I'm out of ideas... help!
Last edited: