I have a FreeNAS system I intend to use as a backup storage. I was trying to get a feel for the file transfer speed entitlement of my hardware when I found a strange (to me at least) difference between Linux+SMB vs Windows+SMB that prevents Linux from matching Windows performance on file transfers to FreeNAS on my system.
My FreeNAS system (standalone, nothing else running on it):
FreeNAS 11.1 - U4
Supermicro X9SCM-F
E3-1240V2
32GB ECC DDR3 1600
Intel 82579LM and 82574 1Gb NICs
WD Red 5400 RPM 4TB drives
Client System:
Linux Mint 18.3 host OS (Ubuntu 16.04 I believe)
Windows 10 VM (KVM/QEMU)
ASUS Server board
dual E5-2670
64GB memory
Intel NICs
All tests were using large seq writes of media files 1GB or larger per file with a total transfer of 20-60GB each time. Used the same dataset in all cases. Share type Unix, compression off, sync=standard.
I am using single disk pools. Yes, I know (and have read) that for most cases this is pointless and defeats some of the ZFS benefits. There is no redundancy, can't fix errors, etc. I do it for the same reason others have (to have scrub capability), home backup system only, not an active file server. I follow 3-2-1 so if I lose a disk I still have the originals plus an offsite copy etc.
Bottom line is that from the Windows 10 VM (SMB 3.1.1) I can get ~108MB/s at least up to the multi-file 60GB total that I tested. The transfer rate stays very steady and smooth throughout the entire transfer. FreeNAS reporting confirms this for both disk and network. This is about the max I would expect from these 5400 RPM drive and 1Gb pipe. When I use samba on the Linux system (I tried SMB 3, 2.1, and 1.0) I can get only 70MB and it is very spikey no matter what I did with mount parms etc.
Note that since this is a WIN10 VM, both WIN10 and Linux are using the same hardware, same NIC, same cable, same switch, network path etc. And is reading the source files from the same physical disk.
I turned on FreeNAS SMB logging and what I see is that the log and timestamps confirm that the WIN10 writes are consistent within a file and file to file transitions. On the other hand, the Linux writes are fast within a file (hitting the same 100MB+) but there is a repeatable ~5 sec delay between files. Top is showing that during writing, smbd is ~25-30% on a single thread (as expected). Between files, smbd drops to zero for about 5 seconds (WIN10 does not do this, smbd is steady). The time stamps in the log (log setting=full) shows the same thing, but there are no errors. For Linux just a ~5 sec (slightly over that) delay from when each and every file gets "opened" and the first writing log entry for that specific file.
If you do the math that 5+ sec delay between file end and next file start, at 1Gb speeds, accounts for the difference between 108GB/s and 70GB/s. So I can clearly see why Linus+SMB is slower, but I don't know what causes that delay between files on FreeNAS when using Linux to connect, but not when using WIN10.
If I had to guess, the time is suspiciously close to the 5 sec txg time. Maybe it is either being flushed, or has to expire. But then again, why only for a Linux SMB connection to the share and not a WIN10 connection?
I am looking into ZFS tunables and may look at newer Ubuntu versions with newer samba versions to see if that makes a difference.
I also looked at NFS, but can't get it above 70MB/s either, even with async,.and it was spiky too,
If anyone knows what might be going on or has any suggests any help would be welcome.
Thanks
My FreeNAS system (standalone, nothing else running on it):
FreeNAS 11.1 - U4
Supermicro X9SCM-F
E3-1240V2
32GB ECC DDR3 1600
Intel 82579LM and 82574 1Gb NICs
WD Red 5400 RPM 4TB drives
Client System:
Linux Mint 18.3 host OS (Ubuntu 16.04 I believe)
Windows 10 VM (KVM/QEMU)
ASUS Server board
dual E5-2670
64GB memory
Intel NICs
All tests were using large seq writes of media files 1GB or larger per file with a total transfer of 20-60GB each time. Used the same dataset in all cases. Share type Unix, compression off, sync=standard.
I am using single disk pools. Yes, I know (and have read) that for most cases this is pointless and defeats some of the ZFS benefits. There is no redundancy, can't fix errors, etc. I do it for the same reason others have (to have scrub capability), home backup system only, not an active file server. I follow 3-2-1 so if I lose a disk I still have the originals plus an offsite copy etc.
Bottom line is that from the Windows 10 VM (SMB 3.1.1) I can get ~108MB/s at least up to the multi-file 60GB total that I tested. The transfer rate stays very steady and smooth throughout the entire transfer. FreeNAS reporting confirms this for both disk and network. This is about the max I would expect from these 5400 RPM drive and 1Gb pipe. When I use samba on the Linux system (I tried SMB 3, 2.1, and 1.0) I can get only 70MB and it is very spikey no matter what I did with mount parms etc.
Note that since this is a WIN10 VM, both WIN10 and Linux are using the same hardware, same NIC, same cable, same switch, network path etc. And is reading the source files from the same physical disk.
I turned on FreeNAS SMB logging and what I see is that the log and timestamps confirm that the WIN10 writes are consistent within a file and file to file transitions. On the other hand, the Linux writes are fast within a file (hitting the same 100MB+) but there is a repeatable ~5 sec delay between files. Top is showing that during writing, smbd is ~25-30% on a single thread (as expected). Between files, smbd drops to zero for about 5 seconds (WIN10 does not do this, smbd is steady). The time stamps in the log (log setting=full) shows the same thing, but there are no errors. For Linux just a ~5 sec (slightly over that) delay from when each and every file gets "opened" and the first writing log entry for that specific file.
If you do the math that 5+ sec delay between file end and next file start, at 1Gb speeds, accounts for the difference between 108GB/s and 70GB/s. So I can clearly see why Linus+SMB is slower, but I don't know what causes that delay between files on FreeNAS when using Linux to connect, but not when using WIN10.
If I had to guess, the time is suspiciously close to the 5 sec txg time. Maybe it is either being flushed, or has to expire. But then again, why only for a Linux SMB connection to the share and not a WIN10 connection?
I am looking into ZFS tunables and may look at newer Ubuntu versions with newer samba versions to see if that makes a difference.
I also looked at NFS, but can't get it above 70MB/s either, even with async,.and it was spiky too,
If anyone knows what might be going on or has any suggests any help would be welcome.
Thanks
Last edited: