I seem to be a glutton for punishment, so here I am, returned once again, with questions about pool performance.
My Setup
I have decided that I wanted to build a single Proxmox box that houses both virtualized FN along with various VMs for media sharing and testing/fun stuff. I building this box to replace a long-running Unraid box and am building it for future use with mGig/10GbE support. Specs for the PM host are:
"Ok, let's see what performance I'm getting!"
* I recognize that, if it is somehow smbd limiting performance, more cores will do nothing as smbd is single-threaded. My thought was something else that was being done by the FreeNAS VM was hogging other resources and causing a bottleneck. One of my next troubleshooting tests is swapping in a CPU with a slightly higher base clock to see if anything changes.
EDIT: ZFS on Proxmox
Out of curiosity, I decided to remove the passthrough of the LSI and onboard SCU, import the pool directly into zfs, run a simple share on the same Movies share used above, and test performance. In doing so, I saw ~850MB/s read/ ~800MB/s write speeds directly to the pool from Proxmox, and transferring files in the same Ubuntu VM maxed out my SSD speeds on "fresh" (ie: files not accessed and thus not cached in ARC). This at least shows it's not a hardware problem.
EDIT: Additional Testing, abridged
I have also tried various combinations of the following, in no particular order:
Problems I want to fix
Out of all of that, the big issue I want to fix is:
Any suggestions or troubleshooting recommendations are welcome with outstretched arms.
Cheers
My Setup
I have decided that I wanted to build a single Proxmox box that houses both virtualized FN along with various VMs for media sharing and testing/fun stuff. I building this box to replace a long-running Unraid box and am building it for future use with mGig/10GbE support. Specs for the PM host are:
- Motherboard: SuperMicro X9SRA
- CPU: Xeon e5 2660
- RAM: 128GB 1600mhz DDR3 ECC
- Boot disk: Samsung 860 EVO 500gb
- Pool disk connections: LSI 9211-8i in IT mode (8 drives) & onboard (mobo) Intel SCU (4 drives)
- ** Note: the 9211-8i is plugged into a 4x PCI-E x4 slot, so I know this will handicap me a bit, so I'm not looking for theoretical max performance (based on drive speeds and ZFS theoretical calculations), only for equivalent performance as on FN VM as between FN and Ubuntu VM.
- Proxmox installed.
- PCI-E passthrough setup.
- FreeNAS installed as VM
- passed 4 cores, 32gb memory, virtio NIC
- Both LSI HBA (20.0.0.7-IT firmware and matching BIOS) and mobo SCU passed through to FN VM
- pool and datasets built (2x 6 disk Raidz2, 12tb shucked easystore drives [~225MB/s read / 200MB/s per write per drive based on previous testing]).
- Datasets shared via SMB
- Ubuntu VM installed and SMB shares mounted via fstab.
- Both VMs share one network bridge (vmbr0) that connects them to my home network (LAN is 1Gbe all around). vmbr0 is set to MTU 9000, because in a previous setup I learned this is crucial for max throughput.
"Ok, let's see what performance I'm getting!"
- DD in FreeNAS host
- write to dataset (no compression, 80GB testfile [bs=2048k count=40000]): ~750MB/s
- read: ~550MB/s.
- hmmmm, odd. Slower reads?
- FOR REFERENCE: These are the speeds that I'm looking for between VMs, but I would like to understand why I'm getting slower reads and how to fix it.
- Iperf3 Between FN VM and Ubuntu VM
- FN as server: 12.7Gbps
- Reversed: 10.3Gbps
- Great, network performance is good, if a smidge asymmetrical, and is not leaving the server to the LAN (again, LAN is all 1gbe).
- File transfer in Ubuntu to/from SMB share of "Movies" dataset, compression=off.
- Read from FN to Ubuntu: 190MB/s
- Wait, what?
- Delete the new local copy file, copy same file again from SMB share to Ubuntu local disk (file is now cached in ZFS ARC): ~420MB/s read.
- Ok, so SMB isn't directly bottlenecking. Once the file is in ARC, I can transfer at near the theoretical write speed of the SSD.
- Rename the now-local file copied previously. Write back to same folder on SMB share: 600MB/s write
- Theoretical limit of SSD read, so that makes sense.
- Read from FN to Ubuntu: 190MB/s
- Run same type of copy with new large file from same dataset (eliminate ARC from the chain) using "pv" from terminal: same results as with "Files"
- DD in Ubuntu onto mounted SMB share of "Movies" dataset (compression still off):
- write=505MB/s
- smbd on FN VM hovered around 75% single-core usage during whole test, so not pegged at 100% but maybe CPU limited?
- read back=228MB/s
- smbd: around 68% +/-5% during whole test
- write=505MB/s
* I recognize that, if it is somehow smbd limiting performance, more cores will do nothing as smbd is single-threaded. My thought was something else that was being done by the FreeNAS VM was hogging other resources and causing a bottleneck. One of my next troubleshooting tests is swapping in a CPU with a slightly higher base clock to see if anything changes.
EDIT: ZFS on Proxmox
Out of curiosity, I decided to remove the passthrough of the LSI and onboard SCU, import the pool directly into zfs, run a simple share on the same Movies share used above, and test performance. In doing so, I saw ~850MB/s read/ ~800MB/s write speeds directly to the pool from Proxmox, and transferring files in the same Ubuntu VM maxed out my SSD speeds on "fresh" (ie: files not accessed and thus not cached in ARC). This at least shows it's not a hardware problem.
EDIT: Additional Testing, abridged
I have also tried various combinations of the following, in no particular order:
- SeaBIOS vs OVMF UEFI
- i440fx vs q35
- changed VM "cpu" type
- Changed some parameters in FSTAB
Problems I want to fix
Out of all of that, the big issue I want to fix is:
- Why is reading from the pool (disks, not ARC) so damn slow, compared to speeds I get from within FN itself?
- It's not the "network" as iperf is >=10Gbps.
- I dont think it's smbd as from ARC to VM is maxing out my SSD.
- It's not disk access as raw speeds reading from disk from within FN are far faster (190MB/s vs 500MB/s).
- It's not the SSD as, again, ARC to SSD maxes out the SSD write speed.
- EDIT: It's not hardware as importing the pool into ZFS on Proxmox and sharing reached max theoretical speeds.
Any suggestions or troubleshooting recommendations are welcome with outstretched arms.
Cheers
Last edited: