Slow random write speeds

[HydroHomie]

Intro/Specs​

I'm setting up TrueNAS for the first time. I'm running it as a VM on Proxmox. Here's my hardware:

• CPU
  • i5-4670k
• Motherboard
  • MSI Z87 G41 PC Mate
  • using the built in gigabit NIC
• RAM
  • 12 GB
    • Started at 8 GB per documentation recommendation, but bumped up to see if it helped
    • Can go higher, if necessary (up to 32 GB)
• Pool
  • 4x 4TB HDDs in RAIDZ1
    • 2x Western Digital WD40EFRX (CMR drives)
    • 2x Seagate ST4000VN000 (CMR drives)
    • These are all connected to the motherboard via SATA cables and then passed to TrueNAS VM in Proxmox.
• Boot
  • Proxmox virtual disk, which is an Intel Optane 660p NVME SSD

And finally, I'm pretty sure I left my dataset options as the defaults, but in case I might've changed something, here they are.

Show : Dataset settings

Speed tests/performance​

TrueNAS​

Here are the results from testing from my Windows PC (also using a gigabit NIC).

The RND4K Q32T1 (3rd row) write results are much lower than I'd expect them to be.

I also tested on an Ubuntu VM that was running on the same Proxmox host as the TrueNAS VM.
I used flexible I/O tester (fio) using the scripts found here.
I'm just pasting the rand read and rand write results below as sequential saturates the gigabit NICs just fine.

	Rand Read (MByte/s)	Rand Write (MByte/s)
TrueNAS - NFS	88.1	0.78
TrueNAS - SMB	62.8	32.2

The results are similar to that on Windows, though the NFS rand write speeds are abysmal, but that might be a question for another thread.

Synology DiskStation​

As a point of comparison, I conducted the same tests on my existing Synology NAS and the results were much higher:

And here are the fio results on the Synology.

	Rand Read (MByte/s)	Rand Write (MByte/s)
DiskStation - NFS	107	90.6
DiskStation - SMB	114	83.8

Questions/Troubleshooting​

Is this kind of performance expected from TrueNAS?
I'm hoping for closer to gigabit speeds on TrueNAS for random writes, because I want to use it as LanCache storage.

What kind of troubleshooting steps can I take to find the issue/bottleneck?

Any settings/configuration that may be causing this?

 

[HydroHomie]

NFS Fix​

I watched a couple of videos (1 & 2) to further my learning on ZFS and learned that adding a ZFS Log device to the pool can help with NFS write caching. So I tested this by adding a single 120 GByte SATA SSD to my existing pool as a ZFS Log device and found that rand write speeds for NFS increased to around what I was seeing on SMB (i.e. 25-30 MByte/sec). But still not in the realm of my Synology DiskStation.

Pool troubleshooting cont.​

Alas, the goal to see desired rand write speeds continues. I proceeded to test different configurations for my pool.
I also increased RAM to 24 GBytes. Throughout the tests it seems TrueNAS generally would only take up to 18 GBytes of RAM.

This time running fio directly on the TrueNAS machine (to avoid any network bottlenecks).
I had to change the ioengine in my original script to posixaio for it to work on the TrueNAS SSH (aka FreeBSD).
But with that change I was able to test directly from the dataset.

• I tested all 4 HDDs striped and found this helped increase write speeds to ~45 MBytes/sec.
• I tested all 4 HDDs striped + the SSD as a ZFS Log and results were the same as above ~50 Mbytes/sec
• I tried lots of different configurations like adding 2 different striped vdevs in a pool (one vdev with WD drives and the other with Seagate drives) with and without the ZFS Log SSD and they all returned similar results to my original post
• The most flabbergasting result was when I had just a single vdev with just the single 120 GByte SATA SSD.
  • This test also yielded similar rand write speeds as my original post (20-30 MBytes/sec)

So I'm still left scratching my head and hoping someone has some more troubleshooting steps for me to take.

 

[Arwen]

I have no information on further testing, but I do want to suggest you review passing the disks through from Proxmox. The "correct" way is to pass the disk controller through, giving ZFS direct access to the disks. By passing only the disks, Proxmox acts as a BIOS for any disk access, (as far as I understand it). That may potentially slow down access, (and potentially lead to risk of data loss). Probably not a problem for the boot drive, (if you have the configuration saved). But data drives, that may be part of your problem.

See this;

Resource - "Absolutely must virtualize TrueNAS!" ... a guide to not completely losing your data.

[---- 2024/01/16: Still relevant. Virtualization really doesn't change much. Updates made as appropriate. ----] [---- 2018/02/27: This is still as relevant as ever. As PCIe-Passthru has matured, fewer problems are reported. I've updated some...

www.truenas.com

Next, I tried to find what network chip was used in that board, neither the system board specifications nor the user manual list that detail. Certain network chips are proven to be of lower performance, (though obviously your local testing is not affected by the NIC). If you can get the make & model of the network chip, we can advise further.

Use of a SLOG for NFS on ZFS has some peculiarities in that you want a SSD with high write speed & high write endurance. Some consumer SSDs claim good performance, but for burst writes they don't perform well. And many consumer SSDs don't have the write endurance ZFS desires, though for SOHO uses write endurance might not be as critical.

Last, I was unable to find which TrueNAS version you were using. This does make a difference as their was an annoying ZFS ARC cache bug in a recent, (but not current), version of TrueNAS Core. (If I remember correctly that it is now fixed in current...)

 

[HydroHomie]

I have no information on further testing, but I do want to suggest you review passing the disks through from Proxmox. The "correct" way is to pass the disk controller through, giving ZFS direct access to the disks. By passing only the disks, Proxmox acts as a BIOS for any disk access, (as far as I understand it). That may potentially slow down access, (and potentially lead to risk of data loss). Probably not a problem for the boot drive, (if you have the configuration saved). But data drives, that may be part of your problem.

See this;

Resource - "Absolutely must virtualize TrueNAS!" ... a guide to not completely losing your data.

[---- 2024/01/16: Still relevant. Virtualization really doesn't change much. Updates made as appropriate. ----] [---- 2018/02/27: This is still as relevant as ever. As PCIe-Passthru has matured, fewer problems are reported. I've updated some...

www.truenas.com

I am not using a controller. All my HDDs are hooked up directly to the motherboard via SATA. I'm then using the instructions here to pass them through to the TrueNAS VM. My CPU doesn't support IOMMU, so I wouldn't be able to pass through a PCI device.

Next, I tried to find what network chip was used in that board, neither the system board specifications nor the user manual list that detail. Certain network chips are proven to be of lower performance, (though obviously your local testing is not affected by the NIC). If you can get the make & model of the network chip, we can advise further.

The lspci command says that my ethernet controller is:

Code:

03:00.0 Ethernet controller: Realtek Semiconductor Co., Ltd. RTL8111/8168/8411 PCI Express Gigabit Ethernet Controller (rev 0c)

Use of a SLOG for NFS on ZFS has some peculiarities in that you want a SSD with high write speed & high write endurance. Some consumer SSDs claim good performance, but for burst writes they don't perform well. And many consumer SSDs don't have the write endurance ZFS desires, though for SOHO uses write endurance might not be as critical.

Interesting point. I tested this by creating a 15 GByte LVM disk from the 1 TByte NVME SSD I have that has Proxmox installed on it. I used this 15 GByte disk as the new ZFS Log device and rand write speeds seemed to increase a bit 50-60 MBytes/sec, but still not as high as I'd like...

Last, I was unable to find which TrueNAS version you were using. This does make a difference as their was an annoying ZFS ARC cache bug in a recent, (but not current), version of TrueNAS Core. (If I remember correctly that it is now fixed in current...)

Woops, sorry about that. I'm on the latest as I just installed within the last week. TrueNAS-13.0-U6.1

 

[Arwen]

I am not using a controller. All my HDDs are hooked up directly to the motherboard via SATA. I'm then using the instructions here to pass them through to the TrueNAS VM. My CPU doesn't support IOMMU, so I wouldn't be able to pass through a PCI device.
...

Even SATA ports on the system board have a controller. If you can't pass that entire SATA controller through to the VM, then that IS A PROBLEM. The note from Proxmox does not take into account ZFS. They may say it is fine, but it does affect performance. See that article I referenced.

...
The lspci command says that my ethernet controller is:

Code:

03:00.0 Ethernet controller: Realtek Semiconductor Co., Ltd. RTL8111/8168/8411 PCI Express Gigabit Ethernet Controller (rev 0c)

...

Yes, that is one of the problematic brands. See;

Is my Realtek ethernet really that bad?

One of the longest running issues faced here in the forums may be people who come to these forums, wanting to recycle an old PC into a NAS. In general, this is a bad idea because typical PC hardware isn't optimized towards the task. However...

www.truenas.com

Theoretically TrueNAS SCALE might be faster with that chip because it uses a different base OS, (Linux), & driver. But, the Open Source information about some of these devices is limited. Then, even with decent information, some driver developers stop when they getting it working without errors. Meaning, they did not go through and try to make the network driver as high as performance as they can.

Intel has probably written many of the Linux, (and possibly FreeBSD), drivers for their network chips, so they have a vested interest in making them as fast as possible. This then encourages people to buy their network chips when looking for a network performance upgrade.

 

[HydroHomie]

Considering I can't pass the SATA controller to the VM because my CPU doesn't support IOMMU (i.e. VT-d) and I might have a problematic NIC, I've decided to forgo storing LanCache's cache data on TrueNAS and just store it directly on the same NVME disk that the LanCache itself is running on.

I obviously have much less capacity for caching now, but it should be fine as my use case is just for my wife I. LanCache will help us avoid our ISP's data caps when we both download the same game to our PCs to play together.

Thanks for the guidance and avoiding further headache!