CORE VM - Slow read speeds

[adirburke]

Hi All,

TLDR :: Promox running a TrueNAS VM with passed through HBA getting very slow NFS speeds. Noob needs help with setup


I am running a proxmox server with the specs:
CPU : Dual Silver 4114
RAM : 10x 16GB ECC 2400 Mhz
MotherBoard : WS C621E SAGE


On the ProxMox I am running a TrueNAS-13.0-U3.1 Vm:
8 Cores
16Gb of ram
q35

Passing through to the TrueNas VM

- Adaptec Asr-71605
- Intel T540-DA2

Connected to the Adaptec
4x Samsung 860 1TB in zfs raidz1 (Called fast) - Compression off
5x 10tb Ironwolf Pro in zfs raidz1 (Called slow) - Compression off

NFSv4 setup to both pools

iPerf3 from pve to nas 20Gb/s
iPerf3 from nas to pve 20Gb/s

Speed over NFS

Code:

Writing Test

root@pve:/mnt/pve/nas# dd if=/dev/zero of=testfile bs=4M count=10000
10000+0 records in
10000+0 records out
41943040000 bytes (42 GB, 39 GiB) copied, 78.0714 s, 537 MB/s

Reading Test

root@pve:/mnt/pve/nas# dd of=/dev/zero if=testfile bs=4M count=10000
10000+0 records in
10000+0 records out
41943040000 bytes (42 GB, 39 GiB) copied, 73.9434 s, 567 MB/s

Speed direct from NAS shell

Code:

Write Test
root@truenas[/mnt/fast/testspeed]# dd if=/dev/zero of=testfile bs=4M count=10000
10000+0 records in
10000+0 records out
41943040000 bytes transferred in 35.452023 secs (1183092991 bytes/sec)

Read Test

root@truenas[/mnt/fast/testspeed]# dd of=/dev/zero if=testfile bs=4M count=10000
10000+0 records in
10000+0 records out
41943040000 bytes transferred in 40.815562 secs (1027623727 bytes/sec)
root@truenas[/mnt/fast/testspeed]# 

 

[adirburke]

iozone -a -s 48g -r 4096

Code:

                                                              random    random     bkwd    record    stride                                   
              kB  reclen    write  rewrite    read    reread    read     write     read   rewrite      read   fwrite frewrite    fread  freread
        50331648    4096  1084374  1065329   911521   930583   497851  1072405   531983   9335051    597734  1058855  1030935   857024   909104

 

[adirburke]

Update:
Upgraded to Scale and now
im getting 1.2GB/s up/down on 10Gb/s link
and 2.0GB/s up/down on 40Gb/s link

 

[jgreco]

Adaptec Asr-71605

This is not suitable for use with TrueNAS. See

What's all the noise about HBAs, and why can't I use a RAID controller?

1) An HBA is a Host Bus Adapter. This is a controller that allows SAS and SATA devices to be attached to, and communicate directly with, a server. RAID controllers typically aggregate several disks into a Virtual Disk abstraction of some sort...

www.truenas.com

 

[WI_Hedgehog]

@jgreco has a point, from what I understand you're not passing through a HBA, it's a Microsemi Adaptec 16-port RAID controller. If I understand correctly Adaptec's closest mode is JBOD, however it still runs through the RAID BIOS and doesn't give TrueNAS direct access to the disks like a HBA would.

The Adaptec RAID cards I have running in DELL servers are rock solid, however running a RAID card or anything but a rock-solid HBA under TrueNAS is probably not a great path to go down.

 

[jgreco]

The big concern with the Adaptec stuff is the use of the aacraid driver, which is known to cause system panicks under certain conditions. Even without that, we don't really know if it implements stuff like device hotswap and disconnection properly, or has other sharp edges, because ZFS is very demanding and particular in what it wants to see out of attached storage devices. Despite the fact that ZFS should theoretically be impervious to random events like power losses, crashes, or system panicks, we strongly encourage users to use a UPS, and it is probably foolish to tempt fate by using a device known not to be completely stable.