Performance problems from virtual machines

[Pelle Hanses]

Hi,
short problem statement. Something is doing VM's working incredible slow. The performance are so bad that if I copy many files in one VM everything else stops.

I'm using Proxmox as virtual environment and using FreeNAS as storage servers for all VM's. We have a couple of Proxmox clusters and three Supermicro servers for FreeNAS. All of the three FreeNAS servers have 36 disk bays, 2 x CPU, 256GB RAM or more, 30Gb/s to core switch. The Proxmox servers have 20Gb/s connection to the core switch.
VM's using other FreeNAS server are working very well. The new server with the performance issue have:

• one pool with 10 14TB HGST Ultrastar HE14 SAS disks in Raid 10 and
• 6 SSD in Raid 10
• there are no log or cache disks.

Memory used for ZFS cache are 240 GB and only 7GB free.

Question: Is it possible to see if it is the lack of log or cache that are causing the performance problems?

I have tested to shut down all VM's and the network speed are correct.
I have some extra SSD I can use for log and cache disks. I have set up many other FreeNAS servers and the log disks have done wonders for performance.

/PelleH

 

[PhiloEpisteme]

• one pool with 10 14TB HGST Ultrastar HE14 SAS disks in Raid 10 and
• 6 SSD in Raid 10

I'm going to nit pick a bit here; but I think it is worthwhile. FreeNAS and zfs has no RAID 10. ZFS uses pools, pools use vdevs, and vdevs are one of the following

• single-disk
• mirror vdevs of 2 or more disks
• raidz1
• raidz2
• raidz3

Raid10 is most similar to a pool composed of two mirror vdevs. raidz1 is most similar to raid5. raidz2 is most similar to raid6. mirror vdevs are most similar to raid1. Raid1 is VERY different from raidz1 but they both read similarly. I think there is benefit to being a bit picky about terminology just to avoid confusion. :)

Sorry to reply with a nit pick and no solution to your problem. Hopefully you'll forgive me.

Edit:

Question: Is it possible to see if it is the lack of log or cache that are causing the performance problems?

Yes, check zilstat and the arc hit ratio. If you're using sync writes and have a lot of zil usage a SLOG may help, especially on that HDD pool. If your arc hit ratio is low you'll want to first max out your ram; and then consider L2ARC devices.

 

[Pelle Hanses]

My bad, the pool consists of 4 vdevs of 2 mirrored disks.

I tested with zilstat on both pools and I get this results - what should I understand from this?
SSDPool consists of SSD and DataPool consists of SAS spinning disks.

Code:

# zilstat -p SSDPool       
   N-Bytes  N-Bytes/s N-Max-Rate    B-Bytes  B-Bytes/s B-Max-Rate    ops  <=4kB 4-32kB >=32kB
         0          0          0          0          0          0      0      0      0      0
      5040       5040       5040      73728      73728      73728      2      0      0      2
     13040      13040      13040      73728      73728      73728      2      0      0      2
     33896      33896      33896     110592     110592     110592      3      0      0      3
     29800      29800      29800     110592     110592     110592      3      0      0      3
    124240     124240     124240     516096     516096     516096     14      0      0     14
         0          0          0          0          0          0      0      0      0      0
         0          0          0          0          0          0      0      0      0      0
         0          0          0          0          0          0      0      0      0      0
    155768     155768     155768     724992     724992     724992     33      0     20     13
         0          0          0          0          0          0      0      0      0      0
    231104     231104     231104     565248     565248     565248     16      0      1     15
    146032     146032     146032     663552     663552     663552     18      0      0     18
         0          0          0          0          0          0      0      0      0      0
      4472       4472       4472      36864      36864      36864      1      0      0      1
         0          0          0          0          0          0      0      0      0      0
     63696      63696      63696     221184     221184     221184      6      0      0      6
^C
    160016     160016     160016     811008     811008     811008     22      0      0     22

# zilstat -p DataPool
   N-Bytes  N-Bytes/s N-Max-Rate    B-Bytes  B-Bytes/s B-Max-Rate    ops  <=4kB 4-32kB >=32kB
    190392     190392     190392     921600     921600     921600     25      0      0     25
    312784     312784     312784    2179072    2179072    2179072     46      0      3     43
    187944     187944     187944     700416     700416     700416     19      0      0     19
    157808     157808     157808     860160     860160     860160     26      0      4     22
    228000     228000     228000     958464     958464     958464     28      0      3     25
    102816     102816     102816     442368     442368     442368     12      0      0     12
    104896     104896     104896     589824     589824     589824     16      0      0     16
    331784     331784     331784    1724416    1724416    1724416     55      0     15     40
    314848     314848     314848    2080768    2080768    2080768     52      0      4     48
    151640     151640     151640     757760     757760     757760     21      0      6     15
    121184     121184     121184    1130496    1130496    1130496     36      0     12     24
     40624      40624      40624     221184     221184     221184     10      0      6      4
^C
    165816     165816     165816     847872     847872     847872     25      0      3     22

 

[Heracles]

Hey Pelle,

How is your storage presented to the VM ? If you are using NFS, you are almost certainly doing Sync Writes with it and that is the culprit.

I would suggest you switch to iSCSI for exposing your storage.

Also, how loaded is your pool ? Performance starts to drop from as soon as 10%. Up to 50, you already lost a big factor compared to an empty pool. Should you be 75 or over, that would definitely hurt a lot.

 

[Pelle Hanses]

The storage are presented over NFS. NFS are much better for Proxmox and from my other storage servers there are no problem.

The DataPool (spinning SAS disks) are only loaded to 17% and SSD-pool to 45%.

I tested to diable atime and the performance get a little better. The strange thing is if I enable atime performance will also be better than before...
A new zilstat test with atime disabled.

Code:

zilstat -p DataPool
   N-Bytes  N-Bytes/s N-Max-Rate    B-Bytes  B-Bytes/s B-Max-Rate    ops  <=4kB 4-32kB >=32kB
    485664     485664     485664    2867200    2867200    2867200     68      0      8     60
    574376     574376     574376    3428352    3428352    3428352     99      0     21     78
    233104     233104     233104    1294336    1294336    1294336     38      0     15     23
    202888     202888     202888     847872     847872     847872     23      0      0     23
    200288     200288     200288     991232     991232     991232     28      0      3     25
    313392     313392     313392    2416640    2416640    2416640     58      0     10     48
    405824     405824     405824    2449408    2449408    2449408     56      0      7     49
   1226992    1226992    1226992    4997120    4997120    4997120    138      0     25    113
    147376     147376     147376     663552     663552     663552     18      0      0     18
    437296     437296     437296    1253376    1253376    1253376     34      0      0     34
    711592     711592     711592    2764800    2764800    2764800     59      0      0     59
   1071608    1071608    1071608    5287936    5287936    5287936    129      0      1    128
    360872     360872     360872    1904640    1904640    1904640     67      0     23     44
    148456     148456     148456     405504     405504     405504     11      0      0     11
    361800     361800     361800    1798144    1798144    1798144     39      0      0     39
    600856     600856     600856    3297280    3297280    3297280     69      0      0     69
     61672      61672      61672     380928     380928     380928     11      0      1     10
^C
   1208552    1208552    1208552    5517312    5517312    5517312    163      0     20    143

 

[Heracles]

The storage are presented over NFS. NFS are much better for Proxmox and from my other storage servers there are no problem.

NFS is doing sync writes be default. That is, the operation is not marked as successful until the full write is done and over. When you need to update such a big file as a VMDK (or whatever virtual hard drive format is used by Proxmox), that is a big write to do.

By doing iSCSI, only the small blocks that need to be updated will be written. That way, it is way easier to handle for ZFS.

Should you wish to risk the integrity of your NFS content, you can disable sync writes and speed will boost by a big factor. But know that you will then be exposed to corruption, so you may loose it all.

That is why I recommend you to do iSCSI for your storage : safer and faster.

 

[HoneyBadger]

That is why I recommend you to do iSCSI for your storage : safer and faster.

iSCSI by default doesn't use synchronous writes, so it is less safe in the out-of-the-box state. Forcing sync=always to regain that safety results in the same performance state as NFS by default.

There are some other differences (eg: MPIO, recordsize tuning for file vs block, iSCSI tending to require more RAM) but the sync/async difference should be highlighted here; there's no free lunch when it comes to sync writes.

 

[Pelle Hanses]

Hi, I want to use NFS bacause simplicity with Proxmox.
I did add 2 log disks (mirrored dev) and here are the results with zilstat:
Before adding log dev:

Code:

zilstat -p DataPool
   N-Bytes  N-Bytes/s N-Max-Rate    B-Bytes  B-Bytes/s B-Max-Rate    ops  <=4kB 4-32kB >=32kB
   1031208    1031208    1031208    4567040    4567040    4567040     91      0      0     91
   1061000    1061000    1061000    5173248    5173248    5173248     95      0      0     95
   1092840    1092840    1092840    5058560    5058560    5058560    107      0      0    107
    871056     871056     871056    4513792    4513792    4513792     94      0      0     94
    446352     446352     446352    2416640    2416640    2416640     46      0      0     46
   1323872    1323872    1323872    5062656    5062656    5062656    100      0      0    100
    962536     962536     962536    4239360    4239360    4239360    107      0      0    107
    814464     814464     814464    4161536    4161536    4161536     80      0      0     80
   1315744    1315744    1315744    5619712    5619712    5619712    108      0      0    108
    612240     612240     612240    2482176    2482176    2482176     54      0      0     54
   1022952    1022952    1022952    4075520    4075520    4075520     83      0      0     83
    981816     981816     981816    3739648    3739648    3739648     81      0      0     81
    928744     928744     928744    4501504    4501504    4501504     99      0      0     99
    836448     836448     836448    4177920    4177920    4177920     84      0      0     84
    676896     676896     676896    3260416    3260416    3260416     60      0      0     60
   1320200    1320200    1320200    5271552    5271552    5271552     95      0      0     95

After adding log dev:

Code:

zilstat -p DataPool
   N-Bytes  N-Bytes/s N-Max-Rate    B-Bytes  B-Bytes/s B-Max-Rate    ops  <=4kB 4-32kB >=32kB
   9919360    9919360    9919360   34025472   34025472   34025472    416      0      6    410
   3219456    3219456    3219456    9547776    9547776    9547776    160      0      0    160
   2433144    2433144    2433144    9900032    9900032    9900032    225      0      0    225
   7268096    7268096    7268096   29626368   29626368   29626368    376      0      0    376
   8144176    8144176    8144176   36941824   36941824   36941824    434      0      0    434
   3467296    3467296    3467296   11792384   11792384   11792384    292      0      0    292
   4907528    4907528    4907528   19697664   19697664   19697664    247      0      0    247
   9748160    9748160    9748160   45682688   45682688   45682688    496      0      1    495
   3564192    3564192    3564192   12390400   12390400   12390400    308      0      0    308
   9014416    9014416    9014416   39817216   39817216   39817216    438      0      0    438
   8892504    8892504    8892504   38215680   38215680   38215680    493      0      0    493
   3415448    3415448    3415448   10862592   10862592   10862592    261      0      0    261

What I can understand is the performance of writing significantly better.

 

[mjt5282]

That is a great result, but keep in mind not all SLOG devices perform the same... Check out Honeybadger's thread about SLOG performance....IMHO if you are doing sync writes, you should invest in a fast SLOG device.

 

[Pelle Hanses]

I'm using two of these as log devices: Intel Solid-State Drive D3-S4610 Series, 240GB 2.5" Serial ATA-600.
Now we have tested with some database servers and statistical calculation programs and it is running much better.

 

[HoneyBadger]

Glad to hear things have improved. The S4610 is a "mixed use" drive so it will be an adequate SLOG. You might see further improvements by going to a "write-intensive" drive - depending on if you have SAS or SATA free can determine your options here.

The best performance available now though is often via NVMe devices like the Intel Optane or DC P-series drives (P3700/P4600) as they have much lower latency and higher performance for sync write workloads. This would require one (if your motherboard does PCI bifurcation) or two PCIe slots though.