Homeassistant VM backup I/O

[daschmidt]

I have a homeassistant on a VM with Truenas Scale running.
Homeassistant make every day at 2am a backup. Every day at the same time I also get this error:

• Device /dev/disk/by-partuuid/6ada7c65-30d8-42e8-bd8a-09df6049c9eb is causing slow I/O on pool tank.

• Device /dev/disk/by-partuuid/1aae3f5f-f1af-4715-bb18-6842dc2ba183 is causing slow I/O on pool tank

this are two 1TB SSD in mirror.



is something misconfigured here or where do I have to look for the error?

 

[daschmidt]

nobody a idea why the I/O is so high?

 

[sretalla]

I don't know how you're "making" your backup... is it a snapshot? followed by a copy? to where? using what protocol/service?

The attached charts show that you're well under the expected maximum for throughput on those disks (although there's a lot of guesswork in that statement as you gave no hardware details at all).

I suspect you're somehow overloaded on IOPS (which is a little concerning as SSDs usually have a vast amount of IOPS capability compared to the requirement that a backup medium would have).

If you can explain a little better what's happening, maybe we can see where the problem is coming from.

 

[daschmidt]

I don't know how you're "making" your backup... is it a snapshot? followed by a copy? to where? using what protocol/service?

it's a build in function in homeassistant. The developer said, the folders where zip. The backupsice is about 3,5GB.

Hardware is a I3 9100 with 32GB ECC Ram.

The VM have 2 virtual CPU's
1 core and
1 thread

 

[sretalla]

The VM have 2 virtual CPU's
1 core and
1 thread

OK, so that's one thing to try...

1 CPU, 2 cores, 1 thread (or 1 CPU, 1 core, 2 threads).

Multi-CPU could be part of the cause in slowing things down.

 

[daschmidt]

OK, so that's one thing to try...

thanks, today night I got no error

 

[daschmidt]

OK, so that's one thing to try...

the joy was short lived. Last night at 02:22 am I got the email with the I/O error again. At 06:23 the e-mail came that the error had been fixed.

 

[sretalla]

a homeassistant on a VM with Truenas Scale running

Can you be a little more detailed on what exactly this is...?

Is it a VM which runs SCALE with another VM inside it running HA or SCALE on bare-metal and HA as a VM on it?

How are you passing the disks in? that format looks strange now that I look more closely:
/dev/disk/by-partuuid/6ada7c65-30d8-42e8-bd8a-09df6049c9eb

 

[daschmidt]

Can you be a little more detailed on what exactly this is...?

I have a server there is truenas scale installed as operating system.
Homeassistant is virtualized on this server.

How are you passing the disks in? that format looks strange now that I look more closely:
/dev/disk/by-partuuid/6ada7c65-30d8-42e8-bd8a-09df6049c9eb

It's a zvol on tank/vm/homeassistant

In the settings it's a VirtIO

 

[sretalla]

OK, well that gives me another idea to try... but you need to understand what's happening before you do that.

You're using a block storage device on an SSD (or a pair of them) to provide disk(s) to a VM.

This will normally be doing sync writes, so adds a lot of load to the write operations to ensure everything written is really on the ssds before confirming OK to continue with the next block.

You can edit the properties of the ZVOL to set it to sync=disabled, which will return those writes while they are still only in RAM... small risk that in a power cut you will lose at most 5 seconds of data... but if that 5 seconds is some critical metadata, the disk may be viewed by the guest as corrupt and not usable.

It's not very likely to be a problem, but if it is, it may represent a total loss of data, so be prepared for that with a download of the HA backup.

 

[daschmidt]

You can edit the properties of the ZVOL to set it to sync=disabled

thanks I will give this a try.
But why I don't see this muche I/O in the graphic? A ssd can writte 100+MB/s

small risk that in a power cut you will lose at most 5 seconds of data

That's not really a problem, there is a usv before the server.

otherwise it's not bad, since a backup is made every day and it's really only about the connected devices in homeassistants

 

[sretalla]

why I don't see this muche I/O in the graphic? A ssd can writte 100+MB/s

The charts are showing throughput, not IOPS.

You can write tons of small files resulting in IOPS for each one or a small number of large files resulting in minimal IOPS, but a lot of data throughput.

It seems reporting doesn't show IOPS in any of the chart options despite calling the throughput chart I/O (which means data in/out, not IOPS)

You can find those metrics in the reporting screen for CORE though:

And you have it on SCALE for NFS (if you were sharing out your ZVOL over NFS).