Intensive IO task running more than 8 hours, on saturdays, each month

[eauque]

Hi all.
I am runing TrueNAS-13.0-U2 with 6 disks (8 TB each). It is accessed via ISCSI connections by two ovirt nodes, running 7 virtual machines.

Each month, on a saturday, some task is running for about 8-10 hours. When I run the gpstat command, can see disks are more than 100% busy. And during this process, my virtual machines run slow and can not even login to them. Once the process finish, all comes to normal again... until the next month.

I have seen in True NAS if there is scheduled tasks, but only find a SCRUB task. But it is scheduled to run in sundays only, and finish in seconds when it runs.

Do you have any idea of what is happening?

 

[jgreco]

Hi all.
I am runing TrueNAS-13.0-U2 with 6 disks (8 TB each). It is accessed via ISCSI connections by two ovirt nodes, running 7 virtual machines.

Each month, on a saturday, some task is running for about 8-10 hours. When I run the gpstat command, can see disks are more than 100% busy. And during this process, my virtual machines run slow and can not even login to them. Once the process finish, all comes to normal again... until the next month.

I have seen in True NAS if there is scheduled tasks, but only find a SCRUB task. But it is scheduled to run in sundays only, and finish in seconds when it runs.

Do you have any idea of what is happening?

You haven't provided much information here, so, only intelligent guesses.

Sounds like you're probably doing something wrong, like too little memory, or using RAIDZ, or filling the pool beyond 40-50% capacity. Please refer to the following resource:

The path to success for block storage

It seems like I haven't written a sticky for awhile, but just in the last week I've had to cover this topic several times. ZFS does two different things very well. One is storage of large sequentially-written files, such as archives, logs, or data files, where the file does not have the middle...

www.truenas.com

which is an absolute must-read for designing an iSCSI based block storage system.

 

[Etorix]

I have seen in True NAS if there is scheduled tasks, but only find a SCRUB task. But it is scheduled to run in sundays only, and finish in seconds when it runs.

There's no way that scrubbing a 6*8 TB pool could "finish in seconds" if the pool holds actual data.
If it's not a scrub, it could be a long SMART test.

 

[eauque]

You haven't provided much information here, so, only intelligent guesses.

Sounds like you're probably doing something wrong, like too little memory, or using RAIDZ, or filling the pool beyond 40-50% capacity. Please refer to the following resource:

The path to success for block storage

It seems like I haven't written a sticky for awhile, but just in the last week I've had to cover this topic several times. ZFS does two different things very well. One is storage of large sequentially-written files, such as archives, logs, or data files, where the file does not have the middle...

www.truenas.com

which is an absolute must-read for designing an iSCSI based block storage system.

Thank you for your response. TrueNas server has 64 GB of memory. I'm going to read the suggested resource. Thank you.
What I am surprised is that between the occurrence of this problem, all tasks runs fine, the performance in virtual machines is also optimal. The ISCSI connections are made via 10GB FC ethernet cards.

 

[eauque]

There's no way that scrubbing a 6*8 TB pool could "finish in seconds" if the pool holds actual data.
If it's not a scrub, it could be a long SMART test.

Yes, I think you are right. But I can not find how to determine what process/task is doing it. And the process always runs on saturday. Once per month.
I do have other trueNAS installation (version12), with 4*4TB disks, also with ISCSI access, 10 GB connections. It has been running for 2 years without this behavior.