Hi,
We are running FreeNAS 11.2-RELEASE-U1 for quite a while.
We have an ESX is running VMs over a lun that is located on the FreeNAS, connected over ISCSI.
Lately, it seems like we have reached some bottleneck in our storage system, causing our VMware VMs to run very slowly.
We tried to identify the root cause of the slowness, but we are still not sure what is the problem.
CPU is around 20% used
RAM is used 7GB out of 64
Network interface uses ~0.7Gb/s (while the connection to the ESX is 10Gb/s)
And "iostat -x 3" shows all the SSDs in the pool are utilized to ~50%
The one suspicious thing that we did was in iostat, but it wasn't the SSDs.
There are some"cbb" devices, and part of them (in our case cbb3 and cbb2) are utilized to ~100%
We suspect this is the issue, but have no idea what are these devices.
Sadly, the internet didn't help us much with understanding what are these devices, and they don't even appear in /dev/.
We did manage to understand that there is one "cbb" per ISCSI extent, but we have no idea what that means, and how can we improve it.
A debug file of the FreeNAS machine attached, as far as I understand, it should also contain the hardware information.
Help anyone? :)
We are running FreeNAS 11.2-RELEASE-U1 for quite a while.
We have an ESX is running VMs over a lun that is located on the FreeNAS, connected over ISCSI.
Lately, it seems like we have reached some bottleneck in our storage system, causing our VMware VMs to run very slowly.
We tried to identify the root cause of the slowness, but we are still not sure what is the problem.
CPU is around 20% used
RAM is used 7GB out of 64
Network interface uses ~0.7Gb/s (while the connection to the ESX is 10Gb/s)
And "iostat -x 3" shows all the SSDs in the pool are utilized to ~50%
The one suspicious thing that we did was in iostat, but it wasn't the SSDs.
There are some"cbb" devices, and part of them (in our case cbb3 and cbb2) are utilized to ~100%
We suspect this is the issue, but have no idea what are these devices.
Sadly, the internet didn't help us much with understanding what are these devices, and they don't even appear in /dev/.
We did manage to understand that there is one "cbb" per ISCSI extent, but we have no idea what that means, and how can we improve it.
A debug file of the FreeNAS machine attached, as far as I understand, it should also contain the hardware information.
Help anyone? :)