iocage list is very slow after upgrade to FN-11.3-RC1

Niels Erik

Dabbler
Joined
Aug 9, 2015
Messages
18
Also posted to: https://github.com/iocage/iocage/issues/1114
iocage list is very slow after upgrade to FN-11.3-RC1
RC 1 contains iocage 1.2
The command 'iocage list' takes 45 seconds to complete on my system.
After a lot of digging I have found out that the time is consumed in
iocage-lib/zfs.py line 8 run(command,**kwargs)
The function performs the command "zfs get -H -o name,property,value all" , that lists all attributes, for all snapshots, for all data-sets.
On my system it generates an output that is 256960 lines long.
I have 183 data-sets on my system, and 2 months daily snapshots.

The main part of my data-sets is not even in the path of iocage.. and should not be of interest to iocage.
There is only 3 jails on my system.
Other iocage commands like console have the same behavior.
Have others experienced this slow behavior?
Should this be reported as a bug?
 
Last edited:

Yorick

Wizard
Joined
Nov 4, 2018
Messages
1,912
Have others experienced this slow behavior?
Should this be reported as a bug?

No, and yes.

I have only one jail, a plex plugin. iocage list returns results immediately.

And yes, that should be reported as a bug. There's some interplay between your setup and the 11.3 release that's causing this, I am thinking. And with a bug, the devs can figure out what it is and account for it / fix it.
 

Snow

Patron
Joined
Aug 1, 2014
Messages
309
I have same Thing going on its just not the J list doing it. When I edit a jail it takes like 1min to load the setting's.
Also my jails are on a Raidz1 array of SSD'S I mean speed of pool is not a problem. Read & wright is in the 1.8Gb/s both way's
I do not have the amount of snapshots you do. I do a 3 week and 2 months snapshot. The 3 week one is Weekly.

Unlike this problem witch I am not having.
My problem in detail.

Plugin also take About 20 secs to load not sure if your also have the same problem.
 
Last edited:

Snow

Patron
Joined
Aug 1, 2014
Messages
309
I Removed all my snapshots to see if this would help it has not. Still take 60sec just to load the jails in the gui.
I am pretty sure this is a bug if more then One person is having this problem.
 
Last edited:
Joined
Jul 2, 2019
Messages
648
I have some slowness with four jails - 3 Emby jails (different subnets) and 1 for the UniFi Controller. It seems that the jail GUI response takes about twice as long to load. Maybe if I have time this weekend I will delete the four jails and reinstall them to see if a "native" installation makes it quicker. I do have all my jails upgraded to 11.3.
 

beedix

Dabbler
Joined
Aug 11, 2017
Messages
20
I also upgraded from 11.2 (latest) to 11.3 today. After upgrade, I have the same issue with extremely slow iocage commands. I have 13 jails. As others describe, the GUI even has a seemingly consistent delay until the jails are even displayed.

Any iocage command seems to have this delay.

Any help on where to go is appreciated.
 

Snow

Patron
Joined
Aug 1, 2014
Messages
309
I do not have all my jails upgraded its about half and half. 11.3/11.2.
 
Joined
Jul 2, 2019
Messages
648
I wonder, could it be where it is reaching out to the iXsystems' official plugins and the new community plugins site?
 

Snow

Patron
Joined
Aug 1, 2014
Messages
309
Here is what happening for me, go to About 00:25 Even deleting jails takes a long time which in 11.2 it was fast. Right after that I add a jail and it takes like 20 secs to just load the setup page.

Well ones a jail is created It should not have to talk to the ixsystems any more, unless you are updating a plugin I think.
 
Last edited:

Snow

Patron
Joined
Aug 1, 2014
Messages
309
Has any one created a bug issues for this yet ?
 

Snow

Patron
Joined
Aug 1, 2014
Messages
309
when you do iocage list is it very slow or fast for you?
 

pkropf

Dabbler
Joined
Jul 15, 2019
Messages
14
I'm seeing the same slowdown after upgrading to 11.3. From the CLI, iocage list shows:

b0210520-fs01% time iocage list
+-----+------------+-------+--------------+-----------------+
| JID | NAME | STATE | RELEASE | IP4 |
+=====+============+=======+==============+=================+
| 12 | bacula | up | 11.3-RELEASE | 192.168.1.94/24 |
+-----+------------+-------+--------------+-----------------+
| 13 | git | up | 11.3-RELEASE | DHCP |
+-----+------------+-------+--------------+-----------------+
| 2 | influxdb | up | 11.3-RELEASE | 192.168.1.80 |
+-----+------------+-------+--------------+-----------------+
| 3 | logging | up | 11.2-RELEASE | 192.168.1.16 |
+-----+------------+-------+--------------+-----------------+
| 9 | mqtt | up | 11.2-RELEASE | 192.168.1.99 |
+-----+------------+-------+--------------+-----------------+
| 5 | mysql | up | 11.3-RELEASE | 192.168.1.55 |
+-----+------------+-------+--------------+-----------------+
| 6 | netbox | up | 11.3-RELEASE | 192.168.1.82 |
+-----+------------+-------+--------------+-----------------+
| 7 | postgres | up | 11.3-RELEASE | 192.168.1.40 |
+-----+------------+-------+--------------+-----------------+
| 8 | zoneminder | up | 11.2-RELEASE | DHCP |
+-----+------------+-------+--------------+-----------------+
18.13 real 3.07 user 15.24 sys
b0210520-fs01%


The GUI is around the same time lag to display.
 

Snow

Patron
Joined
Aug 1, 2014
Messages
309
iocage is fast for me, but the Jails in the GUI take about 30-45secs.
 

pkropf

Dabbler
Joined
Jul 15, 2019
Messages
14
Looks like this may be related to the number of snapshots of the iocage dataset:

root@b0210520-fs01[~]# time zfs get -H -o name,property,value -r all space/iocage > /dev/null
zfs get -H -o name,property,value -r all space/iocage > /dev/null 2.54s user 43.88s system 95% cpu 48.354 total
root@b0210520-fs01[~]#
 

pkropf

Dabbler
Joined
Jul 15, 2019
Messages
14
(Forgot this part)

root@b0210520-fs01[~]# zfs list -t snapshot | grep iocage | wc
10143 50715 1085301
root@b0210520-fs01[~]#
 

pkropf

Dabbler
Joined
Jul 15, 2019
Messages
14
I've pruned down to 40 snapshots on this server and response time from the CLI has dropped. However,, it's still in the 20 - 30 second range from the GUI.


root@b0210520-fs01[~]# time iocage list
+-----+------------+-------+--------------+-----------------+
| JID | NAME | STATE | RELEASE | IP4 |
+=====+============+=======+==============+=================+
| 1 | bacula | up | 11.3-RELEASE | 192.168.1.94/24 |
+-----+------------+-------+--------------+-----------------+
| 2 | git | up | 11.3-RELEASE | DHCP |
+-----+------------+-------+--------------+-----------------+
| 3 | influxdb | up | 11.3-RELEASE | 192.168.1.80 |
+-----+------------+-------+--------------+-----------------+
| 4 | logging | up | 11.2-RELEASE | 192.168.1.16 |
+-----+------------+-------+--------------+-----------------+
| 5 | mqtt | up | 11.2-RELEASE | 192.168.1.99 |
+-----+------------+-------+--------------+-----------------+
| 8 | mysql | up | 11.3-RELEASE | 192.168.1.55 |
+-----+------------+-------+--------------+-----------------+
| 6 | netbox | up | 11.3-RELEASE | 192.168.1.82 |
+-----+------------+-------+--------------+-----------------+
| 9 | postgres | up | 11.3-RELEASE | 192.168.1.40 |
+-----+------------+-------+--------------+-----------------+
| 7 | zoneminder | up | 11.2-RELEASE | DHCP |
+-----+------------+-------+--------------+-----------------+
iocage list 1.59s user 2.09s system 94% cpu 3.913 total
root@b0210520-fs01[~]# zfs list -t snapshot | grep iocage | wc
40 200 4120
root@b0210520-fs01[~]#
 

ornias

Wizard
Joined
Mar 6, 2020
Messages
1,458
Looks like this may be related to the number of snapshots of the iocage dataset:

This has already been known for about a month now (had a chat with iocage folks over on github about it and scolded them about not doing enough research on it, because it slipped past them and thats how it got into FreeNAS 11.3 in the first place.

Most of it has been fixed in 11.3U1.
It's still a tad slow at times, but the minute+ long loadtimes from 11.3 are gone :)
 

pkropf

Dabbler
Joined
Jul 15, 2019
Messages
14
Ok but I'm running 11.3-U1 and still see the dramatic slowdowns if I have a large number of snapshots on my system.
 
Top