Adding/removing an iSCSI extent is very slow

[gregb]

It takes around 5 minutes (I haven't timed it) to get the add dialog up and another 5 minutes to save the config. Other operations seem responsive. I am using the new CTL based target (v9.1.2.7). While the operation is hung, I can see a zfs list operation running as a child of python (is that related?):

|-- /usr/local/bin/python /usr/local/www/freenasUI/manage.py runfcgi method=threaded host=127.0.0.1 port=9042 pidfile=/var/run/django.pid (python2.7)
| `-- /sbin/zfs list -t volume -o name -H​

The process is stuck state 'D' (Marks a process in disk (or other short term, uninterruptible) wait.). What I missed is that the PID keeps changing. It isn't that the 'zfs list' takes a long time, but that the command is executed many times.

If I run the 'zfs list' command it completes successfully with a list of 110 entries in 3-5 seconds.

What can I do to try to track down this issue? I want to add 50+ extents. The 'ctld' is running. No errors are reported in '/var/log/*'. DNS is resolving ok.

Thanks,

Greg.

​

 

[cyberjock]

Can you provide the debug file for your server?

 

[gregb]

Yes I would be more than happy to, but what is a debug file?

[Edit] I found this http://doc.freenas.org/index.php/Freenas-debug and can run it from the command line.

 

[cyberjock]

Actually debug is a button in the WebGUI... probably easier to do that since it will tar.gz the files for you to make it easy to handle. ;)

 

[gregb]

I've cloned FreeNAS and opened the code in PyCharm. Find usages of 'zfs_snapshot_list' brings this code up (below). My suspicion is that there is a nested loop and because I have more than a handful of zfs volumes (74) and an n^2 [O(2)] issue is causing the delay.

'gui/services/forms.py':1004

Code:

        _notifier = notifier()
        for volume in Volume.objects.filter(vol_fstype__exact='ZFS'):
            zvols = _notifier.list_zfs_vols(volume.vol_name)
            for zvol, attrs in zvols.items():
                if "zvol/" + zvol not in used_zvol:
                    diskchoices["zvol/" + zvol] = "%s (%s)" % (
                        zvol,
                        humanize_size(attrs['volsize']))
                zsnapshots = _notifier.zfs_snapshot_list(path=zvol).values()
                if not zsnapshots:
                    continue
                for snap in zsnapshots[0]:
                    diskchoices["zvol/" + snap.fullname] = "%s (%s)" % (
                        snap.fullname,
                        humanize_size(attrs['volsize']))

Is it worth raising an issue of this one?