TRIM on NVMe ZIL makes pool creation very slow

[Lars Jensen]

Hi

I'm trying to create a pool with a Intel P3700 NVMe as ZIL (probably the same issue with any NVMe/SSD and TRIM).

Reading https://www.freebsd.org/doc/handbook/zfs-advanced.html this can take a long time because of TRIM will run on the P3700. And it does, so the "GUI Storage menu" hangs and I see a CLI command run. I assume when the CLI command has finished the GUI Storage menu will become responsive again.

I've added tunable vfs.zfs.vdev.trim_on_init=0 and the pool gets created instant. The long wait is gone!

There a mention on https://www.freebsd.org/doc/handbook/zfs-advanced.html that states using vfs.zfs.vdev.trim_on_init=0 can be added to make creation go faster, if device "has already been secure erased".

Is it safe to use vfs.zfs.vdev.trim_on_init=0 (and wipe disk), or should I wait for TRIM to finish ?

P.S. TRIM is enabled by default with vfs.zfs.trim.enabled= 1

 

[titan_rw]

If you're sure the SSD is erased prior to adding it to the pool, I see no benefit from a trim pass. I'd say it's safe to disable trim on init in this case.

 

[Lars Jensen]

Ok, but what happens if the SSD contains data? (Is full zero wipe data?)


Sent from my iPhone using Tapatalk

 

[Ericloewe]

Ok, but what happens if the SSD contains data? (Is full zero wipe data?)


Sent from my iPhone using Tapatalk

Sub-optimal performance, as the controller will still be juggling stuff until it realizes it's been deleted. The secure erase (or issuing TRIM to every LBA) tells the drive that everything's empty and there's no need to keep whatever had been stored.

 

[Lars Jensen]

Ok thanks, I decided to let the trim process finish on its own. Could be nice if the GUI would tell that this process is going on. I also noticed that zpool status command hangs on CLI when the proces is occuring.


Sent from my iPhone using Tapatalk

 

[cyberjock]

Well, there's no "good" way to identify that TRIM is actually running. A single command is sent to the device to perform the trim operation. You are at the mercy of the device to do as much or as little work is necessary. It is all internal to the drive and is somewhat impractical to get a status from the drive. trim itself doesn't include provisions to monitor the progress.

 

[Lars Jensen]

Ok I see the problem.

If there's no way of getting any information on TRIM process and zpool commands hang a workaround could be probing the path to the new pool (/mnt/MYPOOLNAME once the pool is created the path appears!), and more importantly log the process id of the "zpool create" command somewhere and continue to monitor existence of that process id.
This would require the GUI to stop using "hang" commands in "Storage menu" and first lookup if there's a zpool create process id running AND let the user know how long this process id have been running (maybe with features like "kill zpool create process id" etc.). At least the user knows that there's something going on. Maybe if "zpool history" doesn't hang (haven't checked) output could be parsed to show some info of zpool create processes.

 

[cyberjock]

No, you don't see the whole problem. There's far more at work than just what you think is at work...

For example, if you try to add an slog or l2arc to a zpool that is already storing the logs for the system, services will begin going catatonic and becoming unstable when their logging locations get locked up (and they *will* get locked).

So while it sounds great in theory.. no, it isn't that simple, and there is no simple fix without large redesigns of the system.

This is the result of something that should be handled by a professional that understands everything at work. It's not designed to be super-simple with lots of hooks all over the place to try to figure out if something is going on. The expectation is that the storage admin is aware of how to do all of this stuff themselves, is aware of behavior anomalies (such as trimming of nvme storage), etc.

Any kind of attempt to add in lots of hooks, even to say zpool is busy/not busy, adds complexity that can make problems even worse (imagine if zpool history is run every minute and the system is locked up for 2+ hours.. do you want 100+ threads waiting for I/O from the zpool?)

There's a point at which things are too complex to try to dumb down to something simple. This is one of those situation.

 

[Lars Jensen]

You are correct, I only see the TRIM problem. I was just trying to suggest a fairly simple workaround for the "create pool TRIM problem" and not for problems with all other components and behaviours of FreeNAS/ZFS. I wouldn't expect a large redesign or great complexity but only some adjustments in Django utilizing some commands that don't freeze the FreeNAS Storage menu for several hours.

Surely professionals that work with ZFS and design storage systems must know all the caveats, but that shouldn't disallow FreeNAS users to see what's going on. Also FreeNAS 10 seems to address a lot of the GUI challenges in the 9.x branch http://www.freenas.org/whats-new/2015/06/freenas-10-a-developers-perspective.html.
So maybe a better solution is coming with 10 which is made by professionals. And I believe they always try to make it more simple despite of the underlying complexity