Pool record size and SSDs

[Octopuss]

Does it matter what I set record size to with SSDs? Judging by the documentation I figured it made (or could) a difference with spinners, and since I mostly store backups and media on mine, I set the record size to 1MB (on that dataset at least). But do the same rules/logic apply to SSDs? I couldn't find anything related to this anywhere.

 

[winnielinnie]

SSD, HDD, use the recordsize policy that reflects the files and workload.

The main difference of performance based on the the "sector size" of the underlying block device is the ashift value, which defaults to 12 (4 KiB) in TrueNAS for newly created pools. This is a safe and sensible default.

If you're mainly using your SSD pool to archive files, media, and possibly seed, then a 1 MiB recordsize will do you well. You'll get the most out of inline compression, read throughput, and smaller metadata overhead.

If you're dealing with many small files, then you might benefit from a recordsize that reflects the modifications/writes of the files "in place".

If dealing with database software, set the recordsize accordingly, depending on which software.

Don't worry about "padding" or that you're concerned a 1 MiB recordsize will waste storage: ZFS's inline compression handles it seamlessly.

 

[Octopuss]

I guess I can keep the default 128K but change it on per-dataset basis.

Are there any other important pool settings that might be worth changing with SSDs during the pool creation? It's been two years since I did this....

 

[winnielinnie]

Are there any other important pool settings that might be worth changing with SSDs during the pool creation? It's been two years since I did this....

You can either enable "Auto TRIM" for the SSD pool or set a weekly task that trims the pool.

As long as the pool is exclusively using SSDs, and all the drives support "trim".

Personally, I disabled "Auto TRIM" (under "Pool Options"), and instead use the latter method.

I created a weekly trim job under Tasks -> Cron Jobs

Name: Trim SSD pool weekly
Command: zpool trim ssdpool
Run as user: root
Schedule: Weekly (you can choose a custom the day of the week and hour of the day)
Enabled: yes

(Replace "ssdpool" with the actual name of your pool.)

 

[Octopuss]

My TrueNAS is virtualized. Should I still enable the TRIM?

 

[winnielinnie]

My TrueNAS is virtualized.

I'm the wrong person to ask then.

I suppose if you allow passthrough to the underlying SSD drives, then issuing the trim commands should be safe. But I don't know. I must defer to whomever is using a virtualized instance of TrueNAS and can confirm this is safe.

 

[Octopuss]

I can always enable this later.
Meanwhile someone might step in with some additional tips for setting a SSD pool up.

 

[Octopuss]

So, any input about TRIM on virtualized system? Is this being taken care of by ESXi or not? I would guess it does, but that's just a wild guess.

 

[jgreco]

As long as you have used PCIe passthru (as you should) to pass through the storage controller to the TrueNAS VM, the answers to this question are the same as they would be for a bare metal setup. ESXi will not take care of it for you because it has nothing to do with your storage.

 

[Octopuss]

Good to know.

 

[Davvo]

You can either enable "Auto TRIM" for the SSD pool or set a weekly task that trims the pool.

Personally, I disabled "Auto TRIM" (under "Pool Options"), and instead use the latter method.

Why did you decide so?

 

[winnielinnie]

Why did you decide so?

I carried it over from my practices on desktop Linux, where one can choose to use the (1) discard mount option or (2) the weekly timer (implemented with SystemD's fstrim.timer) to issue trim(s) to all applicable filesystems once per week.

The former has negative performance implications, as each delete operation is followed with a discard. Theoretically this can cause additional wear on the SSD as it requires a higher voltage to reset the memory cells, and might possibly trigger it to do its garbage collection more often.

Whereas a weekly trim is only once per week (rather than multiple times an hour or day, depending on how often you delete files), and thus there is less noticeable performance impact for multiple delete operations and theoretically less wear on the SSD.

Most importantly, I've been happily using weekly trims, not only on my desktop Linux computers, but also on TrueNAS, without noticing any performance losses, whether with file deletions or new writes. On top of this, I have the peace of mind knowing that the SSD will do its internal garbage collection less often. (Hopefully?)

The above might not entirely be true, since garbage collection happens at the firmware level; so it's not like each time a trim/discard is issued that the SSD will immediately run its garbage collection to consolidate "unused" areas so that they can be reset with those "zippity zaps".


EDIT: To add more nuance, I believe there is a "discard=async" mount parameter for BTRFS under Linux (and possibly the other filesystems?), which can help with the previous performance penalty. ZFS may in fact use this method as well, so that if the "Auto TRIM" pool option is enabled, you won't notice the performance impacts.