Sorry to be so thick, I rather ask one time to many, than to assume wrong things. Also my statements are not really statments, more like guesses that are probably wrong so you can correct me.
"... Every question is a cry to understand the world. There is no such thing as a dumb question." - Carl Sagan
A fusion pool and a traditional pool loose a write in the worst case, but if it is sync, at least the client knows it lost a write.
Like if I have a Proxmox Hypervisor with NFS on TrueNAS and suddenly disconnect the LAN cable, there is just a ugly disconnect but the VM would probably survive it because it is just a normal crash. Not a disaster for a normal Windows VM. A database on the other hand would know that it could not finish the write and because of Atomicity and all, rollback. So basically it behaves like an unexpected shutdown, not a huge deal.
A network disconnect is slightly different from a sudden power failure of the storage, so let's use the latter as an example (your TrueNAS system freezes, or loses power due to an internal PSU/motherboard/HBA failure, etc)
In the "worst case", if you are writing async, the client
incorrectly believes that the writes were safe. This is very bad, especially for things like VMs or databases, because from the client's perspective, the writes were safely committed - when they go to read them back, either "on demand" from a DB, or "on boot" from the VM, it won't find the data it expects. This could be a "whoops, your file is missing" or "whoops, your whole VM is corrupt."
If you have sync writes enabled,
none of the above applies. Your client won't get the confirmation of "write safe" until it actually is safely on the ZIL - whether that resides on the pool, or on an SLOG. When your TrueNAS system powers back on, it sees there are pending uncommitted writes in the SLOG, and "completes the writes" to the main pool. (This is the only time an SLOG is read from.) Your VM will react as if it had crashed at the exact time your storage did. A client write or DB commit that was in progress may have been partially completed from the client's OS - but it will be "definitively incomplete" rather than "mistakenly complete" if that makes sense.
If I use SLOG and have a power outage or unexpected shutdown, without "in flight" PLP I risky loosing the whole pool, because some things have not been written to the disk, because the controller basically lied to TrueNAS having it all written to the SSD, while in reality it was only in SSD cache.
I'll talk about "in-flight PLP" in a moment - but the "controller lying" is one of the reasons why RAID controllers are discouraged - they will take a pending write into their cache, and then if that cache is lost for any reason (RAID card failed, power outage exceeds BBWC hold-up time, etc) then that write is also lost forever.
Another reason could be that even if the controller does not lie to TrueNAS, it will probably be slower, because unlike a PLP SSD, it can not cache data if it honest to TrueNAS.
Most post-2014 or so SSDs don't lie about their caching; you're exactly on the mark about a controller that's "honest" about its lack of in-flight PLP being significantly slower, devices that don't have it tend to not be suitably fast SLOGs.
So assuming that my HDD or SSD in my pool do not lie to TrueNAS about writes, there is no real risk of loosing my pool. Which would answer our original question why we don't need PLP on Fusion Pools. We don't because there is no risk of loosing a pool by not using PLP.
Correct. The pending writes to your special ("Fusion Pool") devices are still being protected through the same path as your regular writes. They're async, async, async - until ZFS finally decides to close the transaction, at which point it issues that small burst of sync-writes to the vdevs and says "commit this metadata/uberblock update, and I'm not proceeding until you do" - but similarly to how an SLOG without PLP is slow to do this, so are pool devices. That means that the best special/meta devices share similar characteristics to SLOGs; they don't have quite the emphasis on endurance, but having a low write latency is helpful - or at least, "low relative to your regular pool devices."
Which brings me to the next thing, if I have no SLOG, I basically have ZIL on my pool. If my pool is only HDD or SSD, can't they lie to TrueNAS?
They could, but see above regarding how that's not much of a thing anymore because of the negative repercussions.
Assuming they do lie to TrueNAS, it would be a security gain to add a PLP SLOG drive, because that way I will always know that writes that were promised to be on disk are actually on disk.
Unfortunately not. If your pool vdevs lie about the data being safe when it isn't, you'd open up the same vulnerability when the transaction commits to disk - ZFS would think it's safe, clear out the SLOG, and then the crash happens and it turns out the disks lied. Whoops.
The only real guarantee that ZFS requires for data safety is "power loss protection for data at rest" - devices shouldn't lie about a cache flush, and shouldn't corrupt existing good data when writing new data. See footnote [1] for an old paper from 2013 about the behavior of some SSDs under power fault.
I can take the risk of not mirroring SLOG, because it only gets read from after a power loss. The only scenario were I would matter is if the drive dies, and I have a power outage. Pretty unlikely.
Notably, the drive has to die
at the same time as a power outage. A dead SLOG will cause your sync-writes to revert to the on-pool ZIL. They'll still be safe, just
very slow. If you then have a power failure after the reversion completes, you're still safe, because upon power-up ZFS would replay the transaction from the on-pool ZIL.
I can not take the risk of not mirroring Fusion Pools, because I would loose my pool.
If by "Fusion Pools" you mean the special/meta vdevs - yes, they absolutely require redundancy. They are considered a root vdev, and their loss renders the pool unmountable. If you have mirrored vdevs, use mirrored special devices. If you have RAIDZ vdevs, use at least the same level of parity (RAIDZ1 = 2, RAIDZ3 = 3-way mirror)
and also strongly consider not using a special vdev as it won't be removable, even in a controlled manner, without destroying the pool!
Does that sound more or less right to you guys? Thank you for reading all of this!
You're welcome, and sorry about the post length. :)
[1]
https://www.usenix.org/system/files/conference/fast13/fast13-final80.pdf