Good writeup! Nice to see someone that likes to chat about deep level stuff.
So here's what I know for certain:
1. You always have a ZIL. So even if you have no ZIL device, the pool has it's own ZIL. I'm very fuzzy on what those writes actually are. My guess is that writes fall into 2 categories:
Sync writes: sync writes must be honored so I'm guessing a tx would be opened and then closed and the data written to the pool. (this is a performance nightmware)
non-sync writes: My guess is that there is some degree of caching, but to what extend I'm not sure. But you have to open and close the tx/txg as well as write the data. From what I've seen in iostat is that when ZFS "breathes" it opens the tx/txg, writes the data, and closes the tx/txg fairly rapidly. Like a second or less for the whole thing. The exception are files that are being sent that take longer than a single "breath".
There is something here I'm not sure I understand. zpools can be created with and without log devices. (Could be a singular device but mirrored is far wiser in the context of FreeNAS.) Without log devices, the ZIL info committed to stable storage is written to blocks in the main pool. I'm trying to imagine doing a sync write that is 32K of data. So a ZIL record is created that includes the 32K of data. Because it is a sync write that record gets committed to stable storage. if there is no log devices, that 32K now resides in the main pool, then the actual write needs to occur and that same 32K is rewritten to the pool? Interesting. I'm assuming the committed ZIL info that includes the 32K of data will be slightly larger than 32K. Say for the sake of the argument it is 36K total. That still sets up the a scenario for fragmentation as that 36K area would be large enough to hold a future 32K data write leaving a 4K fragment which is good for not much.
So here's what I know for certain:
1. You always have a ZIL. So even if you have no ZIL device, the pool has it's own ZIL. I'm very fuzzy on what those writes actually are. My guess is that writes fall into 2 categories:
Sync writes: sync writes must be honored so I'm guessing a tx would be opened and then closed and the data written to the pool. (this is a performance nightmware)
non-sync writes: My guess is that there is some degree of caching, but to what extend I'm not sure. But you have to open and close the tx/txg as well as write the data. From what I've seen in iostat is that when ZFS "breathes" it opens the tx/txg, writes the data, and closes the tx/txg fairly rapidly. Like a second or less for the whole thing. The exception are files that are being sent that take longer than a single "breath".