SOLVED Question: Is this SMB write speed expected?

[spiceygas]

Brand new install.

• Pool
  • 1 vdev
    • 9x 16 TB Seagate Exos drives
    • RAIDZ2
• Gigabit network.
• 2x Xeon 2680v3
• 128 GB RAM

Using dd I get write speeds in-line with what I would naively expect. That's about 4.9 gigabit/second, if my math works right.

Code:

sync; dd if=/dev/zero of=tempfile bs=16M count=1024; sync

1024+0 records in
1024+0 records out
17179869184 bytes transferred in 26.679442 secs (643936607 bytes/sec)

For testing purposes, I tried SMB file moves with different sync values on the dataset. Transferring a large file (37 GB) in Windows and monitoring transfer speed via Windows task manager.

• Sync=Always: About 0.34 gigabit/second
• Sync=Standard: About 0.98 gigabit/second

I think that means that if I allow RAM to act as a buffer in received files then it can saturate my network connection, implying that writing directly to file synchronously is the bottleneck? But why? The dd test would imply that the array can handle 5x what the network can provide.

During the sync=always test, top shows that smbd is between 8-11%, so CPU is not a bottleneck. This is a brand new server, so no other process/VMs/whatever are running yet that might interfere with the test. This is literally the only transaction underway.

 

[Yorick]

You are hitting an IOPS wall. When you async write, ZFS will accumulate writes before flushing them to disk - the exact formula is a bit involved and was discussed on the forum recently, and I don't remember all of it, but naively "5 seconds or what the underlying disk can sustain, whichever is lower". Sync, every write needs to get into the ZIL as it is being written. It still gets flushed from there as always, but now those writes are in the ZIL on disk, not RAM. In raidz, your entire vdev has the IOPS of a single disk, or near-as for purposes of estimating performance.

This is why SLOG devices are a thing. They just need to be able to hold 5 seconds of writes, so they're usually small, have a ton of IOPS, and either have PLP or are so fast that separate PLP circuitry isn't needed (Optane).

More to the point: Why sync write over SMB? Say you write a file and the power goes out. With sync: The last 5 seconds of writes are in, but the file is still incomplete, so you have to copy it again. Async: The last 5 seconds of writes are lost, and the file is just as incomplete as with sync, so you have to write it again.

The one use case where sync over SMB makes sense to me is Mac time machine, that actually requires sync.

Usually sync is for block storage (VMs) and databases that rely on every write having been committed.

 

[spiceygas]

That helps a lot. Thank you.

I see similar effects with an 2x SSD-based mirror vdev. I don't have the numbers handy, but with sync it cannot saturate the gigabit network. (It's better than HDD, but still not saturated) Would that be due to a similar problem, just that the underlying media is faster so it's not quite as large of a penalty?

More to the point: Why sync write over SMB? Say you write a file and the power goes out. With sync: The last 5 seconds of writes are in, but the file is still incomplete, so you have to copy it again. Async: The last 5 seconds of writes are lost, and the file is just as incomplete as with sync, so you have to write it again.

You are right. I'm new to FreeNAS and didn't fully understand. I'll change it.

 

[Yorick]

Would that be due to a similar problem, just that the underlying media is faster so it's not quite as large of a penalty?

I'd expect so, but have no experience to back that up. You can find some sample IOPS numbers here: https://en.wikipedia.org/wiki/IOPS

 

[spiceygas]

Thank you. I appreciate you sharing your experience and expertise.

Marking the thread as solved.