Write speed of different datasets and write speed generally.

[calhta]

Hi All,

New to the forum and just looking for some answers. I have looked around for anything to explain this issue but I cannot seem to find anything that references the same situation.

I noticed yesterday that I am getting a really low write speed to one of my datasets. Anything between 8-35 MB/s through windows over a copper 10GbE link, my mac is largely similar over a 10GbE link.

If I create a new dataset or use an alternative existing dataset, certainly of the ones I've tried, I am getting an initial surge at 800-1100 MB/s and then settling down to around 190 - 300 MB/s - which I guess is drive speed. Noting that I sit more at an average of 550 MB/s on a replication to my backup server over its SFP+ Fiber link, I feel that should be more the speed I get over smb but maybe I am wrong.

I want to learn what I can do to get speed up generally but initially my concern is why these, two actually, datasets are performing so much slower than the rest. Obviously, all on the same disks, same pool etc.

They are larger datasets in fairness, the first one I tried is around 6.4TiB - perhaps this is expected behaviour but I cant explain it myself and I cant find any similar discussion. Apologies if I have missed anything.

Thanks

 

[Arwen]

Look at any differences in the ZFS dataset attributes:
zfs get all
It is possible you have extreme compression turned on the slow one, and higher speed compression, (or none), enabled for the faster one. Their are other attributes that contribute to speed, like the cache, both primary & secondary. And yes, those do affect writes indirectly because they would cache directory entries.

Further, you mention these datasets are in the same pool, yet you state that the first one is around 6.4TB. Amount of data in a dataset should not affect it's write speed. Unless you have a flatter directory structure, (too many files in a directory, like 1,000s to 10,000s).

 

[WI_Hedgehog]

Baffling Performance issues with large zfs pool

Just want to start by saying I have searched the forum on performance issues for the past few months and they all come back to failing drives. I have tried all the smart tests etc and all the drives seem healthy. So i feel like its time to reach out to see if anyone can help. The issue at...

www.truenas.com

 

[calhta]

Look at any differences in the ZFS dataset attributes:
zfs get all
It is possible you have extreme compression turned on the slow one, and higher speed compression, (or none), enabled for the faster one. Their are other attributes that contribute to speed, like the cache, both primary & secondary. And yes, those do affect writes indirectly because they would cache directory entries.

Further, you mention these datasets are in the same pool, yet you state that the first one is around 6.4TB. Amount of data in a dataset should not affect it's write speed. Unless you have a flatter directory structure, (too many files in a directory, like 1,000s to 10,000s).

Thanks, interestingly the opposite appears to be the case. The fast dataset has compression and the slow one has nothing. I have created a new dataset and am transferring the data back accross. Staying fast currently - or reasonably at least, I assume the only way I can increase above 200-300MB/s is with some mirrored vdevs? I see conflicting information about people saturating a 10GbE link, some suggest the can - I am only able to saturate for 5 seconds before the ram flushes to the array.

I figured size of a dataset shouldnt make any difference, There are a fair few folders compared to other datasets but its less than 100 at the top level, 7641 files recursivly within that dataset, thoughout a directory structure. Is that enough to cause any issue?

I have restored from my backup server very recently, since I upgraded my 3TB disks to 10TB, rather than spending many days individually resilvering. Perhaps that has caused fragmentation in some way? Not that it explains the also restored datasets that are still fast, just trying to share all detail that may be relevent.

 

[WI_Hedgehog]

As a note, @Arwen is correct, "extreme" compression (somewhere over 9, depending on the system's abilities) can cause massive delays.

@calhta is also correct, "reasonable" compression (2 to 5) with compressible data (documents, Excel spreadsheets, CAD files...) can speed up writes as there are fewer blocks to write+verify + parity+verify.

 

[Arwen]

Yes, reasonable compression can speed up both writes and reads, if the amount of compression and CPU speed is faster than normal disk access.