Explanation on different read and write speeds of identical zpools sought

[saikee]

I have two zpools of identical size of 5x8TB in RAIDZ1. One is a source data of 15.23Tib (54% filled) and the other has been emptied to receive data as a backup. FreeNAS version is 11.2-U5 with the new web interface. Both pools are normal and not encrypted.

For fastest speed I had both pools placed inside my FreeNAS server and mounted both pools in /mnt/source and /mnt/target respectively. The files are mainly large videos. The file transfers involved only the internal Sata controllers and so no traffic in the network card.

For checking the progress of the copying action of a directory1 I used the command

Code:

cp -R mnt/source/directory1/  mnt/target/directory1/

For “each” of the 5 disks I got an “average” read and write speeds of 2.45Mb/s from the source pool and 26Mb/s on the target pool respectively. The average speeds were nearly the same in the 5 disks of each pool.

I was puzzled by FreeNAS’s apparent ability to read one data from a source pool but to write 10 times more (or faster) on a target pool.

Thinking I might be able to speed up the process a little bit I opened up a second web front to the same FreeNAS server and copied the next directory2 while directory1 was still being copied over.

Code:

cp -R mnt/source/directory2/  mnt/target/directory2/

To my surprise the overall average read and write speed changed to 6.8Mb/s and 2.5Mb/s when both directories were being copied over.

The 15.23Tib data took about 17 hours to copied over. The back up copy appears sound and healthy.

My question : Is this normal for FreeNAS to read one data and write 10 times more and then later read one data and write only 1/3 of it? Or the Version 11.2-U5 has a bug in the display?

Since both zpools are identical in geometry (hard disk size, type, number and RAID are the same) should we not expect the average reading speed to match the write speed?

 

[pschatz100]

Are you certain you have your math right? 15.24 Tib copied in 17 hours would be about .896 Tib per hour, which equals 985,033,064,178 bits per hour or about 273,620,295 bits per second (273 Mb/sec) which doesn't seem to be too bad.

 

[Heracles]

Hey Saikee,

There are many factors to consider and many of them you did not provide. One is RAM.

When you did your first copy, your RAM was empty. Thanks to that, everything got written in RAM, so was lightning fast.

For your second copy, the RAM was basically loaded with your first one. FreeNAS was working to move it to disks but was not done yet. For that reason, there was not much that ended up written in RAM and everything has to go straight to disks. That is why it was very slow compared to RAM.

Another reason is that the target pool was empty when receiving your first copy. For that, no matter where the drives were, they could start writting right away. The more and more data you have in your pool, the more and more often the disks must search for some place where more data can be written. Performance are very fast for the first 10% of your pool and decrease exponentially. At 50%, the impact is very real. So because you crossed that 50% mark, again that can explain why you finished slow.

These are but few factors. What are the complete specs of your box ? How big is Directory1 versus Directory2 ? How long between the two commands to copy ?

In all cases, Yes, it is normal for the speed to vary significantly and neutral, objective performance testing is extremely difficult to do.