FN11, 10 & 40Gbe and dismal performance with replication

[Mlovelace]

So, out of curiosity I wanted to see how replication performance was with the 9.10.2-U5 version. It appears to be better than what I was getting with 11.0-U1. The transfer rate was about 3Gbit/sec and the CPU usage was a bit lower. I'll leave these filers on the freeNAS 9.10 version for a couple weeks and see how it looks over time. One bonus is I don't get the false "replication failed" errors with the older build of freeNAS.

 

[cyberjock]

@Mlovelace do you know why you are getting false "replication failed" errors? You shouldn't be getting any false replication errors, and that can be a problem for people that are using the new alert system in FN11...

 

[Mlovelace]

@Mlovelace do you know why you are getting false "replication failed" errors? You shouldn't be getting any false replication errors, and that can be a problem for people that are using the new alert system in FN11...

I summited a ticket and uploaded logs, but I haven't seen progress on the ticket in... 22 days now. The replication does in fact work, it reports as failed and emails as such, and a short time later changes to "up to date". I confirmed the files match on both the source and target systems.

https://bugs.freenas.org/issues/23837#change-132161

 

[cyberjock]

@Mlovelace I'm sending you a PM so we don't clutter this thread with your issue.

 

[HeloJunkie]

OK, so more testing and more weirdness.

So this is what my speed is running replication between my two servers:



[FreeNAS1]-------------10Gbe-------------[FreeNAS2]


No matter what I do, no matter which cards I use (I have tested Intel 10Gbe, Chelsio 10Gbe and Chelsio 40Gbe cards) this is the best I can get utilizing replication.


So then I decided, HEY - why not just use rsync locally (so no ssh). So I mounted an NFS share from the backup server to the primary server and ran this command:

rsync -aAX --numeric-ids --progress --no-whole-file /mnt/vol1/media/ /mnt/plexnas-ii/

and was NOT impressed:



I went from 900Mbit/s to about 600Mbits/s. YIKES.

[FreeNAS1]-------------10Gbe-------------[FreeNAS2]


OK, so what next....?

So I decided to try something else altogether. I have a media management server that is connected via Chelsio 10Gbe cards to both FreeNAS servers. On that server, I mount an NFS share from each FreeNAS server and ran the following command:

rsync -aAX --numeric-ids --progress --no-whole-file /mount/media/ /mount/plexnas-ii/


[FreeNAS1]--------10Gbe--------[Media Server]----------10Gbe---------[FreeNAS2]




Almost 2GBits/s!!


OK, so why in the world would running rsync running through a third machine be faster than running rsync on one of the actual FreeNAS servers? The CPU load on the FreeNAS servers while the rsync is running on them is not very high so I do not think that is it?






I am looking for ANY ideas to try to try and figure this issue out.