FreeNAS Replication stuck repeating after it reaches ~100%

[Alex_K]

I messed some snapshots on source and destination, deleted some... replication stuck

It takes about a day and half to replicate 1.3TB of data over the connection it got

I've terminated replication on source with kill to make changes and restart ( I killed the "/usr/local/bin/ssh -i /data/ssh/replication...." process, or pipe compression process, sometime both - each time I killed one the other closed too )

After it nearly finishes (closest I've seen is 97% but I had to leave before I indeed seen 100), replication fails without explanation and starts anew

Edited replication task so as to replicate to different desto (h4-p0/copy-of-H3) (to preserve what old copy I had in h4-p0/copy-of-h3), it kept restarting
Deleted the old desto h4-p0/copy-of-h3 - it kept repeating
Deleted the old and new destos, reverted replication setting to h4-p0/copy-of-h3, I have not much hope it would complete replication though

I read multiple similar threads but cleaning up snapshots and restarting from scratch worked for them, or it was free space issue, or there were restart involved

I tried everything including cleaning up space, disabling ref/reservation, checked readonly is disabled etc.

Restarting FreeNAS is last resort as they host multiple VMs and not the least, I would lose trained ARC for a year and 3 month uptime.


Could anyone tell why this is happening and/or suggest a solution to make replication work again?

I'm considering zrep as one more last-resort option but would I be able to go on replicating by native/GUI means even if zrep replicates "basis bulk of data" snapshot for me once? Also as these are production systems I'm very hesitant to run anything non-GUI, non-native there at all.

 

[Alex_K]

I apologize if my English may be not very clear, non native speaker here

If anyone has any ideas where to look to make replication work again? Maybe I need to provide some more details? I would do what I can

It keeps restarting

Today it restarted early, so it probably were connection issue, but it did reach 80-90+ several times in two weeks now, but never completed.

 

[Alex_K]

Ok I solved it

TL;DR Connection interrupted during replication.

To fight it, seems it was time for me to dig a bit deeper and understand some of zfs send/recv parameters behind GUI replication, then mix up my own.

In hope it helps the next person, here are the few lines that saved my day there:

1. Source# zfs send -v source_zpool/snapshot_name | mbuffer -s 128k -m 1G | pigz -9c | mbuffer -s 128k -m 128M -q | ssh root@desto_ip "mbuffer -s 128k -m 512m -q | pigz -d | mbuffer -s 128k -m 512m -q | zfs recv -sveF desto_zpool/desto_dataset"

2. Desto# zfs get all| grep receive_resume_token

3. Source# zfs send -vt the_resume_token | mbuffer -s 128k -m 1G | pigz -9c | mbuffer -s 128k -m 128M -q | ssh -root@desto_ip "mbuffer -s 128k -m 512m -q | pigz -d | mbuffer -s 128k -m 512m -q | zfs recv -sveF desto_zpool/desto_dataset"

4. Source# zfs send -vi source_zpool/snapshot_name source_zpool/next_snapshot_name_of_same_dataset | mbuffer -s 128k -m 1G | pigz -9c | mbuffer -s 128k -m 128M -q | ssh root@desto_ip "mbuffer -s 128k -m 512m -q | pigz -d | mbuffer -s 128k -m 512m -q | zfs recv -sveF desto_zpool/desto_dataset"

Elaborating
1 = attempt to replicate the bulk of data. Snapshot may be any existent, not neccessary 1st or last one. But it would be "base" one as we won't be able to subsequently replicate snapshots on Desto. And if it goes away on source before you reach "4", you'd have to start anew.

2 = assuming replication interrupted for some reason. Here we use glorious zfs send/recv resume feature, which we have no access via GUI (at least not in FreeNAS versions I worked with) and retrieve The Token (it was created with zfs recv "-s" key). You should not normally see more then one token in the output, but if you do, use the one corresponding to the dataset you replication. You can clear these tokens and discard the partially replicated data with
zfs recv -A desto_zpool/desto_dataset
"-v" on recv is probably meaningless but I had it so I included it

3 = magic. use the Token to Resume

4 = not neccessary. After 3 is complete, you can go on replicating with GUI. But if you wish you can replicate some more snapshots of same data from console. And you may want to, because console gives you more power to:
a. Choose any compression application with any parameteres you find out to be best for your setup.
b. Use mbuffer. Its magic! Whether its HDDs or connection which is bad, you can smooth it out with some write buffer. You can calculate buffer size knowing channel width, and also use it for diagnostics: if your buffer goes to 0% that means the stream before it is the bottle neck - it was not able to provide data in time.
I used pigz -9 and mbuffers in such a way because it allowed me perfectly to saturate my intermittent 100Mbps conection at maximum compression level machine CPUs could sustain, thuis reaching maximum effective data throughput.

It copied no less then 20% faster then GUI replication with finest tuned parameters. That, not taking into account that GUI replication could not finish at all on that particular connection.

I found it quite safe, too, given you won't mess up the letter or digit somewhere. Replication were interrupted and resumed several times but I was able to run VM from replicated zvol without errors. I CRC testes several big archives inside that encrypted VM, there were no errors found.
I have _no_ base to be sure, however, because I still do not have enough understanding how zfs send/recv resume works and how it ensures data integrity, but being familiar with how "paranoid" level approach to data ZFS in general has, I think I can work with that. And everything used is strictly vanilla FreeNAS binaries.