Help with replicating to 2 targets, maybe with script?

[cyberjock]

Can you post a debug file of this 3rd FreeNAS you are replicating to? Curious as to if the system hasn't figured out it's standing over a rancor pit or if the devs did something to prevent this issue from fubaring the system. ;)

 

[yanir99]

Can you post a debug file of this 3rd FreeNAS you are replicating to? Curious as to if the system hasn't figured out it's standing over a rancor pit or if the devs did something to prevent this issue from fubaring the system. ;)

Here you go, I do hope it's ok...

 

[yanir99]

I've just noticed i missed something in my script - deleting old snapshots!!!
with this script running all the time I will just be sending more and more snapshots.... can someone help with the right way (command) to remove snapshots older then X days?
or simply using the -F on the receive side will do the trick?

 

[yanir99]

Well it seems that the -F tdoes the trick, so I updated my script.
Also so far 3rd machine is working OK inspite the .system replication...

@cyberjock: did you have a chance to look at the debug log?

 

[cyberjock]

Somehow I missed the debug file.

Looking it over I see horrifying things.

• Your zpools (both of them) are clearly running on a hardware RAID. Good luck with that!

• You have lots of snapshots that I'd expect to have expired quite some time ago.. (1w snapshot from March 20th, but the debug is from the 31st or 1st, so they weren't removed when they should have been).
• You have multiple snapshots in the same task and same dataset all labeled as "LATEST" and that's basically impossible.
• You have a crapload of errors in your logs that say "em0: Watchdog timeout -- resetting"
• You have 1/2 of the minimum RAM for FreeNAS (4GB versus 8GB) (this might explain why there are multiple snapshots labeled "LATEST").
• You're running FreeNAS in a VM.

I'd say it is not working "smoothly", and I wouldn't be surprised if you suddenly woke up one morning and found that the replication tasks are throwing errors that they weren't having when you went to bed.

I also won't be too surprised if this whole thing blows up in your face one morning without warning. You've literally taken every recommendation we give in the forum for how to do things properly, then done the exact opposite.

 

[yanir99]

Somehow I missed the debug file.

Looking it over I see horrifying things.

• Your zpools (both of them) are clearly running on a hardware RAID. Good luck with that!

• You have lots of snapshots that I'd expect to have expired quite some time ago.. (1w snapshot from March 20th, but the debug is from the 31st or 1st, so they weren't removed when they should have been).
• You have multiple snapshots in the same task and same dataset all labeled as "LATEST" and that's basically impossible.
• You have a crapload of errors in your logs that say "em0: Watchdog timeout -- resetting"
• You have 1/2 of the minimum RAM for FreeNAS (4GB versus 8GB) (this might explain why there are multiple snapshots labeled "LATEST").
• You're running FreeNAS in a VM.

I'd say it is not working "smoothly", and I wouldn't be surprised if you suddenly woke up one morning and found that the replication tasks are throwing errors that they weren't having when you went to bed.

I also won't be too surprised if this whole thing blows up in your face one morning without warning. You've literally taken every recommendation we give in the forum for how to do things properly, then done the exact opposite.

Hi @cyberjock thanks for going over the debug log, regarding your remarks, this is a DR Site machine, it's not production and never intend to be used for work, that's why it runs as VM (with only 4GB RAM) and the zpools are actually RDM from an HP P2000 ISCSI storage.
The log was before I added the -F to the command in order to destroy old snapshots (now it does), and the freenas:state flag is irrelevant since I'm not using the FreeNAS built in replication mechanism for this machine.

Regarding the errors on em0 I'll check to see what's going on there.

I guess this setup is not exactly by-the-book, but I couldn't find any solution for actually replicating to 2 destinations with the built-in mechanism.

 

[cyberjock]

Hi @cyberjock thanks for going over the debug log, regarding your remarks, this is a DR Site machine, it's not production and never intend to be used for work, that's why it runs as VM (with only 4GB RAM) and the zpools are actually RDM from an HP P2000 ISCSI storage.
The log was before I added the -F to the command in order to destroy old snapshots (now it does), and the freenas:state flag is irrelevant since I'm not using the FreeNAS built in replication mechanism for this machine.

Regarding the errors on em0 I'll check to see what's going on there.

I guess this setup is not exactly by-the-book, but I couldn't find any solution for actually replicating to 2 destinations with the built-in mechanism.

Yeah, I wasn't sure if you were doing it just for testing or not. I didn't tell you to take a hike or anything despite me crying a little because, ultimately, this was all about the learning experience for the 2 destinations. I'm not precisely sure how it is actually working. I tend to think it's just sheer chance its working. As it is, the replication and snapshot code is somewhat fragile and really doesn't take well if things get too far off the beaten path. There are definitely plans to revamp the whole system in 10.x.

I think it's important to note that this thread kind of broke into two different discussions:

1. Replicating from Server-A to Server-B AND Server-C/D/E/F.... I'm pretty sure you can do this, but I haven't tried to do it. (I think this is what you are doing yanir99, but I thought you were doing #2 for some reason).
2. Replicating from Server-A to Server-B, then replicating from Server-B to Server-C. This definitely does NOT work, at all. The replication system can only track from one source or one destination, not one source and one destintion. So if you get A->B to work, then B->C will be broken. And if you get B->C to work, then you break A->B. It is because the FreeNAS tags are meant to be for the source or destination, not source and destination.