replication from 11.2-RELEASE-U1 to 11.2-U2(.1) panics replication target

[styno]

TLDR: see title, am I the only one?
Long version: (this covers a month worth of troubleshooting/fiddling, I'll try to be brief)

As a good measure, I upgrade my backup system before the 'main' one, so a month ago, I did the upgrade to 11.2-U2.

The system was stable until I enabled the replication, this resulted in a panic and an unexpected reboot. (panic: dva_get_dsize_sync(): bad DVA 1937010277:3260866560) An upgrade to 11.2-U2.1 still showed the issue, I opened Bug #78411 (hidden atm due to debug logs). Support suggested a scrub; this finished fine, but replication still resulted in a panic.

At that point I took the plunge and updated the target system with newer hardware (incl. ecc memory) as I wasn't sure I could trust that aging hardware anymore (bought around 2006). It is worth to note that after replacing motherboard, ram & cpu, I also nuked the pool and rebuilt from scratch with a different layout. Replication once again resulted in a panic on the target system. (This step also includes replacing the TIM on the HBA's and adding an extra fan on them just to be on the safe side.)

Now I started to look at the source system and noticed a snapshot count over 50k (hourly, keep for 2 months). I also noticed that the panic happened when the exact same snapshot finished replication. Combining those two elements, all the snapshots till and including the offending one were deleted on the source system. Since that moment, replication no longer resulted in a panic & reboot but a full blown panic & halt. Worst, a manual reboot also results in a panic on zpool import.

Only one option remained: install 11.2-RELEASE-U1 on the target system import config, recreate zpool from scratch and replicate again. For the first time in a month I am looking a a successful backup of my data. I am relieved but not very eager to hit the upgrade button again. So my question: who is replicating from an older version to 11.2-U2+

system specs:
Source: Supermicro X10SL7-F | i3-4160 @ 3.60GHz | 32GB ECC | Boot: 2x SAMSUNG MMCRE64G5MPP | RAIDZ2: 8x WD30EZRX
Target: Supermicro X9SCL-F | Xeon E3-1220 V2 @ 3.10GHz | 16GB ECC | Boot: 2x MZ-5EA1000-0D3 | RAIDZ2: 4*5x HDS722020ALA330 RSD HUA | 2x X-flashed HP H220 | 4x Icy Box IB-565SSK

 

[Redcoat]

So my question: who is replicating from an older version to 11.2-U2+

I am. From FreeNAS-11.2-RELEASE-U1 to FreeNAS-11.2-U2.1 for three/four days now (took my time to upgrade my backup as a test...). No issues with U2.1 or replication so far.

 

[styno]

Thx for the confirmation! Did you upgrade or reinstall?
If I check on all the tests/steps I did, it seems I only missed a new install of 11.2-U2.1 and import the config. But I have little confidence that this would be causing the issue. I guess I'll wait a bit longer and see if someone else can chime in and/or something will happen with the open case.

 

[Redcoat]

Did you upgrade or reinstall?

Upgrade.