ZFS Replication Fails on some datasets only

[esceng]

Moving datasets from FreeNas-11.3-U5 to TrueNAS-12.0-U6.

Forgot to mention! Both systems are on VM

Created 5 replication tasks and 4 of them successfully finished but the last one keep giving error.

Here is the log output

Code:

[2021/11/05 14:38:03] INFO     [Thread-90] [zettarepl.paramiko.replication_task__task_16] Connected (version 2.0, client OpenSSH_8.4-hpn14v15)
[2021/11/05 14:38:03] INFO     [Thread-90] [zettarepl.paramiko.replication_task__task_16] Authentication (publickey) successful!
[2021/11/05 14:38:03] INFO     [replication_task__task_16] [zettarepl.paramiko.replication_task__task_16.sftp] [chan 5] Opened sftp connection (server version 3)
[2021/11/05 14:38:03] INFO     [replication_task__task_16] [zettarepl.replication.run] For replication task 'task_16': doing push from 'Shares/ENG' to 'Aktif/ENG' of snapshot='auto-2021-11-05_08-30' incremental_base=None receive_resume_token=None
[2021/11/05 15:54:43] WARNING  [replication_task__task_16] [zettarepl.replication.run] For task 'task_16' at attempt 1 recoverable replication error RecoverableReplicationError("full send of Shares/ENG@auto-2021-11-05_08-30 estimated size is 544G\nfull send of Shares/ENG@auto-2021-11-05_08-30 estimated size is 544G\nwarning: cannot send 'Shares/ENG@auto-2021-11-05_08-30': Input/output error\ncannot receive new filesystem stream: checksum mismatch or incomplete stream.\nPartially received snapshot is saved.\nA resuming stream can be generated on the sending system by running:\n    zfs send -t 1-f345a81e0-f0-789c636064000310a500c4ec50360710e72765a526973030784f9205abc1904f4b2b4e2d618003903c1b927c5265496a3190dea116d85887457f497e7a69660a0383a1409661bade05630724794eb07c5e626e2a0343704662516ab1beab9fbb43626949beae918191a1aea1a1ae8169bc8185aeb101b2fddc0c08ff24e7e716003516e76723dc0800f7192028")
[2021/11/05 15:54:43] INFO     [replication_task__task_16] [zettarepl.replication.run] After recoverable error sleeping for 1 seconds
[2021/11/05 15:54:45] INFO     [replication_task__task_16] [zettarepl.replication.run] Resuming replication for destination dataset 'Aktif/ENG'
[2021/11/05 15:54:45] INFO     [replication_task__task_16] [zettarepl.replication.run] For replication task 'task_16': doing push from 'Shares/ENG' to 'Aktif/ENG' of snapshot=None incremental_base=None receive_resume_token='1-f345a81e0-f0-789c63606400031....e716003516e76723dc0800f7192028'
[2021/11/05 15:54:45] ERROR    [replication_task__task_16] [zettarepl.replication.run] For task 'task_16' unhandled replication error ExecException(255, "warning: cannot send 'Shares/ENG@auto-2021-11-05_08-30': Input/output error\ncannot receive resume stream: checksum mismatch or incomplete stream.\nPartially received snapshot is saved.\nA resuming stream can be generated on the sending system by running:\n    zfs send -t 1-f345a81e0-f0-789c636064000310a500c4ec50360710e72765a526973030784f9205abc1904f4b2b4e2d618003903c1b927c5265496a3190dea116d85887457f497e7a69660a0383a1409661bade05630724794eb07c5e626e2a0343704662516ab1beab9fbb43626949beae918191a1aea1a1ae8169bc8185aeb101b2fddc0c08ff24e7e716003516e76723dc0800f7192028\n")
Traceback (most recent call last):
  File "/usr/local/lib/python3.7/site-packages/zettarepl/replication/run.py", line 143, in run_replication_tasks
    run_replication_task_part(replication_task, source_dataset, src_context, dst_context, observer)
  File "/usr/local/lib/python3.7/site-packages/zettarepl/replication/run.py", line 210, in run_replication_task_part
    resumed = resume_replications(step_templates, observer)
  File "/usr/local/lib/python3.7/site-packages/zettarepl/replication/run.py", line 315, in resume_replications
    observer_snapshot=resumed_snapshot)
  File "/usr/local/lib/python3.7/site-packages/zettarepl/replication/run.py", line 517, in run_replication_step
    ReplicationProcessRunner(process, monitor).run()
  File "/usr/local/lib/python3.7/site-packages/zettarepl/replication/process_runner.py", line 33, in run
    raise self.process_exception
  File "/usr/local/lib/python3.7/site-packages/zettarepl/replication/process_runner.py", line 37, in _wait_process
    self.replication_process.wait()
  File "/usr/local/lib/python3.7/site-packages/zettarepl/transport/ssh.py", line 136, in wait
    self.async_exec.wait()
  File "/usr/local/lib/python3.7/site-packages/zettarepl/transport/async_exec_tee.py", line 100, in wait
    raise ExecException(exit_event.returncode, self.output)
zettarepl.transport.interface.ExecException: warning: cannot send 'Shares/ENG@auto-2021-11-05_08-30': Input/output error
cannot receive resume stream: checksum mismatch or incomplete stream.
Partially received snapshot is saved.
A resuming stream can be generated on the sending system by running:
    zfs send -t 1-f345a81e0-f0-789c636064000310a500c4ec50360710e72765a526973030784f9205abc1904f4b2b4e2d618003903c1b927c5265496a3190dea116d85887457f497e7a69660a0383a1409661bade05630724794eb07c5e626e2a0343704662516ab1beab9fbb43626949beae918191a1aea1a1ae8169bc8185aeb101b2fddc0c08ff24e7e716003516e76723dc0800f7192028

What could be the problem?