Receiving replication stream fails if any snapshot exceeds quota

F

Fab Sidoli

Contributor
Joined
May 15, 2019
Messages
114
Dear All,

I'm hoping someone can explain to me why the following error occurs (see below) when my nightly replication task runs (user name redacted). I naively assumed replication would blindly plough on through the datasets. Not doing so leaves me vulnerable to not being able to replicate user data over if a user exceeds their quota.

I'm hoping I'm doing something completely stupid with my setup.

Thanks,
Fab

"cannot receive resume stream: destination store/home/UserName space quota exceeded
warning: cannot send 'store/home/UserName@auto-2021-01-09_21-00': signal received."

In /var/log/zettarepl.log I see

"
[2021/01/11 22:56:13] DEBUG [replication_task__task_5] [zettarepl.transport.base_ssh.root@10.0.102.2.shell.93.async_exec.354471] Running ['zfs', 'get', '-H', '-p', 'receive_resume_token', 'store/home/UserName'
[2021/01/11 22:56:13] DEBUG [replication_task__task_5] [zettarepl.paramiko.replication_task__task_5] [chan 8178] Max packet in: 32768 bytes
[2021/01/11 22:56:13] DEBUG [Thread-228] [zettarepl.paramiko.replication_task__task_5] [chan 8178] Max packet out: 32768 bytes
[2021/01/11 22:56:13] DEBUG [Thread-228] [zettarepl.paramiko.replication_task__task_5] Secsh channel 8178 opened.
[2021/01/11 22:56:13] DEBUG [Thread-228] [zettarepl.paramiko.replication_task__task_5] [chan 8178] Sesch channel 8178 request ok
[2021/01/11 22:56:13] DEBUG [replication_task__task_5] [zettarepl.transport.base_ssh.root@10.0.102.2.shell.93.async_exec.354471] Reading stdout
[2021/01/11 22:56:13] DEBUG [Thread-228] [zettarepl.paramiko.replication_task__task_5] [chan 8178] EOF received (8178)
[2021/01/11 22:56:13] DEBUG [replication_task__task_5] [zettarepl.transport.base_ssh.root@10.0.102.2.shell.93.async_exec.354471] Waiting for exit status
[2021/01/11 22:56:13] DEBUG [Thread-228] [zettarepl.paramiko.replication_task__task_5] [chan 8178] EOF sent (8178)
[2021/01/11 22:56:13] DEBUG [replication_task__task_5] [zettarepl.transport.base_ssh.root@10.0.102.2.shell.93.async_exec.354471] Success: 'store/home/UserName\treceive_resume_....1516a71717e36c28d009c612a57\t-\n'
[2021/01/11 22:56:13] INFO [replication_task__task_5] [zettarepl.replication.run] Resuming replication for destination dataset 'store/home/UserName'
[2021/01/11 22:56:13] INFO [replication_task__task_5] [zettarepl.replication.run] For replication task 'task_5': doing push from 'store/home/UserName' to 'store/home/UserName' of snapshot=None incremental_base=None receive_resume_token='1-12cc5c461d-118-789c636064000....6e41516a71717e36c28d009c612a57'
[2021/01/11 22:56:13] DEBUG [replication_task__task_5] [zettarepl.transport.base_ssh.root@10.0.102.2.shell.93.async_exec.354472] Running ['zfs', 'umount', 'store/home/UserName']
[2021/01/11 22:56:13] DEBUG [replication_task__task_5] [zettarepl.paramiko.replication_task__task_5] [chan 8179] Max packet in: 32768 bytes
[2021/01/11 22:56:13] DEBUG [Thread-228] [zettarepl.paramiko.replication_task__task_5] [chan 8179] Max packet out: 32768 bytes
[2021/01/11 22:56:13] DEBUG [Thread-228] [zettarepl.paramiko.replication_task__task_5] Secsh channel 8179 opened.
[2021/01/11 22:56:13] DEBUG [Thread-228] [zettarepl.paramiko.replication_task__task_5] [chan 8179] Sesch channel 8179 request ok
[2021/01/11 22:56:13] DEBUG [replication_task__task_5] [zettarepl.transport.base_ssh.root@10.0.102.2.shell.93.async_exec.354472] Reading stdout
[2021/01/11 22:56:13] DEBUG [Thread-228] [zettarepl.paramiko.replication_task__task_5] [chan 8179] EOF received (8179)
[2021/01/11 22:56:13] DEBUG [replication_task__task_5] [zettarepl.transport.base_ssh.root@10.0.102.2.shell.93.async_exec.354472] Waiting for exit status
[2021/01/11 22:56:13] DEBUG [Thread-228] [zettarepl.paramiko.replication_task__task_5] [chan 8179] EOF sent (8179)
[2021/01/11 22:56:13] DEBUG [replication_task__task_5] [zettarepl.transport.base_ssh.root@10.0.102.2.shell.93.async_exec.354472] Error 1: "cannot unmount 'store/home/UserName': not currently mounted\n"
[2021/01/11 22:56:13] DEBUG [replication_task__task_5] [zettarepl.transport.local.shell.1.async_exec.354473] Running ['zfs', 'send', '-V']
[2021/01/11 22:56:13] DEBUG [replication_task__task_5] [zettarepl.transport.local.shell.1.async_exec.354473] Error 2: 'missing snapshot argument\nusag.... list, run: zfs allow|unallow\n'
[2021/01/11 22:56:13] DEBUG [replication_task__task_5] [zettarepl.transport.local.shell.1.async_exec.354475] Running ['sh', '-c', 'exec 3>&1; eval $(exec 4>&1 >&....] && exit $pipestatus1; exit 0']
[2021/01/11 22:56:14] DEBUG [replication_task__task_5.async_exec_tee.wait] [zettarepl.transport.local.shell.1.async_exec.354475] Error 255: None
[2021/01/11 22:56:14] DEBUG [replication_task__task_5.process] [zettarepl.transport.local.shell.1.async_exec.354474] Error 255: "cannot receive resume stream: ....01-09_21-00': signal received\n"
[2021/01/11 22:56:14] DEBUG [replication_task__task_5.monitor] [zettarepl.transport.local.shell.1.async_exec.354475] Stopping
[2021/01/11 22:56:14] WARNING [replication_task__task_5] [zettarepl.replication.run] For task 'task_5' at attempt 5 recoverable replication error RecoverableReplicationError("cannot receive resume stream: destination store/home/UserName space quota exceeded\nwarning: cannot send 'store/home/UserName@auto-2021-01-09_21-00': signal received")
[2021/01/11 22:56:14] ERROR [replication_task__task_5] [zettarepl.replication.run] Failed replication task 'task_5' after 5 retries
[2021/01/11 22:56:14] DEBUG [Thread-228] [zettarepl.paramiko.replication_task__task_5] EOF in transport thread
[2021/01/11 22:56:13] DEBUG [replication_task__task_5] [zettarepl.transport.base_ssh.root@10.0.102.2.shell.93.async_exec.354471] Running ['zfs', 'get', '-H', '-p', 'receive_resume_token', 'store/home/UserName']
[2021/01/11 22:56:13] DEBUG [replication_task__task_5] [zettarepl.paramiko.replication_task__task_5] [chan 8178] Max packet in: 32768 bytes
[2021/01/11 22:56:13] DEBUG [Thread-228] [zettarepl.paramiko.replication_task__task_5] [chan 8178] Max packet out: 32768 bytes
[2021/01/11 22:56:13] DEBUG [Thread-228] [zettarepl.paramiko.replication_task__task_5] Secsh channel 8178 opened.
[2021/01/11 22:56:13] DEBUG [Thread-228] [zettarepl.paramiko.replication_task__task_5] [chan 8178] Sesch channel 8178 request ok
[2021/01/11 22:56:13] DEBUG [replication_task__task_5] [zettarepl.transport.base_ssh.root@10.0.102.2.shell.93.async_exec.354471] Reading stdout
[2021/01/11 22:56:13] DEBUG [Thread-228] [zettarepl.paramiko.replication_task__task_5] [chan 8178] EOF received (8178)
[2021/01/11 22:56:13] DEBUG [replication_task__task_5] [zettarepl.transport.base_ssh.root@10.0.102.2.shell.93.async_exec.354471] Waiting for exit status
[2021/01/11 22:56:13] DEBUG [Thread-228] [zettarepl.paramiko.replication_task__task_5] [chan 8178] EOF sent (8178)
[2021/01/11 22:56:13] DEBUG [replication_task__task_5] [zettarepl.transport.base_ssh.root@10.0.102.2.shell.93.async_exec.354471] Success: 'store/home/UserName\treceive_resume_....1516a71717e36c28d009c612a57\t-\n'
[2021/01/11 22:56:13] INFO [replication_task__task_5] [zettarepl.replication.run] Resuming replication for destination dataset 'store/home/UserName'
[2021/01/11 22:56:13] INFO [replication_task__task_5] [zettarepl.replication.run] For replication task 'task_5': doing push from 'store/home/UserName' to 'store/home/UserName' of snapshot=None incremental_base=None receive_resume_token='1-12cc5c461d-118-789c636064000....6e41516a71717e36c28d009c612a57'
[2021/01/11 22:56:13] DEBUG [replication_task__task_5] [zettarepl.transport.base_ssh.root@10.0.102.2.shell.93.async_exec.354472] Running ['zfs', 'umount', 'store/home/UserName']
[2021/01/11 22:56:13] DEBUG [replication_task__task_5] [zettarepl.paramiko.replication_task__task_5] [chan 8179] Max packet in: 32768 bytes
[2021/01/11 22:56:13] DEBUG [Thread-228] [zettarepl.paramiko.replication_task__task_5] [chan 8179] Max packet out: 32768 bytes
[2021/01/11 22:56:13] DEBUG [Thread-228] [zettarepl.paramiko.replication_task__task_5] Secsh channel 8179 opened.
[2021/01/11 22:56:13] DEBUG [Thread-228] [zettarepl.paramiko.replication_task__task_5] [chan 8179] Sesch channel 8179 request ok
[2021/01/11 22:56:13] DEBUG [replication_task__task_5] [zettarepl.transport.base_ssh.root@10.0.102.2.shell.93.async_exec.354472] Reading stdout
[2021/01/11 22:56:13] DEBUG [Thread-228] [zettarepl.paramiko.replication_task__task_5] [chan 8179] EOF received (8179)
[2021/01/11 22:56:13] DEBUG [replication_task__task_5] [zettarepl.transport.base_ssh.root@10.0.102.2.shell.93.async_exec.354472] Waiting for exit status
[2021/01/11 22:56:13] DEBUG [Thread-228] [zettarepl.paramiko.replication_task__task_5] [chan 8179] EOF sent (8179)
[2021/01/11 22:56:13] DEBUG [replication_task__task_5] [zettarepl.transport.base_ssh.root@10.0.102.2.shell.93.async_exec.354472] Error 1: "cannot unmount 'store/home/UserName': not currently mounted\n"
[2021/01/11 22:56:13] DEBUG [replication_task__task_5] [zettarepl.transport.local.shell.1.async_exec.354473] Running ['zfs', 'send', '-V']
[2021/01/11 22:56:13] DEBUG [replication_task__task_5] [zettarepl.transport.local.shell.1.async_exec.354473] Error 2: 'missing snapshot argument\nusag.... list, run: zfs allow|unallow\n'
[2021/01/11 22:56:13] DEBUG [replication_task__task_5] [zettarepl.transport.local.shell.1.async_exec.354475] Running ['sh', '-c', 'exec 3>&1; eval $(exec 4>&1 >&....] && exit $pipestatus1; exit 0']
[2021/01/11 22:56:14] DEBUG [replication_task__task_5.async_exec_tee.wait] [zettarepl.transport.local.shell.1.async_exec.354475] Error 255: None
[2021/01/11 22:56:14] DEBUG [replication_task__task_5.process] [zettarepl.transport.local.shell.1.async_exec.354474] Error 255: "cannot receive resume stream: ....01-09_21-00': signal received\n"
[2021/01/11 22:56:14] DEBUG [replication_task__task_5.monitor] [zettarepl.transport.local.shell.1.async_exec.354475] Stopping
[2021/01/11 22:56:14] WARNING [replication_task__task_5] [zettarepl.replication.run] For task 'task_5' at attempt 5 recoverable replication error RecoverableReplicationError("cannot receive resume stream: destination store/home/UserName space quota exceeded\nwarning: cannot send 'store/home/UserName@auto-2021-01-09_21-00': signal received")
[2021/01/11 22:56:14] ERROR [replication_task__task_5] [zettarepl.replication.run] Failed replication task 'task_5' after 5 retries
[2021/01/11 22:56:14] DEBUG [Thread-228] [zettarepl.paramiko.replication_task__task_5] EOF in transport thread"
 
You must log in or register to reply here.

Similar threads

F
ZFS dataset missing after replication task failed to unmount share
Replies
1
Views
1K
curtishall
curtishall
K
Temporary replication failure
Replies
1
Views
1K
Kevin_Lemaire
K
O
Problems with remote replication
Replies
2
Views
628
JulianSimon
J
R
SSH Push Replication Fails against SSHD 8.8
Replies
1
Views
1K
Richard Durso
R
L
Replication error...but no error in log?
Replies
12
Views
1K
leonardorame
L
Top