ZFS replication without using Root user

[potope]

Is it possible to use replication with a standard user?

The documentation hints that is possible but I failed to achieve the desired result.

Go to PULL and click Account → Users → View Users. Click the Modify User button for the user account you will be using for replication (by default this is the root user).

(http://doc.freenas.org/index.php/Replication_Tasks)

I followed the instructions but replication fails and I cannot even succeed with the SSH test from the local to the remote system (documentation refers to the local as push ).

I managed to pass the ssh test only changing the permissions on /data/ssh/replication because the standard user can't even read the private key, it makes sense but I'm just trying to understand what I'm doing wrong, but then FreeNas complains about the permissions on /data/ssh/replication


The last thing I tried was a different certificate but while I can connect through ssh to the remote system without being prompted for a password I am still unable to replicate to the remote system.


ssh with certificate fails

[replicone@ClientNAS /]$ ssh -v -i /data/ssh/replication 192.168.12.90
OpenSSH_6.2p2, OpenSSL 0.9.8y 5 Feb 2013
debug1: Reading configuration data /etc/ssh/ssh_config
debug1: Connecting to 192.168.12.90 [192.168.12.90] port 22.
debug1: Connection established.
debug1: identity file /data/ssh/replication type 1
debug1: identity file /data/ssh/replication-cert type -1
debug1: Enabling compatibility mode for protocol 2.0
debug1: Local version string SSH-2.0-OpenSSH_6.2_hpn13v11 FreeBSD-20130515
debug1: Remote protocol version 2.0, remote software version OpenSSH_6.2_hpn13v11 FreeBSD-20130515
debug1: match: OpenSSH_6.2_hpn13v11 FreeBSD-20130515 pat OpenSSH*
debug1: SSH2_MSG_KEXINIT sent
debug1: SSH2_MSG_KEXINIT received
debug1: kex: server->client aes128-ctr hmac-md5-etm@openssh.com none
debug1: kex: client->server aes128-ctr hmac-md5-etm@openssh.com none
debug1: sending SSH2_MSG_KEX_ECDH_INIT
debug1: expecting SSH2_MSG_KEX_ECDH_REPLY
debug1: Server host key: ECDSA ea:39:d4:40:63:88:04:01:d2:f1:df:7d:3f:de:67:ad
debug1: Host '192.168.12.90' is known and matches the ECDSA host key.
debug1: Found key in /etc/ssh/ssh_known_hosts:1
debug1: ssh_ecdsa_verify: signature correct
debug1: SSH2_MSG_NEWKEYS sent
debug1: expecting SSH2_MSG_NEWKEYS
debug1: SSH2_MSG_NEWKEYS received
debug1: Roaming not allowed by server
debug1: SSH2_MSG_SERVICE_REQUEST sent
debug1: SSH2_MSG_SERVICE_ACCEPT received
debug1: Authentications that can continue: publickey,password
debug1: Next authentication method: publickey
debug1: Offering RSA public key: /data/ssh/replication
debug1: Server accepts key: pkalg ssh-rsa blen 279
*debug1: could not open key file '/data/ssh/replication': Permission denied
debug1: Next authentication method: password
replicone@192.168.12.90's password:

check on file permission

[replicone@ClientNAS /]$ ls -lg /data/ssh/replication
-rw------- 1 root wheel 1679 Jun 1 22:44 /data/ssh/replication
[replicone@ClientNAS /]$ cat /data/ssh/replication
cat: /data/ssh/replication: Permission denied

change permission

[replicone@ClientNAS ~]$ sudo chmod g+r /data/ssh/replication
Password:
[replicone@ClientNAS ~]$ ls -lg /data/ssh/replication
-rw-r----- 1 root wheel 1679 Jun 1 22:44 /data/ssh/replication
[replicone@ClientNAS ~]$ cat /data/ssh/replication
-----BEGIN RSA PRIVATE KEY-----
bla bla bla

ssh again (success)

[replicone@ClientNAS ~]$ ssh -v -i /data/ssh/replication 192.168.12.90
OpenSSH_6.2p2, OpenSSL 0.9.8y 5 Feb 2013
debug1: Reading configuration data /etc/ssh/ssh_config
debug1: Connecting to 192.168.12.90 [192.168.12.90] port 22.
debug1: Connection established.
debug1: identity file /data/ssh/replication type 1
debug1: identity file /data/ssh/replication-cert type -1
debug1: Enabling compatibility mode for protocol 2.0
debug1: Local version string SSH-2.0-OpenSSH_6.2_hpn13v11 FreeBSD-20130515
debug1: Remote protocol version 2.0, remote software version OpenSSH_6.2_hpn13v11 FreeBSD-20130515
debug1: match: OpenSSH_6.2_hpn13v11 FreeBSD-20130515 pat OpenSSH*
debug1: SSH2_MSG_KEXINIT sent
debug1: SSH2_MSG_KEXINIT received
debug1: kex: server->client aes128-ctr hmac-md5-etm@openssh.com none
debug1: kex: client->server aes128-ctr hmac-md5-etm@openssh.com none
debug1: sending SSH2_MSG_KEX_ECDH_INIT
debug1: expecting SSH2_MSG_KEX_ECDH_REPLY
debug1: Server host key: ECDSA ea:39:d4:40:63:88:04:01:d2:f1:df:7d:3f:de:67:ad
debug1: Host '192.168.12.90' is known and matches the ECDSA host key.
debug1: Found key in /etc/ssh/ssh_known_hosts:1
debug1: ssh_ecdsa_verify: signature correct
debug1: SSH2_MSG_NEWKEYS sent
debug1: expecting SSH2_MSG_NEWKEYS
debug1: SSH2_MSG_NEWKEYS received
debug1: Roaming not allowed by server
debug1: SSH2_MSG_SERVICE_REQUEST sent
debug1: SSH2_MSG_SERVICE_ACCEPT received
debug1: Authentications that can continue: publickey,password
debug1: Next authentication method: publickey
debug1: Offering RSA public key: /data/ssh/replication
debug1: Server accepts key: pkalg ssh-rsa blen 279
debug1: read PEM private key done: type RSA
*debug1: Authentication succeeded (publickey).
*Authenticated to 192.168.12.90 ([192.168.12.90]:22).
debug1: Final hpn_buffer_size = 2097152
debug1: HPN Disabled: 0, HPN Buffer Size: 2097152
debug1: channel 0: new [client-session]
debug1: Enabled Dynamic Window Scaling
debug1: Requesting no-more-sessions@openssh.com
debug1: Entering interactive session.
Last login: Sat Jun 21 09:29:16 2014 from 192.168.12.91
FreeBSD 9.2-RELEASE-p4 (FREENAS.amd64) #0 r262572+17a4d3d: Wed Apr 23 10:09:38 PDT 2014
FreeNAS (c) 2009-2014, The FreeNAS Development Team
All rights reserved.
FreeNAS is released under the modified BSD license.
For more information, documentation, help or support, go here:
http://freenas.org
Welcome to FreeNAS
[replicone@ServerNAS ~]$

but FreeNas complains

CRITICAL: Replication zuppa/samba -> 192.168.12.90 failed: @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ @ WARNING: UNPROTECTED PRIVATE KEY FILE! @ @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ Permissions 0640 for '/data/ssh/replication' are too open. It is required that your private key files are NOT accessible by others. This private key will be ignored. bad permissions: ignore key: /data/ssh/replication Permission denied (publickey,password).

restore permission

sudo chmod g-r /data/ssh/replication

Should I try a different certificate?

[replicone@ClientNAS ~]$ cd .ssh/
[replicone@ClientNAS ~/.ssh]$ ls
[replicone@ClientNAS ~/.ssh]$ cd ..
[replicone@ClientNAS ~]$ ssh-keygen -t rsa
Generating public/private rsa key pair.
Enter file in which to save the key (/mnt/zuppa/home/replicone/.ssh/id_rsa):
Enter passphrase (empty for no passphrase):
Enter same passphrase again:
Your identification has been saved in /mnt/zuppa/home/replicone/.ssh/id_rsa.
Your public key has been saved in /mnt/zuppa/home/replicone/.ssh/id_rsa.pub.
The key fingerprint is:
0a:06:b3:11:e4:25:72:82:59:4e:96:11:44:3b:4a:a3 replicone@ClientNAS.local
The key's randomart image is:
+--[ RSA 2048]----+
|+*&+. |
|oO.= |
| oO |
|o..* |
|E . o S |
| . . . |
| . |
| |
| |
+-----------------+
[replicone@ClientNAS ~]$ ls .ssh/
id_rsa id_rsa.pub
[replicone@ClientNAS ~]$ cat .ssh/id_rsa.pub
ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDD6MAtjFgPwAe2UsUf5Xe3r0RpVAmVsD+7l1YjUE0CjcNF8WUZkz1T0ZYLEXfOUI9nCiRI5KEVC7Fhd/X/kUFmoLa2aXYXXmMWSQhk78lZ67OWvOG29IF+e5YNiiglKntIGRPpyq8eUFkxLk/UMKuBGtfhsreuFjnnE8NzZDOSCxBNkDahOfY6y1QonSq/Uh/wE+r+L0V39YeDc/UPnVMHyMARAyJ5GaIxaPz801LwaYQiUn9MEN7pp2UpK8TUKIgsRL60YjrBs8UzZjlaeYIsqVHl25gMeXWKzdVky+ThEDVYUumZNZNcOhUsFW2nG/A3KSscEqRyl3KjqEbUg9bB replicone@ClientNAS.local
[replicone@ClientNAS ~]$ ssh -i ~/.ssh/id_rsa 192.168.12.90
Last login: Sat Jun 21 09:32:21 2014 from 192.168.12.91
FreeBSD 9.2-RELEASE-p4 (FREENAS.amd64) #0 r262572+17a4d3d: Wed Apr 23 10:09:38 PDT 2014
FreeNAS (c) 2009-2014, The FreeNAS Development Team
All rights reserved.
FreeNAS is released under the modified BSD license.
For more information, documentation, help or support, go here:
http://freenas.org
Welcome to FreeNAS
[replicone@ServerNAS ~]$

Still something wrong

CRITICAL: Replication zuppa/samba -> 192.168.12.90 failed: cannot receive new filesystem stream: permission denied

 

[titan_rw]

I'd be interested in how this is setup too.

Currently I'm only doing (root) replication between two local boxes, but am looking at acting as a replication target for a friend. I'd rather not need to have his freenas need root access to mine.

 

[erikiiofph7]

I have finally succeeded having replication as a non-root user. I had a lot of struggle before I got it to work, but it turned out that it isn't that complicated actually.

The thing is that on the transmitting system (called PUSH in the docs) it is the root user that is ssh:ing in to a normal user on the receiving system (called PULL). Therefore you don't need to change any permissions on the replication key.

This is how I did it step by step:

On the receiving system (PULL):

1. Create a normal user johndoe with a corresponding group.
2. Enter the replication key from PUSH in the user settings for johndoe on PULL (Account > Users > johndoe).
3. Create a dataset vol1/dsrep_johndoe where johndoe can have all his datasets.
4. Give the user johndoe permissions to create his datasets on PULL by running:
   zfs allow johndoe create,destroy,diff,mount,readonly,receive,release,send,userprop vol1/dsrep_johndoe
5. Check that the permissions was set by listing the permissions for the dataset:
   zfs allow vol1/dsrep_johndoe
6. In the web interface under System > Sysctls > Add sysctl:
   Variable: vfs.usermount
   Value: 1
   Enabled: yes

And on the transmitting system (PUSH):

1. Define a Periodic Snapshot task under Storage > Replication Tasks in the web interface with these settings:
   Enabled: yes
   Filesystem/Volume: my_vol/ds_test
   Remote ZFS filesys. name: vol1/dsrep_johndoe
   Recursively replicate and remove stale snapshots on remote side: no
   Initialize remote side for once: no
   Limit (kB/s): 0
   Begin: 00:00:00
   End: 23:59:00
   Remote hostname: <IP ADDRESS FOR PULL>
   Remote port: <SSH PORT ON PULL>
   Dedicated user enabled: yes
   Dedicated user: johndoe (i.e. the johndoe user on the PULL system)
   Enable high speed ciphers: no
   Remote hostkey: <fill by clicking the button SSH Key Scan>

Now it should work, it did for me.

Documentation on the zfs allow command: http://docs.oracle.com/cd/E23824_01/html/821-1448/gfkco.html

 

[Nomak]

Thanks a lot for your guide erikiiofph7.
I try to do this configuration but I did not found this step

In the web interface under System > Sysctls > Add sysctl:
Variable: vfs.usermount
Value: 1
Enabled: yes

In the FreeNAS 9.10.2 U1 I did not found the menu System>Sysctls
What is the purpose of this step?

Thanks in advance

 

[erikiiofph7]

Thanks a lot for your guide erikiiofph7.
I try to do this configuration but I did not found this step

In the web interface under System > Sysctls > Add sysctl:
Variable: vfs.usermount
Value: 1
Enabled: yes

In the FreeNAS 9.10.2 U1 I did not found the menu System>Sysctls
What is the purpose of this step?

Thanks in advance

They seem to have moved it to Tunables in the GUI, i.e. go to System > Tunables > Add Tunable. Then fill in this:
Variable: vfs.usermount
Value: 1
Type: Sysctl
Enabled: yes

You need it because to be allowed to create a snapshot as a user you need to be allowed to mount a filesystem.

 

[Nomak]

Thanks a lot

 

[Nomak]

Hi, I run this command
zfs send POOL1/BKUP@auto-20170214.0400-3d | ssh -i /data/ssh/replication HOST IP ADDRESS zfs receive POOL2/BKUP@auto-20170214.0400-3d
It works, but there is a way to see the progress of this operation?

Thabks in advance