Replication task not writing data?

[leonardorame]

Hi I configured two replication tasks on a FreeNas 11.2 server, one points to data/db (80mb) and the other to data/images (1000s of files, ~1Tb approx).

Both tasks are working properly, I can see on the remote server how Storage->Pools are continuously using more and more disk space. The pool's data/db status on the source server is Up to date, and the one for data/images is Sending data/images.

On the destination server, when I go to Storage->Pools I see ~80MB for data/db and ~8gb (and increasing) for data/images.

What makes me wonder if everything is ok, is when I go to the command line and do an ls -lah /mnt/data/db I can see the files stored there, but if I do ls -lah /mnt/data/images the directory exists, but is empty.

Also, if I do du -sh /mnt/data the used space is 80gb, so, it looks like the info I get from the UI on Storage->Pools for data/images is something stored elsewhere (in RAM maybe?). How can I see exactly where the data is being written?.

Leonardo.

 

[artlessknave]

zfs replication doesn't exactly "Write" data in the usual sense, since it is sending snapshots that are independent of the filesystem hierarchy; replication also often doesn't mount, or causes to unmount, the destination filesystem, meaning the data is there because the snapshots are there, but is not visible
you can clone the filesystem and access it through the clone, mount the individual filesystem, or just tell zfs to mount everything (below); this should let you see it with ls, at least until the next time it gets unmounted.

Code:

(sudo) zfs mount -a

 

[leonardorame]

Thanks @artlessknave, I first did a "zfs mount" to check what's mounted and noticed the directory /mnt/data/images wasn't there, so I did a zfs monunt -a, and now that directory is mounted, but empty...

The weird thing is FreeNas shows an ever increasing size on that pool.

 

[artlessknave]

is your replication finished? if it hasn't finished at least one snapshot there will be no data since...there is no data yet. if you have a 1TB snapshot it's gonna be awhile. otherwise, the next most likely thing I can think of is a misconfiguration in the replication, such as overwritting your snapshots.
if you post your replication screenshots for both datasets, as well as the below commands from both servers (in code tags), maybe we can spot something?

Code:

zfs list -t all (could be very long)
zfs mount
zpool list -v
zpool status -v

(if you wrap each output in spoiler/code it can make the post more readable)

Show : example

Code:

[_SPOILER="example"][_CODE]some output[/_CODE][/_SPOILER ]
(remove the _)

(note that the forum rules require including your hardware in either your post or signature, which can also be done with spoiler to make it short (like mine).)

 

[leonardorame]

No, the replication isn't finished, it looks like I'll have to be patient.

 

[leonardorame]

Today I noticed something very weird. The replication of the big dataset looks like having started again, the "used" size of the data/images dataset is now 10Mb while yesterday night it was ~50gb.

Is this possible?.

 

[artlessknave]

the next most likely thing I can think of is a misconfiguration in the replication, such as overwritting your snapshots.

The replication of the big dataset looks like having started again, the "used" size of the data/images dataset is now 10Mb while yesterday night it was ~50gb.

 

[leonardorame]

I checked again the Periodic Snapshot Tasks and Replication Tasks on source server and can confirm both snapshot and replication tasks (data/db and data/images) are configured exactly equal, in both cases the snapshot is kept for 1 week.

The only difference is the data/db dataset is small and finished the first replication. The data/images dataset never finished to replicate, but as it started yust a couple of days ago the snapshot isn't deleted.

 

[leonardorame]

Mm, yesterday 7pm I received this email from the target host:

Hello,
The replication failed for the local ZFS data/images while attempting to
send snapshot auto-20190518.1102-1w to xxx.yyy.zzz.fff

When a replication task fails it deletes de last snapshot it was sending and starts again?.

 

[artlessknave]

if you set data/images and data/db to both replicate to the same place example/destination, they will overwrite each other with every single replication and produce the exact thing you are reporting.
this is why I sugested posting a screenshot of your replications to check the config, but while you haven't done so, they way you worded it tells me this is probably what you did. (I know, cuz I did the exact same thing myself)
you need to do something like
data/images > remote/images
data/db > remote/db

configured exactly equal

 

[leonardorame]

Right, I configured data/db --> data and data/images --> data

I did that because if I do data/db --> data/db and data/images --> data/images in the target server I end with data/data/db and data/data/images.

 

[artlessknave]

well, you have a choice. would you like to actually have the data replicated instead of endlessly overwriting itself...or would you like to have less datasets...
if you can have both, I haven't figured a way to, and since it's a backup...i haven't really bothered to try. as long as it's there somewhere I can find it if it's needed.
and, again, i literally did the exact same thing, which is part of why I suspected it kind of early.

 

[leonardorame]

well, you have a choice. would you like to actually have the data replicated instead of endlessly overwriting itself...or would you like to have less datasets...
if you can have both, I haven't figured a way to, and since it's a backup...i haven't really bothered to try. as long as it's there somewhere I can find it if it's needed.

Yes, indeed I don't care too much about the directory layout right now, I just want to be sure how replication works to plan a production server.

 

[artlessknave]

actually, it occurs to me you could also recursively replicate
data > data
and they would be identical, but you would be unable to replicate any other pools to that same pool (trying would cause the same overwriting problem)

 

[leonardorame]

actually, it occurs to me you could also recursively replicate
data > data
and they would be identical, but you would be unable to replicate any other pools to that same pool (trying would cause the same overwriting problem)

Interesting, I'll try that also.

 

[artlessknave]

what I do is replicate entire pools to datasets
ex
data1 > bkup/data1
data2 > bkup/data2
data3 > bkup/data3
it seems that replicating datasets generates more extra datasets
so for you:
data > bkup/data
which should give you
bkup/data/images
bkup/data/db
etc.

 

[leonardorame]

Great!, I'll do exactly that.