SOLVED Collect multiple periodic snapshot for "Restore previous version" under cifs

Status
Not open for further replies.

pernils

Explorer
Joined
Aug 31, 2015
Messages
87
Default you can only use a single snapshot schema for your "Restore previous version". The problem lays in the sambas shadow:format setting.

Lets examine one of my shares with testparm.

Code:
...
[data]
   path = /mnt/Vol_1/data
   read only = No
   guest ok = Yes
   ea support = No
   case sensitive = Yes
   preserve case = No
   short preserve case = No
   hide dot files = No
   veto files = /.snapshot/.windows/.mac/.zfs/
   map archive = No
   map readonly = no
   store dos attributes = No
   vfs objects = shadow_copy2 zfs_space zfsacl aio_pthread streams_xattr recycle recycle
   zfsacl:acesort = dontcare
   nfs4:chown = true
   nfs4:acedup = merge
   nfs4:mode = special
   shadow:snapdirseverywhere = yes
   shadow:format = auto-%Y%m%d.%H%M-100d
   shadow:localtime = yes
   shadow:sort = desc
   shadow:snapdir = .zfs/snapshot
   recycle:subdir_mode = 0700
   recycle:directory_mode = 0777
   recycle:touch = yes
   recycle:versions = yes
   recycle:keeptree = yes
   recycle:repository = .recycle/%U
...


Pay attention for the shadow:format and shadow:snapdir. From shadow:format we see that samba will only dive into and search snapshot who start with auto- and have YYYYMMDD.HHMM and ends with -100d.
For example ...

auto-20170203.0930-100d
auto-20170204.0930-100d
auto-20170205.0930-100d
auto-20170205.0600-1w <-- this will be skipped.

So why don't we generate a script that takes all the snapshots and normalize them into a single format? As .zfs is write protected we have to make our list else where. It's should be no problem because we can specify the shadow:snapdir.

A tip here when you are writing a script is that call it from another script in a loop. Then you just have to save and bang your changes will be shown in the terminal.
It can be a bit tedious switching terminal window back and forth just to execute it.

Here is my loop script when I was testing and writing my main script
Code:
while true;
do
  clear   # clear screen

  dir='/home/pernils/d/links'
  touch $dir/tmp
  rm $dir/tmp_ln1
  rm $dir/tmp_ln2
  ln -s $dir/tmp $dir/tmp_ln1
  ln -s $dir/tmp $dir/tmp_ln2
  rm $dir/tmp
	 # the above is for generate broken symlink

./make_symlink.sh   # my main script
done


So after long weekend I end up with this ...

Code:
#!/bin/bash

dryrun="true"
# Dryryn simulate mode. For live system set it to false
#dryrun="false"
#

snapshot_dir='/mnt/Vol_1/data/.zfs/snapshot'
# snapshot_dir is the directory where your snapshot is located.
#
# tex /mnt/Vol_1/dataset/.zfs/snapshot
#

link_dir='/mnt/Vol_1/data/.zfs_links'
# link_dir is where your symlinks to your snapshot will be saved.
#
# tex /mnt/Vol_1/dataset/.zfs_links
#


# **************************************************
#
#  Normilize the snapshot name
#
# **************************************************
# Add your snapshot schema suffix to be include
#

function normilize {
   # strip out prefix "auto-" and suffix like "-1d, -1w" etc
   # add your own suffix if needed

   item=$1	# first argument
   item=$(echo $item | sed -e 's/auto-//') # remove prefix auto-

   item=$(echo $item | sed -e 's/-1d//')	# remove trailing -1d
   item=$(echo $item | sed -e 's/-100d//') # remove trailing -100d
   item=$(echo $item | sed -e 's/-1w//')	# remove trailing -1w
   item=$(echo $item | sed -e 's/-2w//')	# remove trailing -2w

# your edit goes in above .....

   item=$(echo $item | sed -e 's/\///') # remove trailing /
   echo $item   # form return value
}


if $dryrun; then
  echo " **** Deletes broken snapshots ****"
fi

# ****************** Main start *********************
#
#				Clenup of broken symlinks
#
# ***************************************************
#
# Broken symlink means snapshot that have been deleted /rotated
#
for item in $(find -L $link_dir -maxdepth 1 -type l )
do
   if $dryrun; then
	  echo "<broken symlink>... rm $item"
   else
	  rm $item   # deletes broken sym link
   fi
done

if $dryrun; then
   echo
   echo " **** Collectings snapshots ****"
fi

# **************************************************
#
#  Collect the snapshot list
#
# **************************************************
# Collecting all snapshots that start with prefix auto-
# This list is store into $snap_array

i=0		#counter for snap_array
snap_array=()   #initilize the array


for item in $(find $snapshot_dir -mindepth 1 -maxdepth 1 -type d );
do
   item=$(echo $item | sed -e "s|$snapshot_dir/||")  # strip off prefix $snapshot_dir

   if grep -q '^auto-' <<< $item; then			 #grep -q quit no output
	  snap_array[$i]=$(echo $item | sed -e 's/\///')  # removes trailing /
	
	  if $dryrun; then
		   echo  "<found snapshot>.." ${snap_array[$i]}
	  fi

	  i=$((i+1))	   # inc counter
   fi
done


if $dryrun; then
   echo
   echo " **** Writing new symlinks ****"
fi

# **************************************************
#
#  Writing new symlinks
#
# **************************************************
# Loops throw $snap_array and wrtite symlink that don't exists
#

for item in ${snap_array[@]}
do
  normitem=$(normilize $item)

   # check if normilized $item -> $normitem can't be found aka !
   if ! test -n "$(find $link_dir/ -name $normitem -type l)"; then
	 if $dryrun; then
		echo "... ln -s $snapshot_dir/$item $link_dir/$normitem"
	 else
		ln -s $snapshot_dir/$item $link_dir/$normitem   # generate symlink
	 fi
   fi
done

if $dryrun; then
   echo
   echo .... waiting 1 sec ....
   sleep 1
fi


First we have the conf.

dryrun=true means that it will only output to terminal what it want to do.

uncomment the dryrun=false to make it run.

snapshot_dir and link_dir should be self explained.
The link_dir you must make your self. Set write permission to what user this script should be run under.

Verify that your snapshot schema is added to the normilize function.

From our testparm we copy the important pieces and paste into the Auxiliary Parameters.
Remeber also to include the VFS Object. For 9.3 it's shadow_copy2.

upload_2017-3-5_22-22-15.png


Remember to edit the shadow:format and shadow:snapdir and also unselect the Periodic Snapshot Task:

Bellow you have an easy copy and paste
.....
shadow:snapdirseverywhere = yes
shadow:format = %Y%m%d.%H%M
shadow:localtime = yes
shadow:sort = desc
shadow:snapdir = .zfs_links

.....


Now you set up a cron job ..
upload_2017-3-5_22-34-30.png


Don't be so foolish as I'am by using the user root.

When all this done you can peek into symlink_directory (the example above is .zfs_link) to see when list is starting populate.

upload_2017-3-5_22-39-59.png


Be a bit patient, for me I have to disconnect the mapped letter (or I was patient ... it finally started show something).

And voila ..

upload_2017-3-5_22-42-11.png


As you can see the last snapshot or Restore previous version slot is way off the other ones (2017-03-05 09:57).

I find this interesting, do you ?.

**** NOTE I take no responsibility for your data while you are using this script. ****

Hmm I think it works ... still testing

... after some testing .. bah it seems to default to first snapshot schema. What you get from
testparm is not necessary the conf. for samba it seems.
... back to ground zero again ...
 
Last edited:

Ericloewe

Server Wrangler
Moderator
Joined
Feb 15, 2014
Messages
20,194
This is on the roadmap to be implemented in 9.10.3 or .4 or something, you might want to take a look at the ticket and engage with the devs.
 

pernils

Explorer
Joined
Aug 31, 2015
Messages
87
I guess you are referring to ticket 21333 -> https://bugs.freenas.org/issues/21333

I can't test it now but if the case is that the conf file /usr/local/etc/smb4.conf in some cases is generate after the samba service have been restart then you can use service samba_server restart to restart samba manually .

Something is fishy thought ..

My testparm..

upload_2017-3-6_7-55-53.png


My symlinks list for snapshots (.zfs_links)

upload_2017-3-6_7-57-6.png


Notify the last ones 20170305.2320 and 20170305.2320

upload_2017-3-6_7-58-49.png


Is nowhere to be found in my previous version list.

And yes I had a cron that edit a file (auto) every 3 minutes.

Code:
#!/bin/sh

ls /mnt >> /mnt/Vol_1/test2/auto


Folks I need more tester on this ......

When we are on it ... I have heard that you shouldn't expose shadow copy due trojans like cryptolocker can write to it. Done some small searches and it seem to be legit when it's a windows server. But in freenas you can't write to the snapshot right? (I can't)
Or is there something vital that I'm missing here?


Ericloewe :

Can you remember and evolve your answer in this thread ...
https://forums.freenas.org/index.ph...revious-versions-over-cifs.17954/#post-167000

Is it final that freenas is defaulting to the first setup snapshot schema nerveless what you have typed in shadow:format ?
 
Last edited:

Ericloewe

Server Wrangler
Moderator
Joined
Feb 15, 2014
Messages
20,194
Is it final that freenas is defaulting to the first setup snapshot schema nerveless what you have typed in shadow:format ?
No, for about a year or more now, FreeNAS sets up the task that you choose (hence why there's a drop-down box for it).
 

pernils

Explorer
Joined
Aug 31, 2015
Messages
87
My test bench is 9.3 due performance reason.

This drop-down box seems to be useless due it will not reflect what you get . Even output from testparm don't correspond to witch periodic snapshot schema you will see.
It's like the only periodic snapshot schema you will be using is the the ever first created.
It doesn't matter how many periodic snapshot you do and select in the drop-down box and verify by running testparm. Only the first created periodic snapshot is the one that will be used.

It's just odd ...
 

Ericloewe

Server Wrangler
Moderator
Joined
Feb 15, 2014
Messages
20,194
My test bench is 9.3 due performance reason.
How do you figure?

It doesn't matter how many periodic snapshot you do and select in the drop-down box and verify by running testparm. Only the first created periodic snapshot is the one that will be used.
I haven't checked recently, but nobody's complained about that. Maybe the build you're running doesn't have the backend implemented yet.
 

pernils

Explorer
Joined
Aug 31, 2015
Messages
87
How do you figure?

It's the problem with Freebsd 10 and samba https://forums.freenas.org/index.php?threads/performance-9-3-vs-9-10-folder-browsing-linux.50744/

I haven't checked recently, but nobody's complained about that. Maybe the build you're running doesn't have the backend implemented yet.

You seems to be right on this. I have no source if 9.3 version of samba is broken, instead I moved my test to 9.10 ...

This is on 9.10.2-U1

upload_2017-3-6_22-28-6.png

Note the two retention periods -1d and -5d. Make also notation for the Refer column witch say that something have been changed.

upload_2017-3-6_22-31-37.png


Okay there are my snapshot with 2 different schema.

upload_2017-3-6_22-35-14.png


Witch have been normilized and symlinked in the .zfs_links folder. Hmm all god so far ...

upload_2017-3-6_22-37-43.png


And there we have them all.

The original snapshots are
Code:
auto-20170306.2147-2d
auto-20170306.2215-2d
auto-20170306.2205-5d
auto-20170306.2220-2d


2146 is from 2147-2d and 2204 is from 2205-5d. Must be round vs truncation of the %S.

Did a fast check on restore point 2146 and 2204 and the file (auto) have a diff.


So good folks .. I think we have a solution for having multi periodic snapshot collected under the same "Restore previous version" facility.


Do I dare to take this into production? The actually snapshots that you can reach via the symlink seems to be non writable so I think it's a go.

You with more intelligence and knowledge do you see any problem?'

** edit **

After some more testing on 9.3 I manage to get it working on one of my shares (Homes). I don't know how...It just started to behave as it should.
But still one of my other shares (data) will not do as I say. Some error message "Unable to set SID to S-1-5-21-1105206447-762845627-1237636618" could be the source.
It's like that you have to clear any connection for your share before samba really reload it's config. That's feeling I get ...

** edit **
 
Last edited:

anodos

Sambassador
iXsystems
Joined
Mar 6, 2014
Messages
9,554
Do I dare to take this into production?
It depends on what you mean by "production". At home? Sure, why not. At a client? Nope. You're better off just presenting a single snapshot task for present and waiting for the desired functionality to be added to FreeNAS. The reason why is that you're ending up with a system that's less maintainable. In the long run such scripts / additions do a disservice to the business client because it will create one more thing for the next guy to have to understand and maintain.

I'd probably just make do with the single snapshot task and keep an eye on the performance regression you spotted in 9.10. Eventually the devs will sort it out. It may just be an issue of waiting for the samba45 port to come down the pipe (or for FreeNAS to switch to FreeBSD 11).

But still one of my other shares (data) will not do as I say. Some error message "Unable to set SID to S-1-5-21-1105206447-762845627-1237636618" could be the source.
That's indicative of a permissions error. Perhaps you set an ACL and then deleted the User. The ACE still exists, but the SID no longer maps to a user... On second thought, that SID looks like a domain / machine SID (lacks the RID component) and so the message might be nothing. FreeNAS does some checks to make sure that the domain / local machine SID hasn't changed. Or it might be that your domain / local machine has changed. In this case, permissions on your system would be in a fairly broken state.

It's like that you have to clear any connection for your share before samba really reload it's config. That's feeling I get ...
That's wrong. There's a difference between reloading the smb.conf file and restarting samba. Reloading the smb.conf file does not affect connections that have been already established. This includes IPC$ connections and mapped network drives. To force all clients to use the new smb.conf file you have to restart the samba server (not just reload the smb.conf file).
 

pernils

Explorer
Joined
Aug 31, 2015
Messages
87
It depends on what you mean by "production". At home? Sure, why not. At a client? Nope. You're better off just presenting a single snapshot task for present and waiting for the desired functionality to be added to FreeNAS. The reason why is that you're ending up with a system that's less maintainable. In the long run such scripts / additions do a disservice to the business client because it will create one more thing for the next guy to have to understand and maintain.

It's at my main work (I'm just a engineer interest in IT).All our drawings and homes is backed up by freenas.

The Vol_1/data share (cad drawings) contains 165 517 files, 1 911 dir approx total 128 GB.
This one "Vol_1/data" will not "work with me" by show the windows VSS. A bit odd because it was no problem implementing this with our Vol_1/Homes.

Freenas is part of the AD and it's when I edit the Vol_1/data share's properties and press ok that this SID problem pops up in the status window in the footer.


That's wrong. There's a difference between reloading the smb.conf file and restarting samba. Reloading the smb.conf file does not affect connections that have been already established. This includes IPC$ connections and mapped network drives. To force all clients to use the new smb.conf file you have to restart the samba server (not just reload the smb.conf file).

I can't restart it now due personal is working against it. I will restart it tonight.

*************

anodos
: thanks for the nice explanation as usually .
 
Last edited:

anodos

Sambassador
iXsystems
Joined
Mar 6, 2014
Messages
9,554
It's at my main work (I'm just a engineer interest in IT).
All our drawings is backed up by freenas. The ./data share (cad drawings) contains 165 517 files, 1 911 dir approx total 128 GB.
This one "/mnt/Vol_1/data" will not "work with me" by show the windows VSS. A bit odd because it was no problem implementing this on our ./Homes.

Freenas is part of the AD and it's when I edit the ./data shares properties and press ok that this SID problem pops up in the status window in the footer.
Well, to get to the bottom of the SID error, you should look at your current local machine SID.
net getlocalsid will display your Samba server's SID. Is this the SID that is referenced in the error message?
net groupmap list will display the SID -> Unix Group mappings. The last group of numbers (usually 4 digits) is the RID for the SID . It is appended to your local server's SID and should be identical to the Unix Group's GID. These SIDs will be unique for every server.
net groupmap list will also display "BUILTIN" Groups / users. These have a different SID structures than user / group SIDs. They are identical across all windows systems (like well-known SIDs such as "world" S-1-1-0).
net usersidlist will display the SIDs associated with all users (including AD users).
Once you look at all this data, try to match S-1-5-21-1105206447-762845627-1237636618 with something on your system. It is probably SID for your samba server's local domain (output of net getlocalsid).

I can't restart it now due personal is working against it. I will restart it tonight.
Indeed. That's the reason why FreeNAS defaults to reloading the smb.conf file rather than restarting Samba.
 

pernils

Explorer
Joined
Aug 31, 2015
Messages
87
net getlocalsid will display your Samba server's SID. Is this the SID that is referenced in the error message?

Yes it is
Code:
[root@thor] /mnt# net getlocalsid
SID for domain THOR is: S-1-5-21-1105206447-762845627-1237636618
[root@thor] /mnt#


net groupmap list will display the SID -> Unix Group mappings. The last group of numbers (usually 4 digits) is the RID for the SID . It is appended to your local server's SID and should be identical to the Unix Group's GID. These SIDs will be unique for every server.

This i didn't get
Code:
root@thor] /mnt# net groupmap list
Administrators (S-1-5-32-544) -> BUILTIN\administrators
Users (S-1-5-32-545) -> BUILTIN\users
[root@thor] /mnt#


net usersidlist will display the SIDs associated with all users (including AD users).
Once you look at all this data, try to match S-1-5-21-1105206447-762845627-1237636618 with something on your system. It is probably SID for your samba server's local domain (output of net getlocalsid).

Will list a ton of SID's for different domain users. My S-1-5-21-1105206447-762845627-1237636618 is nowhere to be found.

anados : thanks for the command and the explanation .. reboot of the box is in progress ......


....... after reboot ..... yepp it works as aspected. I can see the 10 min and the once a day snapshot schema in Restore previous version.
 
Last edited:

pernils

Explorer
Joined
Aug 31, 2015
Messages
87
One thing that puzzle me is:

Is't okay to expose the snapshot directory?
I have read in the past post from cyberjock that for me it was more or less "you can kill your cat but don't expose the snapshot directory".
I guess I can veto my .zfs_links folder but I found rather neat to dive in and recover deleted files this way then having VFS object recycle enabled. One of my sources also pointed out the less VFS object you have enabled the better performance. And we like performance right.

Anyone have more insight in this?

** Edit. Links that touch interesting things related to snapshot **

https://forums.freenas.org/index.ph...2012-windows-server-backup-and-freenas.34465/
https://www.youtube.com/watch?v=Bqu16ZThviI ... (fast forward to 34:50)
https://forums.freenas.org/index.php?threads/dangerous-zfs-snapshot-directories.30828/
https://forums.freenas.org/index.php?threads/share-zfs-snapshots-with-cifs.36990/
https://forums.freenas.org/index.ph...all-locker-ransomware-virus-infections.39591/

** edit **
 
Last edited:

pernils

Explorer
Joined
Aug 31, 2015
Messages
87
I have been running this setup for a while now and suddenly the share started to be slow. After I had view the graphs over Disk I/O and ARC Request I went to inspect physically the box.
upload_2017-3-14_21-25-2.png


upload_2017-3-14_21-25-37.png


The disks was spinning like mad.

The only thing that I could come up with is that clients on the shop floor witch is searching for drawings is doing this from the root of the share /mnt/Vol_1/data and therefore also search in a enumeration of the 300 snapshots in .zfs_links. The snapshot 300 will see all it's 299 predecessor, and each one will see all his.
As the share consist of 120k files this will lead I guess the biggest file search of man kind.
It can also be some dude who have set windows to index the share.

The end story is that could be good idea to veto the dir. I have temporary removed my symlinks to stop this disk spinning madness over the night.

To to this you add it to the Auxiliary Parameters:
Use tesparm from shell to inspect the default veto files setting as you have to include that by your self.
Code:
veto files = /.snapshot/.windows/.mac/.zfs/.zfs_links/


upload_2017-3-14_21-45-16.png


Remember what anodos have said about active connection to your share. It will prevent samba for restarting so best option to be sure it will take effect is to stop/start cifs under the service menu.

The other failure scenario that is on my mind is if the snapshot mechanism is falling on it self. The symlink is pointing to a snapshot back in time which have .zfs_links folder with symlinks to snapshot pointing back in time and so forth. I have sort of ruled this out due this excessive read started in the middle of the day. The 10 min snapshot schema is from 07:00 - 16:00

The .zfs folder is not part of the snapshot mechanics. Is there a editable list where you can exclude directories from being snapshoted?
 
Last edited:
Status
Not open for further replies.
Top