Use USB extrenal drive to make backup of data on Server

[optic]

please help.
I have setup a 8.3.0 Freenas server and it is working. Now I would like to use a extrenal USB drive to make a backup of the data on the server.
How do I go about getting to see the drive in Freenas?
thanx

 

[joeschmuck]

Took me about 10 minutes via search and trying it on my system, maybe you should try searching yourself next time before just posting a question. Took me longer to enter this message.

Here is the answer anyway and it does work for me but this is just how to mount the USB drive which is the most difficult part in the first place and I'm assuming this is a standard external USB drive that is formatted in NTFS (that's all I have to test with) and only has a single partition.

When I inserted my USB drive it was attached as device da2. This may be different for you and if it is the second USB device attached (FreeNAS boot flash being the first) then you're might be da1.

Do not include the single quotes when you type in the commands!

1) Open either the console or Shell.
2) Type 'ls /dev' and it will list a lot of devices. You are looking for everything starting with da. Take note.
3) Plug in your USB drive. Wait 5+ seconds for FreeNAS to recognize it.
4) Repeat step 2 and you should see a new da listing, likely da1 and da1s1 (in my case it was da2 and da2s1). We will assume it is da1 and da1s1 for this example.
5) Type 'mkdir /var/usb'. We must create a place to mount the drive. If I knew more about FreeNAS then I might have a better way of doing it but this worked for me.
6) Type 'ntfs-3g /dev/da1s1 /var/usb'. This will mount partition 1 of da1 in the location /var/usb.
7) Type 'cd /var/usb'
8) Type 'ls' and you should see all the files on the USB drive.

To copy files you will use the 'cp' command. If you don't know how to use it, do a google search on 'freebsd cp command'.

To disconnect, do not just unplug your drive, this is not Windoze.

1) Type 'cd /'. This only ensures you are not in the /var/usb directory.
2) Type 'umount /var/usb'
3) If there was an error you could use 'umount -f /var/usb' but you should give the step 2 command a few more tries first.

 

[JoeAshley]

Backup Script Doesn't Work

Took me about 10 minutes via search and trying it on my system, maybe you should try searching yourself next time before just posting a question. Took me longer to enter this message.

Here is the answer anyway and it does work for me but this is just how to mount the USB drive which is the most difficult part in the first place and I'm assuming this is a standard external USB drive that is formatted in NTFS (that's all I have to test with) and only has a single partition.

When I inserted my USB drive it was attached as device da2. This may be different for you and if it is the second USB device attached (FreeNAS boot flash being the first) then you're might be da1.

Do not include the single quotes when you type in the commands!

1) Open either the console or Shell.
2) Type 'ls /dev' and it will list a lot of devices. You are looking for everything starting with da. Take note.
3) Plug in your USB drive. Wait 5+ seconds for FreeNAS to recognize it.
4) Repeat step 2 and you should see a new da listing, likely da1 and da1s1 (in my case it was da2 and da2s1). We will assume it is da1 and da1s1 for this example.
5) Type 'mkdir /var/usb'. We must create a place to mount the drive. If I knew more about FreeNAS then I might have a better way of doing it but this worked for me.
6) Type 'ntfs-3g /dev/da1s1 /var/usb'. This will mount partition 1 of da1 in the location /var/usb.
7) Type 'cd /var/usb'
8) Type 'ls' and you should see all the files on the USB drive.

To copy files you will use the 'cp' command. If you don't know how to use it, do a google search on 'freebsd cp command'.

To disconnect, do not just unplug your drive, this is not Windoze.

1) Type 'cd /'. This only ensures you are not in the /var/usb directory.
2) Type 'umount /var/usb'
3) If there was an error you could use 'umount -f /var/usb' but you should give the step 2 command a few more tries first.

I figured this was a good place to start as I have successfully completed the steps above but I am trying to automate this...... I have spent hours trying to get a VERY simple script to just do a simple CP to an external USB HD. I've had problems keeping the FSTAB and crontab edited as I left it after reboots... Thats for a different post anyway.......

I can run EVERY single command in the script in order from the console and it work perfectly. When I run the .sh file from the console it creates a different backup folder name. On my HD I have create the test folders qtest, qtest2 and qtest3. In my script when I try to copy from qtest to qtest2 it creates QYUUS~2 and successfully finishes the script. This doesn't work as I am trying to automate cascading backups. I have done this MANY time in batch files, .PS1 and even QIX... its weird. Any help will be appreciated.

 

[joeschmuck]

Two things...

1) What is exactly your script, from top to bottom (post a copy of it, enclose it with "Code" wraped around the test, to do that go to the advance editor, highlight the your code already pasted into the text, look for the hash mark icon and click it).

2) To fix the loss of your CRONTAB and FSTAB, you need to make those changes under /conf/base/etc, first type 'mount -uw /' and then you can make changes to those files. Once done type 'mount -ro /' to close the area properly. Now when you reboot these files will be what you desire. But you should test everything out using the /etc area first and if it appears to work properly then I'd change the /conf/base/etc area files.

 

[JoeAshley]

Here it is JoeSchmuck... Us Joe's have to stick together... Please excuse the "testing" and comments in my script. I have been trying to get this simple thing done and I have hacked it up... Anyway.. When it gets to the part here "cp -r /mnt/ZFSRAID/DATA/qtest /mnt/ZFSRAID/DATA/qtest2" it creates a name of it's own starting with "q" and ending it "2" but garbage in between.

Thanks for the tip about the CRONTAB and FSTAB.. I have seen many posts about this but none that explained how to do it this simply.. I will try it soon......

Ok... I edited CRONTAB and FSTAB exactlly to the letter how you suggested by using 'mount -uw /' Now after a reboot it will not mount the drives at all and asks for the location of /bin/sh. I run FreeNAS form an 8GB thumb drive if this helps. I now have a wife freaking out that the 1000's of pictures are gone. :( This is what I get...

Using my original USB Stick I am getting:
init: /bin/sh on /etc/rc terminated abnormally, going to single user mode
inti: NSSWITCH(_nsdispatch): files, passwd_compat, endpwent, not found, and no fallback provided
Enter full pathname of shell or RETURN for /bin/sh:
​

:)
I fixed it... Was able to revert changes made to FSTAB.

I still have the automated backup issue to fix

Thanks again for the help.

Code:

#!/bin/bash
# this script does a full backup of ALL important data
# from the ZFSRAID_CIFS share to the external USBHD mounted
# at /mnt/USBHD
#
VDATE=`date +%Y%m%d`
VDAY=`date +%a`
VNOW=`date +%k%M`
VSOURCEDIR=/mnt/ZFSRAID/DATA/
VDESTDIR=/mnt/ZFSRAID/DATA/
LOGFILE=/mnt/ZFSRAID/STUFF/Utils/Backup.log
echo Starting Full Backup at >> /mnt/ZFSRAID/STUFF/Utils/Backup.log
sleep 5
date >> /mnt/ZFSRAID/STUFF/Utils/Backup.log
#rm -r /mnt/USBHD/BACKUPS/full_OLD
rm -r /mnt/ZFSRAID/DATA/qtest3
sleep 5

echo Starting move of Full to full_OLD >> /mnt/ZFSRAID/STUFF/Utils/Backup.log
sleep 5
date >> /mnt/ZFSRAID/STUFF/Utils/Backup.log
#mv /mnt/USBHD/BACKUPS/full /mnt/USBHD/BACKUPS/full_OLD
mv /mnt/ZFSRAID/DATA/qtest2 /mnt/ZFSRAID/DATA/qtest3
sleep 5

echo Starting ZFSRAID/DATA copy >> /mnt/ZFSRAID/STUFF/Utils/Backup.log
sleep 5
date >> /mnt/ZFSRAID/STUFF/Utils/Backup.log
#cp -r /mnt/ZFSRAID/DATA /mnt/USBHD/BACKUPS/full
cp -r /mnt/ZFSRAID/DATA/qtest /mnt/ZFSRAID/DATA/qtest2
sleep 5

echo Finished backup >> /mnt/ZFSRAID/STUFF/Utils/Backup.log
sleep 5
date >> /mnt/ZFSRAID/STUFF/Utils/Backup.log
echo . >> /mnt/ZFSRAID/STUFF/Utils/Backup.log

 

[joeschmuck]

First, getting your FreeNAS back online. Your pool should be fine. Did you type 'mount -ro /' when you got done modifying the files? It's too late now if you didn't.

1) Get another USB key and load it with FreeNAS, same flavor you were using.
2) Boot up the new USB key. Hopefully you saved your config file because here is where it will come in handy, restore it and reboot.
3) You should be up and running. If your pool did not automatically import then do an Automatic Import. Now you should be up and running.
4) Make a damn backup of those photos on DVD(s) before your wife kills you!

I simplified it drastically just to verify the cp command works fine...
File Name: backup.sh

Code:

#!/bin/bash
cp -r /mnt/peanuts/DATA/qtest/ /mnt/peanuts/DATA/qtest2/

And this works fine.

I tried the following as well and it worked too.

Code:

#!/bin/bash
VSOURCEDIR=/mnt/peanuts/DATA
VDESTDIR=/mnt/peanuts/DATA
cp -r $VSOURCEDIR/qtest/ $VDESTDIR/qtest2/

(my VM pool is called peanuts)
When I ran them I typed 'sh backup.sh'

Are you trying to copy only the files in "qtest" or do you also want to copy the folder name "qtest" into the "qtest2" directory? My example above does only the files within "qtest" and then copies them to "qtest2". If you try to copy "qtest" itself, you can easily screw up and copy "qtest2" into itself. Pay attention to the "/" marks at the end of the paths.

So does your script work when you command it in an SSH window or does it fail there? Or is it only when you try to run it from a CRON that it fails?

And please keep it simple until you can get the basic function working. I do understand adding in all the extra code but as temperamental as these scripts are it's just easier to keep it simple.

 

[JoeAshley]

I did just as you instructed... I bow down to your 920 post and senior status :) I did remount read only... I was editing my own post on here as I went through and I was able to get it back up and running after removing my apparent BAD entry in my FSTAB. '/dev/da1 /mnt/USBHD ufs rw 0 0'

So does your script work when you command it in an SSH window or does it fail there? Or is it only when you try to run it from a CRON that it fails?
​

I haven't tried this... My heart is still racing after fixing the pending divorce ;)

Are you trying to copy only the files in "qtest" or do you also want to copy the folder name "qtest" into the "qtest2" directory? My example above does only the files within "qtest" and then copies them to "qtest2". If you try to copy "qtest" itself, you can easily screw up and copy "qtest2" into itself. Pay attention to the "/" marks at the end of the paths.
​

My real goal (commented out in the script) is to make a copy of all content under DATA (in test it is qtest) to a mounted USB HD. I have done it manually but it takes FOREVER... Not a huge deal just unbelievably slow. After that I will rotate backups via the script so I have about 2 fulls and daily differentials for 2 weeks.

I am very familiar with other script languages but the syntax of any language is something to practice with :) ... BASH is a little weird.

I really appreciate the help. oh and sorry for the ugly unreadable reply with indents and stuff.... Still figuring out the editor.

 

[joeschmuck]

Also, I'd recommend keeping a backup USB drive on hand with your setup loaded and ready to go if you are going to be developing on your machine and not a VM machine.

You can build a VM machine for free, setup a RAIDZ, even attach your real external HD if you like, all without destroying your real data. Just something else to consider.

Wow, I didn't know I had that many postings. But I'm not all knowing, if I were then I'd know the next lottery numbers and go enjoy some much needed travel.

So making the script do what you want shouldn't bee too hard. I'm not sure why you have some weird file names. Are these file names only on the external hard drive or also if you point to an internal drive? I ask because what if the external HD is formatted in a not so compatible format. NTFS was only supported to allow a user to copy data to the FreeNAS device, not for routine use, although I don't see why it wouldn't work.

P.S. Glad your wife hasn't killed you yet (yet I said) :)

 

[JoeAshley]

I also wondered if the weird folder name would happen on the ZFSRAID and it did. I just don't get it...It works if I do the commands manually. I have had a similar issue in DOS when the files being copied were not released yet from the previous process but I don't think that is the case here.

Your suggestion about the backup USB drive is a good one.... I know I could find it here in the forums but is it as easy and copying the data from one thumb drive to another or is there something more complicated? Could I automate that during a clean shutdown? I am an automation freak. :)

If I should find it elsewhere just tell me :) I have seen some guys on here get VERY upset if you get repetitive in your questions

 

[joeschmuck]

I wouldn't worry about repeat questions. Sometimes I don't have good luck with searching these forums either so I resort to posting a question periodically as well.

I do not know of any way to copy the USB drive to another drive but that is only because I haven't done that before. I just load up another USB Flash drive with a fresh image and then restore the configuration file. It's probably faster this way anyway, only about 6 minutes to create it + a few more minutes to restore the config + a few more minutes to actually test it and verify it will work when it's crunch time.

You say it works when you do the commands manually. Pretend I'm an idiot, it helps me understand. Does that mean you are at an SSH window and you type in 'cp -R /mnt/ZFSRAID/DATA/qdata/ /mnt/ZFSRAID/qdata2/' and it copies fine? Or does it mean that it also works if you run the script manually by typing 'sh backup.sh' (for my example above of course)?

The filename you are stating looks like a DOS 8.3 name which is odd since the directory name is not a long name.

Another question, how was the directory created? In FreeBSD, in Windoze?

Also, what version of FreeNAS are you running?

Something about the cp command... Use -R, not -r. They are not the same and the -r is not really supported and has issues apparently, but I doubt that will fix your problem.

I'm about out of options right now because I cannot replicate your problem.

While in the SSH window, type in 'finger -m username' (where username is the user you are logged in as). Look for the Shell, what does it say?

 

[JoeAshley]

Ok I'll try that with the USB drive.

Only when I type the command lines individually does it work correctly. All of my testing with the script has produced the results I am describing.

I tried it by creating the original folders both in windows and freeBSD. same issue. I am running FreeNAS 8.2 x64.

The finger -m root gives me Shell: /bin/csh

Again.. Thanks for the help.

 

[joeschmuck]

What I was looking at is if your shell was causing the issue. The script uses bash while you are using csh. For the commands you are running it shouldn't make a difference. I am running 8.3.0-Release but I don't see why that would make a difference.

In the batch file replace the cp line with this and let me know if it works, this will force use of the c shell...

Code:

csh -c "cp -R /mnt/ZFSRAID/DATA/qdata/ /mnt/ZFSRAID/qdata2/"

 

[JoeAshley]

I'll give it a try. I don;'t think it will work either because the MV command doesn't find the qtest3 folder even though it is right there :(

Thanks again

 

[joeschmuck]

The other thing you can do is create your destination folder up front in the script and then copy to it, this should work.

 

[[XAP]Bob]

The other thing you can do is create your destination folder up front in the script and then copy to it, this should work.

I'd also suggest rsync if FreeNAS supports it - far better for backups than cp
See http://www.mikerubel.org/computers/rsync_snapshots/