clarify / sugest - Copy command to use in Shell

[TDPsGM]

I am moving data from an old zpool called "Volume1" to a 4x2TB Raidz2 array.

With a clean install of release v8.3.0 p1 I was going to import the first zPool (volume1) and create the new array Raidz2 array and call it: bin1

SIDE NOTE: all the data in volume1 is actually in a dataset called 'admin' ===> don't ask, it's just what happened as I was learning the system.

The command I was thinking of using was:

cp -r /mnt/volume1/admin/ /mnt/bin1

Now for my questions as I have never really messed with shell, the cp command or the options that are available:

Q1) Would cp -a /mnt/volume1/admin/ /mnt/bin1
not be a better option? As I understand it, it preserves much of the file attributes, permissions, etc.

Q2) I have seen cp -a /mnt/volume1/admin/* /mnt/bin1
. . . use of the asterisk. Like I said I am just looking for guidance on this.

Q3) http://en.wikipedia.org/wiki/Cp_(Unix) suggest that there are slight differences between how the copy command is handled in a GNU system verses a BSD system. Look for "however, on a BSD system" and you will see it on the page --- I think there was a slight typo on that example too, which is why I am asking here.

Q4) The trailing '/' on the target . . . is that not required?

Q5) In windows there are a couple of programs out there that run hash checks on the data copied from one location to another. Is there a way to do this with the copy command in shell?

Thanks for the help.

 

[jgreco]

q1) Yes, that's fine and maybe better.

q2) The asterisk is a shell expansion; the cp command does not see it. If your "/mnt/volume1/admin" has files "larry" and "curly" and "moe" in it, this expands to "cp -a /mnt/volume1/admin/larry /mnt/volume1/admin/curly /mnt/volume1/admin/moe /mnt/bin1"

The usual new UNIX user mistake is to enter a command like "cp -a /mnt/volume1/admin/*" and then be totally puzzled why the contents of larry and curly ended up in moe. Again, it's because the UNIX shell is doing the expansion - not the command itself.

q3) I'm pretty sure - but too lazy to verify - that the issue is summarized at the top of the article you quote. Specifically, "The two major specifications are POSIX cp and GNU cp. GNU cp has many additional options over the POSIX version." I would be shocked if the BSD behaviour isn't POSIX-compliant. The GNU people, on the other hand, have a tendency to do whatever they like, and don't always care about standards they didn't write.

Personally I think it is way cool that you can select either mode of operation merely by intelligent specification of the pathname...

q4) See q3.

q5) No, not really. This is UNIX. There are tools to do just about anything you can imagine, but you usually have to put them together. You can easily make a little shell script to run "cp", "md5", and then "diff" on the data or something like that. If you're just looking for verification that the data copied, you don't even need to hash it, you can just run "cmp" or "diff" or whatever.

 

[fracai]

q2) be careful with shell expansion like this. I've done this to copy the contents of my home folder before and forgotten that the * will not grab dot files. It's safer to to copy the containing folder, specify '.*' as well, or use another tool.

q5) use rsync instead of cp. Basically, "rsync -Pva /mnt/volume1/admin/ /mnt/bin1/" will put the contents of "admin" in "bin1". If you want the "admin" folder to end up in "bin1" (ie. "bin1/admin") then use: "rsync -Pva /mnt/volume1/admin /mnt/bin1/". As far as I'm aware, there isn't a difference on BSD and GNU systems. The "-P" means that you can resume the transfer if you need to stop for some reason, and also prints status info, and the "-a" is similar to the "-a" of "cp". The "-v" just spits out more info during the transfer. Note that rsync will skip files that it sees as being the same size and date as existing files. There are flags to disable this and rely on checksumming if you want to use that to check that everything has been tranfserred correctly.

 

[TDPsGM]

Thanks for the tips fracai.

q2) be careful with shell expansion like this. I've done this to copy the contents of my home folder before and forgotten that the * will not grab dot files. It's safer to to copy the containing folder, specify '.*' as well, or use another tool

Just to make sure I understand it the code would have looked like this:

Code:

cp -a /mnt/volume1/admin/.* /mnt/bin1

is that correct?

I take it that is different from what they were discussing here:
http://en.wikipedia.org/wiki/Cp_(Unix)

Where it says:

The same happens in both GNU and BSD systems if the path of the source directory ends in . or .. (with or without trailing slash).

I didn't understand what they were getting at there with the single '.' or double '..' per chance can you elaborate on it for me?

. . . q5) use rsync instead of cp. Basically, "rsync -Pva /mnt/volume1/admin/ /mnt/bin1/" . . . AND . . . The "-P" means that you can resume the transfer if you need to stop for some reason, . . .

I was actually thinking about just taking the content of the 'admin' data set and just putting it into "bin1" but I am still trying to decide if that is wise as I noticed just the other day that there are "hidden" files in that directory. I am wondering to myself if maybe I SHOULD put a 1 directory buffer between them and the data in case I screw up and or someone some how is able to get at theme and delete them ---- I am guessing that would cause me some significant grief.

Regardless, If I wanted to put the data into the 'bin1' volume directly I'd use:

Code:

rsync -Pva /mnt/volume1/admin/ /mnt/bin1/

====> with rsync the trailing slash after 'bin1' as the target is important?

I didn't use it with the cp command, and I just want to make sure that I understand the difference.

also the -P means you can pause and resume . . . what would you enter in order to do that?

. . . Note that rsync will skip files that it sees as being the same size and date as existing files. There are flags to disable this and rely on checksumming if you want to use that to check that everything has been transferred correctly.

Do you have a link that shows some one how to do that?
I could google it I guess, but I just thought I'd ask in case you know of a great "how to" somewhere - would save me the time to search it out. If not I'll look when I have a little more time.

Thanks for the tips!

 

[fracai]

You should probably have a trailing slash on that example, but yes, that would copy all the hidden files.


'.' means 'current directory'
'..' means 'parent directory'

I assume this means that blah/blah/blah/. is the same as blah/blah/blah/./ and the same for blah/.. equalling blah/../


You could always use the one directory buffer and then move files up a directory once the transfer is complet. When the buffer directory is empty, including dotfiles, you're done and the buffer can be deleted.


The rsync command looks fine to me. Pause and resume is as simple as killing the process (Ctrl-C) and running the command again. Rsync won't retransfer data that looks to be complete already.


My reference for rsync checksumming would be the man page. So: http://linux.die.net/man/1/rsync
The flag would be: '-c' or '--checksum'.

 

[TDPsGM]

That's Awesome fracai. Thank You!