SOLVED rsync chokes on foreign characters

[zafiro17]

All - I've been running BSD servers and desktops for 14 years and have even contributed to documentation on occasion. But I'm stumped here. I'm running FreeNAS 9.2-RELEASE on dedicated hardware, an iXsystems miniNAS (lovely little box), with all data on volume /mnt/sharkstuff/...

I'd like a cron script that mounts an external USB harddrive that I've plugged into the USB port, and rsyncs a bunch of data to it. Once I know the script works I'll let cron do its work, but for now rsync is giving me trouble. It's backing up 90% of my data, but skipping any file with foreign characters in it, and my music has quite a bit of that. Here's the little script:

Code:

#!/bin/sh
mount_msdosfs -o large  /dev/da1s1 /mnt/external
rsync -av  --no-o --no-g /mnt/sharkstuff/randymon /mnt/external 2> rmonbkup-error.log
umount /mnt/external

So, every Sunday I plug in the USB drive, sudo mkdir /mnt/external, and then run the script: sudo ./bkup-my-crap.sh

The mount isn't giving me any trouble, it's a FAT32 drive formatted under Windows (I don't want to format it as UFS or anything else because the purpose of the drive is to give me access to a copy of my stuff from other (Windows, Linux, Mac) machines as FAT32 is the lowest common denominator. And I don't need time stamps or ownership/groups to be preserved (therefore: --no-o and --no-g options on rsync).

I can't figure out how to get rsync to read the files with accented characters. The FreeNAS box is UTF-8 and every machine to which it serves files is able to read access and modify those files with no problem. But rsync barfs with errors like this one:

Code:

rsync: recv_generator: failed to stat "/mnt/external/Documents/Budingen/._Bu\#314\#210dingen.pdf": Invalid argument (22)

That's a dotfile with an umlaut in it.

Not sure if the trouble is in mounting the FAT32 file system? I've had no luck with any version of
mount_msdosfs -L utf-8 so I think the problem is rsync. rsync is version 3.0.9 protocol level 30, which is supposed to be able to do --iconv translations when necessary, but I'm not convinced that's the solution.

Any ideas? Am I barking up the wrong tree?

 

[cyberjock]

Yeah, I remember having this problem myself. The solution I found.... don't use rsync.

Read my whole summary about my migration from Server 2008 R2 to FreeNAS here... http://forums.freenas.org/index.php...indows-server-2008-r2.10031/page-2#post-46971

 

[zafiro17]

Hmmm ... 'don't use rsync' is going to put a crimp in the backup/disaster recovery plans of a lot of people. Rsync is basically irreplaceable on a *nix system - nothing else can play the same role in a server system.

I'm currently looking into Tarsnap, which obliges me monthly fees and a reliance on cloud hosting. Suppose I could use a simple 'cp' command, but imagine syncing ISOs and large video or audio files. This is exactly the sort of thing where rsync should be the preferred solution and is not.

 

[cyberjock]

There "may" be a fix. I don't know. My experience with rsync isn't all it's cracked up to be but if you read through the thread I could never solve the problem. And if memory serves me right I've never seen someone else that had a fix for the problem (although many have complained about it).

 

[mjws00]

Does --iconv fail? The last post on this thread mentions: 'added “charset = utf-8” into /etc/rsyncd.conf'

You'd think there would be a significant group unwilling to just punt rsync. Good luck, I'm sure you and cyber know more about it than I do, just an interesting question from my perspective.

 

[zafiro17]

I did read your doc (and your guide); thanks, they were both helpful. I'm going to reformat one of the USB drives as UFS and see if that helps in any way, but I'm not expecting it to change anything at all. I'm also going to try the regular mount command instead of mount_msdosfs with the iocharset option to see if that helps. If anyone else has any ideas, they'd be appreciated.

 

[cyberjock]

You'd think there would be a significant group unwilling to just punt rsync.

Well, most people use ZFS replication. It's extremely fast and only the changed data blocks are transferred. You also get all of the benefits of snapshots and such. When being put against rsync the winner is pretty clear. Of course there's a problem if your source and destination aren't ZFS though. ;)

 

[zafiro17]

I've made some progress and will label this thread 'solved' in hopes others find this info useful. I still don't know why rsync wouldn't transfer files with accented file names to a FAT32 drive, but it seems the FAT32 filesystem is at least part of the problem.

1) Last night I opened up an account at http://rsync.net (how nice to work with real Unix guys that know their stuff), and rsync on my FreeNAS was able to transfer absolutely everything with no problems - even files with Russian titles. It was as simple as rsync -av folder/location me@rsync.net:

2) I also reformatted one of my external USB hard drives to UFS2, and rewrote the script to mount the drive as UFS rather than FAT32. (I'll paste it below). And that too worked absolutely flawlessly. So the error has something to do with FAT32 not liking the accented characters, not rsync. So I conclude, "you've got to sync to a Real Man's file system"

By backing up to a UFS2 file system I lose the ability to read those external drives on Mac, Linux, even Windows machines, since none of them reads UFS file systems without complaining. But maybe I'll consider that a security feature - if someone ever steals that hard drive they're going to have to put together a FreeBSD system in order to read it. Ha ha!

Code:

#!/bin/sh
mount /dev/da1s1a /mnt/external
rsync -av /mnt/sharkstuff/randymon /mnt/external 2> rmonbkup-error.log
umount /mnt/external

 

[zafiro17]

Update - I don't see a way to label this thread 'solved' but if a moderator could do that I think everyone would appreciate it.

 

[mjws00]

If you format with UFS2, you might as well format with ZFS and gain replication, snapshots, and bit-rot advantages. You completely give up portability anyway so why bother? It seems though if the rsync.net guys had no ideas for you either it may be difficult or they defaulted to 'this is easier'. Nice support, but you sure do pay for it with the cost of their service.

So cyberjock punched out. You punched out. Possibly rsync.net punched out, or they don't have a use case as their back-end supports different char sets. Bit of a shame as it looked to me like you may have an excellent solution to the portability problem, and possibly the knowledge to address it.

Glad you found something that works.

 

[zafiro17]

Thanks, but I have no interest at all in running ZFS on an external USB drive - wrong tool for the job. I can do ZFS snapshots and replicate the Zpools to other NAS devices or rsync.net, but that wasn't the purpose of this exercise. If you search this forum you'll find resounding disdain for the idea of ZFS on USB devices, and I kind of agree with them.

And anyway, we all kind of suspected hte FAT32 filesystem was a toy in the first place.

 

[mjws00]

Can't fault you for that. :) Admittedly I find the poor external device support, and lack of filesystem portability a bother. I can't escape cross-platform so whenever I see a possibly elegant solution I perk up.

 

[cyberjock]

The other thing is UFS support is already removed from FreeNAS. So even doing UFS as external media isn't an option if you ever plan to upgrade FreeNAS again. :P

 

[zafiro17]

have they stripped everything but ZFS? Any support for ext3 or something? I suppose 5 years from now my media need will have grown so that external drives are no longer a useful solution and I will just use twin NASes. But it seems shortsighted to remove this functionality!

Sent from my Nexus 7 using Tapatalk

 

[cyberjock]

ext2/3 still works as does NTFS. Of course, when we say 'works' it's probably not what you want.

Long story short, the only file system FreeNAS will be expected to support, use, recommend, etc for long-term use is ZFS. ext2/3 and NTFS are supported only to the extent that you can copy your data from your disks to the pool. But if you read around you'll find that they can be buggy and they have been known to trash file systems.

 

[zafiro17]

It's been a year since this thread. Just for the sake of anyone reading this thread in 2015, I'm happy to report my system is still working great on FreeNAS 9.3. UFS support is still definitely present in FreeNAS 9.3, and I've been able to back up to external hard drives formatted to UFS for a year now with zero problems. In fact, it's lovely.