Backing Up to External USB from FreeNAS

seb101

Contributor
Joined
Jun 29, 2019
Messages
142
From reading all the Googled threads on this topic, I feel like I'm going to get a pile of abuse for this post... but stick with me.

Backing up to an external USB drive directly from the FreeNAS host hardware is a sought after feature and a real use-case for home users. It has two advantages:
1) it is a backup, and if the sh*t hits the fan in your main Zpool any backup is better than no backup
2) it is easily man-portable and can be swapped in and out to store off-site or quickly grabbed in the case of a disaster.

The typical community response at this point is that plugging USB drives into FreeNAS should be avoided and you should do the backup from a client PC on the network. Fine, but some people don't have PCs always-on in their house, or perhaps their only PC is the laptop that they occasionally need to take out of the house yet still want backups to continue. For whatever reason, lets keep the faith that as many threads suggest, backing up to a locally attached USB drive is actually a sought after feature for home users no matter how unsuitable it might be for enterprise level deployments.

So the next, often touted, issue is that supposedly ZFS over USB is baaaaaaaaad. Though even the experts don't seem to agree on exactly why this is. However the argument that ZFS is not particularly compatible with any subsequent data recovery use-case (reading the files on your Windows laptop for example) is compelling enough to rule out ZFS as the file-system of choice for this purpose. In any case, it probably makes more sense in a home-brew one-shot backup solution to have the backup on a different file-system to avoid a fs level corruption ruining everything.

So, the feature request is now getting narrowed down. Can we backup to a non-ZFS, locally attached USB device?

Most people now seem to turn to ext3/4, the Linux-Lover's fs of choice. There is a lot of commentary out there from FreeNAS ultras about why ext is baaaaaaaaad. However the simple fact that modern ext4 is not supported on the FreeNAS FreeBSD current kernel is enough to rule this out too.

What's left? Is there a file system that FreeNAS/BSD can mount, is generally considered to be 'okay' for use on USB drives and readable by most consumer OSes?

Ruling out NTFS for Linux/BSD write compatibility and FAT32 for the file-size limits, it would appear to leave one contender... exFAT.
Supported natively in Windows and MacOS, supported in Linux with exfat-fuse and supporting in FreeBSD with fusefs-exfat.

So, the ultimate ask:
Is it possible to configure FreeNAS, somehow, to mount an exFAT formatted USB drive and create an Rsync type cron backup for:
a) an entire pool
b) an entire dataset
c) specific paths in a dataset


Thanks for reading, I hope this turns into a useful discussion.
 
Last edited:

seb101

Contributor
Joined
Jun 29, 2019
Messages
142
I keep doing the research on this.

Installing fusefs-exfat in a jail or backing up via a VM is not an option as the USB passthrough does not work.

It would seem to need fusefs-exfat installed on the FreeNAS base system. Which is going to need either a custom compile of FreeNAS, or for fusefs-exfat to be included in future releases. With my limited experience of compiling embedded BusyBox systems I think I can probably get this up and working, however it's not ideal as it will require constant maintenance to keep it aligned with the main FreeNAS releases.

How can I get this added as a feature request for future FreeNAS releases?
 

garm

Wizard
Joined
Aug 19, 2017
Messages
1,555
Is there a file system that FreeNAS/BSD can mount, is generally considered to be 'okay' for use on USB drives and readable by most consumer OSes?
Ya, ZFS. Only one “consumer” OS product is unable to read it. Launching a Ubuntu live USB should be easy on any laptop
Is it possible to configure FreeNAS, somehow, to mount an exFAT formatted USB drive
No, FreeNAS doesn’t support it and it would be a poor backup “tape”.

The reason people frown on USB is because it dosent include the whole sata bus. ZFS is an awesome filesystem when it is allowed to control the disks, take that away and it is crippled.

Having said that, I still believe it’s far superior to anything else and there is a resource page here about having a USB drive backup pool. There are some considerations to account for such as scrubs and how to safely detach it.
 

seb101

Contributor
Joined
Jun 29, 2019
Messages
142
Could you link to the resource page? Thanks.
 

kingc

Dabbler
Joined
Jul 2, 2019
Messages
17
I think garm is referring to this: https://www.ixsystems.com/community/resources/how-to-backup-to-local-disks.26/

Can we backup to a non-ZFS, locally attached USB device

I expect the response would be that FreeNAS is an appliance that doesn't officially support this. And this would be correct.

That being said, I agree that many home users running smaller systems need to back up to usb disks. I'm one of them. Having spent a fair amount of time looking at options to achieve this, there are only 2 possibilities I've come up with to back up to non-zfs drives:

1. Pull the data off your FreeNAS using another device. This could be a simple device, like a Raspberry Pi. This gives you a lot of flexibility (use whatever drive/filesystem you want), but may be slow because you may be limited by your network transfer speed and small devices like RPi's are typically further constrained in terms of network and usb performance.

2. You can actually mount NTFS-formatted drives on a FreeNAS host. This uses the built-in ntfs-3g and fuse. Performance probably won't be all that great (haven't tested this). If you think of FreeNAS as an appliance, this is a hack that may stop working at any point.

Nevertheless, to do this, add the following to /boot/loader.conf to automatically load the fuse ko:
fuse_load="YES"

Reboot.

Then, you can create a mountpoint (note: the mountpoint probably won't persist across reboots):
mkdir /mnt/<your_mountpoint>

And mount:
ntfs-3g -o rw /dev/<your_device_slice> /mnt/<your_mountpoint>
 
Last edited:

seb101

Contributor
Joined
Jun 29, 2019
Messages
142
To be clear, my use-case for this is as follows. I have about 10TB of data, this roughly breaks down as follows:
A) 5TB of stuff that would be annoying to lose but relatively easy to replace (rips of media, software library, things that can be re-ripped or re-downloaded pretty easily).
B) 4TB of stuff that would be really hard to replace but possible with time and effort (rips of my home-video tapes, old digitized photos, the seemingly bottomless pool of 'unsorted' digital photographs from the last 10 years that are also still on SD/CF cards somewhere)
C) 1TB of stuff that is truly irreplaceable (documents, the 'family album' digital photographs, code projects etc).

A - I don't backup, I have a mirrored pool to mitigate the pain-in-the-ass that losing this data would be, but I don't want to spend any money on it.
C - Is on a mirrored pool and I am remotely syncing this with S3 in two AWS regions. This needs a proper off-site and I'm willing to pay for it.

It's the data in B that I look to the USB solution for. I don't care enough to pay for offsite storage, but I care enough that I want to be able to add one more layer of protection from a NAS corruption, and rescue this easily in the event of a fire/disaster.

My old Netgear NAS, despite all it's other failings, had a good solution to this. You could schedule backups jobs to NTFS/FAT USBs from the UI. It was robust enough that it would fail gracefully with an alert if you'd forgotten to attach the USB drive for the backup window.
 

seb101

Contributor
Joined
Jun 29, 2019
Messages
142

This guide doesn't mention USB, rather e-SATA. I presume the principles would be the same.

I think the main issue with ZFS for USB backup is that it invisibly performs a lot of maintenance on the FS which is great in a high-availability system, but not so much when you might pull the drive at any moment. If you're not actively writing to an exFAT mount then it is, for all intents and purposes, fairly safe to hot-pull the drive at any moment. With ZFS you never know whether the drive is in the middle of a scrub or another maintenance task and could be corrupted by a hot-pull at that very moment. Or am I misunderstanding ZFS? Is it safe to hot-pull drives like this?
 

garm

Wizard
Joined
Aug 19, 2017
Messages
1,555
Is it safe to hot-pull drives like this?
Neither systems are safe to just yank out of the bus.

This guide doesn't mention USB, rather e-SATA
eSATA is definitely better, I use that exclusively for disks outside of the case as for instance if I run disk cloning. Had I done backups to external disks I would use eSATA.
 

seb101

Contributor
Joined
Jun 29, 2019
Messages
142
Neither systems are safe to just yank out of the bus.
Not safe but safer. If your USB attached exFAT drive is just doing a short nightly rsync task, then it's likely that for the rest of the time it's gone into power saving mode, retracted the drive heads and spun down. You're much better off pulling this drive in an emergency than one that might be doing a maintenance task.
 

seb101

Contributor
Joined
Jun 29, 2019
Messages
142
So putting my thoughts together on this.

Would it be possible to have a pair of ZFS formatted USB drives that have the same gptid/zpool config (perhaps initially cloned via DD?) so to the OS, look like they are the same Drive then write some scripts that on a cron:

mounts the USB drive
imports the zpool from the USB drive
rsyncs the required files
exports the zpool
unmounts the drive

That way, you could be sure if you pull your USB outside of a backup window, it's pretty safe. You can swap the drives around on a schedule and offsite one at a time. You could even work into the script a weekly scrub.

Would the mount points be predictable enough to fully automate that?
 

kingc

Dabbler
Joined
Jul 2, 2019
Messages
17
There shouldn't be anything to do to mount the usb drive, you just need to import the pool.

The script in the resource linked to is quite a fitting example. The author has two different disks with different pool names (you could probably create 2 pools with the same name since you wouldn't have them both imported at once; I wouldn't clone the disks though). You can see how the mountpoints are derived etc.

There's no reason you can't automate the entire process. Depending on your schedule, maybe add in some notifications (email) to let you know once the backup starts and finishes so you know when it's safe to pull the disk.
 

seb101

Contributor
Joined
Jun 29, 2019
Messages
142
OK. I'm new to ZFS and wasn't sure if there is any persistent state management done on the host side that might cause it to derp if a zpool with the same name but a different disk ID was imported.

I will give this a go.
 

InfoSecMan

Cadet
Joined
Sep 7, 2019
Messages
1
OK. I'm new to ZFS and wasn't sure if there is any persistent state management done on the host side that might cause it to derp if a zpool with the same name but a different disk ID was imported.

I will give this a go.

Did you get anywhere with this?
 

vafk

Contributor
Joined
Jun 22, 2017
Messages
132
Good morning everybody. I also like to ask on this subject:

There is a given pool on FreeNAS which contains images (pictures with SMB share linked to Windows Pictures).
The external hard drive formatted with NTFS is mounted on mnt/usb, readable and writable from console.
The task be done is to sync this folder with the external USB hard drive.

What commands or even better GUI settings have to be done that:

1) pool mnt/nas1/pictures replicates (or copies) to /mnt/usb/
and
2) from time to time when the USB drive is reattached to FreeNAS the folder pictures gets synced with the pool

To make the answer simple for, this is not to discuss opinions on safety, whether or not it is recommended or not, or that if it is a bad idea using USB hard drives (as in the answers above) but to answer how to tell FreeNAS to do this lets call it backup or even simpler copy-to-USB-task.

To anticipate why not use ZFS on the Harddrive because in case one travels likes to take the HDD and access it with Windows laptop. To anticipate why not do the copy from Windows laptop because it was seen at a small company where they had a small QNAP and each Friday the IT guy swapped the HDD and the system automatically replicated the predefined directories. Yes, this was a different filesystem (Windows SMB Server) and it suited the purpose but I wonder if FreeNAS can handle it also that easy. Perhaps there is a simple backup software which can be run on BSD, Jail or VM and access the picture pool and do the desired copy/compare/delete.

I am also not asking to discuss all consequences that could happen and interfere with FreeNAS, but to get input for a bit of experimenting. Thank you!
 

Adrian

Contributor
Joined
Jun 29, 2011
Messages
166
Your first task - mount a USB external NTFS formatted disk as read / write and test it.
I used to do just this for my backups, but switched to ZFS pools on external disks as it was becoming difficult.
As far as I know, the only NTFS support in FreeNAS is for importing (copying content of) a NTFS disk into FreeNAS.
You didn't ask for this advice, but an incremental copy on a Windows machine (e.g. robocopy source target /IR /J /XJ /FFT /R:2 /W:1 /COPY:DT) might well achieve your backup quite easily.
 

sretalla

Powered by Neutrality
Moderator
Joined
Jan 1, 2016
Messages
9,702
There is a given pool on FreeNAS which contains images (pictures with SMB share linked to Windows Pictures).
The external hard drive formatted with NTFS is mounted on mnt/usb, readable and writable from console.
The task be done is to sync this folder with the external USB hard drive.

What commands or even better GUI settings have to be done that:

1) pool mnt/nas1/pictures replicates (or copies) to /mnt/usb/
and
2) from time to time when the USB drive is reattached to FreeNAS the folder pictures gets synced with the pool
I will do as you ask and give no advice on data security...

If all of your conditions are met (by you), then all that's needed is a script to launch an appropriate rsync command (https://www.freebsd.org/cgi/man.cgi...ath=FreeBSD+8.0-RELEASE+and+Ports&format=html) which should check for the existence of your mounted backup media, then execute rsync if it's there or quit if not.

Then schedule that script to run periodically as a task in the GUI.
 

vafk

Contributor
Joined
Jun 22, 2017
Messages
132
@Adrian
Thanks for the note of robocopy on a windows machine and I will definitely check this option if I do not get it working on FreeNAS. Menawhile I am doing it with Midnight Controller on the FreeNAS and it works.
 

vafk

Contributor
Joined
Jun 22, 2017
Messages
132
@sretalla
Thanks for rsync recommendation however I understand that rsync won't write to a NTFS formatted drive connected to FreeNAS.

I am getting at a point where I would like to understand why FreeNAS does not offer the opportunity from GUI or as a plugin to copy a given directory to a external HDD. Besides all the pros why to use ZFS on the target, if we just put that aside when we want to export a given part of the data stored in a ZFS pool to another system (NTFS) why is this not supported by BSD?

I can connect the drive to my PC but since I have the machine with FreeNAS running 24h why should it not be desired to connect any type of external media and copy to it?

Or would be the workaround to install a lite-Windows/Linux in FreeNAS WM with any type of Backup that then copies the desired directory to the external HDD?
 

Adrian

Contributor
Joined
Jun 29, 2011
Messages
166
I believe that iXsystems make their money by supplying servers, many to enterprise customers with high capacity and expensive storage systems. The sort of machine that few people are allowed to get near, let alone touch. These customers would have very little interest in attaching USB devices or Windows formatted disks at all. If any of their users wanted (and are allowed) to copy their data to external Windows formatted disks, they would do it through a Windows machine.

Many of the server customers will run TrueNAS, FreeNAS's big brother. It seems that the code bases are now merged, but exclusive TrueNAS facilities are disabled in FreeNAS. I very much doubt that many TrueNAS users are at all adventurous in adopting new releases. So FreeBSD could be said to be an extended beta test for TrueNAS.

iXsystems thus have little incentive to expend their paid developers time on keeping writing to exFAT or NTFS disks easy and working (and as I recall it has been problematic over the years for FreeBSD). The same is true of anything which is not going to end up supported in TrueNAS. Some argue that a NAS should not support anything but network storage - no plugins, jails or VMs. iXsystems obviously consider that some of their paying customers do have requirements for more than just a NAS.

You could infer from exFAT or NTFS formatted disks being not being easily written under FreeNAS that it is unsupported, may not work, and possibly hazardous.

You could convert your system to FreeBSD, carefully ensuring that you did not destroy your pool in the process, and be able to do whatever FreeBSD and its package collection support, but administration will be a lot more work, and you will really miss the ease and elegance of FreeNAS. You might find it easier to use one of the desktop FreeBSD variants, or even a LINUX distribution, but you are still going to have a lot of work.

I used to run my own FreeBSD workstations (with SMB shares and NFS exports) and FreeBSD firewalls. Then I saw the light and switched to FreeNAS and pfSense giving myself a lot less grief and much more free time. I still run a couple of stock FreeBSD systems - as Hyper-V VMs accessed via SSH. No desktop. One is where I read and write most of my email (mutt), the other is mainly a testbed. Both are set up for binary OS and package updates. Both too are used for some file manipulation tasks which are more difficult under Windows.
 

NAK

Dabbler
Joined
Feb 5, 2020
Messages
16
As I wanted to make a larger pool I created an pool with 2 usb drives as mirror. Worked for me.
 
Top