Importing external usb drive (NTFS) to FreeNAS pool

[JohnB]

Hello All, I'm a new FreeNAS user and this is my first post.

Up to now I've been using a WD My Book 4TB external USB 3.0 drive, formatted NTFS, for backups. My goal has been to use the new FreeNAS Mini XL+ as my primary backup solution going forward. I created a dataset "wdmybook", connected the external drive to the front panel USB 3.0 port, and used the Import Disk feature to copy the backup files (~1.8 TB) to the newly created dataset.

I monitored the progress of the import over several hours, and estimated the completion time to be sometime around 3:00 AM. As you can see from this console output snippet it was pretty close:

Code:

Dec 24 15:46:13 freenas fuse-freebsd: version 0.4.4, FUSE ABI 7.8
Dec 24 15:46:14 freenas ntfs-3g[4315]: Version 2017.3.23 external FUSE 29
Dec 24 15:46:14 freenas ntfs-3g[4315]: Mounted /dev/da0s1 (Read-Only, label "My Book", NTFS 3.1)
Dec 24 15:46:14 freenas ntfs-3g[4315]: Cmdline options: ro
Dec 24 15:46:14 freenas ntfs-3g[4315]: Mount options: ro,allow_other,nonempty,relatime,fsname=/dev/da0s1
Dec 24 15:46:14 freenas ntfs-3g[4315]: Ownership and permissions disabled, configuration type 1
Dec 24 16:12:17 freenas smartd[3086]: Device: /dev/da0 [SAT], Temperature 42 Celsius reached critical limit of 40 Celsius (Min/Max 4
2!/42!)
/***additional temperature warnings removed to save space***/
Dec 25 00:00:00 freenas syslog-ng[2402]: Configuration reload request received, reloading configuration;
Dec 25 00:00:00 freenas syslog-ng[2402]: Configuration reload finished;
/***additional temperature warnings removed to save space***/
Dec 25 02:56:07 freenas ntfs-3g[4315]: Unmounting /dev/da0s1 (My Book)

But when I checked first thing this morning, the FreeNAS GUI was showing this pop-up message:

Code:

Error: Traceback (most recent call last):
  File "/usr/local/lib/python3.6/site-packages/middlewared/job.py", line 333, in run
    await self.future
  File "/usr/local/lib/python3.6/site-packages/middlewared/job.py", line 364, in __run_body
    rv = await self.method(*([self] + args))
  File "/usr/local/lib/python3.6/site-packages/middlewared/schema.py", line 664, in nf
    return await f(*args, **kwargs)
  File "/usr/local/lib/python3.6/site-packages/middlewared/plugins/pool.py", line 488, in import_disk
    raise Exception("rsync failed with exit code %r" % rsync_proc.returncode)
Exception: rsync failed with exit code 23

Aside from that I could find no evidence of any problems. I remounted the external drive, ran a 'du' on the new dataset and the external drive, and found a fairly large discrepancy (the ZFS dataset being about 25 percent smaller). Of course I wouldn't expect them to match, but I didn't expect them to be that far apart either.

The questions I now have are:
1) Is there a best way to verify the fidelity of the copy?
2) Is there a faster and/or better way to copy the external drive to the ZFS pool?
3) Any other general advice?

I did search both the Internet and the forum, but couldn't find the answers I was looking for.

Many thanks in advance for any and all help for this FreeNAS newbie.

 

[Kris Moore]

ZFS has compression on by default, so there's a chance the 25% smaller size is legit, depending on what kind of data you were importing.

There's a bunch of ways to compare if the files match, personally I'd do the following:

Mount the external storage, then run a 'diff' between the two directories.

# diff -ruN /mnt/<pathtodevice> /mnt/<pathtomydir>

That command should print out any differences between the files, however it'll need to do a full-scan, so it'll take a while. You can do similar things with rsync and other utils, so by no means is this the only way.

 

[JohnB]

ZFS has compression on by default, so there's a chance the 25% smaller size is legit, depending on what kind of data you were importing.

There's a bunch of ways to compare if the files match, personally I'd do the following:

Mount the external storage, then run a 'diff' between the two directories.

# diff -ruN /mnt/<pathtodevice> /mnt/<pathtomydir>

That command should print out any differences between the files, however it'll need to do a full-scan, so it'll take a while. You can do similar things with rsync and other utils, so by no means is this the only way.

Thanks Kris! I did consider compression, but that dataset only shows 1.14x compression ratio in the GUI, so I was still suspicious.

I ran the diff you suggested, and it bombed on an Acronis backup image file that contains a space and parentheses in the file name. That file had not been successfully imported to the FreeNAS dataset, but interestingly 4 others with the same naming scheme made it. Nevertheless I think I'm on the right path.

Thanks again!
-John

 

[TonNAS]

Thanks Kris! I did consider compression, but that dataset only shows 1.14x compression ratio in the GUI, so I was still suspicious.

I ran the diff you suggested, and it bombed on an Acronis backup image file that contains a space and parentheses in the file name. That file had not been successfully imported to the FreeNAS dataset, but interestingly 4 others with the same naming scheme made it. Nevertheless I think I'm on the right path.

Thanks again!
-John

@JohnB Were you able to get this resolved? I encountered a similar issue this morning and have not been able to find much information on how to remedy this.

 

[JohnB]

@JohnB Were you able to get this resolved? I encountered a similar issue this morning and have not been able to find much information on how to remedy this.

Yes, I did get it resolved. If I remember correctly, I just slogged through it in brute force fashion, manually copying directories and files that were missed. I'll review any notes I kept and will post a follow-up with any helpful information I find.