Initial data import

[hell0un1verse]

Hello FreeNAS community,

I am new to FreeNAS, this is my first post on the forum. I have just finished assembling the hardware and planning to build a 4TBx6 RAIDZ2 pool to store all my data, which are about 3-4TB now, sitting on a couple hard drives after the Synology unit failed a month ago. I read various forum posts, stickies and FreeNAS documentations, it looks like in order to fill the data into the newly created zpool, one would usually do import disk, but my understanding is it would copy everything on the disk into the pool, my question is what if I only want to copy certain folders? How should I import folders individually instead of whole disk, since I have other stuff on it that I don't want to save to the pool.

Also, after I create empty RAIDZ2 pool, I will create empty datasets before I import data, they are mainly going to be CIFS shares, the dataset names and structure will match the folder tree of the data. My question is when importing data into pool, can I import at the root directory and the sub-folders will naturally be copied into matching dataset or do I have to import at each folder level into corresponding dataset?

Thanks in advance!

 

[Robert Trevellyan]

one would usually do import disk

A lot of people seem to run into trouble with this failing for no obvious reason.

I will create empty datasets before I import data, they are mainly going to be CIFS shares, the dataset names and structure will match the folder tree of the data

I suggest copying the data over the network after you have the shares set up.

 

[hell0un1verse]

Thanks a lot for the reply. So I guess I will just have to copy the files over from a LAN cable connected machine into each individual dataset on the FreeNAS box.

 

[Robert Trevellyan]

At least this way you'll have full control of the process ... and you only have to do it once.

 

[philhu]

yes, I have 2 nas devices I am retiring, qnap and synology

I mounted both and the freenas on a centos box and rsync'd the syn and qnap to folders on the freenas.

44TB moved in 6 days at 1G

 

[hell0un1verse]

Thanks for the replies!

Is rsync recommended way to copy into FreeNAS pool or regular "cp -rp" will do, or TeraCopy from Windows machine? Just looking for safest and recommended way to copy, since I really don't want to lose the data (family pictures/videos) during the copy process.

Thanks again.

 

[philhu]

This is the command I used for each nas:
rsync -aAXv --numeric-ids --exclude=#recycle* --exclude=#snapshot* --exclude=@eaDir* --exclude=PlexBck* --exclude=@Recycle* /NFS2/data/ /NFS-FN/data/

NFS2 was the QNAP, NFS-FN is the mounter Freenas
The exclude are due to wierd ways the qnap ans synology store snapshots, recycle bins, etc.
The exclude PlexBck is my way to NOT sync the plex media server until I am ready to do it by itself. Mine config is 3 MILLION files and takes 11 hours to copy

The --numeric-ids is forcing the same user/group as the old unit. You should then build a user with the uid/gid from the moved files on the NAS with an add user from the GUI

 

[hell0un1verse]

Thanks Phil!

The @eaDir was annoying on my failed Synology unit, guess Qnap has its own flavor of it...

Getting ready now, and will try to set up everything and feeling excited:)

 

[hell0un1verse]

Apologies that this might not be exactly related to FreeNAS.... But to my surprise, when I do a straight copy with "rsync -avh <src> <desc>" between two sets of local hard disks, I only get about 10MB/s transfer rate, is it normal?

I booted up the newly built machine with Ubuntu 16.04 LTS and did the rsync on it, the filesystem of both source and destination are ext2.

Just wondering if I missed anything obvious that causes it to be this slow.

Thanks.

 

[philhu]

It depends on file sizes. 1000 1k files will take a long time to copy, while 2 300M files will fly.

The file structure overhead is the killer.