Maintaining Hardlinks when migrating data from Synology to FreeNAS?

[kars85]

I'm really close to pulling the trigger and moving over to FreeNAS. This is either a really simple (stupid?) question or one that opens a can of worms for me to look into.

I have a substantial amount of media that is hardlinked - will that need to be taken into consideration when I rsync/migrate the data to the FreeNAS box? Am I correct in that the hardlink metadata is stored on the inode of the actual file(s)?

 

[Chris Moore]

Why is it that you have hardlinks?

Sent from my SAMSUNG-SGH-I537 using Tapatalk

 

[kars85]

I have data that needs to be appear to be in two places - space constraints didn't allow me to have two actual copies of the data.

 

[Elliot Dierksen]

Why not use a symlink? I believe that it is still true that you cannot hard link across different file systems.

 

[kars85]

Why not use a symlink? I believe that it is still true that you cannot hard link across different file systems.

The media management apps I use don't have an option for symlink when they post process the data. They do have an API, so theoretically anything is possible, but between Hardlink/Move/Copy as my options, hardlink has worked good so far. I know it's minuscule in comparison to data hoarders, but right now I am hardlinking about 12TB of data. Move is out of the question, and Copy didn't scale well when I was first getting started with this endeavor, so I took hardlink. In hindsight, it's saved me over $600 bucks in drive space since I don't have to actually have two sets of the data.

All of my hardlinked data is under /mnt/nfs/shared, so I was thinking something like this would work?

Code:

rsync -az -H --delete --progress --numeric-ids /mnt/nfs/shared destinationIP:/mnt/nfs/shared/

1. -a : Archive mode (i.e. recurse into directories, and preserve symlinks, file permissions, file modification times, file group, file owner, device files & special files)
2. -z : Compress file data during the transfer
3. -H : Preserve hard links (i.e. copy hard links as hard links)
4. --delete : Delete extraneous files from the receiving side (ones that aren't on the sending side), but only for the directories that are being synchronized i.e. keep exact replica of your /raid6/rsnapshot directory.
5. --numeric-ids : Transfer numeric group and user IDs rather than using user and group names and mapping them at both ends.
6. --progress : Show progress during transfer.

 

[Chris Moore]

I have data that needs to be appear to be in two places - space constraints didn't allow me to have two actual copies of the data.

I know what a link is. I guess I could have given a more detailed question. I am trying to understand what makes you need a link at all, because it is possible that with the way FreeNAS is able to work, you may not need to do links any more. What software are you using? Are you willing to consider using something else? If you run a media manager application in a jail inside FreeNAS, that jail can mount a directory on the FreeNAS storage that appears to the jail to have one path where inside FreeNAS it has another path and in a different jail the same file in FreeNAS storage can appear to have yet another path, but there are no links being used. There are ways of doing things in FreeNAS that may eliminate the problem. Please explain the cause of the problem so we can discuss solutions.
The solution is not to keep doing this the same way that you did them before.

 

[Elliot Dierksen]

Is it a file (like a database store) that is hard linked, or a directory? I don't do as much Unix sysadmin as I used to, but I do remember hard links being a real problem when data was moved. My simplest suggestion (assuming you are migrating to FreeNAS) would be to remove the hard links before you migrate the data, and then recreate them after the migration is complete. If my memory serves me correctly, cpio dealt with hard links properly but tar did not. Perhaps the preserve hard link option of rsynch would work, but I would still go for the delete and recreate option.

 

[kars85]

I know what a link is. I guess I could have given a more detailed question. I am trying to understand what makes you need a link at all, because it is possible that with the way FreeNAS is able to work, you may not need to do links any more. What software are you using? Are you willing to consider using something else? If you run a media manager application in a jail inside FreeNAS, that jail can mount a directory on the FreeNAS storage that appears to the jail to have one path where inside FreeNAS it has another path and in a different jail the same file in FreeNAS storage can appear to have yet another path, but there are no links being used. There are ways of doing things in FreeNAS that may eliminate the problem. Please explain the cause of the problem so we can discuss solutions.
The solution is not to keep doing this the same way that you did them before.

I didn't mean to slight your knowledge - I assumed you knew what a link is. I was just typing what came to mind at the time :) I could be more forthcoming with specific details, I apologize. I think at this point that'd help. The apps I'm using are Radarr/Sonarr/Plex/Deluge, all within respective docker containers on a separate bare metal Ubuntu server. Relevant volume mappings point to /mnt/nfs/shared passed through the host OS into the containers, and I drill down as each app requires (deluge for incomplete/complete download handling, radarr/sonarr for post-processing and creating a hardlink to the /mnt/nfs/shared/Movies or /mnt/nfs/shared/TVShows with the correct naming structure so the Plex Media Scanner can pick up the correct metadata).

I've been doing this for a few years, and through trial and error, I've landed on this setup. It is the most straight forward I've configured, but that's not to say there isn't a better mousetrap. Does that help clarify?

Is it a file (like a database store) that is hard linked, or a directory? I don't do as much Unix sysadmin as I used to, but I do remember hard links being a real problem when data was moved. My simplest suggestion (assuming you are migrating to FreeNAS) would be to remove the hard links before you migrate the data, and then recreate them after the migration is complete. If my memory serves me correctly, cpio dealt with hard links properly but tar did not. Perhaps the preserve hard link option of rsynch would work, but I would still go for the delete and recreate option.

They're just .mkv files. There seems to be little on this topic, so I may just have to give rsync a try for an hour and check some file inodes before continuing on.

 

[Elliot Dierksen]

They're just .mkv files. There seems to be little on this topic, so I may just have to give rsync a try for an hour and check some file inodes before continuing on.

Do watch the space consumed. The reason I brought it up was that my recollection was that things that didn't deal correctly with hard links would end up creating a copy of the file for each hard link.

 

[wblock]

rsync will preserve hard links, but not by default. The -H option does this, but there are lots of other options and getting it to preserve everything can be tricky. Plan on doing at least a limited test run before doing the whole thing, and spend some time reading the rsync(1) man page.

 

[Chris Moore]

The apps I'm using are Radarr/Sonarr/Plex/Deluge, all within respective docker containers on a separate bare metal Ubuntu server. Relevant volume mappings point to /mnt/nfs/shared passed through the host OS into the containers, and I drill down as each app requires (deluge for incomplete/complete download handling, radarr/sonarr for post-processing and creating a hardlink to the /mnt/nfs/shared/Movies or /mnt/nfs/shared/TVShows with the correct naming structure so the Plex Media Scanner can pick up the correct metadata).

All of those applications (unless I am mistaken) have versions that run in jails under FreeNAS, so the entire reason for the hardlinks might be avoided through the directory mappings that jails use.
I think @danb35 uses those apps in his system, perhaps he could comment?

 

[Chris Moore]

Additional plus to running those in jails, you only need one system instead of two. It might cut your electrical bill by a dollar.