SOLVED My files disappeared on reboot.

[Joe Goldthwaite]

Hi,

This is probably an issue that my FreeNAS box inherited from my previous configuration. I originally built the box using Ubuntu with ZFS on Linux. I created a single pool called mediapool. Underneath that I created two file systems, "backups" and "media". Somehow I managed to bork things up so that when I would reboot Ubuntu, everything in /mediapool/media would disappear.

I think the problem is that when I copied the data to /mediapool/media, the media folder wasn't mounted so the data got copied to a folder called "media" in the root of mediapool. When I would reboot, and empty folder got mounted to /mediapool/media and all my data disappeared. Every time I would reboot the machine I would manually execute "zfs umount mediapool/media" and all my data would show up again.

While this was going on, I built a second machine using FreeNAS. I liked it so much that I decided to upgrade the first machine from Ubuntu over to FreeNAS. I disconnected the Ubuntu boot drive and booted FreeNAS from a USB drive. I was able to import the mediapool and everything was golden. The mount issues seemed to go away, at least until today.

Today I powered the server down so I could blow out the built up dust. When I booted it back up, the /mnt/mediapool/media directory is empty again. The backup directory, /mnt/mediapool/backup is fine. It looks like my old issue has reared its ugly head again.

I tried my old fix "zfs umount /mnt/mediapool/media" but I get a message "cannot umount '/mnt/mediapool/media': Device busy". I've tried the -f option but that doesn't seem to work with a busy device. I've tried finding what process has it in use but it's not finding anything.

Sorry this has gotten so long. I'm not even sure what question to ask. How do I get past the device busy issue? Does anyone know how I would keep the mount from happening during system startup? I think either of those would help me trace it down.

Let me now if there's any other information I can provide that would help trace the issues down.

 

[depasseg]

Can you paste the output of zfs list in CODE tags? Thank you!

 

[Joe Goldthwaite]

I'd be happy to! The media server is at home and I'm normally at work so I wasn't able to reply until this evening. Just in case, I included all the other mount related commands I could think of.

Code:

[root@mediaserver2 /mnt/mediapool/backups]# zfs list   
NAME  USED  AVAIL  REFER  MOUNTPOINT
freenas-boot  530M  27.1G  31K  none
freenas-boot/ROOT  521M  27.1G  25K  none
freenas-boot/ROOT/default  521M  27.1G  521M  legacy
freenas-boot/grub  6.78M  27.1G  6.78M  legacy
mediapool  15.0T  3.08T  9.93T  /mnt/mediapool
mediapool/.system  414M  3.08T  322M  legacy
mediapool/.system/configs-5ece5c906a8f4df886779fae5cade8a5  329K  3.08T  329K  legacy
mediapool/.system/cores  72.1M  3.08T  72.1M  legacy
mediapool/.system/rrd-5ece5c906a8f4df886779fae5cade8a5  329K  3.08T  329K  legacy
mediapool/.system/samba4  15.4M  3.08T  15.4M  legacy
mediapool/.system/syslog-5ece5c906a8f4df886779fae5cade8a5  4.15M  3.08T  4.15M  legacy
mediapool/backups  5.05T  3.08T  5.05T  /mnt/mediapool/backups
mediapool/jails  8.61G  3.08T  393K  /mnt/mediapool/jails
mediapool/jails/.warden-template-pluginjail  923M  3.08T  922M  /mnt/mediapool/jails/.warden-template-pluginjail
mediapool/jails/plexmediaserver_1  7.71G  3.08T  8.61G  /mnt/mediapool/jails/plexmediaserver_1
mediapool/media  274K  3.08T  274K  /mnt/mediapool/media

[root@mediaserver2 /mnt/mediapool/backups]# zfs get mounted   
NAME  PROPERTY  VALUE  SOURCE
freenas-boot  mounted  no  -
freenas-boot/ROOT  mounted  no  -
freenas-boot/ROOT/default  mounted  yes  -
freenas-boot/grub  mounted  yes  -
mediapool  mounted  yes  -
mediapool/.system  mounted  yes  -
mediapool/.system/configs-5ece5c906a8f4df886779fae5cade8a5  mounted  yes  -
mediapool/.system/cores  mounted  yes  -
mediapool/.system/rrd-5ece5c906a8f4df886779fae5cade8a5  mounted  yes  -
mediapool/.system/samba4  mounted  yes  -
mediapool/.system/syslog-5ece5c906a8f4df886779fae5cade8a5  mounted  yes  -
mediapool/backups  mounted  yes  -
mediapool/jails  mounted  yes  -
mediapool/jails/.warden-template-pluginjail  mounted  yes  -
mediapool/jails/.warden-template-pluginjail@clean  mounted  -  -
mediapool/jails/plexmediaserver_1  mounted  yes  -
mediapool/media  mounted  yes  -

[root@mediaserver2 /mnt/mediapool/backups]# zpool history mediapool   
History for 'mediapool':
2014-11-06.12:52:28 zpool create mediapool raidz2 sda sdb sdc sdd sdf sdg sdh sdi sdj -f
2014-11-06.12:57:10 zpool export mediapool
2014-11-06.12:57:40 zpool import -d /dev/disk/by-id mediapool
2014-11-06.13:02:56 zfs create mediapool/backups
2014-11-06.13:03:06 zfs create mediapool/media
2014-11-06.13:39:14 zfs set mountpoint=/zfs mediapool/backups
2014-11-06.13:39:22 zfs set mountpoint=/zfs mediapool/media
2014-11-06.13:41:50 zfs set mountpoint=/zfs mediapool/media
2014-11-06.13:42:00 zfs set mountpoint=/zfs mediapool/backups
2014-11-06.13:48:41 zfs set mountpoint=/mnt mediapool/backups
2014-11-06.13:49:11 zfs set mountpoint=/mnt mediapool/media
2014-11-06.13:58:23 zfs set mountpoint=/mnt mediapool/media
2014-11-06.14:09:59 zfs set mountpoint=/mediapool mediapool/media
2014-11-06.14:10:05 zfs set mountpoint=/mediapool mediapool/backups
2014-12-16.20:59:58 zpool import -N mediapool
2014-12-16.21:42:41 zpool import -N mediapool
2014-12-16.21:47:57 zpool import -N mediapool
2014-12-17.21:18:28 zpool import -N mediapool
2014-12-20.11:20:20 zpool import -N mediapool
2014-12-27.11:04:02 zpool import -N mediapool
2015-01-01.16:14:17 zpool import -N mediapool
2015-01-04.21:09:01 zpool import -N mediapool
2015-01-12.17:28:39 zpool import -N mediapool
2015-01-22.21:18:08 zpool import -N mediapool
2015-04-26.14:11:26 zpool import -N mediapool
2015-07-26.21:44:27 zpool import -N mediapool
2015-08-01.09:09:49 zpool import -N mediapool
2015-08-01.09:36:54 zpool import -N mediapool
2015-08-01.13:01:06 zfs inherit mountpoint mediapool/backups mediapool/media
2015-08-01.13:37:33 zfs inherit mountpoint mediapool
2015-08-01.13:39:11 zfs set mountpoint=/mediapool mediapool
2015-08-01.14:53:04 zpool export mediapool
2015-08-01.14:54:08 zpool import -N -d /dev/disk/by-id mediapool
2015-08-01.14:54:39 zfs inherit -r mountpoint mediapool
2015-08-01.14:58:13 zfs inherit -r mountpoint mediapool
2015-08-01.15:05:41 zpool export mediapool
2015-08-01.15:06:32 zpool import -N -d /dev/disk/by-id mediapool
2015-08-01.15:07:02 zfs inherit -r mountpoint mediapool
2015-08-01.15:17:54 zpool export mediapool
2015-08-01.15:18:51 zpool import -N -d /dev/disk/by-id mediapool
2015-08-01.15:19:04 zfs inherit -r mountpoint mediapool
2015-08-01.19:35:59 zpool import -N mediapool
2015-11-07.15:56:29 zpool import -f -R /mnt 10878629456800793981
2015-11-07.15:56:29 zfs inherit -r mountpoint mediapool
2015-11-07.15:56:29 zpool set cachefile=/data/zfs/zpool.cache mediapool
2015-11-07.15:56:29 zfs set aclmode=passthrough mediapool
2015-11-07.15:56:34 zfs set aclinherit=passthrough mediapool
2015-11-07.15:57:24 zfs create -o mountpoint=legacy mediapool/.system
2015-11-07.15:57:24 zfs create -o mountpoint=legacy mediapool/.system/cores
2015-11-07.15:57:25 zfs create -o mountpoint=legacy mediapool/.system/samba4
2015-11-07.15:57:25 zfs create -o mountpoint=legacy mediapool/.system/syslog-5ece5c906a8f4df886779fae5cade8a5
2015-11-07.15:57:25 zfs create -o mountpoint=legacy mediapool/.system/rrd-5ece5c906a8f4df886779fae5cade8a5
2015-11-07.15:57:30 zfs create -o mountpoint=legacy mediapool/.system/configs-5ece5c906a8f4df886779fae5cade8a5
2015-11-09.18:27:27 zpool upgrade mediapool
2015-11-11.20:38:10 zfs create mediapool/jails
2015-11-11.20:39:42 zfs create -o mountpoint=/mediapool/jails/.warden-template-pluginjail -p mediapool/jails/.warden-template-pluginjail
2015-11-11.20:41:36 zfs snapshot mediapool/jails/.warden-template-pluginjail@clean
2015-11-11.20:41:44 zfs clone mediapool/jails/.warden-template-pluginjail@clean mediapool/jails/plexmediaserver_1
2016-01-31.14:49:30 zpool import -c /data/zfs/zpool.cache.saved -o cachefile=none -R /mnt -f 10878629456800793981
2016-01-31.14:49:30 zpool set cachefile=/data/zfs/zpool.cache mediapool
2016-01-31.20:52:25 zpool import -c /data/zfs/zpool.cache.saved -o cachefile=none -R /mnt -f 10878629456800793981
2016-01-31.20:52:25 zpool set cachefile=/data/zfs/zpool.cache mediapool


[root@mediaserver2 /mnt/mediapool/backups]# df /mnt/mediapool   
Filesystem  512-blocks  Used  Avail Capacity  Mounted on
mediapool  27950393274 21332983523 6617409751  76%  /mnt/mediapool

[root@mediaserver2 /mnt/mediapool/backups]# df /mnt/mediapool/media   
Filesystem  512-blocks Used  Avail Capacity  Mounted on
mediapool/media 6617410300  549 6617409751  0%  /mnt/mediapool/media

[root@mediaserver2 /mnt/mediapool/backups]# df /mnt/mediapool/backups
Filesystem  512-blocks  Used  Avail Capacity  Mounted on
mediapool/backups 17472194530 10854785492 6617409038  62%  /mnt/mediapool/backups

 

[Joe Goldthwaite]

That was all probably too much information. I'll try to simplify the question. This is what I think is going on;

1. I created a mediapool and then I created a media folder in it.
2. I copied a bunch of data into /mnt/mediapool/media
3. At some point I think I managed to get an empty folder mounted at /mnt/mediapool/media. When that folder is mounted, I can't see any of my data.

So the question is, how can I stop the mounting of the empty folder in /mnt/mediapool/media. I've done some searching but I haven't been able to figure out where in the boot process the mounting of secondary file systems occurs. I checked /etc/fstab. This is what's in it;

Code:

[root@mediaserver2 ~]# cat /etc/fstab
freenas-boot/grub /boot/grub zfs rw,noatime 1 0 

I think the mounting is being done in ZFS but I don't know where. Can anyone point me in the right direction?

Thanks

 

[Joe Goldthwaite]

I got my data back. I had been playing with the PlexMediaserver plugin. I had to stop it and stop the jail I created when I set it up. After that I was able to do the "zfs umount /mediapool/media" command. It completed and now all my data is visible again. I know how to manually fix it in the future. I guess eventually I'll back up everything and rebuild it from scratch.

Thanks

 

[SweetAndLow]

I would rename the folder 'media' to be something else, then reboot and move the data in your folder into the dataset you created. You should store your files in a dataset under the root dataset not in a folder in the root dataset.

 

[Joe Goldthwaite]

That's a great idea! I should have thought of that. Thanks.