Hi all,
I've got a slightly odd spin on the traditional 'where did my data go' question and wonder if anyone can help me pinpoint where I might have gone wrong.
I recently upgraded my FreeNAS installation from 11.1-U3 to 11.2-U7. Before doing this, I decided to do something I've done several times in the past without incident, wanting to copy some data from a multiple redundant RAIDZ2 pool to a single stripe volume. This is to allow a new pool to be created on the new installation (basically to allow more volumes to be added to a vdev). Not a lot of spare storage around and I wanted as many drives in the new vdev as possible, so I did the following:
1) add a spare drive into the 11.1-U3 system
2) create it as a simple stripe volume (mounted as /mnt/wd4Tb )
3) rsync data onto it from the existing RAIDZ2 pool (mounted as /mnt/tank ). This done with this command: rsync -avri /mnt/tank/dir1/* /mnt/wd4Tb/
4) install FreeNAS 11.2-U7, create a new RAIDZ2 pool newtank (with more drives in the vdev), and also import the simple stripe wd4Tb volume on the new installation.
5) rsync from the stripe volume back onto a completely new created pool. This done with a similar command: rsync -avri /mnt/wd4Tb/* /mnt/newtank/dir1/
So far, so amateur, but this worked fine to shunt the data around and allow me to get the data onto a new vdev with more devices. The above worked fine, but the following produced a different result (please note that I did both the above and below 'Step 3' rsyncs from 11.1-U3 before upgrading to 11.2-U7).
1) add a second spare drive into the 11.1-U3 system
2) create it as a simple stripe volume (mounted as /mnt/wd750_1 )
3) rsync data onto it from the existing RAIDZ2 pool (mounted as /mnt/tank ). This done with this command: rsync -avri /mnt/tank/dir2/* /mnt/wd750_1/
4) install FreeNAS 11.2-U7, create a new RAIDZ2 pool newtank with more drives in the vdev, and also import the simple stripe wd750_1 volume on the new installation.
5) rsync from the stripe volume back onto a completely new created pool. EXCEPT...
When I imported the second single stripe volume wd750_1 onto 11.2-U7 and tried to view the contents, there was nothing on the filesystem. Just the . and .. directories visible with an ls -a. I almost thought that I must not have run the rsync command to populate the data onto the wd750_1 drive, so to prove I wasn't imagining things I loaded the old FreeNAS installation and checked the bash history - the rsyncs are there, along with various other commands I ran such as du -h /mnt/tank/wd750_1/ repeatedly, as I checked the data being copied successfully. There is no doubt in my mind that I ran the rsyncs and that there was data on the drive when the volume was last seen on the 11.1-U3 install. But after adding the smaller drive to the 11.2-U7 installation, it's as though there's nothing on the drive at all. The drive even appears empty if I try to view the contents on the old FreeNAS 11.1-U3 installation. It's as though the act of importing the drives into the new FreeNAS installation has somehow zeroed the filesystem.
Some other possible causes - when rsyncing onto the drives initially, they were connected over HBA, but when restoring, they were connected directly onto the SATA headers on the motherboard. I also didn't export the volumes from the old FreeNAS installation before importing to the new installation, but if this was the issue, I should have seen the wd4Tb stripe volume have the same issue as the wd750_1 stripe volume.
So, if what I'm describing is what's happened, is there any way of getting the data from the wd750_1 volume? There hasn't been a failure of any kind and I haven't created a new volume on this disk so hopefully no superblock destruction - is this an issue with the ZFS cache or something? The data I copied onto this drive isn't irreplaceble but it's an awkward job to restore it, so would appreciate any suggestions as to why the filesystem seems to be empty, but only for the second stripe volume.
Board: Supermicro X10SLL-F-O
CPU: Xeon E3-1231 V3
RAM: Crucial 2x 8GB ECC / DDR3-1600/ PC3-12800 / CL11
HBA: Dell H310 w/LSI 9211-8i P20 (IT mode)
Storage: WD Red 4Tb / Samsung 750Gb
Old version: FreeNAS-11.1-U3
New version: FreeNAS-11.2-U7 STABLE (FreeNAS.amd64) #0 r325575+c9231c7d6bd(HEAD): Mon Nov 18 22:46:47 UTC 2019
Thanks for any advice!
Rob
I've got a slightly odd spin on the traditional 'where did my data go' question and wonder if anyone can help me pinpoint where I might have gone wrong.
I recently upgraded my FreeNAS installation from 11.1-U3 to 11.2-U7. Before doing this, I decided to do something I've done several times in the past without incident, wanting to copy some data from a multiple redundant RAIDZ2 pool to a single stripe volume. This is to allow a new pool to be created on the new installation (basically to allow more volumes to be added to a vdev). Not a lot of spare storage around and I wanted as many drives in the new vdev as possible, so I did the following:
1) add a spare drive into the 11.1-U3 system
2) create it as a simple stripe volume (mounted as /mnt/wd4Tb )
3) rsync data onto it from the existing RAIDZ2 pool (mounted as /mnt/tank ). This done with this command: rsync -avri /mnt/tank/dir1/* /mnt/wd4Tb/
4) install FreeNAS 11.2-U7, create a new RAIDZ2 pool newtank (with more drives in the vdev), and also import the simple stripe wd4Tb volume on the new installation.
5) rsync from the stripe volume back onto a completely new created pool. This done with a similar command: rsync -avri /mnt/wd4Tb/* /mnt/newtank/dir1/
So far, so amateur, but this worked fine to shunt the data around and allow me to get the data onto a new vdev with more devices. The above worked fine, but the following produced a different result (please note that I did both the above and below 'Step 3' rsyncs from 11.1-U3 before upgrading to 11.2-U7).
1) add a second spare drive into the 11.1-U3 system
2) create it as a simple stripe volume (mounted as /mnt/wd750_1 )
3) rsync data onto it from the existing RAIDZ2 pool (mounted as /mnt/tank ). This done with this command: rsync -avri /mnt/tank/dir2/* /mnt/wd750_1/
4) install FreeNAS 11.2-U7, create a new RAIDZ2 pool newtank with more drives in the vdev, and also import the simple stripe wd750_1 volume on the new installation.
5) rsync from the stripe volume back onto a completely new created pool. EXCEPT...
When I imported the second single stripe volume wd750_1 onto 11.2-U7 and tried to view the contents, there was nothing on the filesystem. Just the . and .. directories visible with an ls -a. I almost thought that I must not have run the rsync command to populate the data onto the wd750_1 drive, so to prove I wasn't imagining things I loaded the old FreeNAS installation and checked the bash history - the rsyncs are there, along with various other commands I ran such as du -h /mnt/tank/wd750_1/ repeatedly, as I checked the data being copied successfully. There is no doubt in my mind that I ran the rsyncs and that there was data on the drive when the volume was last seen on the 11.1-U3 install. But after adding the smaller drive to the 11.2-U7 installation, it's as though there's nothing on the drive at all. The drive even appears empty if I try to view the contents on the old FreeNAS 11.1-U3 installation. It's as though the act of importing the drives into the new FreeNAS installation has somehow zeroed the filesystem.
Some other possible causes - when rsyncing onto the drives initially, they were connected over HBA, but when restoring, they were connected directly onto the SATA headers on the motherboard. I also didn't export the volumes from the old FreeNAS installation before importing to the new installation, but if this was the issue, I should have seen the wd4Tb stripe volume have the same issue as the wd750_1 stripe volume.
So, if what I'm describing is what's happened, is there any way of getting the data from the wd750_1 volume? There hasn't been a failure of any kind and I haven't created a new volume on this disk so hopefully no superblock destruction - is this an issue with the ZFS cache or something? The data I copied onto this drive isn't irreplaceble but it's an awkward job to restore it, so would appreciate any suggestions as to why the filesystem seems to be empty, but only for the second stripe volume.
Board: Supermicro X10SLL-F-O
CPU: Xeon E3-1231 V3
RAM: Crucial 2x 8GB ECC / DDR3-1600/ PC3-12800 / CL11
HBA: Dell H310 w/LSI 9211-8i P20 (IT mode)
Storage: WD Red 4Tb / Samsung 750Gb
Old version: FreeNAS-11.1-U3
New version: FreeNAS-11.2-U7 STABLE (FreeNAS.amd64) #0 r325575+c9231c7d6bd(HEAD): Mon Nov 18 22:46:47 UTC 2019
Thanks for any advice!
Rob
