How to copy one pool into another

[jyavenard]

Hi there.

Is there a way using freenas' GUI to copy one pool (A) (with lots of snapshots, and lots of dataset), into another (B) (currently empty)...

I wish to retain the entire content of pool A, including snapshots and datasets. E.g. once the operation is complete, pool A and pool B will be identical content wise (except that pool B is encrypted)

pool A is made of 6 disks in RAIDZ2 and so is pool B. 12 disks in total, all within the same chassis.

Thanks
JY

 

[Dusan]

You already mentioned you are familiar with zfs send/receive. Any reason why this should not work?

Code:

zfs snapshot -r pool_A@migrate
zfs send -R pool_A@migrate | zfs receive -F pool_B

EDIT: I should read more carefully, you are asking for a GUI method. Hmm, maybe you could use the the auto replication for that, but I'm not sure.

 

[jyavenard]

I was just curious on how to do it via the GUI rather than the command line.
I did:

send -Rv pool@clone1 | zfs receive -F -d pool2

it took:
real571m43.088s
user0m0.784s

sys104m46.892s

the pool is 11.7TB according to zpool list; so that's 357MB/s
actual content sent by the command zfs send: 8.17TB, so 250MB/s

that's from an unencrypted pool to an encrypted one...
I actually never paid particular attention of zfs send | receive speed; as I only ever done it over a network, and never locally (that I can remember). So not sure if that goes at full disk speed always, or there's an overhead somewhere.

Earlier during the copy (went to bed straight after starting it), CPU was showing only 11.4% idle...

 

[cyberjock]

There is overhead. The speed is definitely pretty good. I know I did 120MB/sec non-stop for 5 days straight when I migrated from one FreeNAS server to another over Gb LAN. Not really sure if the MB/s you got really say anything except that you get those speeds though. The overhead is a big unknown because of the encryption and other factors.

Cool to see nonetheless.

 

[jyavenard]

Problem is that I have no idea how long it would have taken to copy the same data to an unencrypted pool... 9 hours is a long time to wait to find out :)

 

[cyberjock]

I totally agree. That's why I said there's lots of factors. ;)

I think the only way we'd really be able to do a comparison is to setup a test bed with 2 rather large pools and do some detailed analysis. I doubt anyone has 10 disks sitting around and is willing to spend the time to find out. I'm really waiting for the RELEASE of 9.2.0 so I can do a scrub and see how scrub times change for my encrypted pool.

 

[Brosif_My_Nif]

I was just curious on how to do it via the GUI rather than the command line.
I did:




it took:
real571m43.088s
user0m0.784s

sys104m46.892s

the pool is 11.7TB according to zpool list; so that's 357MB/s
actual content sent by the command zfs send: 8.17TB, so 250MB/s

that's from an unencrypted pool to an encrypted one...
I actually never paid particular attention of zfs send | receive speed; as I only ever done it over a network, and never locally (that I can remember). So not sure if that goes at full disk speed always, or there's an overhead somewhere.

Earlier during the copy (went to bed straight after starting it), CPU was showing only 11.4% idle...

I was wondering if you could please explain a little more thoroughly how you accomplished copying between your two local pools. I am trying to accomplish the same task. I have an old pool (x12 2TB Disks) that I am trying to copy over to a new pool (x6 4TB Disks) on the same FreeNAS server. I am sorry if the answer is right in front of my eyes, but I tried some of the commands you listed and I couldn't get it. I am really only familiar with the basic shell commands, but trying to learn more.

I have tried searching the forums, but haven't found much luck on this particular topic.

Thank you for any assistance you can offer!

 

[jyavenard]

I was wondering if you could please explain a little more thoroughly how you accomplished copying between your two local pools. I am trying to accomplish the same task. I have an old pool (x12 2TB Disks) that I am trying to copy over to a new pool (x6 4TB Disks) on the same FreeNAS server. I am sorry if the answer is right in front of my eyes, but I tried some of the commands you listed and I couldn't get it. I am really only familiar with the basic shell commands, but trying to learn more.

assuming the two pools are mounted (imported) into your machine, and the pool you want to copy to is empty: If it's not empty, I suggest you detach and erase that pool and restart from scratch.

First, go into the GUI into the Storage screen, and select the pool you want to copy from.
Click at the bottom of the screen: "Create Snapshot" , make sure "Recursive Snapshot" is checked. Use the default snapshot name, or create your own: but whichever you do, write it down.

Now, open a shell to the box, either ssh to it, or use the console (don't use the web GUI Shell)
I assume the first pool is named pool1 (the one you want to copy from) and the second pool is named pool2 (the one you want to copy to)

Type:
zfs send -Rv pool1@snapshot_name | zfs receive -Fdu pool2

replace "snapshot_name" with the name of the snapshot you've just done.

Now go and do something else, it make take a while :)

 

[Brosif_My_Nif]

Thank you! It's currently transferring right now. Greatly appreciate your assistance and taking the time to write that all out for me.

 

[arnicks]

I am about to do similar task.

I am concerned however that due to other comitments and the elapsed time it will take for the data to copy, that it could be a few days/weeks before I have time to go back and swap the pools over, making the new pool the active one.

Hence, I am thinking that I want to add a step and do something like the following:

• create recursive snapshot "@migration_base"
• zfs send -Rv pool1@migration_base | zfs receive -Fdu pool2
• wait... get distracted for a week... find lots of new data on old pool...
• create recursive snapshot "@migration_mopup" to include any changes
• zfs send -Rv -i migration_base pool1@migration_mopup | zfs receive -Fdu pool2
• don't have to wait long as only incremental... complete rest of my upgrade and export old pool from system...

As far as I can make out from docs, this will send just the changes as an incremental snapshot so the new pool is fully up to date, and I don't need to do anything differently on the receive end for an incremental stream.

Is there anything I misunderstood?

Cheers
Andrew

 

[dlavigne]

Did that solution work for you?

 

[MaIakai]

I'm currently mirroring a pool, going slow but it should finish in a few hours. But so far its working, thanks for the help.

Moving my data from a 512 aligned pool to a 4k pool.
Seems to be doing 450GBs every 40 minutes.

 

[Bonnie Follweiler]

Using the GUI
I have 3 pools: tanka, tankb, tankc
I set up a periodic snapshot for tanka/graphics
I then set up a replication task to go to tankc/multmedia
(make sure to copy the Temp Auth Token before you try to set up and use the Semi_automatic set up mode)
set it up using the the ip address in the Remote Host Name and the Temp Auth Token

*please note that none of my volumes are encrypted. I will try that and let you know if it works

 

[danb35]

Holy necro-thread, Batman!

 

[Stux]

Its worth pointing out that you can do the first transfer... and then do an additional transfer to pick up any changes that happened during the very long first transfer... repeat until you're happy with the state on your destination pool.

The command line does vary for the additional ones, but following the GUI method will do it.

 

[George777]

zfs send -Rv pool1@snapshot_name | zfs receive -Fdu pool2

Thank you! I had some weird hanging failure 80% nine hours in to a local transfer and learned that if I had added a -s to the receive command, I could have resumed with a second command using the -t flag using a new-ish feature (FreeNAS 10.x, 11). Doing it over again, but I figured I'd mention that for the next person to read this post. I'd use this command:

zfs send -Rv pool1@snapshot_name | zfs receive -Fdus pool2

From the man page:

Code:

-s If the receive is interrupted, save the partially received state,
		 rather than deleting it. Interruption may be due to premature
		 termination of the stream (e.g. due to network failure or failure of
		 the remote system if the stream is being read over a network
		 connection), a checksum error in the stream, termination of the zfs
		 receive process, or unclean shutdown of the system.

		 The receive can be resumed with a stream generated by zfs send -t
		 token, where the token is the value of the receive_resume_token
		 property of the filesystem or volume which is received into.

 

[BanksiaBoy]

USING MBUFFER TO SPEED UP SLOW ZFS SEND | ZFS RECEIVE

Good article - mbuffer is already installed in FreeNAS 11.2, but not in 8.30.

So I am going to copy my 8.30 2 x 640GB disk mirror to a new disk installed in the 8.30 box using send.
Removing, and Importing that new disk in my Ubuntu 18.04 with zfs.
Removing the mirror disks from my 8.3 and putting them on the shelf.
Doing a fresh install of 11.2 to the box, with no data disks. Upgrade too messy. Lost the login. God knows what else.
Shutdown, put 4 new disks in now 11.2 box.
Make a new RAID-Z2 of those 4 x 3TB disks.
Send the pool from Ubuntu zfs to the z2 pool on FreeNAS 11.2 using mbuffer.
Upgrade the pool
Simulated all of this using VMware player.

Could have done it with 2x2x mirrors, but I want safety