NFS does *not* support server-side copy with ZFS? :(

[winnielinnie]

I was excited that TrueNAS Core 13 would ship with NFS version 4.2, which supports "server-side copy". This means that copying a file within an NFS share is done on the server's side (i.e, no "round trips" over the network just to copy a file.)

However, when I tested this out, copying a large file was in fact doing a round trip from server -> client -> server, which was substantially slower than doing the same copy operation while SSH'd into the TrueNAS server.

After much reading, the only references of using server-side copy over NFS is in regards to non-ZFS filesystems on the servers (e.g, Ext4 and XFS).

I'm having trouble wrapping my head around this, since server-side copy has been implemented elsewhere, including over SMB and NFS for a while now (with traditional filesystems on the server.)

My simple setup has all the prerequisites:

• Server is using NFS 4.2
• Client is using NFS 4.2
• Client has coreutils 9.1 (9.0+ is required)
• TrueNAS Core 13.0-U1.1 (FreeBSD 13.1-RELEASE)

Here is how I mounted the share, which "works" for the most part, except for server-side copies.

This is the output of "nfsstat -m" from my client:

Code:

/mnt/nfs from 192.168.1.100:/mnt/test-pool/playground
 Flags: rw,relatime,vers=4.2,rsize=131072,wsize=131072,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,clientaddr=192.168.1.20,local_lock=none,addr=192.168.1.100

The "mount" command I issued for this test:

Code:

mount -t nfs 192.168.1.100:/mnt/test-pool/playground /mnt/nfs/

I lack the technical savvy to understand why this is the case or if I am overlooking something?

Is it true that for server-side copy to work over NFS 4.2, the underlying filesystem on the server must support reflinks?

Is there development in the works to support server-side copy over NFS if the server is using ZFS? (Perhaps by an alternative method?)



I believe this issue of server-side copy is its own topic, which I'm continuing from this older thread:

NFS Version 4.2 support

Will TrueNAS be getting NFS 4.2 support? NFS 4.2 would bring many performance increases especially for VM storage and HPC. Some info linked and attached. Network File System (NFS) Version 4 Minor Version 2 Protocol

www.truenas.com

Has anyone else tried server-side copies over NFS with TrueNAS Core 13? Any luck?

 

[Kris Moore]

Did some asking around here, not sure its supported in FreeBSD yet, might be some upstream work happening to fix this. Worth testing on SCALE though, may already work there out of box, but we're going to see about adding a new unit test eventually to ensure this is working on SCALE. If somebody else tests in the meantime before we get to it, let me know!

 

[awasb]

The server side seems to be ok, as far as FreeBSD is concerned. All that's needed is the support of the copy_file_range syscall. Once in the system sources, the base distribution will support it, AFAIK. nconnect=16 is supported now by TrueNAS Core. That's for sure.

On Linux it seems to be different, since it's just the kernel, that's centrally maintained; not all packages in user space of all distributions are "coordinated" and compiled to support that syscall. Just an example: From coreutils 9.0 onwards copy_file_range is supported. But my actual Debian 11/Devuan 4.0 boxes show coreutils-8.30-3 respectively 8.32-4. So ... it's gonna take some time, until things are upstream. (Not to mention the more "abstract" packages like DM file managers etc.)

@winnielinnie: You could try and just compile the current coreutils on a Linux client machine, just to see with some basic

Code:

cp

what is going to "happen". (Or You could install funtoo ... )

 

[winnielinnie]

From coreutils 9.0 onwards copy_file_range is supported.

I already have coreutils 9.1, fully updated Manjaro client. (Arch Linux based distro.)

My simple setup has all the prerequisites:

• Server is using NFS 4.2
• Client is using NFS 4.2
• Client has coreutils 9.1 (9.0+ is required)
• TrueNAS Core 13.0-U1.1 (FreeBSD 13.1-RELEASE)

A basic "cp" takes a round trip (as is obvious by the time-to-completion and speed).

Code:

cd /mnt/nfs
time cp bigfile.img bigfilecopy.img

The same operation on the TrueNAS Core server itself (via SSH) happens as fast as you would expect from a local copy operation.

@awasb: Are you currently using SCALE by any chance to test this on? Just read your signature. Since you're on Core 13.0-U1.1, have you tested if server-side copy works for you over NFS? Perhaps with a live ISO of an updated distro, such as Manjaro, that ships with coreutils 9.1?

* The "bigfile.img" is random data, which takes compression out of the equation.

 

[awasb]

Sorry. Overlooked that.

And again: No. Sorry. All filers here are running CORE. (And the day, SCALE will be the unified TrueNAS „solution“, I‘ll switch back to Solaris.)

Edit: I will try on a funtoo laptop and some raspian-raspis lying around "somewhere"; just need to find them. I'll report back.

 

[awasb]

Well, freshly built from git (with standard ./configure no special options)

cp (GNU coreutils) 9.1.48-e3d60

won't do. Neither on x86_64 (5.10.0-16-amd64) nor on armv7l (5.15.56-v7+). Maybe, we need to test a FreeBSD-client. (Setting up a jail won't do.)

 

[winnielinnie]

Well, freshly built from git (with standard ./configure no special options)

cp (GNU coreutils) 9.1.48-e3d60

won't do.

Maybe, we need to test a FreeBSD-client. (Setting up a jail won't do.)

I figured the quickest way to test this is with a live session of a FreeBSD-based system. I booted into the latest live ISO of GhostBSD and MidnightBSD.

What's astonishing is that even though it reports the kernel as 13.1-RELEASE, it does not support NFS version 4! Not 4.2, not 4.1, not 4.0. Only NFS version 3.

Without having to resort to installing FreeBSD onto a virtual machine for a very basic/quick test, what other options are there?


However, I highly doubt that server-side copy will work in such a test (even with a FreeBSD client), because my hunch is:

1. This is a server issue (not a client issue), or...
2. NFS 4.2 supports "server-side copy", but not if ZFS is the underlying filesystem

Did some asking around here, not sure its supported in FreeBSD yet, might be some upstream work happening to fix this.

My excitement was based on the song-and-dance about NFS 4.2 de-facto supporting "server-side copy", and TrueNAS Core 13 will follow FreeBSD 13.x releases, and thus de facto will support "server-side copy" over NFS.

 

[winnielinnie]

Dang it. I hate it when this happens...

If I didn't know any better, I'd assume that barely anyone uses ZFS on their file servers. As if ZFS is this "new kid on the block" with very little mileage and not enough real-world scenarios nor combinations of setups.

 

[awasb]

SEO @ its best!

However, I highly doubt that server-side copy will work in such a test (even with a FreeBSD client), because my hunch is:

1. This is a server issue (not a client issue), or...
2. NFS 4.2 supports "server-side copy", but not if ZFS is the underlying filesystem

I'd follow route #1. But not #2.

 

[anodos]

Code:

openat(AT_FDCWD, "/tmp/nfs4bsd/stuff", O_RDWR) = 3
openat(AT_FDCWD, "/tmp/nfs4bsd/stuff2", O_RDWR) = 4
copy_file_range(3, NULL, 4, NULL, 5, 0) = 5

With TrueNAS 13 mounted via modern Linux kernel NFS client:

Code:

192.168.0.156:/mnt/dozer/NFS on /tmp/nfs4bsd type nfs4 (rw,relatime,vers=4.2,rsize=131072,wsize=131072,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,clientaddr=192.168.0.154,local_lock=none,addr=192.168.0.156)

copy_file_range() clearly works on FreeBSD 13 NFS server.

If in doubt dtrace the relevant providers (or be lazy and use `fbt::*copy_file_range:entry`):

Code:

  0  46768     nfsrvd_copy_file_range:entry time 2022 Aug 17 07:57:00
  0  46992         vn_copy_file_range:entry time 2022 Aug 17 07:57:00
  0  16603     vop_stdcopy_file_range:entry time 2022 Aug 17 07:57:00
  0  46518 vn_generic_copy_file_range:entry time 2022 Aug 17 07:57:00

 

[winnielinnie]

copy_file_range() clearly works on FreeBSD 13 NFS server.

Did you try a test by "timing" the copy of a very large file (preferably uncompressible)? Or even with many, many small/medium sized files?

I created the share and mounted it almost identical to your above example (seen in my earlier post). Whether or not the server might report that it supports this, the copy operations still make a round-trip over the gigabit network (and took substantially longer than a local copy operation when SSH'd into the server.)

@awasb noticed the same thing, even with a Linux client using coreutils version 9.1

 

[anodos]

Did you try a test by "timing" the copy of a very large file (preferably uncompressible)? Or even with many, many small/medium sized files?

I created the share and mounted it almost identical to your above example (seen in my earlier post). Whether or not the server might report that it supports this, the copy operations still make a round-trip over the gigabit network (and took substantially longer than a local copy operation when SSH'd into the server.)

@awasb noticed the same thing, even with a Linux client using coreutils version 9.1

Code:

root@truenas[/tmp/to_build]# time ./a.out
XXX: success
./a.out  0.00s user 0.00s system 0% cpu 1.088 total

That was copy of 800MiB file. These are two VMs connected via gigabit ethernet. Your application is probably not using copy_file_range(2).

 

[winnielinnie]

Your application is probably not using copy_file_range(2).

My application is "cp". Plain old "cp" of coreutils 9.1.

@awasb also tried it from a freshly compiled coreutils 9.1.48-e3d60 in post #6 earlier above. Same issue.

From the coreutils 9.0 release announcement back in September 2021:

This is to announce coreutils-9.0, a stable release.

This is a new major release, with these significant changes:
- cp has changed how it handles data
- enables CoW by default (through FICLONE ioctl),
- uses copy offload where available (through copy_file_range),
- detects holes differently (though SEEK_HOLE)
- This also applies to mv and install.

 

[anodos]

My application is "cp". Plain old "cp" of coreutils 9.1.

@awasb also tried it from a freshly compiled coreutils 9.1.48-e3d60 in post #6 earlier above. Same issue.

From the coreutils 9.0 release announcement back in September 2021:

Well, you'd have to strace your application and see whether it's working correctly.

 

[anodos]

To clarify, the above example was tested with Linux NFS client and confirms that when application makes correct system call on the correct type of mount, the correct kernel function is called on the FreeBSD NFS server. Beyond those steps, there is nothing else that can be done server-side.

 

[awasb]

[...]
With TrueNAS 13 mounted via modern Linux kernel NFS client:
[...]

What _exactly_ do you concern "modern"? The System call got implemented with kernel 4.x (don't remember the exact minor). Please share the secret of your kernel source's version, eventually patches and distribution.

Just to try to "re-build" it.

 

[anodos]

What _exactly_ do you concern "modern"? The System call got implemented with kernel 4.x (don't remember the exact minor). Please share the secret of your kernel source's version, eventually patches and distribution.

Just to try to "re-build" it.

root@truenas[/tmp/to_build]# uname -a
Linux truenas 5.15.45+truenas #1 SMP Fri Jul 29 06:08:31 UTC 2022 x86_64 GNU/Linux

I used the NFS client in a SCALE nightly. I did not test with `cp` because I was isolating the particular system call.

 

[awasb]

That makes perfectly sense. Thank you.

 

[anodos]

That makes perfectly sense. Thank you.

If you want to do your own testing, you can use the os module in python 3.8:

os — Miscellaneous operating system interfaces

Source code: Lib/os.py This module provides a portable way of using operating system dependent functionality. If you just want to read or write a file see open(), if you want to manipulate paths, s...

docs.python.org

This is probably the simplest way to test whether the operation works on your particular client + server combo.

 

[winnielinnie]

Well, you'd have to strace your application and see whether it's working correctly.

More testing + strace revealed something... really, really bizarre. I was even able to reproduce this issue consistently and I'm not sure what to make of it.


----------


@awasb you will be interested in this as well:

For some reason "cp" (from coreutils 9.1) inspects the filetype before the actual copy operation.

Certain filetypes it will use the slow, round-trip over the network with read/write (I could reproduce this in 100% of the tests with .mp4 videos.)

Other filetypes (such as .tar.gz), it will always use copy_file_range.


----------


What the heck? I thought "cp" was a simple utility. It's as if it tries to do more than what the user asks of it. Why should it care if it's copying a video file or tarball? (That's why I'm curious if others can reproduce this as well.)

It seems to inspect the filetype (rather than simply read the file extension), since renaming the extension from .mp4 to .dat it will still use read/write. The inverse is also true. Renaming .tar.gz to .mp4 it will still use copy_file_range.


----------


@awasb you can test this yourself to see if you get the same results.

Repeat this test twice (once with a legit .mp4 video, and again with a legit .tar.gz file.)

time cp archive.tar.gz archive2.tar.gz
Really fast, uses copy_file_range (server-side copy)

strace cp archive.tar.gz archive2.tar.gz
strace reveals it uses copy_file_range

time cp video.mp4 video2.mp4
Slow, uses read/write (round-trip over network)

strace cp video.mp4 video2.mp4
strace messages are flooded with incessant read/write


----------


I'm not sure why my initial test with random data did not use copy_file_range.