Faster method of transferring lots of small files.

[ShimadaRiku]

4TB Mirror, encrypted, 30% full
16GB ECC
xeon 1231
CIFS
1Gbps LAN

Lots of small documents: ~ 200KB/s - 2MB/s
Large files : ~60MB/s to 90MB/s

I understand CIFS processes 1 file at a time, there are overhead associated with small file transfer with zfs etc and disk i/o. So, what are alternatives to transferring small files faster to FreeNAS?

 

[Bidule0hm]

Do a tarball with the files, transfer it, extract the files :)

 

[DifferentStrokes]

Do a tarball with the files, transfer it, extract the files :)

Good idea.

 

[ShimadaRiku]

Do a tarball with the files, transfer it, extract the files :)

I could use tar over ssh to pack, send, and unpack seamlessly on a *nix machine, but is there a windows utility to facilitate this? Maybe I'll have to use 7z to pack and manually extract on freenas after.

 

[Bidule0hm]

Yep, just use some archive software, any archive software should be able to do tar archives, if it can't then delete it...

On the server side you can use tar -xf the_tarball_name.tar

 

[cyberjock]

Except you'll lose time doing the tarball. If you're just trying to get the data copied to the server, just copy it. Having to read it so you can make the tarball, write the tarball, then read back the tarball is not going to be faster than simply reading the files once.

 

[Bidule0hm]

Of course it depends on the time you need to make the tarball and then extract the files. I guess it can be useful is some cases but not others.