RSNAPSHOT backup to FreeNAS 11-Release NFS share from Ubuntu Server Random Hangs

[peterfarrow]

I am using FreeNAS 11 - Release.
Exported a NFS share to backup Ubuntu and Raspbian servers

Symptom Rsnapshot backup Rsync hangs indefintiely at random points in the backup, soon after it starts.

Environment: Ubuntu 16.04 LTS server and Rasbian Wheezy server - both have the same issue, using Rsnapshot defaults.

First problem: "Operation Not permitted" on first pass caused by ACLs on ZFS, created all empty files, second run attempt, filled those files with data - so turned off ACLs at the root of the dataset on ZFS setacl -bn - that problem was fixed.
Second problem: rsync hangs randomly never completes when backing up to NFS share from freenas.

Test NFS share from second ubuntu server - backup runs fine - problem is linked to FreeNAS.

So

(i) Replaced network cards in FreeNAS server - problem remained
(ii)Installed FreeNAS on new hardware - problem remained
(iii) tried no-lock and investigated all kinds of NFS options - problem remained
(iv) Rsync from USB drive (ext4 format) in Ubuntu Server to FreeNAS server - WORKED

Ubuntu servers running EXT4, raspbian server running both FAT32 and EXT4 problem persists.

Blew out all the ACL stuff at the top of the mounts from Freenas and the problem still remained, but it did copy much faster before it hangs.

Built an Ubuntu 16.04LTS server with NFS shares and the problem goes away.
Installed ZFS on Ubuntu 16.04 and NFS shared from there with ACLS - problem goes away.

This is a FreeNAS issue, possibly a bug? If anyone has any ideas let me know, the FreeNAS box is about to be replaced by Ubuntu/ZFS if this problem is not resolved, I have spent over a week on this, and its a little frustrating because Freenas looks so good so far.

Pete

 

[joeschmuck]

Please list all your hardware. While I'm not an expert on what you are doing, it sounds could have a network issue. Are your NICs all Intel brand and also try to connect the devices directly to each other to see if that helps the hanging issue.

 

[peterfarrow]

It's not a network issue. Look at the details of my post above.

The problem is specific to Freenas. Same hardware running zfs of Ubuntu does not have this issue. It's at HP Ml110 G6 with 16G of Ram all Intel Pro 1000 nics with hp 1820G switches.

 

[joeschmuck]

It's not a network issue. Look at the details of my post above.

While I agree it's not a network hardware issue once you disclosed that you are using Intel NICs, the reason I asked is becasue non-Intel NICs are basically poorly supported by FreeBSD. While a RealTek NIC may work fantastic under Ubuntu or Windows, it works like crap under FreeBSD.

Also, while I can't say it will help, try FreeNAS 11.0-U1 which came out last night, see if that changes things. I'm not an expert on ACLs so I'm of not help I think to you.

Good luck and I hope you are able to solve this problem.

 

[peterfarrow]

SOLVED:

After several days playing with FreeNAS trying to resolve these issue I finally went for solution, as a *nix consultant my time is money at at over £75/hour the savings and benefits of FreeNAS have been completely eroded by the time spent dealing with this problems which won't go away and which are, to my mind indicative of software developers that have little experience of enterprise environments and the requirements of consultants like myself need from their software on a day to day basis.

To that end, decided that its too time-costly to be dealing with the FreeNAS issues which should not be issues, they should not be happening (I can't imagine grappling with these problems in a live environment and thankfully its just my test environment.)

Grappling with ACLs is just like picking selective seeds out of a pomegranate with a pair of tweezers, these seem absolutely like a solution looking for a problem, grappling with NFS problems and the backwards not-code-mature nature of FreeBSD which is years behind Linux (FreeBSD has dreadful hardware support) is all a bit too time costly and I don't believe its anywhere near production ready., when the cp command won't work because of ACLs this software is heading for the recycle bin.

The solution for me: Ubuntu LTS 16.04 solved all these issues, it seems fair to me that I would want to support the clear leader in this field and that is Linux rather than FreeBSD based offerings. Ironically the cash equivalent of the time I wasted grappling with bugs and problems in FreeNAS I could have bought a really nice NAS, maybe not an EMC but certainly a high end QNAP or Synology with cache SSDs and guaranteed reliability,

I did try with FreeNAS and I wanted it to be reliable and work - but it didn't. The market really doesn't need FreeNAS and I think FreeNAS really isn't ready for the market. If that sounds harsh then consider this: I can set up Ubuntu LTS to do the job of a FreeNAS box from bare metal in under 60mins, or I can buy a Synology or QNAP and have that up and running just as quick. Time is money. FreeNAS costs a lot of time, which means in real terms its very expensive, and it has problems which are not readily fixable.

 

[Ericloewe]

Given the rather large pool of rsync users on FreeNAS (almost all of whom use some sort of Linux on the other end), your "FreeNAS is to blame" thesis doesn't stick very well.

the backwards not-code-mature nature of FreeBSD

Really? Damn, I guess that makes Linux something like "experimental".

FreeBSD has dreadful hardware support

Maybe on clients, but most server hardware is well-supported (Adaptec being one major exception, but it's their loss).

it seems fair to me that I would want to support the clear leader in this field and that is Linux rather than FreeBSD based offerings.

The same Linux that took years to finally include ZFS because of "licensing" issues, instead wasting developer time on the sorry excuse for a filesystem that is btrfs, which doesn't even work properly when (not if) disks develop problems, according to its own wiki.

as a *nix consultant my time is money at at over £75/hour

Ironically the cash equivalent of the time I wasted grappling with bugs and problems in FreeNAS I could have bought a really nice NAS, maybe not an EMC but certainly a high end QNAP or Synology with cache SSDs and guaranteed reliability,

So... More than ten hours and this is what you have to show for it? Not much of a Unix consultant, then.

 

[DrKK]

FreeBSD and FreeNAS are not for everyone sir. Good luck with Ubuntu.

 

[danb35]

I can set up Ubuntu LTS to do the job of a FreeNAS box from bare metal in under 60mins

With a feature-complete web GUI? Impressive.