NFS with VMWare Hypervisor (ESXi) 5.1 - cannot get attributes

[stubert]

Hello,

I've been scratching my head on this one ... I have been trying to connect an NFS datastore on two ESXi 5.1 boxes. The connection is successful and I am able to browse the datastore and upload/download files, but the speed is incredibly slow (very, very....VERY slow). In the midst of troubleshooting, I've updated my FreeNAS box to the latest release candidate, and installed all of the available patches for ESXi (I've also tried downgrading to earlier versions).

Here's what I know...when I log the NFS traffic on the ESXi boxes (both behave similarly when trying to connect to the FreeNAS NFS share) I get the following errors:

Code:

2013-03-02T10:08:02.571Z cpu0:2899)WARNING: NFS: 1990: Failed to get attributes 
(I/O error)
2013-03-02T10:08:02.571Z cpu0:2899)NFS: 2053: Failed to get object 36 7ec0861b b
c711aa4 75e5ba38 b77930a 4000a 0 15ded1 0 0 0 0 0 :I/O error

Code:

2013-03-02T11:09:14.224Z cpu0:2050)NFS: 4544: Received nfsstat3 code 791 for the
 procedure 3 client 0x410015601d20 sock 0x4100180b1100 server's universal addres
s 10.0.0.10.0.111

2013-03-02T11:09:14.340Z cpu0:2050)NFS: 4544: Received nfsstat3 code 2049 for th
e procedure 3 client 0x410015601d20 sock 0x4100180b1100 server's universal addre
ss 10.0.0.10.0.111

2013-03-02T11:09:14.356Z cpu0:2050)NFS: 4544: Received nfsstat3 code 2 for the p
rocedure 3 client 0x410015602090 sock 0x410018036cd0 server's universal address 
10.0.0.10.8.1

When I run nfsstat on my FreeNas box, there's an incredible number of getattr:

Code:

Client Info:
Rpc Counts:
  Getattr   Setattr    Lookup  Readlink      Read     Write    Create    Remove
        0         0         0         0         0         0         0         0
   Rename      Link   Symlink     Mkdir     Rmdir   Readdir  RdirPlus    Access
        0         0         0         0         0         0         0         0
    Mknod    Fsstat    Fsinfo  PathConf    Commit
        0         0         0         0         0
Rpc Info:
 TimedOut   Invalid X Replies   Retries  Requests
        0         0         0         0         0
Cache Info:
Attr Hits    Misses Lkup Hits    Misses BioR Hits    Misses BioW Hits    Misses
        0         0         0         0         0         0         0         0
BioRLHits    Misses BioD Hits    Misses DirE Hits    Misses Accs Hits    Misses
        0         0         0         0         0         0         0         0

Server Info:
  Getattr   Setattr    Lookup  Readlink      Read     Write    Create    Remove
     3781        60       163         0      4909     65837         8         6
   Rename      Link   Symlink     Mkdir     Rmdir   Readdir  RdirPlus    Access
        0         0         0         0         0         0        43      1682
    Mknod    Fsstat    Fsinfo  PathConf    Commit
        0        68         4         0         0
Server Ret-Failed
              119
Server Faults
            0
Server Cache Stats:
   Inprog      Idem  Non-idem    Misses
        0         0         0         0
Server Write Gathering:
 WriteOps  WriteRPC   Opsaved
    65821     65837        16

Here is my FreeNAS /etc/exports:

Code:

/mnt/Vol1/VM1/VM  -alldirs -mapall=root:wheel 10.0.0.22 10.0.0.21
/mnt/Vol1/CrashPlanBackup  -alldirs -mapall=root:wheel 192.168.1.23
/mnt/Vol1/Homes  -alldirs -mapall=root:wheel 192.168.1.23
/mnt/Vol1/Media  -alldirs -mapall=root:wheel

I have a second NIC that is configured to handle just the NFS share for the VMs ... I have also tried binding this to the other primary NIC, and the data speed issue still existed.

Debug file is available here: https://www.dropbox.com/s/mzroi9qxq1xdzjv/debug-FILES1-20130302100535.txt?m


Since both independent ESXi boxes are behaving similarly, it seems to point that the FreeNAS NFS configuration may be a bit off. However, I have a few separate NFS shares that some Ubuntu instances connect to and experience very fast, good speeds.

I'm not sure how to log NFS traffic or errors on the FreeNAS box, this seems to be the next step in order to troubleshoot, at least. Are there specific settings I can adjust for caching to help with all of the getattr calls from the ESXi boxes...?