SOLVED NFS shares no longer mountable

[entyrion]

Hi all,

For some reason, my nfs shares have started timing out when I try to mount them. I can use "showmount -e" to see the shares, but "mount -t nfs ip:/mnt/sharepoint /mountpoint" results in "rpc mount export: RPC: Timed out." After receiving the timeout error, showmount -e also returns the same timeout error. Restarting the NAS or the NFS service will make showmount work again, but volumes will still not mount, then showmount breaks after the mount attempt.

This same problem occurs when trying to mount on both my laptop and desktop (both of which have been able to mount in the past). I had previously been able to mount these nfs shares, but just noticed this problem today. I have refreshed my USB install with no effect and added the ip and hostname of the NAS in Network > Global Configuration per the manual with no effect. I've also tried including "-o tcp" in the mount command, with no effect.

Is there something I'm missing here? I got a few leads from old forum posts leading me to the above, but no dice so far and I'm not sure about my next move.... :(

Thanks for any help and advice you can add!

 

[dlavigne]

Have you tried the tips in http://doc.freenas.org/index.php/Unix_(NFS)_Shares#Troubleshooting?

 

[entyrion]

Have you tried the tips in http://doc.freenas.org/index.php/Unix_(NFS)_Shares#Troubleshooting?

Yes. No effect.

 

[leenux_tux]

entyrion,

Are the shares timing out after the same amount of time, everytime they time out ? Say after 15 minutes (for example).

Just wondering if disk-spindowns might be causing some "funnies"

 

[entyrion]

Hi tux,

The amount of time tends to be about the same, perhaps 60-90 seconds or so after entering the mount command on the clients until I receive the timeout message. showmount seems to return the timeout message sooner, within about 20-30 seconds. Also as an additional note, the error messages from each are different.

showmount -e ipaddress works after rebooting or restarting the NFS service until I attempt to mount a share on a client. When I try to mount, I receive the error "mount.nfs: Connection timed out."

Then, when I try subsequent showmounts, I receive the message "rpc mount export: RPC: Timed out."

 

[leenux_tux]

ouch, that's low. kinda blown my theory out of the water !!!

Have you tried (purely as a test) sharing the same folder using CIFS at the same time. I know this can be done as I use this setup for an ISO-IMAGES folder on my FN system. Might help in pointing you in the right direction of hardware (disk problems) or software (NFS playing up).

 

[entyrion]

ouch, that's low. kinda blown my theory out of the water !!!

Have you tried (purely as a test) sharing the same folder using CIFS at the same time. I know this can be done as I use this setup for an ISO-IMAGES folder on my FN system. Might help in pointing you in the right direction of hardware (disk problems) or software (NFS playing up).

Sorry to disappoint ;) I'll try setting up CIFS when I get home (though I'm a noob and the reason I went with NFS in the first place is b/c it was easier to set that up with by Debian client computers). One question: could a problem with one of the physical disks cause something like this to occur? Of my array, 5/6 are brand new WDs, but one is an old 2TB Hitachi. It runs hot, is a bit noisy (has been for at least a year) and is probably on its last legs, but it still passes SMART tests so I figured that I would just run it until it dies to get as much life out of it as possible since I have RAIDZ2 and don't mind rebuilding at some point in the future anyways. Call me stingy, but I figure that every month of life I get out of the old Hitachi is an extra month of life I'll get out of its successor down the road :P

Now, I understand that the official party line would be to recommend that I not be such a cheap bastard and replace the darn drive, but could a problem with that one old drive REALLY have anything to do with this NFS issue? Also, I'll run short and long SMART tests on that drive when I get home too, just for an additional data point.

Thanks for your thoughts and help!!

 

[Yatti420]

Sounds like stale NFS data which can cause timeouts huge pauses etc.... Old drives can cause all sorts of problems..

 

[entyrion]

Sounds like stale NFS data which can cause timeouts huge pauses etc.... Old drives can cause all sorts of problems..

Interesting... is there some way to confirm that creeping hardware failure/stale nfs data is the cause, or other options to try before resorting to swapping out hardware? I'll still run manual SMART tests to gather additional data, and I've also considered deleting and re-creating new nfs shares with the same info, names, etc, would that perhaps help? I'll try the CIFS idea too, if I can figure out how to get all that cifs stuff working on my Debian PC....

On the other hand, is there legitimately a good chance that swapping drives could just fix this issue in one fell swoop? I'm a bit reticent not only because I want to keep the old drive in service as long as possible for cheapness' sake, but also because the backup 6th WD Green that I will swap in is a low mileage hdd from my desktop that was the former home of my entire media/data library. The same drive that I copied my entire media/data library off of, removed from the PC and is now serving as my last line of backup defense if FreeNAS blows up in my face b/c I'm a noob...

 

[Yatti420]

I would probably put all the greens in your nas.. The Hitachi for desktop use.. I have one Hitachi and it has one bad sector (1tb) probably from me hitting the computer with my chair.... Other then that no problems with it so far.. I prefer 7200 drives versus the "green" 5x00 rpm drives..

 

[entyrion]

I might bite the bullet and do it tonight. I'm getting desperate; what's the point of the NAS if I can't mount shares on my home computers? Otherwise it's little more than a Plex and Transmission server :( Thanks for your input and advice!!

 

[entyrion]

So, as an update, I ran a short SMART test on the Hitachi drive and it returned no errors, the long test said it would take about 6 hours, so I didn't bother with that. I tried deleting one of the NFS shares and recreating it, with no effect. Then I removed the old Hitachi drive, added in my new WD Green, resilvered, and the issue still persists after rebooting the NAS... no change to the symptoms. I can use showmount to see the shares until I attempt to mount, then mount and showmount time out after that, returning the error messages I mentioned before.

At this point, I'm pretty lost... I'll give the CIFS test a shot to see if the problem persists with different protocols, but I suspect this is a software issue with NFS, not hardware-related. Does anyone have any other ideas as to what might be causing this problem?

Edit: I found this guide here to clear the NFS cache... would something along these lines be worth a try? http://unixadminguide.blogspot.com/2012/07/how-to-clear-aix-nfs-cache-on-server.html

Edit2: After I found http://www.tldp.org/HOWTO/NFS-HOWTO/troubleshooting.html#SYMPTOM3 I checked the server and client with rpcinfo -p and I believe I identified the source of the problem. The server returns ports mapped to mountd, nfs, portmapper, but the client has no entry for mountd or nfs, so I believe that if I can get the nfs service running on the client, the problem should (hopefully) be solved! Now back to the manual....

 

[entyrion]

So, I finally fixed it, and it wasn't a client-side issue after all. After re-reading http://doc.freenas.org/index.php/Unix_(NFS)_Shares#Troubleshooting, I added the ip and hostname of the client to Network -> Global Configuration -> Host name database and the problem is solved! Thanks to everyone for your help and support! Now to try an figure out how to update the flag for this thread....

 

[Christopher Joyson]

Any chance of giving the syntax you used to add the client ip and hostname?

 

[leenux_tux]

Christopher,

The ip addresses(s) are added via the GUI. Check out the link " http://doc.freenas.org/index.php/Unix_(NFS)_Shares#Troubleshooting" then scroll towards the top of the web page. in the section "Creating NFS Shares" where you can find information about what to put in and where (i.e. do you want to specify a single machine or a hole network).

 

[Christopher Joyson]

I am after the syntax of how to enter the following:
If the client receives timeout errors when trying to mount the share, add the IP address and hostname of the client to the "Host name data base" field of Network → Global Configuration.

e.g. is xxx.xxx.xxx.xxx www.somesite.com correct?