zpool import not able to mount

[Shadowgundam2725]

OS: FreeNAS-9.3-STABLE-201602031011
Mobo: Asus P5B-Deluxe
Processor: Intel Quad2Core Q6600 2.4 GHz
RAM: 4GB DDR2 Pc800 (will be ordering more ram but have to wait for the moment)
HDD: 2TB SSHD Seagate drive x2

I was running 9.2.1.5 as a local plex server, when a power surge came through and rebooted the system. After the reboot I could not access plugins and researched it on forums. The conclusion I kept finding was to reinstall the OS and start over. I had a friend help me with the install and he suggested I go ahead and upgrade to 9.3 stable since I was starting from scratch essentially. He has more experience with FreeNAS than I do, so I thought this was a good suggestion. After installing 9.3 everything was working ok, until we tried to import the volume from the old OS.

The wizard saw the volume and said it would import it, but once the wizard was done the volume was not there. His said this was ok, we just needed to upgrade the zpool. After upgrading the zpool we exported the volume from the shell and tried to import it again from the GUI. The Gui sees the volume and tries to import it and refreshes the screen. Again there is still nothing showing in volumes. At this point he had to leave and I am on my own for now. I have been reading forums for three days now trying to solve this, I see 8GB is the recommend RAM and plan to upgrade but cannot at this exact moment as the power surge took out my router and modem (these were on a different power-strip) and I just replaced those, so money is tight at the moment. I see a lot of post similar to mine but most are resolved off site or have things I have tried above work, but not for me.

When I run zpool import Media get:
[root@freenas ~]# zpool import Media
cannot mount '/Media/.system': failed to create mountpoint
cannot mount '/Media/.system/cores': failed to create mountpoint
cannot mount '/Media/.system/samba4': failed to create mountpoint
cannot mount '/Media/.system/syslog-901a1ff993dd48fd941728573b836c3f': failed to create mountpoint
cannot mount '/Media/Movies': failed to create mountpoint
cannot mount '/Media/TVshows': failed to create mountpoint
cannot mount '/Media/pluginjails': failed to create mountpoint
cannot mount '/Media/pluginjails/.warden-template-pluginjail-9.2-x64': failed to create mountpoint
cannot mount '/Media/pluginjails/plexmediaserver_2': failed to create mountpoint

afterwards when I run zpool status I get:
[root@freenas ~]# zpool status
pool: Media
state: ONLINE
scan: scrub repaired 0 in 4h14m with 0 errors on Sun Mar 13 05:15:00 2016
config:

NAME STATE READ WRITE CKSUM
Media ONLINE 0 0 0
mirror-0 ONLINE 0 0 0
gptid/676b26ad-c3eb-11e5-9ca8-0018f3844dcc ONLINE 0 0 0
gptid/67aaeb5b-c3eb-11e5-9ca8-0018f3844dcc ONLINE 0 0 0

errors: No known data errors

pool: freenas-boot
state: ONLINE
scan: none requested
config:

NAME STATE READ WRITE CKSUM
freenas-boot ONLINE 0 0 0
gptid/5820669c-eb15-11e5-b06e-0018f3844dcc ONLINE 0 0 0

errors: No known data errors

Thank you for any help you can offer this is my first post, also please shoot it to me straight I am big boy and can handle it.

 

[cyberjock]

Lucky you, you get a response from me (for better or for worse). Here's "me giving it to you straight".

The "cannot mount" messages are normal. You did the command "zpool import Media" and that's not the proper way to mount a zpool in FreeNAS. Normally FreeNAS would mount the zpool with a command like "zpool import -f -R /mnt Media". Since you didn't try to mount the zpool to the correct location (/mnt) the mountpoints will fail to be created. So you can relax with those errors. They aren't a sign of significant problems.

Side story: Here's where things get messy, and here's why you need 8GB of RAM. The middleware is python code. It's RAM hungry, and if it tries to do something and there isn't enough RAM, then processes get terminated. There is no feedback to the user that says "Sorry, the process to mount the zpool failed". So it literally silently fails. I've seen silent fail with most anything, but predominantly with zpool mounts, attempts to do upgrades in the WebGUI, and attempts to create or delete boot environments. Those 3 issues account for probably 75% of middleware behaving in very bizarre ways. I'm n ot sure why those 3 are more susceptible to problems, but they are.

Ok, back to your issue.

So we've got a few things we know of:

1. Scrub shows the zpool is healthy and no errors found. So the likelihood of zpool corruption is very, very unlikely.
2. You can mount the zpool manually, and the zpool does mount. If it weren't mounting, causing a lockup or kernel panic then we'd have a totally different problem.
3. You cannot mount the zpool from the WebGUI.

Now, since we know the zpool *will* mount if the zpool import command is issued, it is safe to say that the zpool import command is NOT being executed. Why would it not be executed? The two most likely causes I can think of would be:

1. Python code is being terminated in a very unusual way.
2. There is some kind of fatal flaw in FreeNAS' middleware (aka python code) that is so broken it is quietly failing.

It's pretty unlikely that #2 is the case because if we couldn't mount zpools the forums and TrueNAS users would likely be overrunning this place with complaints, and we'd have a fix out the door (and it would be a well known bug). Obviously this isn't happening, so isn't too likely.

But... we do know that insufficient RAM *does* lead to middleware getting upset and those processes being terminated inappropriately. So it seems very, very likely that #1 is your problem. Python is a memory hog. I've seen very simply middleware threads take up more than 400MB of RAM. Considering the system uses just over 2GB for RAM disks, you've got only 2GB of RAM left for *all* of the other processes plus the kern in your system. That's assuming you don't have 500MB or so for reserved memory space, which is therefore unusable for the system. Some systems may even have as much as 1GB for reserved memory.

Now for the bigger problem. How do we work around this until you get 8GB of RAM?

Well, you can't. At least not particularly easily. See, even if you mimick the exact command FreeNAS uses, the config file will not have the proper entries to think the zpool is mounted. So CIFS won't start because the shares can't work because the mount location won't exist because the link in the config file to the zpool isn't there. You're really caught in this nasty catch-22. I think you need to upgrade your system to 8GB of RAM before you can truly access your data via your normal CIFS/NFS/iSCSI shares.

If you have 8GB of RAM that you can shove in the system long enough to do a zpool import and get the config file to accept the zpool information, that would get you by. But, that's not to say you won't have other problems that could get ugly. Middleware code terminating can create problems, including corrupting the config file. Scenarios like the middleware updating the config file and then updating with corrupt (or missing) information creates issues that are only resolved by going with a brand new config. This is why I'm so onerous about the 8GB of RAM minimum. It creates a situation where you are potentially having very vital processes in FreeNAS have to deal with being terminated suddenly and without warning, and the FreeNAS code cannot be easily coded to deal with every possible scenario of a python process being terminated for any reason.

Running with less than 8GB of RAM is like having a little gremlin running in your system that can decide, for any reason it wants, to terminate your processes. That gremlin might decide to terminate your processes regularly, or it might decide it wants to sleep all day. Can you imagine having a system that is reliable with that kind of gremlin running around? I bet you'd get really frustrated with that system in short order.

But I think that's your issue. The good news is I think your data is safe (for now). I'd certainly upgrade to 8GB of RAM before trying to use the system. If you have data on it that you absolutely need to get access to until you can upgrade to 8GB of RAM permanently, upgrade the system to 8GB and copy off the data you need. Even if you could get the zpool to import with 4GB of RAM, I'd be wary of potential corruption or lost data from starving a system of RAM.

I don't have a working fix for the issue unfortunately. The fix is to not paint yourself into this corner to begin with. ;)

 

[joeschmuck]

Sounds like your friend gave you a raw deal to me. The first was bad advice to upgrade to a FreeNAS version which requires more RAM to operate. The second bad advice was to upgrade your pool. If you hadn't upgraded your pool then you would be able to roll back to 9.2.1.5.

What you should take away from this experience is:
1) Use an UPS and have it configured to shut your machine down properly. A power strip is good at best for power surges but not power failures.
2) When you have a corrupt OS, repair your system with the same version. The old version is still online and available and if you can't find it, just ask here in the forums and someone will either direct you where you can download it or give you a copy they have.
3) NEVER upgrade your pool until you are 100% certain you do not want to roll back to an older version of FreeNAS. Odds are that upgrades to your pool are not going to benefit you greatly, if at all. I haven't upgraded my pool in years but eventually I will when it makes sense to.

Good luck and I hope you can get your hands on that extra RAM so you can bring your system back to life.

 

[Shadowgundam2725]

Ok, I have a gaming PC with a i7-4770k and 16GB of RAM. Is it possible to move the HDDs (Data) and Thumb drive (OS) to this PC mount the drives and transfer the data to another device(PC, USB HDD, etc.) until the NAS system RAM can be increased?

 

[JDCynical]

Ok, I have a gaming PC with a i7-4770k and 16GB of RAM. Is it possible to move the HDDs (Data) and Thumb drive (OS) to this PC mount the drives and transfer the data to another device(PC, USB HDD, etc.) until the NAS system RAM can be increased?

In theory, yes. Assuming that the system can boot on the hardware and the necessary parts work under the BSD core, the only thing you would have to 'fix' would be the network interface MAC address.

But this is in theory, as many consumer level 'gaming' boards add in all kinds of crap that works only under windows, has a weird firmware, kinda works under non-windows environments but is not stable, etc.

 

[joeschmuck]

Ok, I have a gaming PC with a i7-4770k and 16GB of RAM. Is it possible to move the HDDs (Data) and Thumb drive (OS) to this PC mount the drives and transfer the data to another device(PC, USB HDD, etc.) until the NAS system RAM can be increased?

Absolutely, that is one of the key features to using a Software RAID NAS like this. Here is how I'd suggest you do that...

1) Remove your gaming system hard drive(s) and place them somewhere safe so you don't accidentally erase them.
2) Plug in all your NAS drives and your bootable USB Flash Drive.
3) Boot from the Flash Drive and everything should come up roses.
4) You may need to reconfigure your network configuration if it matters to you, the boot screen will have your IP address for the system.
5) Copy off your data.

You can run your system like this however hopefully you have heard about all the ECC RAM concerns and I'm certain your gaming rig doesn't have ECC RAM.

 

[Shadowgundam2725]

Sorry for the long delay in responding, today is the first day I have been able to look at my system.
I have moved everything to my extra PC the new specs are:
Processor:I7-4770k
MOBO: MSI z97 PC Mate LGA 1150
RAM: 8GB G.Skill Ripjaws X series 8GB (2x 4GB) DDR3 1866mhz
HDD: Seagate 2TB SSHD x2
OS: FreeNAS-9.3-STABLE-201602031011

After moving everything the network connection setup was quick and no problems occurred. I still cannot get the GUI to import the volumes. I get the exact same issue as I did before. Little more information, I don't know if this matters. The two drives were originally set in a Mirror Raid and I have not imported any drives at this time.

 

[depasseg]

I would try installing 9.2.1.5.

 

[Shadowgundam2725]

I installed 9.2.1.5 on a different thumb-drive and started it up. 9.2.1.5 does not see the pool at all:
zpool status returns:
No Pools Available
And Auto volume import does not see it at all either.
The pools were updated so this does not surprise me from what I have been reading.

 

[depasseg]

I missed that. Yes that makes sense.

So what is the current status? Is anything showing up in the storage tab of the GUI? If not, I would try to export via the command line, and then reattempt import via GUI.

 

[Shadowgundam2725]

Bolding text so you can see what I said versus outputs. The Export works and then when I attempt to import through the GUI I get nothing. I have a monitor hooked up so I am seeing this error on the screen when it tries to import:
freenas manage.py: [middleware.notifier:3517] Importing Media [2788235721685255697] failed with:
cannot mount '/Media/.system': failed to create mountpoint
cannot mount '/Media/.system/cores': failed to create mountpoint
cannot mount '/Media/.system/samba4': failed to create mountpoint
cannot mount '/Media/.system/syslog-901a1ff993dd48fd941728573b836c3f': failed to create mountpoint
cannot mount '/Media/Movies': failed to create mountpoint
cannot mount '/Media/TVshows': failed to create mountpoint
cannot mount '/Media/pluginjails': failed to create mountpoint
cannot mount '/Media/pluginjails/.warden-template-pluginjail-9.2-x64': failed to create mountpoint
cannot mount '/Media/pluginjails/plexmediaserver_2': failed to create mountpoint

Even though it fails zpool status returns:
[root@freenas ~]# zpool status
pool: Media
state: ONLINE
scan: scrub repaired 0 in 4h14m with 0 errors on Sun Mar 13 05:15:00 2016
config:

NAME STATE READ WRITE CKSUM
Media ONLINE 0 0 0
mirror-0 ONLINE 0 0 0
gptid/676b26ad-c3eb-11e5-9ca8-0018f3844dcc ONLINE 0 0 0
gptid/67aaeb5b-c3eb-11e5-9ca8-0018f3844dcc ONLINE 0 0 0

errors: No known data errors

pool: freenas-boot
state: ONLINE
scan: none requested
config:

NAME STATE READ WRITE CKSUM
freenas-boot ONLINE 0 0 0
gptid/5820669c-eb15-11e5-b06e-0018f3844dcc ONLINE 0 0 0

errors: No known data errors

Also getting another error before and after:
freenas ntpd_initres[2046]: host name not found: 0.freebsd.pool.ntp.org
freenas ntpd_initres[2046]: host name not found: 1.freebsd.pool.ntp.org
freenas ntpd_initres[2046]: host name not found: 2.freebsd.pool.ntp.org

 

[depasseg]

In my 9.3, all my datasets are mounted under/mnt/.

You could try changing the mountpoint manually of Media from /Media to /mnt/Media in the CLI (zfs set mountpoint=/mnt/Media Media)

And the NTP issue is probably a combination of DNS servers and Default Gateway not being set.

 

[Shadowgundam2725]

I exported the Media pool and tried this and got.
[root@freenas ~]# zpool import -f -R /mnt Media
cannot mount '/mnt/Media/.system': failed to create mountpoint
cannot mount '/mnt/Media/.system/cores': failed to create mountpoint
cannot mount '/mnt/Media/.system/samba4': failed to create mountpoint
cannot mount '/mnt/Media/.system/syslog-901a1ff993dd48fd941728573b836c3f': failed to create mountpoint
cannot mount '/mnt/Media/Movies': failed to create mountpoint
cannot mount '/mnt/Media/TVshows': failed to create mountpoint
cannot mount '/mnt/Media/pluginjails': failed to create mountpoint
cannot mount '/mnt/Media/pluginjails/.warden-template-pluginjail-9.2-x64': failed to create mountpoint
cannot mount '/mnt/Media/pluginjails/plexmediaserver_2': failed to create mountpoint

 

[depasseg]

Don't use the CLI, try import with the GUI. And never force. Does /mnt exist?

 

[cyberjock]

Did someone change the permissions of /mnt?