Help with server not starting up on reboot please.

[pix]

Hello I am on FreeNAS 9.2.1 RELEASE x64 still. I have had no issues for months since I built my system.

However today I discovered that I could not get plugin info (installed or available just showed NO ENTRY FOUND)
Turns out I could not ping www.google.com from my FreeNAS box. So I changed the interface setting to DHCP (it had a static) and shutdown my server from the SHUTDOWN button on the webgui.

Now when I press the power button on my server I get the ASROCK splashscreen and
then the usual freeBSD boot screen and then it just keeps putting out the hashtags. I currently have about 20 lines of hashtags.

Code:

F1 FreeBSD
F2 FreeBSD

F6 PXE

Boot: F2 #############################################
######################################################
######################################################
######################################################
######################################################

You get the picture. This is still going on 20 minutes later. My case shows no HDD activity during this and only shows HDD activity when I do a full shutdown and power on and it immediately goes to the load screen.


Here is my build.

Code:

CASE: Fractal Arcmini R2
HDD: 6x2TB WD GREENS. RaidZ2. 7.1TB usable.
CPU: Intel Pentium G3220
RAM: Crucial 16GB (2 x 8GB) 240-Pin DDR3 SDRAM 
   DDR3 1600 (PC3 12800) ECC Unbuffered Server Memory Model 
   CT2KIT102472BD160B
PSU: SeaSonic SSR-360GP 360W ATX12V v2.31 80 PLUS GOLD 
   Certified Active PFC Power Supply New 4th Gen CPU 
   Certified Haswell Ready
Motherboard: ASRock E3C226D2I Mini ITX Server Motherboard 
   LGA 1150 Intel C226 DDR3 1600/1333

Does anyone have any idea what's going on? I keep track of any changes I make on servers as a habit (it's my job) so as stated above - the only change I made was to check off the DHCP box on interface IGB0 so it would pull from the router and then I pressed the SHUTDOWN button on the WebGUI. Waited a few minutes then logged into the IPMI GUI and powered on the server from there.

Any help is greatly appreciated.

 

[jgreco]

That's consistent with a rotted-out USB stick. Try installing the same version of FreeNAS onto a fresh USB flash stick. If that boots, DON'T DO ANYTHING and we can walk you through recovering your configuration from your old USB.

 

[pix]

Oh that sucks. I use a mushkin USB drive. I'll a different one next. All my local stores are closed today. I'll see if can find one in a drawer somewhere.

 

[jgreco]

There's a reason I suggest having three USB flash modules handy. One to run off of, one for an upgrade, and one for the inevitable problem that pops up.

 

[pix]

I appreciate your help with this.
I just finished writing the 9.2.1.3 RELEASE image to a v165w 4GB drive I have. I believe I can get more of exactly this model to so I can have spares.
When I boot this usb stick what's the next step in recovering?

 

[pschatz100]

There is nothing wrong with Mushkin. USB sticks, by their nature, are not supremely reliable - even the good ones. Reading is generally more reliable than writing, which is why they can be used as boot devices, but I would not use one for jails (I made that mistake early on.)

I have had USB sticks go bad, and now make it a practice to use a new flash drive each time I do a major update. This has yielded two benefits: 1) reduces the chance of a flash drive going bad, and 2) provides a backup of the previous installation. Haven't decided, yet, what my strategy will be for 9.3 with its new update system.

Good luck.

 

[pix]

I decided to not write the 9.2.1.3 iso image to USB. I loaded the ISO virtually using IPMI and installed it to the new 4GB exactly how I had done it before.
I currently have the server off.

 

[jgreco]

You'll want to boot into the new image. Once there, the rest of this is shell work, so go to a shell. Haven't done this in a really long time so I'm more or less just guessing here. Stop and ask questions if something seems amiss. Probably my error.

If you do "camcontrol devlist" it should show your USB disk as da0.

VerySuperImportant to make sure we're using the right devices ... if you have other disks attached hopefully they're showing up as ada0...whatever

So then also plug in your old USB stick and if you redo "camcontrol devlist" that should be da1.

If anything is not as described - STOP.

Now do "mount -f -o ro /dev/da1s4 /mnt"

That should make your old /data directory available on /mnt; you should verify that /mnt/freenas-v1.db exists.

Copy that to a temp on /data: "cp /mnt/freenas-v1.db /data/recovery.db"

Unmount the old flash: "umount /mnt" then remove it.

Now the problem here is that we don't really know that the db is in good condition. So now type "sqlite3 /data/recovery.db" and at the sqlite prompt type "pragma integrity_check;" (underscore and semicolon are significant). If it says "ok" then we're probably in luck. If not - you can basically stop and reconfigure FreeNAS from scratch, sorry.

So exit sqlite and type "halt". When the system has halted, press space to begin the reboot, and then be ready at the loader prompt to stop FreeNAS from booting. You want to select the singleuser boot mode.

Once in singleuser, you're once again at a command prompt. Type "mount -a" to mount everything.

Go over into /data: "cd /data"

You can just write over the FreeNAS config because it should not have anything of importance: "cp recovery.db freenas-v1.db"

Reboot. Recovered.

After that, make sure you consider using one of the various scripts to make automatic configuration backups. ;-)

 

[pix]

I guess this little HP usb is borked too. Just as well I found it in a drawer didn't know how good it was.
the message I'm getting is.

Code:

GEOM: da0 : the primary GPT table is corrupt or invalid

I'll keep trying usb drives until I find one that atleast boots up into the prompt.

 

[pix]

well I got some HPs to finally start booting. unfortunately i can't get into my system because it stops at the mDNS responder error and stays there with the AUTHRECORD already in the list message and doesn't move past this. everytime i hard shutdown the system to get away from this message any usb drive i have ends up becoming unbootable and i have to install FreeNAS all over again.

i remember getting past this error when i first set all this up, but i don't remember what it was. i know it was a pretty prevalent bug that got fixed in 9.2.1.6. after i get all this sorted maybe i'll do a few updates.

 

[pix]

....And I'm back up and running. Thank you for all the help. first thing i did was create a local backup of the config.
You have no idea how grateful I am.

my final question - should i update one by one or should i just jump to the last 9.2* release? (going to read up a bit more on this later.)

 

[pschatz100]

From 9.2.1.3, you can update to the last 9.2.1.9 release. Be aware that there are some significant updates to a variety of services, so it would be wise to make certain all is running well under 9.2.1.9 before going further. From there, you can make the jump to 9.3.

One of the enhancements with 9.3 is an update to ZFS. By default, the update to 9.3 won't update your ZFS pool but you will get a warning that ZFS updates are available. Do not update your pool until you've been running 9.3 for a while, just in case you have to go back to 9.2 for some reason. Also, note that 8gb is the recommended minimum for a USB boot device on FreeNAS9.3. If you are planning to buy a new flash drive, consider 16gb for a little extra head room.