Mayday! I broke my TrueNAS CORE Server!

[ryandasilva]

So I’ve been running TrueNAS since i dont even know which version of FreeNAS 11.something. I dutifully upgraded to each version along the way.

Somewhere along in the last few updates, I had trouble updating thru the web interface so I started using the CLI. Then that stopped working I too.

I decided to migrate to SATA disks thinking maybe my USB drives were having issues or were full. I also learned that USB boot isn't recommended anymore.

So I took one of my mirror’d USB boot disks out of the machine but stupidly formatted it thinking it was a blank USB. Shouldnt be a problem though, right?

It’s a mirror and was going to be replaced by a SATA Disk anyways, plus there’s the other functioning USB.

Well, I forgot to mention that I was starting to have an issue where if i rebooted TrueNAS there was a 50/50 chance it would boot up properly, or it would boot up into some blank config mode where I would get a login prompt instead of the menu and if I typed in username root, it wouldn't even ask for a password it would just log me in. The config was blank and web services weren't up.

But if i rebooted again, sometimes selecting the boot normally option #1 seemed to help, I might get back into my proper system config. I’m not sure what exactly was the straw that broke the camels back... i suspect it was when i purchased an 32gb SATA M.2 drive and installed TrueNAS onto it with a blank config and attempted to restore a slightly older backup of my config that i had in my downloads folder, although that shouldn't have affected the USB because it wasn't even plugged in when I did that. Bottom line though, I can no longer boot from USB at all.

I posted to reddit and eventually they referred me to here because the data is so sensitive and nobody knew how to help me. Basically I am able to restore the most recent backup config that I can find without any errors to TrueNAS CORE and I also decided to try TrueNAS SCALE , cuz , why not if the systems down might as well try it. I really liked Scale and think I might end up using that in the end but right now I need my data back. This data is so important to me, as I have most of my family videos on one of those pools which includes video of my late wife and if i lost that forever i don't know what i'd do. I am not panicking yet because i think the USB key if somehow repaired would work, or i could simply extract the decryption key from the USB. I had literally just purchased faster Fiber internet (1.5down / 1.0 up) because i was trying to back my NAS up to iDrive and my cable provider's 30 meg upstream is garbage for the 10 terabytes of data all of my pools consist of.

So to summarize that again, The config restores fine, and one of my pools comes back up... its the one and only pool that isn't encrypted. The other pools are all encrypted and wont come up.. I have backup files called {POOLNAME}-GELI.key but i didn't have to manually enter a decryption key every time so i think the key is somewhere on that unbootable USB key or possibly within my backup files. Not sure where the .key file comes into play.. is that a recovery key or something ?

I would have thought restoring the .TAR file that is backed up would include the decryption keys but i dont have any experience with the backup/restore of FreeNAS config.

Here’s a pic of the foobar'd TrueNAS instance on the USB.. Not sure if i even want to attempt to rescue this or if i should just try to pull the decryption keys off it.


https://imgur.com/a/TzdkNo8

As i mentioned I've tried restoring to both CORE and SCALE; Scale is the most recent. Here's a screenshot of the pools. It looks pretty much the same on CORE. If it makes more sense to go back to Core for now i can... but i would like to think that I just need to find the decryption key on the USB or use one of the key files , or maybe it's in the .TAR file? And where does that decryption key go?

Just wanted to clarify.. 2 of the 3 pools that are offline are encrypted ... but the most important one isn't. I must have mixed that fact up in the worry about my data. Now im scratching my head even more why an unencrypted pool won't come up. Either way, how do I get the 2 encrypted and 1 non-encrypted pool back up

Here is a screencap of my Pools screen:

imgur.com

Discover the magic of the internet at Imgur, a community powered entertainment destination. Lift your spirits with funny jokes, trending memes, entertaining gifs, inspiring stories, viral videos, and so much more from users.

imgur.com

Thanks in advance for your help! Stay safe. Cheers.

Thanks

 

[ryandasilva]

Just wanted to add that i feel like I'm guessing in the dark here due to not having any kind of error messages or anything to trouble shoot of. Where would i find a log file that would have the error message saying the reason the pools are offline.

Also wanted to add that i decided to dig into the disks screen and noticed that a bunch of my disks are missing. Going to dig further on that. Any help in the meantime is much appreciated.

 

[sretalla]

Where would i find a log file that would have the error message saying the reason the pools are offline.

zpool status -v
dmesg
/var/log/messages

 

[ryandasilva]

zpool status -v
pool: TIMECAPSULE1
state: ONLINE
status: Some supported features are not enabled on the pool. The pool can
still be used, but some features are unavailable.
action: Enable all features using 'zpool upgrade'. Once this is done,
the pool may no longer be accessible by software that does not support
the features. See zpool-features(5) for details.
scan: scrub repaired 0B in 04:52:43 with 0 errors on Sun May 16 04:52:44 2021
config:

NAME STATE READ WRITE CKSUM
TIMECAPSULE1 ONLINE 0 0 0
40849465-2c37-11eb-bf81-001018efc630 ONLINE 0 0 0

errors: No known data errors

pool: boot-pool
state: ONLINE
status: Some supported features are not enabled on the pool. The pool can
still be used, but some features are unavailable.
action: Enable all features using 'zpool upgrade'. Once this is done,
the pool may no longer be accessible by software that does not support
the features. See zpool-features(5) for details.
scan: scrub repaired 0B in 00:00:09 with 0 errors on Mon Jun 14 03:45:11 2021
config:

NAME STATE READ WRITE CKSUM
boot-pool ONLINE 0 0 0
sda3 ONLINE 0 0 0

errors: No known data errors

 

[ryandasilva]

its not even showing my pools listed in the gui. haven't had time to open it up and just double check that no stupid mistakes were made like accidently unpluggedin a mini-sas cable that connects to 4 drives or whatever. lol.

 

[sretalla]

dmesg might give you a hint, but if you have unplugged things, the screenshot (which BTW you should have inserted in your message rather than link to external) makes sense.

 

[ryandasilva]

dmesg might give you a hint, but if you have unplugged things, the screenshot (which BTW you should have inserted in your message rather than link to external) makes sense.

sorry about posting the pics wrong. im coming from reddit so keep that in mind ;)

I think im going to reproach this .. forget scale for now. go back to core. the config imports fine. assuming all my drives show up in Core but the problem still remains what would be my next course of action? will post results once reinstalled.

 

[sretalla]

sorry about posting the pics wrong. im coming from reddit so keep that in mind ;)

No problem, you'll know for next time.

the problem still remains what would be my next course of action?

Check your hardware/cabling.

 

[ryandasilva]

ok some positive news guys!! But not there 100% though...

So after fiddling with a Molex power cable all it seems like all my drives were showing up and my biggest, most important pool with the irreplaceable family videos /pics mays it online but om an unhealthy state, while the rest of my pools are offline still. BUT.. I can see the shares and folder structure of VIDEO STORE and right there.. this.. is considered a 100% recovery to me. The rest is just gravy... The VIDEOSTORE pool is all tha really matters becasue it'swhere my late wife's digital soul to resides in hundreds of hours of videos that i've never seen but i was the film man. This whole scare was a sobering learning lesson that has ended well. It could have been worse. My lessons learned are that it must exist in 3 places, i need to learn more about how ZFS encryption works .. like the decryption key stuff. i still dont get that. I think i just got lucky in that it wasn't too overly encrypted...and the more valuable data is.. the more likely it's going to fail. Seriously... its so fucking true!!
Not all the pools are back though ... so i just want to pause and say thank you all for the help so far.! Im going to kick off an SCP job to rip that entire pool an external USB drive plugged into my macbook before i do anything else.. if it all falls apart trying to fix the rest of the pools, or trying to decrypt the video store pool , at least i have a good working copy of the most crucial data... I'l be back in a bit. THANKS EVERYONE!!! Much love.

 

[ryandasilva]

Sorry about the delay in responding.. here is my storage screen. i find it very strange how the one pool i need THE MOST is the only one that came back online . The other pool called timecapsule is a single disk pool used just for time machine backups. we have cloud which i use as my primary OS back up mechanism. I don't really want to use my NAS too much for backup data to be completely honest.

SO what is making these pools not come online? As you can see, all the disks show but sadly they're not forming the pools which they need to form, which is scary AF. Why?? Where do i look for more info on what might have happened here? i to form SDB and SDC back into the Encrypted 4TB Mirror pool listed above (which is offline) but it looked destructive so i aborted. I don't want/cant just start over. erverythigns there's except for the pointers to make those raw disks into the high data riddled Pools that they were not long ago? I refuse to touch them further out of fear of losing the remaining data pools; it literally keeps warning me about losing data so id be a total fool ignore that kind of warning.

TLDR; Guys why is my super important 12TB encrypted pool coming up (thank god), but not the rest of my pools, some of which arent even encrpyted?