Truenas Core 12u8 Access lost to files and UI

[bucu321]

Thank you in advance for your input to this problem and your help and advice figuring it out. I pretty much feel like a newb, well I guess I am a newb. I'll own it.

The problem started yesterday when my AC had a fault and caused my breaker box fuse to trip cutting off power to the house. I have a UPS running my networking gear and NAS to mitigate any downtime for power glitches. However this time the UPS didn't engage and there was an unscheduled reboot.

Now I can't log in to TRUENAS from PC. (Funny thing is my cell phone can log in and see the UI)
Can't access any files on any of the shares. Extremely slow. I can open the share and see files and folders but then it gets really slow. Any file transfer is impossible; always has timeout error.

I can ssh into the server and check all the smart data on the drives (all good).
I can check status of the pool: healthy.
I ran a scrub of the main pool and the boot-pool. Both returned no errors.

Like I said earlier, I'm a bit of a newb and I feel lost. I've spent 6 hours reading forum posts and searching but can't seem to find anyone with the exact problem and a solution to it. I'm not sure what to test to determine the error outside what I already did. I'm pretty sure the files are fine and the pool is fine but something happened to truenas and its ability to operate.

(Go easy on me this was my first time building my own nas after my Synology got old. Obviously I learned somethings about how I set it up after the process of building and using and if I did a new build in the future I would make changes. But this is what I got now.)

Setup: (its been a while let's see if I can remember it all)
i7-9700
ab350m (asrock) I think
32gb RAM
2x Adata 256 SSD's running mirrored for host OS (pluged into the sata ports on mobo)
Intel Ethernet 10gb NIC card
LSI2008 HBA card for drives for the pool in Truenas
Proxmox (host os)
Two vms: pfsense and truenas
Truenas is running Core 12-U8

Originally built in 2021 -- only issues since were a couple bad drives that were replaced as usual wear and tear

I'm not sure exactly what else to provide that would be helpful.

Currently, I shutdown the VM and I plan to turn off the whole proxmox rig this afternoon so I can open it to clean it out and check all the wiring. Just as a basic physical inspection (not that the machine has been moved, bumped, or otherwise jostled that would cause any wires to be loose), but I'll use the opportunity to cleaning it from any dust build up that might have generated excess heat from impacted airflow.

Again thanks for any help you can provide.

 

[bucu321]

Re-reading forum rules I should have also provided the pool setup:

Those 6 drives are in 3 stripped mirrors in zfs. (3 mirrored pairs) (the LSI2008 is running as an HBA and isn't doing any raid configuration at all. Just to clarify and try to put this in line with forum rules. I'm new here so I'm learning what is expected.

 

[sretalla]

Now I can't log in to TRUENAS from PC. (Funny thing is my cell phone can log in and see the UI)

So what's different between those clients?

Can't access any files on any of the shares. Extremely slow. I can open the share and see files and folders but then it gets really slow. Any file transfer is impossible; always has timeout error.

A bit conflicting... you can access things, but then you can't.

How are you trying to transfer files? have you tried SCP?

I can ssh into the server and check all the smart data on the drives (all good).
I can check status of the pool: healthy.
I ran a scrub of the main pool and the boot-pool. Both returned no errors.

Looks like network should be the focus of the troubleshooting.

Poor connectivity could somehow explain some of what you're describing... have you checked your network cabling/connections?

 

[bucu321]

So what's different between those clients?


A bit conflicting... you can access things, but then you can't.

How are you trying to transfer files? have you tried SCP?


Looks like network should be the focus of the troubleshooting.

Poor connectivity could somehow explain some of what you're describing... have you checked your network cabling/connections?

Sorry for not being clear. By PC I meant a separate computer (laptop or my main desktop) not the machine running proxmox and hosting the vms.

I think you are right that it is network related.
After continuing to work on it all day, I currently have access again to both WebUI and file shares. Still a little slower but the only thing I've done today is clean the dust out, inspect the hardware inside, and change one of the network cables. I'm not sure if it is a bad network cable or the NIC is having problems. The host is using a 4 port Intel 10gbit pci adapter. So I switched to a unused port and plugged it in to my switch in a normal 1gbit port rather than the 10gbit. After that change, I got connection to the DHCP and to the webui.

I am not sure if this is just a temp fix or if it was the problem. But I can at least get an updated backup run before doing more troubleshooting.

 

[bucu321]

Further review looks like one of the pins in the network cable rj45 connector melted that was connected to the SFP+ port on the switch. The dang thing runs so hot. But still doesn't fully explain why the phone could access and not my main pc or laptop. Oh well. Continue to observe and maybe get some little heatsinks and a fan to mount on that SFP RJ45 adapter.

Thank you Sretalla for your comments and for those who took the time to read my problem. Till next time.

 

[artlessknave]

note that proxmox is generally not considered a good host for truenas.

if you have your truenas boot disks on proxmox vmware disks, you could very likely see intermittant errors, as the disks may not be available properly as zfs expects.