FreeNAS Mini on 11.2 freezes / locks up

Meyers

Patron
Joined
Nov 16, 2016
Messages
211
Greetings,

I've had a FreeNAS Mini 2 for a couple of years now running 24x7 with few issues. This system is my main home file storage so it's critically important. It serves up AFP and SMB shares.

Recently I upgraded to 11.2, installed a qbittorrent plugin, and decided to seed a bunch of open source torrents. I have a lot of bandwidth and storage so I thought it'd be nice to use some of it for this purpose.

Running FreeNAS-11.2-RELEASE-U1.

Hardware is:

Motherboard: C2750D4I
OS disk: 16GB SATA DOM
Data Disks: 4 x WD Red 8TB
Memory: 32GB
Official 64GB ZIL and 128GB L2ARC installed.
Pool is encrypted.

Code:
# zpool status
  pool: freenas-boot
state: ONLINE
  scan: none requested
config:

        NAME        STATE     READ WRITE CKSUM
        freenas-boot  ONLINE       0     0     0
          ada3p2    ONLINE       0     0     0

errors: No known data errors

  pool: storage
state: ONLINE
  scan: scrub repaired 0 in 0 days 04:11:16 with 0 errors on Tue Jan 15 08:11:26 2019
config:

        NAME                                                STATE     READ WRITE CKSUM
        storage                                             ONLINE       0     0     0
          mirror-0                                          ONLINE       0     0     0
            gptid/f78b68d8-b265-11e6-a9ad-d05099c04440.eli  ONLINE       0     0     0
            gptid/f8ae227c-b265-11e6-a9ad-d05099c04440.eli  ONLINE       0     0     0
          mirror-1                                          ONLINE       0     0     0
            gptid/f9f5b277-b265-11e6-a9ad-d05099c04440.eli  ONLINE       0     0     0
            gptid/fa82d11a-b265-11e6-a9ad-d05099c04440.eli  ONLINE       0     0     0
        logs
          gptid/a56a2569-19f8-11e9-94fb-d05099c356a2.eli    ONLINE       0     0     0
        cache
          gptid/a34a1689-19f8-11e9-94fb-d05099c356a2.eli    ONLINE       0     0     0

errors: No known data errors


Seeding generates a bit of traffic but nothing overwhelming. However, every few days the filesystem freezes, which causes the SMB shares to freeze, my Windows clients to freeze, etc. I have to hard reset the box.

I see nothing in the logs. In fact, for some reason the logs are getting overwritten on boot and I can't figure out why. This only seemed to start when I upgraded to 11.2.

This last time the system even lost its hostname for some reason. It's coming up named "freenas." I've looked all over the place to try and figure out WTF is going on but I'm at a loss here.

I think I'm going to remove the ZIL for now. It's overkill for this system. The L2ARC is getting a lot of use however. Otherwise I'm stumped as to what's causing this and what to do about it.

Would really appreciate some help with this. Thanks.
 
Last edited:

Meyers

Patron
Joined
Nov 16, 2016
Messages
211
By the way, this is a fresh install of 11.2-U1 as of a few weeks ago.
 

toadman

Guru
Joined
Jun 4, 2013
Messages
619
So it's a system lock up? i.e. no response what-so-ever? Or is the GUI and/or console accessible?

If it's a lock up my first thought would be memory. Or you'd have to see what drivers changed in 11.2 from previous version I suppose. That might tell you where else to look.
 

Meyers

Patron
Joined
Nov 16, 2016
Messages
211
Still pingable but the UI / console isn't accessible. The pool completely freezes. Once I was able to log in and when I tried to access the pool the command froze forever. No logs, nothing in dmesg, etc. I have to hard reset it.
 

Chris Moore

Hall of Famer
Joined
May 2, 2015
Messages
10,080
Motherboard: C2750D4I
Did you get the updated board to fix the bug?
Still pingable but the UI / console isn't accessible. The pool completely freezes. Once I was able to log in and when I tried to access the pool the command froze forever. No logs, nothing in dmesg, etc. I have to hard reset it.
My first thought would be overheat. These little systems don't have very good cooling and some of the forum members that have them have had to retrofit fans into the case to provide better cooling.
 

Meyers

Patron
Joined
Nov 16, 2016
Messages
211
Did you get the updated board to fix the bug?

Yep. Then both my mini's boards died and I had to replace them.

My first thought would be overheat. These little systems don't have very good cooling and some of the forum members that have them have had to retrofit fans into the case to provide better cooling.

Interesting. It's in my closet on the floor where it's cold (< 70 F). But I did just change the fan and I have it hard set to a lower fan setting because of noise.

This doesn't feel like a lockup to me. Usually that's the whole system locking up hard in my experience. This seems like more of a filesystem issue, but then I know that could be tied to memory and heat issues which I have seen before.

That said, I'm going to run CPU burn or something and really heat it up and see what happens. Might just have to increase the fan speed.

Plus I removed the ZIL because it's really not needed and to reduce the complexity of the situation.
 

Chris Moore

Hall of Famer
Joined
May 2, 2015
Messages
10,080
By the way, this is a fresh install of 11.2-U1 as of a few weeks ago.
Since the new version of FreeNAS, many people appear to have had failed boot media. That can cause all sorts of strange issues. What are you using for your boot pool?
 

Redcoat

MVP
Joined
Feb 18, 2014
Messages
2,924
I am one of those who's experienced overheating on a C2750D4I on a Mini2 and installed two additional fans at the lower front of the case and a fan on top of the CPU passive heat sink as well as upgrading the standard case fan in order to mitigate the effects.

Have you looked at the ASRock IPMI history to see what's been recorded for the mobo sensor history? When I first did I was shocked to see the reports of CPU temperature excesses.
 

Meyers

Patron
Joined
Nov 16, 2016
Messages
211
Since the new version of FreeNAS, many people appear to have had failed boot media. That can cause all sorts of strange issues. What are you using for your boot pool?

The 16GB SATA DOM that came with the mini like I said.

Have you looked at the ASRock IPMI history

No but I'll definitely do that. What kinds of symptoms were you seeing?
 

Meyers

Patron
Joined
Nov 16, 2016
Messages
211
I am one of those who's experienced overheating on a C2750D4I

OK - been running stress for a few hours to max out all cores. Temps aren't getting above 66-67C which appears to be well within spec. I don't think the issue here is heat. I will keep digging.
 

Redcoat

MVP
Joined
Feb 18, 2014
Messages
2,924
Good luck.
My Mini's satadom bit the dust a couple of weeks ago. It surfaced at the same time as I was dealing with an intermittent powersupply issue that also looks like it took out the mobo. I don't know what came first.
I'm up and running again now - first new mobo I received had two bad sata ports so this turned in to a ~3-week exercise in total by the time that was RMA'd for an exchange. If I hadn't been under pressure at the time of the failure and so looked for the least-stressful way out I'd likely have swallowed hard and switched platforms. On reflection I wish that I had taken the opportunity...
I'm booting off a laptop hard drive now.
 

Chris Moore

Hall of Famer
Joined
May 2, 2015
Messages
10,080

Meyers

Patron
Joined
Nov 16, 2016
Messages
211
Good luck.
My Mini's satadom bit the dust a couple of weeks ago.

Already had one DOM fail and it was unmistakable. The system crashed hard. This one seems fine. The DOM has nearly zero I/O on it either way.

When the system seizes up it's the data pool that does it, not the OS drive. I know this because sometimes I can log in. When I do any kind of operation on the data pool however, the command will hang forever.
 

Meyers

Patron
Joined
Nov 16, 2016
Messages
211
Well I manually set the fan to full on and I haven't had a freeze now in over a week at least. It's possible that heat was indeed the issue.
 

Meyers

Patron
Joined
Nov 16, 2016
Messages
211
Spoke too soon. Started backing up over SMB and had it freeze again today. Someone else reported this happening while using NFS too so it might not just be me. I'll open a ticket.
 

Meyers

Patron
Joined
Nov 16, 2016
Messages
211

wdanen

Cadet
Joined
Mar 14, 2019
Messages
3
Did you ever get any further with this? (The bug report you linked is private)

I'm having a similar issue on my system, where after a while (usually a day or so) I get what I presume to be some sort of pool lockup. Can still ping the server's IP, but that's it; can't even get into shell from IPMI console, logging in via SSH just hangs forever, can't browse to UI website, etc. Have to hard reboot via IPMI to recover, but then can't find anything interesting at all in the logs.

This is on 11.2-U2.1, and when I revert to an 11.1-U7 environment I have no issues.

Motherboard: Supermicro X10SDV-4C+-TLN4F
Boot pool: 250GB Samsung 860 Evo sata SSD (2, mirrored) connected via LSI2008 (cross-flashed intel card, v20 FW)
Data Disks: 5 x WD Red 8TB
Memory: 32GB ECC
 

Stux

MVP
Joined
Jun 2, 2016
Messages
4,358
How is your swap usage when you freeze?

Your results sound to me like they could be caused by a memory leak.

Although, I’d expect kernel error messages on the console.

So, the HDs are on the intel SATA ports
 

wdanen

Cadet
Joined
Mar 14, 2019
Messages
3
How is your swap usage when you freeze?

Your results sound to me like they could be caused by a memory leak.

Although, I’d expect kernel error messages on the console.

So, the HDs are on the intel SATA ports

As far as I know, swap usage on this box is minimal. The historical data I have (for a year of 11.1-Ux versions) indicates it never rose above about 1GB of swap used. I confess, however, that I might not know the best way to capture that information, especially since the system becomes completely unresponsive when it "freezes". What would be the best way to capture the swap usage information just prior to a lock-up?

Yes, the data disks are on the chipset (intel) ports, with one of the chipset ports unused. I suppose I could try moving them over to the LSI HBA to see what happens. Never really occurred to me to try that since I've had completely flawless usage on this box via the chipset SATA ports throughout the 11.1 series.
 

Meyers

Patron
Joined
Nov 16, 2016
Messages
211
Did you ever get any further with this? (The bug report you linked is private)

I made it public. One response has some commands to run for when this happens. It might be helpful if you would run those and update the ticket next time it happens. I would keep a console open in anticipation of this.

This has only happened once in the past few weeks for me. Only thing I did was increase the fan speed to max but I don't think that had anything to do with it. 11.1 worked fine, 11.2 freezes. Nothing else has changed on this server in 18 months.
 
Top