Register for the iXsystems Community to get an ad-free experience and exclusive discounts in our eBay Store.

ARC completely missing on one of two systems?

Patrick M. Hausen

Neophyte Sage
Joined
Nov 25, 2013
Messages
1,480
Hi all,

I now have two systems running this identical software:
Code:
12.0-MASTER-202006040424
FreeBSD freenas-pmh.local 12.1-STABLE FreeBSD 12.1-STABLE 34b610ce0f2(freenas/12-stable) TRUENAS  amd64


System 1, top:
Code:
last pid:  8870;  load averages:  0.91,  1.15,  1.06                                                        up 1+01:32:09  16:39:05
149 processes: 1 running, 147 sleeping, 1 zombie
CPU:  1.1% user,  0.0% nice,  4.2% system,  0.1% interrupt, 94.7% idle
Mem: 6503M Active, 9815M Inact, 10G Laundry, 33G Wired, 3480M Free
ARC: 27G Total, 7061M MFU, 19G MRU, 34M Anon, 1467M Header, 1296M Other
     24G Compressed, 31G Uncompressed, 1.26:1 Ratio
Swap: 16G Total, 600M Used, 15G Free, 3% Inuse


System 2, top:
Code:
last pid:  5819;  load averages:  0.49,  0.41,  0.27                                                        up 0+00:13:39  16:44:06
37 processes:  1 running, 36 sleeping
CPU:  1.9% user,  0.0% nice,  1.9% system,  0.0% interrupt, 96.2% idle
Mem: 994M Active, 91M Inact, 924M Wired, 14G Free
Swap: 16G Total, 16G Free


Note the complete absence of ARC stats. Likewise the second system reports 0 values via collectd --> Influx --> Grafana. (Which is where I noticed it first). First system looks good.
The hit ratio in Grafana is broken for both, but that's a secondary problem.

To file a proper issue in JIRA I would need some help. What lower level information (sysctl?) to look for on both systems?

Kind regards,
Patrick
 

Patrick M. Hausen

Neophyte Sage
Joined
Nov 25, 2013
Messages
1,480
Well it looks like the stats are missing because the ARC is missing:
Code:
root@freenas2-pmh[~]# sysctl kstat.zfs.misc.arcstats.size
kstat.zfs.misc.arcstats.size: 0


Any idea what could lead a system to behave like this? Even with just 16G of RAM but no services running besides smartd and sshd there should be plenty of room for cache ...

Just slightly less than a G in "wired" state hints that there is indeed no ARC at work ...
 

Yorick

Neophyte Sage
Joined
Nov 4, 2018
Messages
1,251
I’d file this with a debug and output from arc_summary.py (or was it arc-summary.py?)
 

Patrick M. Hausen

Neophyte Sage
Joined
Nov 25, 2013
Messages
1,480

Ericloewe

Not-very-passive-but-aggressive
Moderator
Joined
Feb 15, 2014
Messages
16,259
Probably worth a bug report, just to make sure it doesn't get forgotten.
 

Patrick M. Hausen

Neophyte Sage
Joined
Nov 25, 2013
Messages
1,480
See the Twitter discussion, please. It's fixed in upstream.
 

Ericloewe

Not-very-passive-but-aggressive
Moderator
Joined
Feb 15, 2014
Messages
16,259
I did, I mean that the ticket may be important to avoid the fix being left behind for a while because it didn't make it into the next pull from upstream or whatever.
 

Patrick M. Hausen

Neophyte Sage
Joined
Nov 25, 2013
Messages
1,480
Top