ARC completely missing on one of two systems?

Patrick M. Hausen

Hall of Famer
Joined
Nov 25, 2013
Messages
7,737
Hi all,

I now have two systems running this identical software:
Code:
12.0-MASTER-202006040424
FreeBSD freenas-pmh.local 12.1-STABLE FreeBSD 12.1-STABLE 34b610ce0f2(freenas/12-stable) TRUENAS  amd64


System 1, top:
Code:
last pid:  8870;  load averages:  0.91,  1.15,  1.06                                                        up 1+01:32:09  16:39:05
149 processes: 1 running, 147 sleeping, 1 zombie
CPU:  1.1% user,  0.0% nice,  4.2% system,  0.1% interrupt, 94.7% idle
Mem: 6503M Active, 9815M Inact, 10G Laundry, 33G Wired, 3480M Free
ARC: 27G Total, 7061M MFU, 19G MRU, 34M Anon, 1467M Header, 1296M Other
     24G Compressed, 31G Uncompressed, 1.26:1 Ratio
Swap: 16G Total, 600M Used, 15G Free, 3% Inuse


System 2, top:
Code:
last pid:  5819;  load averages:  0.49,  0.41,  0.27                                                        up 0+00:13:39  16:44:06
37 processes:  1 running, 36 sleeping
CPU:  1.9% user,  0.0% nice,  1.9% system,  0.0% interrupt, 96.2% idle
Mem: 994M Active, 91M Inact, 924M Wired, 14G Free
Swap: 16G Total, 16G Free


Note the complete absence of ARC stats. Likewise the second system reports 0 values via collectd --> Influx --> Grafana. (Which is where I noticed it first). First system looks good.
The hit ratio in Grafana is broken for both, but that's a secondary problem.

To file a proper issue in JIRA I would need some help. What lower level information (sysctl?) to look for on both systems?

Kind regards,
Patrick
 

Patrick M. Hausen

Hall of Famer
Joined
Nov 25, 2013
Messages
7,737
Well it looks like the stats are missing because the ARC is missing:
Code:
root@freenas2-pmh[~]# sysctl kstat.zfs.misc.arcstats.size
kstat.zfs.misc.arcstats.size: 0


Any idea what could lead a system to behave like this? Even with just 16G of RAM but no services running besides smartd and sshd there should be plenty of room for cache ...

Just slightly less than a G in "wired" state hints that there is indeed no ARC at work ...
 

Yorick

Wizard
Joined
Nov 4, 2018
Messages
1,912
I’d file this with a debug and output from arc_summary.py (or was it arc-summary.py?)
 

Patrick M. Hausen

Hall of Famer
Joined
Nov 25, 2013
Messages
7,737

Ericloewe

Server Wrangler
Moderator
Joined
Feb 15, 2014
Messages
20,175
Probably worth a bug report, just to make sure it doesn't get forgotten.
 

Patrick M. Hausen

Hall of Famer
Joined
Nov 25, 2013
Messages
7,737
See the Twitter discussion, please. It's fixed in upstream.
 

Ericloewe

Server Wrangler
Moderator
Joined
Feb 15, 2014
Messages
20,175
I did, I mean that the ticket may be important to avoid the fix being left behind for a while because it didn't make it into the next pull from upstream or whatever.
 

Patrick M. Hausen

Hall of Famer
Joined
Nov 25, 2013
Messages
7,737
Top