Monitoring ZFS from Nagios - a problem with jails

[ggallo]

Hi All!

I'm in a building of a FreeNAS based NAS for a client. The main duty will be storing backups made by Bacula Community. I'm proceeding well, almost everything works as expected. But I have a problem that I can't solve myself, in turn I read all the pages Google shown me, without solution.

The problem is that I don't know how to monitor the ZFS pool from Nagios.

I created a 'portjail' jail, installed NRPE and FreeIPMI. The Nagios instance successfully queries the FreeNAS system's main parameters over NRPE, OS stats by classic plugins and hardware stats through IPMI.

But I can't monitor the ZFS pool itself, because it's stats inaccessible within a jail.
I don't want to install custom software or configuration on the root FS, because that's not recommended and hard to re-implement on upgrades.

I read everything about SNMP in FreeNAS, but this documentation is very limited. I can't found what MIBs I must use to decode the SNMPwalk output, can't find that are tehe ZFA info available over SNMP, and so on.

I also read every word about the possibilities to access zpool list/status from the jail, without finding any working solution.

Are there any way to solve this situation? Who uses Nagios/Icinga for monitoring FreeNAS and how do you do this?

Some details about the sytem:
Intel R2000 base 2U rackmont server with a S2600 MB, one E5-2603 CPU, 32GB ECC RAM
LSI 9211-4i SAS HBA (until Intel RMS25JB080 arrives, thats an LSI 9207-8i SAS HBA from Intel) plus an Intel RES2SV240 SAS Expander
4x Intel 1GbE NIC (igb driver), first 2 used in an LACP setup (this HP switch enables only 2 way LACP)
OS (FreeNAS 9.2.1.2) runs from a Kingston 8GB USB drive
12x 2TB Seagate Constellation CS SATA HDDs, arranged in one ZFS pool, 2x 6 disk RAIDZ2

I'm very happy with the performance at first sight: non-compressed dataset reads/writes about 950-980MB/s, NFS speeds about 90-100MB/s both ways (with FreeBSD 10 client), CIFS reads 60MB/s writes 80-100 MB/s (Windows 7 64 bit, SSD). Fortunately, FreeNAS 9 has Bacula-SD plugin, works well, test backups made at 108MB/s speed with it over the LAN.

 

[Rand]

I am not using it yet, but i would have expected that a standard FreeBsd9.2 MIB should contain all relevant values? Does it not?

I dont think the 'access pools' from jail idea is going to prove successful, they are *supposed* to be isolated...

 

[ggallo]

I don't know FreeBSD 9.2 MIB what contains. I found only one MIB named FreeBSD, but it contain only developer and organisation names (about 4 of them), no useful data modelled.

I know that jails are for isolation. But I can't install Nagios NRPE and plugins in base system/outside the jails - it's not supported by design. So I must work in a jail, but from there I can't ask ZFS to present status/health info for me...

I could SSH in, commanded by Nagios for this one check. But I think that's not better than querying pool status from a jail...

 

[Rand]

Well a dedicated user only for execution of one script and no login shell possibly would be ok from a security point of view.
Depending on whether you need to gather statistics as well...

Have you checked
http://serverfault.com/questions/391167/monitor-disk-via-snmp-polling?rq=1
http://serverfault.com/questions/262829/how-to-monitor-zfs-with-snmp-in-freebsd
http://jose-manuel.me/2013/09/how-to-extend-snmp-in-nexentastor-to-get-zfs-stats/


Depending on *what exactly* you need to check - disk space, pool status etc a script (or several) might be the easiest route.

 

[ggallo]

Thanks for the links, I already saw mostly all of them.
I can find the devices under HOST-RESOURCES-MIB, but there are only type and size data, that I can access other ways, too. But there aren't usable status informations. The Nexenta's ZFS extensions cannot be used with FreeNAS, because FreeNAS's SNMP agent don't have the required modules to minotor ZFS, I think.

 

[ggallo]

I downloaded the Nexenta ZFS extensions, and I found that those are simple Python scripts to extend local SNMP agent. I try to adapt those for FreeNAS.
If it's possible, I will share the resulting script(s) and MIB definition(s).

 

[Rand]

SNMP Extension is a feasible way to run the scripts without using ssh if thats not possible - good luck :)

 

[Rand]

So have you gotten it running?:)

 

[ggallo]

Yes, I am. But not with SNMP, instead with cron and some shell scripts.

I get a ZFS check script from Nagios Exchange, and modified (extended) it to save the results to a file. I run this as a cron job on FreeNAS. This file accessible from the Nagios jail and I made a wrapper script that reads this saved results file, and send the contents and exit code to Nagios (through NRPE).
I made the reverse path, too. The wrapper script gets the check script's parameters from Nagios, save to a(n another) file where the cron-started main check script can take those for the next run.

If anyone interested, I can share there scripts somewhere.