FreeNAS + ESXI Lab Build Log

[Maelos]

I am less concerned with the drives at the moment as many are not in the system and none of them are in use. I want to get baseline monitoring and alerts for the board, processors, fans, etc. Does it appear that I am doing something wrong with the IPMI setup? Is it just due to to the BIOS being out of date? Should I drop it and go for IPMIView, FreeNAS, ESXI, or other tool?

Also, when looking at Priviledged Access Management (PAM) solutions, has anyone here tried one of these?

1. http://www.keycloak.org/
2. https://evolveum.com/
3. http://www.soffid.com/
4. https://www.gluu.org/
5. https://github.com/osiam/osiam

pfSense + LetsEncrypt? Hmm

 

[Stux]

Ipmitool can provide a list of all the temps etc. could email that periodically.

 

[Maelos]

I may resort to using ESXI or Freenas with IPMI tool, but I'd like to stay independent of an OS if possible. This is to learn the full use of IPMI as well as keeping apart from the higher operating levels. I updated the IPMI this morning from home and configured the the settings again, but still no luck. I will take screenshots and show below. Of note, I made a new gmail account without 2FA and kept the password without special characters (IPMI warned me not to use them, which was a nice improvement after the updated firmware). I am not sure about the IP in the alert settings.

I found it, the IPMI had a weird DNS set (0.1.1.0). Once I corrected it (thank you Goggle for 8.8.8.8) it worked. I also had to allow access to Gmail from "less secure apps".


Now I need to go back to Stux's thread to figure out how to lower the thresholds, or remove them, for the FANs. All I really care about at the moment is temperature.

//Other thoughts for identity and access management - LDAP and LinOTP with Google Authenticator

 

[Chris Moore]

Now I need to go back to Stux's thread to figure out how to lower the thresholds, or remove them, for the FANs. All I really care about at the moment is temperature.

The drives generate a lot of heat. You can't really figure out what the temperatures are going to be until you have them in. Also, any drive trays that don't have a drive installed need to have a airflow diverter installed. Sometimes called a plug. Looks like this:

It makes a huge difference. Honest.

You can get them here if you need any:
https://www.ebay.com/itm/Supermicro...t-Swap-Caddy-05-SC82708-Lot-of-4/222515802292

 

[Maelos]

Ugh, I am going to need a lot of those. Bummer. The PCH on the board is registering over 40 degrees C, but the processors remain under 35 degrees. Im not sure what the *temp on the* HDDs are, but the data is likely on on FreeNAS as it has the HBA passed directly to it. Where is the best place to run these monitoring/reporting scripts and use things like IPMI tool? I know the most likely answer here will be FreeNAS, but are there any advantages to doing it through ESXI, IPMI, or another VM, if possible?

Actually now that I think about it, I answered my own question. I will be looking into making a script that will alert if temperatures reach a certain threshold with both the temperatures and the fan speeds as well as send scheduled messages. The only way to do this with the HDD temps is through FreeNAS. Can FreeNAS IPMI get the temperature data on drives directly conected to the mobo - aka the SSD and SATA-DOM next to the mobo?

Also, does ESXI not have the ability to see the dedicated IPMI interface? Not a big deal I suppose, just curious.

Chris, why that dummy drive over others that block the flow through the bottom and sides?

 

[Chris Moore]

The PCH on the board is registering over 40 degrees C

PCH temp of 44 ºC is nothing to worry about. If I recall correctly, it is good up to 95 ºC.

Im not sure what the HDDs are

HDD = Hard Disk Drive, kind of like SSD = Solid State Drive.

Where is the best place to run these monitoring/reporting scripts

I monitor my drive and CPU temps from FreeNAS using the scripts I gave you a link to on github. I don't worry about the PCH and other temperatures because they usually stay within acceptable range as long as there is enough air flow to keep the drives cool. In my experience.

I know the most likely answer here will be FreeNAS, but are there any advantages to doing it through ESXI, IPMI, or another VM, if possible?

Sorry, I have not tried because I found the monitoring from FreeNAS sufficient for my purposes.

Chris, why that dummy drive over others that block the flow through the bottom and sides?

That is the one that comes with the server from Supermicro, if you buy it new and it is intended to ensure that enough air is blocked from those bays without drives that the bays with drives still get enough air flow to cool the drives because the heat of the drives is the biggest issue after the CPU, but you have the CPU handled with those Noctua coolers. You can find a lot of places selling the ones they remove when they put drives in. The 24bay chassis I bought didn't come with drive trays, so had to buy trays and the dummy part too.

 

[Maelos]

Alright, I am in a bit of IPMI overload (what do all of these acronyms mean!?). At this point I think it is a given that I will need to invest some time in learning IPMI and how to script what I want. I am gonna have to look through the manuals to figure out what AMI, SMT are and what the different tools can do. I know some of this has already been posted, but I'm going to need to understand a good bit of it to mash together the scripts to report as I like. I would like both a regular reporting script as well as a script that reacts off of temperature alerts.

Poor wording on the HDD line, I meant their temps. Corrected.

As far as the drive bays, the one you show has a wide open hole in the bottom and vents on the front. Why would that be used as opposed to https://www.ebay.com/itm/112412099589 with a solid shell. I imagine they work like the 1U..."blockers" used in racks to help direct airflow where it is needed most.

In summary, my current frustration/challenge is that with the undervolting of the chassis fans via the 3 pin resistor wires, I will get excessive alerts from the default IPMI settings. This means that I will need, at min, to change those thresholds. I would also like to set up scheduled emails displaying the drive and CPU temp, perhaps even with averages, max, and mins. Furthermore I would like to create an alert, similar to what IPMI does, that incorporates the temps of the HDDs, SSD (may not be warranted), and processors. I believe much of this has been done before through various scripts in the forums, it is just a matter of combining them, researching the correct thresholds, and implement/testing. I still need to choose what tool to use within FreeNAS.

Also Chris I will take a look back at Untangle for identity and access management.

 

[Stux]

Here’s the solution to empty bays that I used on my Norco build...

https://forums.freenas.org/index.ph...o-x10-sri-f-xeon-e5-1650v4.46262/#post-315996

 

[Maelos]

I could do that. I recently brought home a good bit of foam that was being trashed at the data center. I could use that in place of the packaging as I'd like to keep it for a future server move.

The question still remains, what is the difference between the full block fills and that Chris suggested with vents. Do they have different use purposes, and if so, what are they?

 

[Chris Moore]

Poor wording on the HDD line, I meant their temps. Corrected.

I like to keep my drives between 30 ºC and 40 ºC but lower is better. Most drives I have read specs on are rated to 55 ºC or even up to 65 ºC, but I wouldn't want them that hot. Hot drives don't last as long. In my 48bay server, I have an average input air time of 77 ºF, and the hottest drives are the boot drives that are all the way at the back of the chassis. Those two run about 42 ºC but they are laptop drives and I know they are rated to a pretty high temp. All the drives in the storage bay are between 28 ºC at the front to 36 ºC at the back and I am satisfied with that.

Here’s the solution to empty bays that I used on my Norco build...

Since the Supermicro drive trays don't have a bottom like the Norco trays, that might be a little harder to pull off. You might be able to tape it in place.

 

[Chris Moore]

As far as the drive bays, the one you show has a wide open hole in the bottom and vents on the front. Why would that be used as opposed to https://www.ebay.com/itm/112412099589 with a solid shell. I imagine they work like the 1U..."blockers" used in racks to help direct airflow where it is needed most.

That is for a different model chassis. Totally different drive tray.

 

[Maelos]

That is for a different model chassis. Totally different drive tray.

Where did you find that information? I cannot seem to find anything specifying what dummy or plug trays go to which chassis, either on the website or within the manuals. My only knowledge is gained from looking at pictures.

 

[Chris Moore]

Where did you find that information? I cannot seem to find anything specifying what dummy or plug trays go to which chassis, either on the website or within the manuals. My only knowledge is gained from looking at pictures.

I learned from looking at these things in the datacenter for the past two decades.
The documentation is probably out there it is the finding of it that is difficult.

 

[Maelos]

Fair enough. I will order some soon.

Code:

#!/bin/sh

### Parameters ###

# Specify your email address here:
email=":-)"

freenashost=$(hostname)
smartctl=/usr/local/sbin/smartctl
logfile="/tmp/temp_report.tmp"
dt=$(date -u)
subject="Temperature Report for ${freenashost} on ${dt} "

# We need a list of the SMART-enabled drives on the system. Choose one of these
# three methods to provide the list. Comment out the two unused sections of code.

# 1. A string constant; just key in the devices you want to report on here:
#drives="da1 da2 da3 da4 da5 da6 da7 da8 ada0"

# 2. A systcl-based technique suggested on the FreeNAS forum:
#drives=$(for drive in $(sysctl -n kern.disks); do \
#if [ "$("${smartctl}" -i /dev/${drive} | grep "SMART support is: Enabled" | awk '{print $3}')" ]
#then printf ${drive}" "; fi done | awk '{for (i=NF; i!=0 ; i--) print $i }')

# 3. A smartctl-based function:
{	
		
gs_smartdrives=""	
		
gs_drives=$("$smartctl" --scan | awk '{print $1}')	
		
for gs_drive in $gs_drives; do	
		
	gs_smart_flag=$("$smartctl" -i "$gs_drive" | grep "SMART support is: Enabled" | awk '{print $4}')	
		
	if [ "$gs_smart_flag" = "Enabled" ]; then	
		
	 gs_smartdrives="$gs_smartdrives $gs_drive"	
		
	fi	
		
done	
		
echo "$gs_smartdrives"	
		
}	
		
drives=$(get_smart_drives)				

# end of method 3.

### Set email headers ###
(
  echo "To: ${email}"
  echo "Subject: ${subject}"
  echo "Content-Type: text/html"
  echo "MIME-Version: 1.0"
  printf "\r\n"
) > ${logfile}

### Set email body ###
echo "<pre style=\"font-size:14px\">" >> ${logfile}

#### Start of CPU and drive temps ####
####							  ####

# Optionally uses IPMI to report temperatures of the system CPU(s)
#
# If IPMI is disabled (see 'use_ipmi' below) then the script uses
# sysctl to report the CPU temperatures. To use IPMI, you must
# provide the IPMI host, user name, and user password file.

# IPMI support: set to a postive value to use IPMI for CPU temp
# reporting, set to zero to disable IPMI and use 'sysctl' instead:
use_ipmi=1

# IP address or DNS-resolvable hostname of IPMI server:
ipmihost=:-)

# IPMI username:
ipmiuser=:-)

# IPMI password file. This is a file containing the IPMI user's password
# on a single line and should have 0600 permissions:
ipmipwfile=/root/ipmi_password

# Full path to 'ipmitool' program:
ipmitool=/usr/local/bin/ipmitool

#############################
# CPU temperatures:
#############################

(
if [ "$use_ipmi" -eq 0 ]; then
  cpucores=$(sysctl -n hw.ncpu)
  echo '=== CPU (%s) ===\n' "$cpucores"
  cpucores=$((cpucores - 1))
  for core in $(seq 0 $cpucores); do
	temp=$(sysctl -n dev.cpu."$core".temperature|sed 's/\..*$//g')
	if [ "$temp" -lt 0 ]; then
	  temp="--n/a--"
	else
	  temp="${temp}C"
	fi
	printf 'CPU %2.2s: %5s\n' "$core" "$temp"
  done
  echo ""
else
  cpucores=$("$ipmitool" -I lanplus -H "$ipmihost" -U "$ipmiuser" -f "$ipmipwfile" sdr elist all | grep -c -i "cpu.*temp")
 
  echo '=== CPU (%s) ===\n' "$cpucores"
  if [ "$cpucores" -eq 1 ]; then
	temp=$("$ipmitool" -I lanplus -H "$ipmihost" -U "$ipmiuser" -f "$ipmipwfile" sdr elist all | grep "CPU Temp" | awk '{print $10}')
	if [ "$temp" -lt 0 ]; then
	   temp="-n/a-"
	else
	   temp="${temp}C"
	fi
	printf 'CPU %2s: %5s\n' "$core" "$temp"
  else
	for core in $(seq 1 "$cpucores"); do
	  temp=$("$ipmitool" -I lanplus -H "$ipmihost" -U "$ipmiuser" -f "$ipmipwfile" sdr elist all | grep "CPU${core} Temp" | awk '{print $10}')
	  if [ "$temp" -lt 0 ]; then
		 temp="-n/a-"
	   else
		 temp="${temp}C"
	  fi
	  printf 'CPU %2s: [%s]\n' "$core" "$temp"
	done
  fi
fi) >> ${logfile}

#############################
# Drive temperatures:
#############################
echo "" >> ${logfile}
echo "=== DRIVES ===" >> ${logfile}

for drive in $drives; do (
  serial=$("$smartctl" -i "$drive" | grep "Serial Number" | awk '{print $3}')
  capacity=$("$smartctl" -i "$drive" | grep "User Capacity" | awk '{print $5 $6}')
  temp=$("$smartctl" -A "$drive" | grep "194 Temperature" | awk '{print $10}')
  if [ -z "$temp" ]; then
	temp=$("$smartctl" -A "$drive" | grep "190 Airflow_Temperature" | awk '{print $10}')
  fi
  if [ -z "$temp" ]; then
	temp="-n/a-"
  else
	temp="${temp}C"
  fi
  dfamily=$("$smartctl" -i "$drive" | grep "Model Family" | awk '{print $3, $4, $5, $6, $7}' | sed -e 's/[[:space:]]*$//')
  dmodel=$("$smartctl" -i "$drive" | grep "Device Model" | awk '{print $3, $4, $5, $6, $7}' | sed -e 's/[[:space:]]*$//')
  if [ -z "$dfamily" ]; then
	dinfo="$dmodel"
  else
	dinfo="$dfamily ($dmodel)"
  fi
  printf '%6.6s: %5s %-8s %-20.20s %s\n' "$(basename "$drive")" "$temp" "$capacity" "$serial" "$dinfo"
) >> ${logfile}
done

### End HTML Style ###
echo "</pre>" >> ${logfile}

### Send report ###
if [ -z "${email}" ]; then
  echo "No email address specified, information available in ${logfile}"
else
  sendmail -t < ${logfile}
  sendmail ${email} < ${logfile}
#  rm ${logfile}
fi

I am getting an error message from gmail, but still getting the content (to be updated in a second from outside the VM).

Code:

---------- Forwarded message ----------
From: :-)
To: :-)
Cc:
Bcc:
Date: Fri, 30 Mar 2018 16:36:49 -0000
Subject: Temperature Report for HomeLab on Fri Mar 30 16:36:44 UTC 2018
=== CPU (%s) ===\n 2
CPU %2s: [%s]\n 1 28C
CPU %2s: [%s]\n 2 31C

=== DRIVES ===
%6.6s: %5s %-8s %-20.20s %s\n da2 -n/a-   

I realize some of this may be because I don't have smartctl installed (not sure here), I have mashed together two scripts (took parts from SMART report and parts from temp script), or some other reason. I am just happy to have a script working from FreeNAS atm. My Linux ability is painfully minimal, but getting there one large rock at a time.

--I think I figured out the error...gmail.com and google.com are not the same...Bing, fixed. Now I just have to figure out the formatting. I believe the replacement of printf with echo may have thrown of the options presented, though not sure. Looking at it again now outside of the virtualized world I see that I kept the smart_drives fuction from the SMART report where I should have copied over the one meant for the temperature report. Will look into this.

Fixed it all. It now outputs

Code:

=== CPU (2) ===
CPU  1: [28C]
CPU  2: [30C]

=== DRIVES ===
   da2:   32C [160GB]  4LS2RGHL			 Seagate Barracuda 7200.9 (ST3160812AS)

That's all I need for now. I can add more later. YAY!

One last question before I go - because I am emailing from and to the same email address, I am getting duplicates in the inbox. How would I change the code to fix this? It only seems to do duplicates when I run the script manually, not with the CRON job. Figured that out, I uncommented two of the email commands on the bottom. I'm not sure what -t does, but I left the second one working.

 

[Maelos]

Current predicament - getting a Prometheus Node_Exporter to run on FreeNAS. I think my lack of Linux-fu (and FreeBSD-fu) is impairing my ability to understand and implement the needed fix. I feel like I need some help, possibly even a step by step approach to getting this fixed. Here comes a bunch of links that I have found regarding the issue (still trying to piece it all together):

https://groups.google.com/forum/#!topic/prometheus-developers/Hdqebk-nfhk
https://github.com/prometheus/node_exporter/issues/293
https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=212467
https://www.freshports.org/net-mgmt/prometheus
https://github.com/prometheus/node_exporter/tree/master/vendor
https://github.com/rfrail3/grafana-dashboards.git

https://svnweb.freebsd.org/ports?view=revision&revision=424002 <- Does this mean that there is a tool already in FreeNAS for this? I will continue to research but hope someone can help guide me a bit. I tried downloading, removing from the tar, and running the program, but no go. The error presented is "ELF Binary '3' not known". I looked and the 64bit Linux bin is not runnable (https://forums.freebsd.org/threads/cannot-run-linux-application-on-freebsd-amd64.29389/). The file is also unreadale so I am guessing that the system cannot read the language as a start.

P.S. - This hopefully will be the last hurdle before I can get to proper FreeNAS testing and setup.

Well I got Go installed on the FreeNAS server and was able to do the sample script here https://golang.org/doc/install. Not sure what to do with the node exporter, nothing works yet.

----

I think I've got a better understanding now. There should be a port (package) I can do for Free bad that has a working node exporter. Now I just need to find the location and get it installed.

----
Trying in IRC to get some help.
[11:01] <Maelos> Can someone help me figure out and define the OS version of FreeBSD running on FreeNAS 11.1. I am trying to use the port/sysutil/node_exporter for Prometheus/Grafana monitoring
[11:01] <Maelos> my FreeBSD fu is not strong
[11:03] <Maelos> when trying to use "make install" within the /usr/ports/sysutils/node_exporter/ directory I get an error stating "make: 'usr/ports/Mk/bsd.pork.mk' line 1171: Unable to determine OS version. Either define OSVERSION, install /usr/include/sys/param.h or define SRC_BASE.
[11:03] <Maelos> any ideas

I made the OSVERSION tp 1101502 (https://www.freebsd.org/doc/en/books/porters-handbook/book.html#versions-11) as it is the closest to the uname -a result I got (r321665). The command "make install" ran but threw a few errors. More fun...

 

[Chris Moore]

Current predicament - getting a Prometheus Node_Exporter to run on FreeNAS.

I wish I knew more about this Prometheus Node_Exporter software, but it sounds like you are trying to install it directly inside FreeNAS and I am pretty sure that isn't going to work out. The NAS systems I use at work get nothing at all added to them and the ones at home only have Plex added in a Jail. I would guess that if you want to run this package, you will need to install it in a Jail instead of trying to install it directly in FreeNAS.
This might be a question for some one like @Jailer , @danb35 , @jgreco or @dlavigne ... If you have a question about something specific like this, you might want to start a new thread for it with a subject that would draw people in to look at it.
Sometimes these build log threads get overlooked but if a new topic comes up, those threads get more attention.

 

[Maelos]

I actually got it working, though more work needs to be done. Thanks to Vall and SweetAndLow on the IRC. There obviously needs to be some fine tuning done in multiple parts, but it IS communicating. Most of the credit goes to https://blog.yo61.com/installing-prometheus-node_exporter-on-freenas/ who gave me the method that actually worked.

Well, I feel like I am stuck again trying to figure out where to configure it. I might have to back out to the FreeBSD OS and re-code it from there before packaging for FreeNAS. I hope with the bash scripts that I have mashed together and a little bit of general coding knowledge I will be able to stumble to a workable result.

 

[Chris Moore]

View attachment 23698

I actually got it working, though more work needs to be done. Thanks to Vall and SweetAndLow on the IRC. There obviously needs to be some fine tuning done in multiple parts, but it IS communicating. Most of the credit goes to https://blog.yo61.com/installing-prometheus-node_exporter-on-freenas/ who gave me the method that actually worked.

Well, I feel like I am stuck again trying to figure out where to configure it. I might have to back out to the FreeBSD OS and re-code it from there before packaging for FreeNAS. I hope with the bash scripts that I have mashed together and a little bit of general coding knowledge I will be able to stumble to a workable result.

Please share your results as it might help others in the future.

 

[Maelos]

I will make a procedure with all the steps to share. I think I have an idea on how to add in the metrics I got through bash scripts without needing to modify Prometheus or it's node extractor. Instead of having the output jnto an email, I will have my script append the info to the text file that the node exporter already uses. I will need to learn the format and make sure that the node extractor does not overwrite the appnded daya, or it somehow is added consistently after each update. More to come.

Found https://rsmitty.github.io/Prometheus-Exporters/ today. Found https://github.com/prometheus/client_golang/issues/269 yesterday and a few more links that I added on my mobile favorites.
Also found https://github.com/prometheus/node_exporter/blob/master/text_collector_examples/smartmon.sh <- basically what I want to do, though not sure I want all those args. It could be useful to those of you running smart tests.

I still have a bit of work to go and only seem to get a few minutes a day (also my choice). I need to print out, sit down, and go through the .go code to really get a better grasp. It has been over a decade since I have done any serious coding (in Java) and its frustratingly slow getting back into it. I need to either use a bash script to append the CPU information to the file (default location is ...) in the correct format or I need to modify/write a new <collector>.go file looking for specifically what I want before compiling the package for FreeNAS. Both are temporary measures until I can script an entire exporter from scratch OR figure out how to make the application stick with FreeNAS through updates. More to come as I get time to research, hopefully later this week.

Bingo - https://gobyexample.com/spawning-processes. I can use this with a go file to run external processes. This follows the idea of making the Go program OS independent as mentioned here: https://www.youtube.com/watch?v=COCUqAwAbD0

4/19/18 - Just an update for anyone watching. I am taking a Go course on Udemy (https://www.udemy.com/learn-how-to-code/) and getting back into the swing of programming. It is an exciting shot at redemption. I changed majors from Computer Science at RIT over 10 years ago and thought I'd never be back. I hope to learn enough to not only make a custom collector for FreeNAS, but to use it to help at my work and grow into a DevOps/SRE professional. Thanks to all of you're help. I am still here, just been busy. I set a date for early June to be done with the course and have the collector ready, as well as finish studying and test for my CCENT (planned to finish that before December...).

5/9/18 - I created a second thread devoted to the custom collector I have made but still need to run. Not much traction there or anywhere else I have asked for help. I more or less finished a Go class on Udemy which has helped, and I will get back to it when I have the chance. https://forums.freenas.org/index.ph...collector-for-prometheus-node-exporter.63462/

 

[Maelos]

I am proud enough of my little accomplishment to make this a bump. Life has been busy but I got my FreeNAS custom exporter for Prometheus working. You can see it at www.github.com/maelos/freenas_exporter . While it is basic, it is working and is the start of something much grander. I dream of making an exporter that can handle all of the reports that the community has developed, presenting them via Grafana, with a flexible alert manager. This will take the humble daily email up to an enterprise standard of monitoring metrics, alerting, and insight. Yaay...