Resource icon

How To: Change IPMI Sensor Thresholds using ipmitool

Many server motherboards, including the Supermicro X10 series, have their fans monitored by the BMC, which automatically attempts to recover stalling (of the motor, not in the aerodynamic sense) fans by spinning them up to maximum (PWM set to 100% speed). This behavior is determined by the sensor thresholds.

Supermicro X10 motherboard fan profiles are controlled via IPMI. Unfortunately, only very basic functions are exposed via its web interface and Supermicro's own IPMI utility is superbly well hidden on Supermicro's website and uses Java. The pragmatists among us would complain about Java, use the application, and move on - however, for some reason (*cough* Java *cough*) Supermicro application and its nice GUI do not always work properly. Thus, third-party universal IPMI tools are needed.

This guide should also be helpful for owners of other motherboards with similar problems or for those trying to learn how to interact with the motherboard's sensors using IPMI.

This is Part 1 of this guide and covers how to perform these operations “locally” (in other words, from FreeNAS’ console). Part 2 covers how to perform these operations remotely using ipmiutil, running on a client machine.

In this guide, I will be using ipmitool, running on the server – all interactions are via a remote terminal session.
Ipmitool can also be used remotely over the network, in the same way that ipmiutil is used in Part 2. If you're interested in this usage scenario, please read the relevant man pages.

Ipmitool forces the user to jump through more hoops in order to accomplish some tasks, but is included in FreeNAS and allows setting thresholds for sensors that are not attached.
Ipmiutil is somewhat more straightforward, but requires a remote client and refuses to change thresholds on sensors that are not attached.

While I will provide the necessary steps and explain some options, I recommend you read through the appropriate man page sections to learn about the various options.

Please note that all input may be case sensitive. For details, check the relevant man pages. Particularly, sensor names are case-sensitive.

Start by loading ipmitool:

~# kldload ipmi.ko

Now, run the following command to list all sensors:

~# ipmitool sensor list all

This will list the system’s sensors, current readings and current thresholds, as exemplified below:

Code:
[root@freenas] ~# ipmitool sensor list all
CPU Temp         | 37.000     | degrees C  | ok    | 0.000     | 0.000     | 0.000     | 95.000    | 98.000    | 100.000
System Temp      | 30.000     | degrees C  | ok    | -9.000    | -7.000    | -5.000    | 80.000    | 85.000    | 90.000
Peripheral Temp  | 40.000     | degrees C  | ok    | -9.000    | -7.000    | -5.000    | 80.000    | 85.000    | 90.000
PCH Temp         | 48.000     | degrees C  | ok    | -11.000   | -8.000    | -5.000    | 90.000    | 95.000    | 100.000
VRM Temp         | 39.000     | degrees C  | ok    | -9.000    | -7.000    | -5.000    | 87.000    | 92.000    | 97.000
DIMMA1 Temp      | na         |            | na    | na        | na        | na        | na        | na        | na
DIMMA2 Temp      | 31.000     | degrees C  | ok    | 1.000     | 2.000     | 4.000     | 80.000    | 85.000    | 90.000
DIMMB1 Temp      | na         |            | na    | na        | na        | na        | na        | na        | na
DIMMB2 Temp      | 32.000     | degrees C  | ok    | 1.000     | 2.000     | 4.000     | 80.000    | 85.000    | 90.000
FAN1             | na         |            | na    | na        | na        | na        | na        | na        | na
FAN2             | na         |            | na    | na        | na        | na        | na        | na        | na
FAN3             | 900.000    | RPM        | ok    | 300.000   | 400.000   | 500.000   | 3700.000  | 3800.000  | 3900.000
FAN4             | 900.000    | RPM        | ok    | 300.000   | 400.000   | 500.000   | 3600.000  | 3700.000  | 3800.000
FANA             | 1100.000   | RPM        | ok    | 400.000   | 600.000   | 800.000   | 25300.000 | 25400.000 | 25500.000
Vcpu             | 1.755      | Volts      | ok    | 1.242     | 1.260     | 1.395     | 1.899     | 2.088     | 2.106
VDIMM            | 1.453      | Volts      | ok    | 1.096     | 1.124     | 1.201     | 1.642     | 1.719     | 1.747
12V              | 12.000     | Volts      | ok    | 10.164    | 10.521    | 10.776    | 12.918    | 13.224    | 13.224
5VCC             | 4.969      | Volts      | ok    | 4.225     | 4.380     | 4.473     | 5.372     | 5.527     | 5.589
3.3VCC           | 3.344      | Volts      | ok    | 2.804     | 2.894     | 2.969     | 3.554     | 3.659     | 3.689
VBAT             | 3.015      | Volts      | ok    | 2.400     | 2.490     | 2.595     | 3.495     | 3.600     | 3.690
AVCC             | 3.329      | Volts      | ok    | 2.399     | 2.489     | 2.594     | 3.494     | 3.599     | 3.689
VSB              | 3.284      | Volts      | ok    | 2.399     | 2.489     | 2.594     | 3.494     | 3.599     | 3.689
Chassis Intru    | 0x0        | discrete   | 0x0000| na        | na        | na        | na        | na        | na


The thresholds listed are, in order: lnr, lcr, lnc, unc, ucr, unr

These are acronyms for:

Lower Non-Recoverable
Lower Critical
Lower Non-Critical

Upper Non-Critical
Upper Critical
Upper Non-Recoverable

Take note of the sensor names for the sensors you want to change (if you don't know which fans you want to edit, I recommend you check the IPMI log in the web interface to see which fans have sensor events being asserted).

Now, calculate the lowest and highest angular velocities your fan is rated to run at (check the manufacturer's specs). For instance, my Noctua NF-F12 IndustrialPPC 3000 PWM are rated at 750RPM +-20% at the low end, so 600RPM or less is an appropriate lower non-critical threshold value. To get the other values, subtract 100 for the lower critical and 200 for the lower non-recoverable.

Take note of your results.


To set the lower thresholds, use:

~# ipmitool sensor thresh "*sensor name*" lower *lnr* *lcr* *lnc*

Replacing *sensor name*, *lnr*, *lcr* and *lnc* with the appropriate values.


To set the upper thresholds, use:

~# ipmitool sensor thresh "*sensor name*" upper *unc* *ucr* *unr*

Replacing *sensor name*, *unc*, *ucr* and *unr* with the appropriate values.

It has come to my attention that ipmitool will gladly set the thresholds in absurd orders (lower non-critical lower than lower non-recoverable, for instance). The effects of this are unknown. In any case, I recommend that attention be paid to the proper order to avoid potential future issues.

Please note that invalid inputs will be rounded to the nearest valid value. On my system, for instance, setting a fan threshold to 540 will actually set it to 500. 550 is set as 600.

Repeat this for all fans whose thresholds you wish to change.
Author
Ericloewe
Views
131,243
First release
Last update
Rating
5.00 star(s) 9 ratings

More resources from Ericloewe

Latest updates

  1. Initial migration

    This is the original migration of the document to the Resources section. The original changelog...

Latest reviews

Very helpful explanation. Works in SCALE as well, I did it via root ssh.

I am a bit confused on the purpose of setting the maximum thresholds? Is there any reason to modify this? To let IPMI know how fast a fan can really go? Just trying to understand in the context of the terminology. Let's say my maximum RPM for my fan is 2200rpm. What should my numbers look like? Just a bit confused by the terminology of Non-Critical vs Critical vs Non-Recoverable with respect to an "upper" number. Would Upper Non-critical be the maximum rated RPM of my fan and then non-critical and non-recoverable higher numbers? I just see these numbers are reported with IPMITool show all three upper numbers for my fan to be beyond what is rated for maximum rpm.

Or does it not really matter and it'll just spin the fan as fast as possible when needed as long as all those values are above the maximum rated RPM?

One additional comment. I assume these changes will stick regardless of reboot or complete shutdown of the system? Even if BMC loses power?
Using an X12, just had to run:
sudo ipmitool sensor thresh FANA lower 50 100 150
for all my fan sensors and then changed the fans from full speed to standard, my office sounds great again. So grateful
Superb. Fixed the problem of my fans ramping up and down in speed and endless log entries in the SEL. Only issue I had was that after setting the thresholds my fans got stuck in the error state and ran at full speed, possibly they were in error at the moment I send the command. Rebooting the IPMI fixed this.
Helped me fix some log spam for one of my fans. Thank you.
Easy to follow, ear-saving guide.
Great help, but I do have one question. My Noctua NF-S12A fans are rated w/ min. rotational speed @ 300 RPM +/- 20%. So I set lower noncritical to 240 RPM, which the ipmi on my SMC X11SSH-F-O motherboard set down to 200 RPM. Following your instructions, I set *lc* to 140 and *lnr* to 40, and the motherboard reset them to 100 & 0 RPM respectively. Is this right, or should I have set them to 200 & 100?
Ericloewe
Ericloewe
It's unlikely to make a difference really. I wouldn't worry.
TOP NOTCH!
I rely on this from time to time when my BMC gets reset. Thank you!
Simply awesome. Thanks
Top