Supermicro fan control, IPMIUTIL or FEEIPMI?

[vikingboy]

Looks like we are both 'right' - although in the spirit of friendly internet banter...I think I'm more right :)

from the X9SRL manual,
Page 1-5, "FANA Thermal fan header for I/O add-on cards"
Page 2-23, " Fan1~Fan4: System Fans Fan A: I/O Fan"

from the x10-slm+-F manual
Page 2-25 "E. FAN A (CPU Fan)

However from an engineering perspective, I'd suggest your manual is wrong as no engineer would put the CPU fan header at the other side of the motherboard. FAN4 is clearly ideally placed for an exhaust fan and it would make sense to tie that in with the CPU1/2/3 headers, not FANA.

EDIT: Out of curiosity I downloaded a number of manuals and checked them all. Its inconsistent at best in the x10 range. Either trust the manual or verify what happens when you stop the CPU if FAN A adjust itself when set to optimal. I'll verify mine for peace of mind when its apart again when I experiment with some fans that are coming in the next couple of days.

 

[pschatz100]

According to the manuals on many of the single CPU motherboards (no, I did not check them all) and confirmed by posts in the Supermicro FAQ, Fans 1-4 are controlled by CPU temperature, and Fan A is controlled by system temperature. Think of it as two PWM "zones": one for Fans 1-4 and one for FanA. Fans 1-4 are all controlled together - they are not individually controlled.

The CPU fan should be plugged into one of the fan headers 1-4 (on some specific motherboards, they recommend Fan1.) FanA is for an add-on card, although it can also be used for a case fan. The Optimum setting runs at 30% PWM, the Standard setting runs at 50% PWM, and the Full speed setting runs the fans at max speed. I'm not certain how PWM values relate to specific fan rpm as various fans appear to act differently.

It makes sense. Just look at your motherboard - often, the FanA header is in an inconvenient location but it is always near the PCIe slots. Putting the CPU on FanA will work as long as the fan is spinning fast enough to keep the CPU at a reasonable temp, but FanA is not monitoring your CPU temperature.

 

[vikingboy]

According to the manuals on many of the single CPU motherboards (no, I did not check them all) and confirmed by posts in the Supermicro FAQ, Fans 1-4 are controlled by CPU temperature, and Fan A is controlled by system temperature. Think of it as two PWM "zones": one for Fans 1-4 and one for FanA. Fans 1-4 are all controlled together - they are not individually controlled.

The CPU fan should be plugged into one of the fan headers 1-4 (on some specific motherboards, they recommend Fan1.) FanA is for an add-on card, although it can also be used for a case fan. The Optimum setting runs at 30% PWM, the Standard setting runs at 50% PWM, and the Full speed setting runs the fans at max speed. I'm not certain how PWM values relate to specific fan rpm as various fans appear to act differently.

It makes sense. Just look at your motherboard - often, the FanA header is in an inconvenient location but it is always near the PCIe slots. Putting the CPU on FanA will work as long as the fan is spinning fast enough to keep the CPU at a reasonable temp, but FanA is not monitoring your CPU temperature.

Absolutely agree with everything you said re dual zones etc. The only thing that doesn't tally with what I'm seeing here and part of the reason for my confusion is what I expected to happen when selecting Optimum or Standard.
I think this statement is wrong though and Ive read it in a few other places too. "The Optimum setting runs at 30% PWM, the Standard setting runs at 50% PWM, and the Full speed setting runs the fans at max speed."
Like you said, I expected under optimum the fans to run at a around 35% of max rpm under PWM control, and 50% when setting standard.
They don't though, not on my x9srl anyway. What happens is that with a cool PC, (i.e one not under heavy load etc), regardless of Optimum or Standard, all my fans (I tied Acrtic F9/F12, Noctua F12 and Coolermaster Jetflo) run at their minimum speed and ramp up according to system temperature.
Minimum speed is not affected by fan speed setting. What I understood from what Supermicro said is that STANDARD adjusts all fans depending on the summary of system temps where as with OPTIMAL, FANA is not affected by an increase in say CPU temps so can continue idling away saving a few $$$/£££ more.

 

[Pharfar]

In the SuperMicro X10SLM-F manual, SuperMicro doesn't specify whether or not the FANA header is meant for CPU. But when you buy the motherboard you get a large "layout" diagram, which states:

I took a leap of faith and used FANA for CPU

 

[pschatz100]

@Pharfar:
Fan1-Fan4 and Fan A are for System/CPU Fan headers - correct. That does not mean that FanA is the best choice for your CPU fan.

MANY OF YOU ARE THINKING LIKE DESKTOP SYSTEM BUILDERS, NOT LIKE ENTERPRISE SERVER BUILDERS.

Desktop fans are designed to be quiet. Server class fans are designed to move air. These are not the same priorities. When you use quiet fans with a server motherboard, you need to think about your cooling. For most NAS applications the stock CPU cooler will be OK when plugged into any of the fan headers, but if you build a system that is going to run your CPU at full load then you need to make certain your fan is running from a fan header that monitors the CPU.

I will summarize for Supermicro boards:

1. There are two PWM zones on most SuperMicro boards: One PWM zone monitors system temp, and that zone controls FanA. The other PWM zone monitors CPU temp, and that zone controls Fan1-4. Fans1-4 are controlled as one group, although all fans report their RPM individually. If you want your CPU fan to be responsive to CPU temp, then it should be connected to one of the Fans1-4 but it doesn't matter which one (except for specific motherboards that designate Fan1.)

2. PWM can be used to control fan speed, but it does not control RPM directly. The exact methodology is outside the scope of this thread. Just keep in mind that 100% is full speed, and 50% is less - but 50% does not mean 50% of the RPM. Do not expect a fan with 2000 RPM at 100% to run at 1000 RPM when set to 50%. PWM does not work that way. The lower the PWM percentage, the less predictable a specific fan's actual performance will be. Below 30%, many fans will not run reliably which is why you never see PWM power settings below 30%.

3. SuperMicro's fan settings are designed for enterprise server fans, which run at very high RPM's compared to quiet desktop fans. Enterprise server fans are designed to move air, and noise is not a big concern. So, at the end of the day, do not expect the fan controls to behave the same way with our quiet low RPM desktop fans as they do with high RPM enterprise class server fans. This is one of the reasons our low RPM fans run at minimum RPM when set to Standard mode (50%) or Optimum mode (30%.) The PWM controls on Supermicro boards are not designed for low RPM fans.

A last thought:
It doesn't really matter which fan header you use for a CPU fan as long as you have enough cooling. However, if you want the CPU fan to be responsive to CPU temp, then it should be in FAN 1-4. Using a utility to alter the fan speed alerts by lowering speed thresholds will make the alarms go away, but won't alter the basic way the fans are controlled.

If you want fans to run at a specific RPM, you need to use 3 pin fans, or use some type of a manual controller.

Another last thought:
Check your disk temps. On my system, I just discovered one disk that is running at 46c, the others are OK. I'm going to revise my fan strategy.

 

[Pharfar]

@pschatz100
I'm thinking like an enterprise builder. Im using a server case, the SuperMicro SuperChassis 822T-400LPB. And it makes a some noise, but not so much, that it matters. All fans are from SuperMicro.
But I agree with you, desktop fans wouldn't be great with my setup

 

[pschatz100]

MANY OF YOU ARE THINKING LIKE DESKTOP SYSTEM BUILDERS, NOT LIKE ENTERPRISE SERVER BUILDERS.

@Pharfar

I wasn't specifically referring to you. It is frustrating when folks refuse to understand how these things work. I will say that Supermicro did not do themselves any favors with their unclear documentation.

Good luck with your system. It certainly sounds like a promising setup.

 

[Pharfar]

I agree. Documentation is at best "unclear". And yet again I agree: We are not building desktop machines. Maybe that should be the first line in the FreeNas manual

 

[DJABE]

Yesterday I upgraded firmware on my SM X10SLM-F. (it came with 1.12 and now I got 1.35).
What is wrong afterwards is that my fans are going up and down as described previously in this topic. Root cause of this behaviour is one of my system fans, going below 600 RPM. I don't bother with that since I'll be running the machine at full speed fans anyway.
The odd thing is, that with previous IPMI firmware I didn't experienced this issue?! Also, I cannot read PCH nor CPU temps anymore!? It just says N/A. I've tried things mentioned here and elsewhere with power cycle etc. but it didn't helped. IPMI "home" page is completely empty, it seems that it cannot read firmware & BIOS version, MAC addresses etc.

I guess this is the old one: "Do not fix something that it's working..."
BIG fail for SuperMicro. This is what they call "enterprise" hardware and firmware?! LOL


P.S. any thoughts how to downgrade IPMI / Firmware? I tried to find SM archives but no luck...

 

[Ericloewe]

Yesterday I upgraded firmware on my SM X10SLM-F. (it came with 1.12 and now I got 1.35).
What is wrong afterwards is that my fans are going up and down as described previously in this topic. Root cause of this behaviour is one of my system fans, going below 600 RPM. I don't bother with that since I'll be running the machine at full speed fans anyway.
The odd thing is, that with previous IPMI firmware I didn't experienced this issue?! Also, I cannot read PCH nor CPU temps anymore!? It just says N/A. I've tried things mentioned here and elsewhere with power cycle etc. but it didn't helped. IPMI "home" page is completely empty, it seems that it cannot read firmware & BIOS version, MAC addresses etc.

I guess this is the old one: "Do not fix something that it's working..."
BIG fail for SuperMicro. This is what they call "enterprise" hardware and firmware?! LOL


P.S. any thoughts how to downgrade IPMI / Firmware? I tried to find SM archives but no luck...

https://forums.freenas.org/index.php?threads/supermicro-x10-motherboard-faq.23332/

This sticky answers all your questions. Be sure to inquire with Supermicro for details if you need to use recovery mode.

 

[Z300M]

Yesterday I upgraded firmware on my SM X10SLM-F. (it came with 1.12 and now I got 1.35).
What is wrong afterwards is that my fans are going up and down as described previously in this topic. Root cause of this behaviour is one of my system fans, going below 600 RPM. I don't bother with that since I'll be running the machine at full speed fans anyway.
The odd thing is, that with previous IPMI firmware I didn't experienced this issue?! Also, I cannot read PCH nor CPU temps anymore!? It just says N/A. I've tried things mentioned here and elsewhere with power cycle etc. but it didn't helped. IPMI "home" page is completely empty, it seems that it cannot read firmware & BIOS version, MAC addresses etc.

I guess this is the old one: "Do not fix something that it's working..."
BIG fail for SuperMicro. This is what they call "enterprise" hardware and firmware?! LOL


P.S. any thoughts how to downgrade IPMI / Firmware? I tried to find SM archives but no luck...

I think that all updating the IPMI firmware has done is reset the various sensor thresholds to the default -- or maybe a new default. Rather than downgrade the firmware, I would simply reset the thresholds to levels that suit the fans you're using. The information is all here somewhere, although I can't give you a specific link. Search for IPMITOOL.

 

[DJABE]

Thanks for a reply folks. I solved issue with the 'low speed' fan with ipmiutil sensor cmd tool. Another big fail with Java (ugggghhh! xD) and SuperMicro in this case - their tool does NOT work as it should, and we need open source projects in order to do the job... it just does not make any sense to me :)

 

[pjc]

FYI, I did confirm with SuperMicro that FAN A is on a separate PWM, based on system temperature. FAN 1-4 are tied to CPU temperature, at least on the X9SCA-F.

Since I wanted all three mid-plane fans to be controlled separately from CPU temperature, I used a PWM splitter (powered by molex to avoid overloading the header on the motherboard): http://www.frozencpu.com/products/10350/cab-183/Akasa_PWM_Splitter_-_Smart_Fan_Cable_AK-CB002.html.

The down side is that I can't monitor all three fans. But the alternative was to put two on the CPU PWM and only one on the chassis PWM.

 

[HolyK]

I have no idea what happened, but the FAN suddenly started again acting crazy. So i checked sensors and they were back on original/default values, so the Noctua FAN was dropping under the critical value. With the serious "WTF" face i was going to update the ipmi-sensor-config as i did last time. Following scenario happened:

- Started temporary jail with installed ipmi config
- Issued "ipmi-sensors-config --filename=ipmi.config --checkout" but no file was created.
- Power-cycled the IPMI via webgui and tried the same command again
- Got error "SDR Cache '/root/.freeipmi/sdr-cache/sdr-cache-temp001.localhost' out of date: Please flush the cache and regenerate it"
- So i did via "ipmi-sensors -f" and got "Flushing cache: /root/.freeipmi/sdr-cache/sdr-cache-temp001.localhost"
- Issued the checkout again and this time file was created and got "Caching SDR repository information: /root/.freeipmi/sdr-cache/sdr-cache-temp001.localhost" and "Caching SDR record 25 of 25 (current record ID 1679)" message
- Modified the config file as i did last time (300;150;0 for FAN1 and 450;300;150 for the rest)
- Issue commit command "ipmi-sensors-config --filename=ipmi.config --commit" and got following
Value '150.000000' for key 'Lower_Critical_Threshold' in section '607_FAN1' cannot be encoded accurately, try another value
Value '450.000000' for key 'Lower_Non_Critical_Threshold' in section '674_FAN2' cannot be encoded accurately, try another value
Value '150.000000' for key 'Lower_Non_Recoverable_Threshold' in section '674_FAN2' cannot be encoded accurately, try another value
Value '450.000000' for key 'Lower_Non_Critical_Threshold' in section '741_FAN3' cannot be encoded accurately, try another value
Value '150.000000' for key 'Lower_Non_Recoverable_Threshold' in section '741_FAN3' cannot be encoded accurately, try another value
- Even more serious WTF face ...
- Just gave a shot and changed first 150 to 100 and committed ... viola ! that one is proper ...
- Changed the values to 100;200;300 and 100;300;400 and committed afgain ... no error at all
- Checked the status via "ipmitool sensor" from the host system and got confirmation about new values set
FAN1 | 300.000 | RPM | nc | 100.000 | 200.000 | 300.000
FAN2 | 1000.000 | RPM | ok | 100.000 | 300.000 | 400.000
FAN3 | 1000.000 | RPM | ok | 100.000 | 300.000 | 400.000
- FAN1 is not acting crazy anymore.

So, what the hell happened? Some fcking auto-update/reset of the BMC/ipmi with a new acceptable values? I know that i did a system shutdown/startup but that should not cause this kind of odd acting, or?

 

[Ericloewe]

I have no idea what happened, but the FAN suddenly started again acting crazy. So i checked sensors and they were back on original/default values, so the Noctua FAN was dropping under the critical value. With the serious "WTF" face i was going to update the ipmi-sensor-config as i did last time. Following scenario happened:

- Started temporary jail with installed ipmi config
- Issued "ipmi-sensors-config --filename=ipmi.config --checkout" but no file was created.
- Power-cycled the IPMI via webgui and tried the same command again
- Got error "SDR Cache '/root/.freeipmi/sdr-cache/sdr-cache-temp001.localhost' out of date: Please flush the cache and regenerate it"
- So i did via "ipmi-sensors -f" and got "Flushing cache: /root/.freeipmi/sdr-cache/sdr-cache-temp001.localhost"
- Issued the checkout again and this time file was created and got "Caching SDR repository information: /root/.freeipmi/sdr-cache/sdr-cache-temp001.localhost" and "Caching SDR record 25 of 25 (current record ID 1679)" message
- Modified the config file as i did last time (300;150;0 for FAN1 and 450;300;150 for the rest)
- Issue commit command "ipmi-sensors-config --filename=ipmi.config --commit" and got following
Value '150.000000' for key 'Lower_Critical_Threshold' in section '607_FAN1' cannot be encoded accurately, try another value
Value '450.000000' for key 'Lower_Non_Critical_Threshold' in section '674_FAN2' cannot be encoded accurately, try another value
Value '150.000000' for key 'Lower_Non_Recoverable_Threshold' in section '674_FAN2' cannot be encoded accurately, try another value
Value '450.000000' for key 'Lower_Non_Critical_Threshold' in section '741_FAN3' cannot be encoded accurately, try another value
Value '150.000000' for key 'Lower_Non_Recoverable_Threshold' in section '741_FAN3' cannot be encoded accurately, try another value
- Even more serious WTF face ...
- Just gave a shot and changed first 150 to 100 and committed ... viola ! that one is proper ...
- Changed the values to 100;200;300 and 100;300;400 and committed afgain ... no error at all
- Checked the status via "ipmitool sensor" from the host system and got confirmation about new values set
FAN1 | 300.000 | RPM | nc | 100.000 | 200.000 | 300.000
FAN2 | 1000.000 | RPM | ok | 100.000 | 300.000 | 400.000
FAN3 | 1000.000 | RPM | ok | 100.000 | 300.000 | 400.000
- FAN1 is not acting crazy anymore.

So, what the hell happened? Some fcking auto-update/reset of the BMC/ipmi with a new acceptable values? I know that i did a system shutdown/startup but that should not cause this kind of odd acting, or?

Did you unplug the server?

 

[HolyK]

You mean power cord? Yea. Well, i shut my UPS off (after system shut down) since there was a planned power outage and last time they power-cycled the powernetwork like 10times in a row, so i shut down everything to avoid damage.

 

[Ericloewe]

You mean power cord? Yea. Well, i shut my UPS off (after system shut down) since there was a planned power outage and last time they power-cycled the powernetwork like 10times in a row, so i shut down everything to avoid damage.

It seems that in some IPMI implementations, sensor thresholds are not written to non-volatile memory.
There may be a way to do so manually, but it would require research.

 

[pjc]

Separately, I once had IPMI get itself confused where a reset via network claimed to work but had no effect. Thankfully, using the ipmi kernel module and ipmitool succeeded in resetting it.

 

[Ericloewe]

Separately, I once had IPMI get itself confused where a reset via network claimed to work but had no effect. Thankfully, using the ipmi kernel module and ipmitool succeeded in resetting it.

In my experience, a full reset and power cycle is necessary, for some reason.

 

[pjc]

That may often be true, but in at least one instance local (kernel-based) reset worked when remote reset didn't.

So long as you can still log into your server, in some cases you can reset IPMI when it otherwise seems like it would need a power cycle.