High Disk Temps on TrueNAS Mini X+ with WD Red Plus drives.

[Cold Ben]

I recently purchased a TrueNAS Mini X+ and have filled the five 3.5" bays with 14TB WD Red Plus drives. I am seeing fairly high temps while the disks are at idle, as can be seen in the attached screenshot. After an hour of idle, the disks are averaging 42C-46C, in a 21C room. This seems very high. So high that I have not tried stress-testing the disks yet. I have since shut the system down and will not be running it until the idle temps can be better controlled.

I have noticed that the "FANA" is always spinning at 600 rpm and never seems to speed up. Is there a way to better control this through ipmitool? (Output included below). Various forum posts here have not provided a solution. If I wet a finger and stick it behind the fan, I can feel very minimal air flow. The fan is spinning and is not making any odd noises that would indicate bad bearings.

Here is the output of ipmitool sensor list all:

Code:

CPU Temp         | 39.000     | degrees C  | ok    | 0.000     | 0.000     | 0.000     | 88.000    | 93.000    | 93.000
System Temp      | 40.000     | degrees C  | ok    | -10.000   | -5.000    | 0.000     | 80.000    | 85.000    | 90.000
Peripheral Temp  | 40.000     | degrees C  | ok    | -10.000   | -5.000    | 0.000     | 80.000    | 85.000    | 90.000
MB_10G Temp      | 56.000     | degrees C  | ok    | -5.000    | 0.000     | 5.000     | 95.000    | 100.000   | 105.000
DIMMA1 Temp      | 43.000     | degrees C  | ok    | -5.000    | 0.000     | 5.000     | 80.000    | 85.000    | 90.000
DIMMA2 Temp      | na         |            | na    | na        | na        | na        | na        | na        | na
DIMMB1 Temp      | 46.000     | degrees C  | ok    | -5.000    | 0.000     | 5.000     | 80.000    | 85.000    | 90.000
DIMMB2 Temp      | na         |            | na    | na        | na        | na        | na        | na        | na
FAN1             | na         |            | na    | na        | na        | na        | na        | na        | na
FAN2             | na         |            | na    | na        | na        | na        | na        | na        | na
FAN3             | 3700.000   | RPM        | ok    | 300.000   | 500.000   | 700.000   | 25300.000 | 25400.000 | 25500.000
FANA             | 600.000    | RPM        | ok    | 200.000   | 300.000   | 500.000   | 25300.000 | 25400.000 | 25500.000
12V              | 12.384     | Volts      | ok    | 10.144    | 10.272    | 10.784    | 12.960    | 13.280    | 13.408
5VCC             | 4.948      | Volts      | ok    | 4.246     | 4.298     | 4.480     | 5.390     | 5.546     | 5.598
3.3VCC           | 3.367      | Volts      | ok    | 2.789     | 2.823     | 2.959     | 3.554     | 3.656     | 3.690
VBAT             | 3.175      | Volts      | ok    | 2.392     | 2.479     | 2.595     | 3.494     | 3.581     | 3.697
Vcpu             | 1.062      | Volts      | ok    | 0.441     | 0.450     | 0.495     | 1.296     | 1.422     | 1.440
VDIMM            | 1.236      | Volts      | ok    | 0.948     | 0.975     | 1.047     | 1.344     | 1.425     | 1.443
PVCCSRAM         | 1.015      | Volts      | ok    | 0.655     | 0.664     | 0.745     | 1.195     | 1.294     | 1.303
P1V05_A          | 1.050      | Volts      | ok    | 0.870     | 0.897     | 0.942     | 1.194     | 1.221     | 1.248
5VSB             | 4.870      | Volts      | ok    | 3.960     | 4.038     | 4.350     | 5.624     | 5.936     | 6.040
3.3VSB           | 3.334      | Volts      | ok    | 2.603     | 2.671     | 2.875     | 3.708     | 3.912     | 3.980
PVNN             | 0.838      | Volts      | ok    | 0.568     | 0.568     | 0.640     | 1.225     | 1.351     | 1.360
PVPP             | 2.696      | Volts      | ok    | 2.174     | 2.228     | 2.390     | 3.092     | 3.254     | 3.308
P1V538_A         | 1.563      | Volts      | ok    | 1.320     | 1.347     | 1.401     | 1.644     | 1.671     | 1.698
1.2V BMC         | 1.218      | Volts      | ok    | 1.020     | 1.047     | 1.092     | 1.344     | 1.371     | 1.398
PVCC_REF         | 1.286      | Volts      | ok    | 1.043     | 1.097     | 1.142     | 1.394     | 1.421     | 1.448
Chassis Intru    | 0x0        | discrete   | 0x0000| na        | na        | na        | na        | na        | na

Does anyone have any suggestions for why "FANA" might be stuck at 600 rpm? Any help is greatly appreciated.

 

[darkfiberiru]

Hello,

Can you post a screenshot of your ipmi web interface the tab under Configuration(top bar) -> Fan Mode.

If your currently set to optimal I would try first standard (Move baseline from 30% pwm to 50% pwm) and then after that if you still need more speed you can go to heavy/io which will move rear fan to ~70% pwm min and main fan to ~50% pwm.

Finally there's full but that will probably be to loud setting both cpu and rear fan to 100%.

 

[tim64]

Hello Ben,

I have a TrueNAS Mini X+ too and have filled the five 3.5" bays with 12TB WD HGST He12 drives and the TrueNAS Mini X+ is standing in a room that has an air temperature of 20C-21C.

I use the 12.0-RELEASE software "out of the box" without any optimizations of temperature, power-consumption or fan speed.

Here are my temperatures and fan speeds for comparison when the disks are idle (no data transfer, scrubbing, etc.) for several hours:

Code:

CPU Temp         | 36.000     | degrees C  | ok    | 0.000     | 0.000     | 0.000     | 88.000    | 93.000    | 93.000
System Temp      | 36.000     | degrees C  | ok    | -10.000   | -5.000    | 0.000     | 80.000    | 85.000    | 90.000
Peripheral Temp  | 37.000     | degrees C  | ok    | -10.000   | -5.000    | 0.000     | 80.000    | 85.000    | 90.000
MB_10G Temp      | 52.000     | degrees C  | ok    | -5.000    | 0.000     | 5.000     | 95.000    | 100.000   | 105.000
DIMMA1 Temp      | 39.000     | degrees C  | ok    | -5.000    | 0.000     | 5.000     | 80.000    | 85.000    | 90.000
DIMMA2 Temp      | na         |            | na    | na        | na        | na        | na        | na        | na
DIMMB1 Temp      | 42.000     | degrees C  | ok    | -5.000    | 0.000     | 5.000     | 80.000    | 85.000    | 90.000
DIMMB2 Temp      | na         |            | na    | na        | na        | na        | na        | na        | na
FAN1             | na         |            | na    | na        | na        | na        | na        | na        | na
FAN2             | na         |            | na    | na        | na        | na        | na        | na        | na
FAN3             | 3900.000   | RPM        | ok    | 300.000   | 500.000   | 700.000   | 25300.000 | 25400.000 | 25500.000
FANA             | 600.000    | RPM        | ok    | 200.000   | 300.000   | 500.000   | 25300.000 | 25400.000 | 25500.000

I have noticed that the "FANA" is always spinning at 600 rpm and never seems to speed up. Is there a way to better control this through ipmitool? (Output included below). Various forum posts here have not provided a solution. If I wet a finger and stick it behind the fan, I can feel very minimal air flow. The fan is spinning and is not making any odd noises that would indicate bad bearings.

Does anyone have any suggestions for why "FANA" might be stuck at 600 rpm? Any help is greatly appreciated.

My FANA is also at 600rpm, but my FAN3 is running slightly higher than yours at 3900 rpm. When I stick a finger behind the Mini/fan I can barely feel the air flow.

The disk in the middle, ada2, was and is the hottest of my drives and it reached 50C under heavy load in the first days but runs now always below 45C even under load.

Maybe your WD 14 TB drives just produce more heat and how tight the disc enclosure is build results in slightly warmer drives and the fan speeds need to be increased for drives that generate more heat?

I do not know how to increase the fan speeds more when the drives run hotter, but maybe this comparison still helps somehow.

 

[hescominsoon]

log into the ipmi then use that to reboot the mini and get into the bios..then increase the fan speed. right now it comes locked into "quiet" mode which reduces airflow. Raise the fan speed and the temps will go down.

 

[Chris Moore]

I like to keep the drive temperatures around 35°C, or less, but never more than 40°C. Alarms go off at 45°C...
High drive temperatures impact drive life.

 

[jenksdrummer]

That's not what I'd consider high temps.

What kills disks is not high temps, it's temp swings. Absurdly high temps, sure, like 100F ambient and 50C at the disk...

 

[Chris Moore]

That's not what I'd consider high temps.

Based on what? Everyone has an opinion... I like to keep my house around 76°F, but some people like 68°F...
Not judging, but why...

 

[tim64]

log into the ipmi then use that to reboot the mini and get into the bios..then increase the fan speed. right now it comes locked into "quiet" mode which reduces airflow. Raise the fan speed and the temps will go down.

I am not sure if the fan speed can be changed in the bios of the TRUENAS-MINI-3.0-X+. I rebooted the machine and went into the bios via IPMI but did not found a fan speed setting in the bios.

However, during my search I noticed, there is a fan mode setting in the IPMI interface itself, so the fan speed setting can be changed without reboot. I should have taken a closer look into the IPMI interface earlier. :)

I changed the fan mode from "Standard" to "Optimal", because "optimal" sounds optimal :) but I really don't know what exactely the difference is.

I can guess what "Full Speed" means, but is there some info/documentation about the differences between "Standard", "Optimal" and "Heavy IO" ?

 

[Redcoat]

I changed the fan mode from "Standard" to "Optimal", because "optimal" sounds optimal :) but I really don't know what exactely the difference is.

I can guess what "Full Speed" means, but is there some info/documentation about the differences between "Standard", "Optimal" and "Heavy IO" ?

You need to get hold of a copy of the motherboard manual for your box. I don't know whose/what motherboard it is but a look under the hood should find that, then go to mfr site and get it - also the ipmi manual, too. If it's a SuperMicro board, both will be available I imagine.

Also, the IPMI information will have the board designation.

 

[tim64]

You need to get hold of a copy of the motherboard manual for your box. I don't know whose/what motherboard it is but a look under the hood should find that, then go to mfr site and get it - also the ipmi manual, too. If it's a SuperMicro board, both will be available I imagine.

Thanks.

The TRUENAS-MINI-3.0-X+ has a Supermicro A2SDi-H-TF Board. The motherboard manual says nothing about the fan mode, but the IPMI says:

• [Standard Speed]: to use the standard fan speed setting for power savings.
• [Full Speed]: to use the full speed setting for optimal system performance.
• [Optimal Speed]: to use the optimal fan speed setting, which will adjust the fan speed by balancing the needs between system performance and power savings.
• [HeavyIO Speed]: to use the heavy I/O speed setting, which will boost cooling to the add-on card zone.

I changed the fan mode from "Standard" to "Optimal", because "optimal" sounds optimal :) but I really don't know what exactely the difference is.

The change from "Standard" to "Optimal" did not change any idle disk temperature. Then later I changed the setting from "Optimal Speed" to "Full Speed" and the idle disk temperatures dropped below 34C within one hour (and stayed there even with a little bit of disk load).

It seems setting the fan mode is more like an on / off switch and there is no way to fine-tune the fan control.

 

[Redcoat]

Well, it sounds like you have the support resources in hand and some positive results from testing. Good job! Mid 30's is where I like to see my drives.

I had the earlier iteration of the Mini and found internal temps to be a problem - I modified the case to install two small fans at the front to blow over the mainboard, added a fan to the CPU heatsink, beefed up the rear fan and took off the front door, to keep things cool. That was my first and last "compact case" server experience.

 

[darkfiberiru]

As mentioned above:

If your currently set to optimal I would try first standard (Move baseline from 30% pwm to 50% pwm) and then after that if you still need more speed you can go to heavy/io which will move rear fan to ~70% pwm min and main fan to ~50% pwm.

Also as a fyi all Gen3 Mini's have active cpu cooling vs Gen2 having passive cooler.

 

[tim64]

I did some more tests with the fan mode settings and the resulting fan speeds when the server and disks were idle:

Code:

Fan Mode            FAN3    FANA
--------------       ----        ----
Full Speed          7000    1400
HeavyIO Speed    6500    1200
Optimal Speed     3100     600
Standard Speed   3100     600

There was no difference in disk temps between "Full" and "Heavy IO" and between "Optimal" and "Standard" and the disk temps of "Optimal/Standard" were around 5-12 C higher than the disk temps at "Heavy IO/Full".

Later I found an interesting article about the Supermicro fan modes:

Reference Material - Supermicro X9/X10/X11 Fan Speed Control

Reference material regarding fan speed control on SuperMicro X9/X10/X11 motherboards. This has been pieced together over some time and I finally got that "aha!" moment yesterday where it all came together. A bit of testing confirmed that this...

forums.servethehome.com

This does mean for theTRUENAS-MINI-3.0-X+ (if I did not mix things up);

FAN3 is the CPU-fan (Zone 0) and FANA (Zone 1) is the fan at top of the back of the case behind the drives.

Standard-Mode: BMC controls FAN3 based on CPU-temp and FANA based on chipset(PCH)-temp
Optimal-Mode: BMC controls FAN3 based on CPU-temp and FANA is fixed at low speed (ca. 43%)
Heavy-IO-Mode: BMC controls FAN3 based on CPU-temp and FANA is fixed at 85%
Full-Speed-Mode: FAN3 and FANA are fixed at 100%

The mainbord is at the bottom of the case. The airflow around the disks (from front to back) does not touch any of the temp sensors of the mainboard. This means there is no temp sensor of the mainboard/BMC that checks somehow the airflow temp of the disks.

I then did a test in "standard speed mode": I blocked the bottom of the front opening with a piece of paper. So the airflow to the mainboard would
be reduced and the airflow to the disks would be increased:

The Result: The BMC sensors reported higher temperatures, the fans ran faster (FAN3: 4000 rpm, FANA: 700 rpm) and the disks became a little bit cooler. When the MB_10G temp reached 57C I stopped my test and removed the paper.

It is somehow a little bit disappointing that it seems that there is no automatic fan control in the MINI-3.0-X+ that increases the fan speed(s) if the disk temps increase.

I guess the "appropriate" setup would be in my case a setting of "Optimal/Standard" for FAN3 and a fixed speed between "Optimal" and "Heavy" for FANA.

Because TrueNAS-Core knows the disk temps: Is it possible that TrueNAS-Core can control the fans based on disk-temperatures?

 

[darkfiberiru]

Also just so you know all of those drives are warrantied by iX(if bought from us) and WD to be supported up to 65C(this . As vendors are going bigger and cheaper on drives the power need and in conjunction the thermal output of drives have climbed and so have supported max temperatures.

There have been a mix of white papers correlate drive temperatures to failure rates many of which have contradictory findings.

Hard Drive Temperature—Does It Matter?

How much does operating temperature affect the failure rates of disk drives? Not much. The unlimited online backup service provided by Backblaze requires

www.backblaze.com

Based upon the mixed data, the fact that the data is gathered over a much larger population sample then the 8 drives or so a small nas will have and the fact that newer drives have been designed for higher temperatures I personally do not get concerned about drive temps as long as they are within manufacturer operating specs as it's impossible to tell if your making a positive benefit.

On the point of new drives having higher tolerances and design changes a lot of new drives are rated at 65c vs the older standard of 55c which is a pretty significant increase.

Don't get me wrong we hear you at iX about fan control limitations as it's all controlled currently by bmc which has limited access to sensors and limited tuning options. I'm investigating improvements but it's a situation to step carefully in to not cause new problems either thermally or acoustically.

 

[joeschmuck]

So I'm going to take an out of the box approach to this cooling problem, because that is the issue here.

First of all, I prefer my hard drive temps to be under 40C when possible, 45C is my alarm set point to figure out what is going on, and 50C is here I've had a failure (in my mind) and I need to fix it or shut it down. That is my opinion and everyone is allowed to have one.

Myself, I'm all over having great and purposeful air flow. You can obtain a smoke stick and watch to see how the smoke moves from the intake to the exhaust and ensure it is taking a path to cool the hot components. If you find out that the flow should be better directed then you could create simple air flow ducting out of simple thin cardboard (like from a cereal box) and properly secure them into the case. I'd also seal up any air intakes that do not serve a purpose in order to increase air flow across the hard drives (as identified in the above posting) but I'd seal these off from the inside of the case for aesthetics (Gorilla makes some great black tape that will not come off easily and makes it all reversible).

Disclaimer: Do at your own risk, may void warranty.
If you are not comfortable doing the next step, then don't. You MUST remove everything from the case, it must be bare bones metal shell when proceeding. This is not for everyone and again, warranty?

The other thing I personally would do is to completely gut the case and cut out the exhaust fan grill which has a lot of air flow restriction and replace it with a 120mm wire grill (It's a bit old school) that offers almost no air flow restriction, here is an example only. Now when you cut out the old grill, do a nice professional looking job and file the edges so they are not sharp and you can buy a plastic edging that you could place on the new hole to protect your fingers, Here is a similar product and there are others made of rubber. And for goodness sake, if you replace the 120mm fan with a more powerful one (more CFM) make sure that the power supply is exhausting air once done. It is possible to place enough of a low pressure within the case that the power supply fan is not flowing enough air and the air is actually flowing backwards. It's not something common to see however it could happen and if you are redesigning a case then you must consider all airflow.

AGAIN - DISCLAIMER - DO AT YOUR OWN RISK.