Register for the iXsystems Community to get an ad-free experience and exclusive discounts in our eBay Store.

Thermal and Accoustical Design Validation

dj_jazzn

Newbie
Joined
Apr 14, 2014
Messages
27

GrumpyBear

FreeNAS Experienced
Joined
Jan 28, 2015
Messages
141
Would it be possible to indirectly influence the fans speeds by lowering one or more of the temperature sensor thresholds? For example, the closest temperature sensor on the X10SL7-F to my HDD cages is the RT1 System sensor. Although not a direct measure of the disk temperature, could it be influenced by disk temperature and the direction of air flow through the case? Perhaps some experimenting is in order while stress testing the system.
My experience is that with adequate low speed fans running at their lowest speed (say 120mm or 140mm running at 600-800rpm) will provide enough airflow to keep the drive at or below 40C when the CPU is at or above 87% utilization and the disks are running 5 simultaneous high volume transfers (about as rough as it can get as it is five large sequential transfers started a few minutes apart so lots of seeking going on).

The only way to influence the thresholds would be by hacking the BIOS as that is what controls the registers for the fan controllers. This I have NO desire to do as the potential for dramatic unintended consequences is huge!

Unfortunately as I mentioned in another thread the RT1 sensor seen here:


is unduly influenced by the CPU heat (the stock Intel coolers blow air down and out) and the heatsink between the CPU and the sensor is the LSI2308 RAID/HBA controller which runs very very hot. In my testing I observed that RT1 tended to follow the CPU temperature whereas the peripheral sensor and the back (I have no cards installed) shows less increase as the CPU utilization and temp increases.

After looking at the issues that the user in the thread had with the LSI controller dropping out during scubs I revised my cooling strategy and reinstalled the Noctus iPPC fans as they move more air and so will hopefully keep the LSI chip cooler.

One other potential fix for the LSI would be to install a fan in the bottom case position to draw cool air in.
 

diedrichg

FreeNAS Expert
Joined
Dec 4, 2012
Messages
1,212
Stellar documentation! Nice job.
 

dj_jazzn

Newbie
Joined
Apr 14, 2014
Messages
27
Stellar documentation! Nice job.
I am in total agreement! This post has been a great resource for my own build.


After looking at the issues that the user in the thread had with the LSI controller dropping out during scubs I revised my cooling strategy and reinstalled the Noctus iPPC fans as they move more air and so will hopefully keep the LSI chip cooler.
I am curious to know where you set your fan threshold limits for the 140mm fans. I have also upgraded cooling and installed two NF-A14 fans. According to the Notua specs. the upper rotation speed is 3000±10% and MIN rotation speed is 800±20%.

I am currently using the following thresholds:

upper 3400 3500 3600
lower 300 400 500
 

GrumpyBear

FreeNAS Experienced
Joined
Jan 28, 2015
Messages
141
I am in total agreement! This post has been a great resource for my own build.




I am curious to know where you set your fan threshold limits for the 140mm fans. I have also upgraded cooling and installed two NF-A14 fans. According to the Notua specs. the upper rotation speed is 3000±10% and MIN rotation speed is 800±20%.

I am currently using the following thresholds:

upper 3400 3500 3600
lower 300 400 500
Setting the fan threshold is where art meets science. The manufacturer's specifications are the starting point. What I found was there are variations within fans and the rated speeds are often out of tolerances. I suspect the amount of back pressure on the fans will slow them down and this will be due to the impedance of airflow and will vary based on the number of fans, their directions (pushing versus pulling) and how airtight the case itself is.

The 3 Noctua NF-A14 iPPC-300 PWM 140mm fans are rated at a maximum speed of 3000 rpm +/- 10% and a minimum speed of 800rpm +/- 20%. The single Noctua NF-F12 iPPC-3000 PWM 120mm fan is likewaise rated at a maximum speed of 3000 rpm +/- 10% but it's minimum speed is 750rpm +/- 20%

So I've just set the thresholds for the Case Fans (FAN2 - 4 and FANA) thus:

FAN2 | 1000.000 | RPM | ok | 200.000 | 300.000 | 400.000 | 3000.000 | 3200.000 | 3500.000
FAN3 | 1000.000 | RPM | ok | 200.000 | 300.000 | 400.000 | 3000.000 | 3200.000 | 3500.000
FAN4 | 1000.000 | RPM | ok | 200.000 | 300.000 | 400.000 | 3000.000 | 3200.000 | 3500.000
FANA | 900.000 | RPM | ok | 200.000 | 300.000 | 400.000 | 3000.000 | 3200.000 | 3500.000

and the CPU fan to this based on the specs for the Intel stock cooler for the 1231E3-v2 and observation.
FAN1 | 1100.000 | RPM | ok | 500.000 | 600.000 | 700.000 | 2500.000 | 2600.000 | 2700.000

The important point is to have the minimum threshold set a fair bit below the lowest observed speed (leaving room below this for the recoverable and unrecoverable thresholds remembering that entered values will be rounded up or down to fit the limited number of bits in the registers used) so you don't trip the unrecoverable alarm and force the fans into high speed and likewise to set the high speeds above the highest observed speeds. These thresholds in no way influence the hysteresis or the curve defining the speed which the fans operate at a specific temperature. That is hard-coded in the BIOS and is the "Secret Sauce". These values just tell the BIOS at which observed fan speeds it should start to detect that a fan has failed either by stalling or bearing failure (low speed) or its TAC sensor being defective (an unrealistic high speed - say reporting 40,000rpm)
 
Joined
Jul 8, 2015
Messages
6
One thing I didn't see you talk about, in what is an amazingly thorough thread, is the static pressure of fans. While I'm not a fan or physics expert (just starting physics 2), my understanding is for applications were the fan air flow flow is restricted, a fan with high static pressure will be superior to one designed to move the most air. From what I've read 120mm fans are better in general for this application as static pressure is increased by lowering fan diameter and increasing thickness. There are fans that particularly state they're high static pressure and are marketed for use with radiators although I couldn't tell you how much of that is marketing vs actual design optimization. Again, I don't much experience and I don't know what the performance differences would be, but I thought it might be of interest to someone researching this.
 

Ericloewe

Not-very-passive-but-aggressive
Moderator
Joined
Feb 15, 2014
Messages
16,076
One thing I didn't see you talk about, in what is an amazingly thorough thread, is the static pressure of fans. While I'm not a fan or physics expert (just starting physics 2), my understanding is for applications were the fan air flow flow is restricted, a fan with high static pressure will be superior to one designed to move the most air. From what I've read 120mm fans are better in general for this application as static pressure is increased by lowering fan diameter and increasing thickness. There are fans that particularly state they're high static pressure and are marketed for use with radiators although I couldn't tell you how much of that is marketing vs actual design optimization. Again, I don't much experience and I don't know what the performance differences would be, but I thought it might be of interest to someone researching this.
It's actually fairly easy to spot quiet high airflow fans (as opposed to louder, static pressure -oriented fans): They have many small blades and they have an angle of attack oriented for the steady-state of "lots of air coming at me".
 

cyberjock

Moderator
Joined
Mar 25, 2012
Messages
19,151
One thing I didn't see you talk about, in what is an amazingly thorough thread, is the static pressure of fans. While I'm not a fan or physics expert (just starting physics 2), my understanding is for applications were the fan air flow flow is restricted, a fan with high static pressure will be superior to one designed to move the most air. From what I've read 120mm fans are better in general for this application as static pressure is increased by lowering fan diameter and increasing thickness. There are fans that particularly state they're high static pressure and are marketed for use with radiators although I couldn't tell you how much of that is marketing vs actual design optimization. Again, I don't much experience and I don't know what the performance differences would be, but I thought it might be of interest to someone researching this.
There are lots of relationships that affect fan design. I touched on these in my thread about overheating in my Norco chassis. For some of the more industrial fan brands like Supermicro they provide performance curves where you can determine the flowrate and backpressure relatively easily. The bad news is that the stuff you buy at Newegg and Amazon never give you that kind of information, and they don't *want* to give you that information either.
 

GrumpyBear

FreeNAS Experienced
Joined
Jan 28, 2015
Messages
141
One thing I didn't see you talk about, in what is an amazingly thorough thread, is the static pressure of fans. While I'm not a fan or physics expert (just starting physics 2), my understanding is for applications were the fan air flow flow is restricted, a fan with high static pressure will be superior to one designed to move the most air. From what I've read 120mm fans are better in general for this application as static pressure is increased by lowering fan diameter and increasing thickness. There are fans that particularly state they're high static pressure and are marketed for use with radiators although I couldn't tell you how much of that is marketing vs actual design optimization. Again, I don't much experience and I don't know what the performance differences would be, but I thought it might be of interest to someone researching this.
As cyberjock says there are lots of inter dependencies and when you start getting into static pressure of fans and the airflow design it is quite technical and would greatly bloat something already really long (and when you start talking thermogodamnamics and physics the average persons eyes glaze over. My aim was to attempt to build a NAS with adequate cooling under heavy that was also reasonably quiet. I quickly came to the conclusion that there were too many variables and the only way to do this was to build, test, tweak, repeat.

From experience I can say that the stock Fractal Fans (0.58mm H2O) don't keep the disks as cool as the Cougar Fans (2.2mm H2O) and the Noctua Fans (10.52mm H2O) are even better but there are no performance curves and they all run at differing speed ranges with differing tolerances and the stock fans don't even have PWM control.

Compounding this we are stuffing a server class motherboard designed to be stuffed into a rackmount chassis with fans that sound like jet engines and shoe horn this motherboard into a chassis designed for gaming and maximizing the cooling for a radiator. On top of this many of us also want the NAS to run as quietly as possible.
 

HardChargin

Newbie
Joined
Jul 19, 2015
Messages
49
@GrumpyBear Just want to say thank you for the awesome, informative, and helpful post.
 

GrumpyBear

FreeNAS Experienced
Joined
Jan 28, 2015
Messages
141
I'm not certain if we need another thread for this but while trolling the SuperMicro Support FAQs on another issue I came across this FAQ: X10 Fan speed control by raw command
The command for x10 is as below:
0x30 0x70 0x66 Fan Control Manually (X10)
Data1 0/1 (Get/Set)
Data2 0/1 Region
Data3 0-0x64 Duty cycle
Duty cycle is a percent in hex, therefore, max is 0x64, half is 0x32.
In theory this might allow manual control of the chassis fans while the CPU fan is still controlled by the CPU temperature.

A command like:
Code:
ipmitool raw 0x30 0x70 0x66 0x01 0x01 0x32

should set the fans in region 2 to 50% PWM. Region 2 is likely the FAN-A header which is normally controlled by the Peripheral (Chassis Temp) sensor.

However, it is uncertain if this command will work in any fan mode (Standard/Full/Optimal/Heavy I/O) of if the fan mode must first be set to a mode that supports independent control of both zones (either Optimal or Heavy I/O) as outlined in this FAQ.
Code:
ipmitool raw 0x30 0x45 0x1 0x0 (Set Mode to Standard)
ipmitool raw 0x30 0x45 0x1 0x1 (Set Mode to Full)
ipmitool raw 0x30 0x45 0x1 0x2 (Set Mode to Optimal)
ipmitool raw 0x30 0x45 0x1 0x4 (Set Mode to Heavy I/O)
ipmitool raw 0x30 0x45 0x0 (Get Fan Mode)


I have a bad disk in my ZFS3 array and when I get a replacement I will poke around a bit and report on the results. If this works it should allow a system to have the chassis fans forced to a adequate speed to keep the disks at or below a target temperature under their heaviest load.

The ultimate solution might involve building a Arduino-based controller with a thermistor that can be bonded to one of the disks which controls the chassis fans via PWM. I did a bit of research on this while helping another user out off-line and it looks promising.
 

Ericloewe

Not-very-passive-but-aggressive
Moderator
Joined
Feb 15, 2014
Messages
16,076
I'm not certain if we need another thread for this but while trolling the SuperMicro Support FAQs on another issue I came across this FAQ: X10 Fan speed control by raw command

In theory this might allow manual control of the chassis fans while the CPU fan is still controlled by the CPU temperature.

A command like:
Code:
ipmitool raw 0x30 0x70 0x66 0x01 0x01 0x32

should set the fans in region 2 to 50% PWM. Region 2 is likely the FAN-A header which is normally controlled by the Peripheral (Chassis Temp) sensor.

However, it is uncertain if this command will work in any fan mode (Standard/Full/Optimal/Heavy I/O) of if the fan mode must first be set to a mode that supports independent control of both zones (either Optimal or Heavy I/O) as outlined in this FAQ.
Code:
ipmitool raw 0x30 0x45 0x1 0x0 (Set Mode to Standard)
ipmitool raw 0x30 0x45 0x1 0x1 (Set Mode to Full)
ipmitool raw 0x30 0x45 0x1 0x2 (Set Mode to Optimal)
ipmitool raw 0x30 0x45 0x1 0x4 (Set Mode to Heavy I/O)
ipmitool raw 0x30 0x45 0x0 (Get Fan Mode)


I have a bad disk in my ZFS3 array and when I get a replacement I will poke around a bit and report on the results. If this works it should allow a system to have the chassis fans forced to a adequate speed to keep the disks at or below a target temperature under their heaviest load.

The ultimate solution might involve building a Arduino-based controller with a thermistor that can be bonded to one of the disks which controls the chassis fans via PWM. I did a bit of research on this while helping another user out off-line and it looks promising.
Very interesting. It would remove the greatest complexity from an external arduino fan controller (the actual PWM hardware).
 

yourmate

Newbie
Joined
Apr 4, 2016
Messages
92
This is very throughout - it's an excellent read, thank you @GrumpyBear

I will definitely use this when setting up my build as I'll use an R5 case (which is literally the same). I wonder how much of this should be modified when using with an X9 board?!?
 

GrumpyBear

FreeNAS Experienced
Joined
Jan 28, 2015
Messages
141
This is very throughout - it's an excellent read, thank you @GrumpyBear

I will definitely use this when setting up my build as I'll use an R5 case (which is literally the same). I wonder how much of this should be modified when using with an X9 board?!?
You're welcome

How much adjustment is required for an X9 series motherboard likely depends on the board and its components that generate heat, the modes supported by the fan controller and whether it has IPMI but the process should be the same (i.e. Monitor the HDD and System temperatures during 100% CPU and heavy disk activities to ensure the temperatures remain in a range you are comfortable with.

After writing this HowTo I determined threat the board I was using was overkill and so I swapped out the X10 for an X9SCL I had. I was confident the Noctua fans in the R4 case would work fine but with a lack of IPMI and a different set of fan modes on the motherboard a quick test running 100% CPU with heavy disk activity for half an hour validated this assumption.

A big part is determining the best fans and the required IPMI thresholds for minimum speed. If the design objective is super quite then fans with a larger diameter supporting smaller minimum RPM (say 140mm versus 120mm and 300rpm versus 600rpm) might be best.

I also found a wide variation in a fan's ability to move air based on the Static Pressure of the fans (higher is better) and significant variation in the upper speed of fans. Considering two fans with identical minimum speeds but differing maximum speeds the unit with the higher maximum speed will likely be noisier at lower PWM duty cycles but YMMV.

What you should get out of this is to establish your design goals then use the methodology outlined to validate the goals and tweak if necessary.
 

yourmate

Newbie
Joined
Apr 4, 2016
Messages
92
Man, I hope you're a teacher ;)

How much adjustment is required for an X9 series motherboard likely depends on the board and its components that generate heat, the modes supported by the fan controller and whether it has IPMI but the process should be the same (i.e. Monitor the HDD and System temperatures during 100% CPU and heavy disk activities to ensure the temperatures remain in a range you are comfortable with.
Yeah, I had a quick look of the board when it came and it has the same FAN1-4 + FANA connector layout but did not have time to check properly. I ordered a splitter and will modify it while I am waiting for the rest of the stuff to arrive.


A big part is determining the best fans and the required IPMI thresholds for minimum speed. If the design objective is super quite then fans with a larger diameter supporting smaller minimum RPM (say 140mm versus 120mm and 300rpm versus 600rpm) might be best
I do want quiet as I don't have a cellar to hide the server but I don't want to sacrifice reliability either. Is there any sense to go for 30-33C for the HDDs (when stress tested) or a 37-39 is more of a reality (so when the system is near idle then it'll drop to 30-33C)?


I also found a wide variation in a fan's ability to move air based on the Static Pressure of the fans (higher is better) and significant variation in the upper speed of fans. Considering two fans with identical minimum speeds but differing maximum speeds the unit with the higher maximum speed will likely be noisier at lower PWM duty cycles but YMMV.
I will keep this in mind.
What are the reliable brands for 140mm fans (apart from Noctua)?
I found a Noctua NH-D9L for my CPU. Is this enough or should I go for a bigger size fan (and sink) for a Xeon E5-2670?
 

GrumpyBear

FreeNAS Experienced
Joined
Jan 28, 2015
Messages
141
I do want quiet as I don't have a cellar to hide the server but I don't want to sacrifice reliability either. Is there any sense to go for 30-33C for the HDDs (when stress tested) or a 37-39 is more of a reality (so when the system is near idle then it'll drop to 30-33C)?
I'd aim for under 40. These drives, even at 5400rpm get hot when under load.
I will keep this in mind.
What are the reliable brands for 140mm fans (apart from Noctua)?
The Cougar Vortex 140mm Fans I tried first worked OK. The temps were a little high but if you don't have a very hot SAS Controller on the motherboard like I did they may work well. They are not that much cheaper than the Noctua iPPC fans though. The Noctuuas are overkill somewhat but spending and extra $150 to protect $1500 - $3000 worth of hardware ... The Cougar fans are, IMHO, ugly and do not have the oomph the Noctuas or Delta Bearing Fans do. Most of the Linux-based appliances I deal with at work use tiny Delta Fans in 1U or 2U rack-chassis and sound like a jet engine when starting.
I found a Noctua NH-D9L for my CPU. Is this enough or should I go for a bigger size fan (and sink) for a Xeon E5-2670?
That's Quite the processor. If you are just planning a simple NAS then it might be overkill. If your looking at virtualization and possible multi-processor then it might be OK. Others have more experience than I in this area. The last Multi-CPU system I built was a Dual Pentium 166 on a Tyan for Quake in 1996 :D. That processor has a TDP of 120W which is just under the TDP guideline of 140W max for that cooler but you should be OK as your not overclocking but if I was spending north of a grand on a CPU I'd probably go for a cooler designed for Server and Workstation use like the NH-U9DX i4 or the NH-U12DX i4 with a 120mm fan which should be quieter. Either of these should fit in the Define R5 no problem.

Remember that all motherboards are usually only rated for 1.5A per fan header and that assumes that you are not using all the headers. Using a modified splitter for the chassis fans will mean just the CPU fan will draw off the motherboards 12V rail though so the usual limit of 3-4A total should handle a couple of fans on a CPU cooler easily.

With the i3-i7 and e3 Xeons I just use the stock Intel coolers and they work well enough.
 

yourmate

Newbie
Joined
Apr 4, 2016
Messages
92
The Cougar fans are, IMHO, ugly and do not have the oomph the Noctuas
if I was spending north of a grand on a CPU I'd probably go for a cooler designed for Server and Workstation use like the NH-U9DX i4 or the NH-U12DX i4 with a 120mm fan which should be quieter
I totally agree - the Cougar would look nice in my son's gaming rig ;)
Decision made I'll shop around for a NH-U12DX i4 and 4 x NF-A14 PWM
 

tmsmith

Newbie
Joined
Apr 28, 2014
Messages
44
First off, great write up and read. I'm getting ready to migrate my server into a new-to-me 846. I now plan to spend a little more time tweaking the fan speeds when I originally built it. This sentence brought up a little concern for me though.

Remember that all motherboards are usually only rated for 1.5A per fan header and that assumes that you are not using all the headers. Using a modified splitter for the chassis fans will mean just the CPU fan will draw off the motherboards 12V rail though so the usual limit of 3-4A total should handle a couple of fans on a CPU cooler easily.
I will be installing 3x 120mm Noctua IPPC-3000 PWM into a modded fan wall and will likely use 2x consumer Nocuta fans for the exhaust. After looking at the manual for my MB, I won't have enough fan headers. Is my only option to purchase a splitter with molex connector (similar to what you linked)?
 

yourmate

Newbie
Joined
Apr 4, 2016
Messages
92
Is my only option to purchase a splitter with molex connector (similar to what you linked)?
To the PWM work with this setup GrumpyBear outlined you have to use a modified splitter anyway. Power from a molex should be fine but control signal should come from the mobo.

Edit: you need 3 of these connectors (or 4 if you have 5 fan headers on your mobo and want to use 5 fans)
 
Last edited:

tmsmith

Newbie
Joined
Apr 28, 2014
Messages
44
I would actually order two sets of cables as you need some extra fan connectors making the mod. Of course if you've got some extra laying around... ;)
Can you explain little further? I should be able to use the 1 splitter for the fan wall and plug the exhaust fans directly to the empty fan headers? This, of course, limits me to controlling all 3 fans on the fan wall to one PWM.
 
Top