Freenas server making whining noise, cannot determine origin

freenas-supero

Contributor
Joined
Jul 27, 2014
Messages
128
Hello all,

The problem:
In the last few months my Freenas server has required some parts be replaced because of noise and old age. Unfortunately, even after replacing the power supplies and most fans, it is now making a constant annoying medium/high pitched whine. See attached recording and please wear headphones as it is difficult to capture with a cell phone. The whine is not the classic "coil whine" which I am well aware of. This whine "oscillates" as it is not a perfectly "flat" tone line tinnitus or a coil whine so I believe it is of mechanical origin.

Just to clear the obvious, I confirmed that its really this server and not something else in my rack, and also confirmed that other people can hear it which they can very clearly.

I proceeded by elimination until I narrowed down the cause to what I believe to be the mid-wall intake fans. Unfortunately it is impossible to know if this whine was present before I replaced the old PSU and fans because the PWS-800-1R were so loud they noise-drowned everything else..... o_O

In the last few months, I:
  • replaced the two PWS-801-1R with 2x PWS-920P-SQ's
  • replaced the intake fans with brand new FAN-0094L4 (the exhaust fans were always unplugged)
  • replaced the Noctua Nh-D9L CPU cooler fan
  • unplugged each of the power supplies one by one to confirm they are not responsible.
  • removed the 3 mid-wall fans to confirm the CPU cooler fan is not responsible. The server is dead quiet which pretty much eliminates coil whine (electronic origin) and PSU fan whine...
I believe the noise is caused by the intake fans spinning at a speed where airflow turbulence causes this whining, or possibly because they are running at a speed where some sort of resonance occurs with the chassis. In order to confirm this, I removed the chassis top cover and sure enough, the noise is much more bearable (can still hear it but its very faint).

Naturally I tried to manually change the fan speeds using web IPMI (ver. 3.52), but I cannot see an option to do so. Using IPMIView under "IPMI device > Fan Speed Mode", I tried changing the setting but the fans are not changing speed (do I need a reboot?).

Next I tried changing the speeds via ipmitool, but got various errors like:

Code:
Unable to send RAW command (channel=0x0 netfn=0x30 lun=0x0 cmd=0x91 rsp=0x82): Unknown (0x82)


After researching a bit, I think these X9 boards (Nuvoton NCT6776F) do not take granular ipmitool commands.... Only fan mode can be set with something like:

Code:
ipmitool raw 0x30 0x45 0x01 0x01


Which similarly to changing the fan mode via IPMIView does nothing....

The mid-wall fans are running at 2400RPM (max 6300RPM). System temps are ALL "acceptable" (ambient temp is 79F/26C)

Code:
=== CPU (4) ===
CPU  0:   38C
CPU  1:   26C
CPU  2:   28C
CPU  3:   29C

=== DRIVES ===
   da0:   31C [2.00TB] ML0221F306AUSD       Hitachi Deskstar 5K3000 (Hitachi HDS5C3020ALA632)
   da1:   34C [2.00TB] S1E1RH1L             Seagate Barracuda 7200.14 (AF) (ST2000DM001-1CH164)
   da2:   31C [2.00TB] 5YD9RJXW             Seagate Barracuda Green (AF) (ST2000DL003-9VT166)
   da3:   32C [2.00TB] ML0220F31B18RD       Hitachi Deskstar 5K3000 (Hitachi HDS5C3020ALA632)
   da4:   31C [2.00TB] ML4220F318UPDK       Hitachi Deskstar 5K3000 (Hitachi HDS5C3020ALA632)
   da5:   37C [3.00TB] PK2234P8JYHX5Y       HGST Deskstar NAS (HGST HDN724030ALE640)
   da6:   35C [3.00TB] PK1234P8K1GMLP       HGST Deskstar NAS (HGST HDN724030ALE640)
   da7:   32C [3.00TB] WD-WCC7K3KL0CV0      Western Digital Red (WDC WD30EFRX-68N32N0)


I wrote to Supermicro but unfortunately since the motherboard and chassis are old and not under warranty, they have been of very little help... I also would like to keep original Supermicro parts if possible and avoid "ghetto style" modifications :cool:

My questions:
  1. Is it possible to manually change the fan speeds on these X9SCL-F boards?
  2. Has anyone had problems like this before?
Server specs:
  • Chassis: Supermicro SC-836 (3U 16 bays)
  • Motherboard: Supermicro X9SCL-F (IPMI 3.52 BIOS 2.3a)
  • CPU: Intel Xeon E2-1220v2
  • RAM: 32GB ECC (4x 8GB Kingston)
  • PSU: 2x redundant PWS-920P-SQ
  • PDU: PDB-PT825-N24
Looking forward to your recommendations and feedback of experience!
 

Attachments

  • recording.zip
    293.9 KB · Views: 128
Last edited:

rvassar

Guru
Joined
May 2, 2018
Messages
972
Find a medium sized #2 Phillips screwdriver with an insulating plastic handle. Carefully place the tip against the main frame of drive "da5", and then place your head against the handle of the screwdriver such that the end of the handle is pressed against your ear, just forward of the Tragicus, so that the sound conduction is imparted just behind your jawbone joint. Observing proper safety restrictions, repeat with any other suspect components until the sound source is located.

No, I'm not kidding... :smile:
 
Last edited:

Spearfoot

He of the long foot
Moderator
Joined
May 13, 2015
Messages
2,478
Hello all,

The problem:
In the last few months my Freenas server has required some parts be replaced because of noise and old age. Unfortunately, even after replacing the power supplies and most fans, it is now making a constant annoying medium/high pitched whine. See attached recording and please wear headphones as it is difficult to capture with a cell phone. The whine is not the classic "coil whine" which I am well aware of. This whine "oscillates" as it is not a perfectly "flat" tone line tinnitus or a coil whine so I believe it is of mechanical origin.

Just to clear the obvious, I confirmed that its really this server and not something else in my rack, and also confirmed that other people can hear it which they can very clearly.

I proceeded by elimination until I narrowed down the cause to what I believe to be the mid-wall intake fans. Unfortunately it is impossible to know if this whine was present before I replaced the old PSU and fans because the PWS-800-1R were so loud they noise-drowned everything else..... o_O

In the last few months, I:
  • replaced the two PWS-801-1R with 2x PWS-920P-SQ's
  • replaced the intake fans with brand new FAN-0094L4 (the exhaust fans were always unplugged)
  • replaced the Noctua Nh-D9L CPU cooler fan
  • unplugged each of the power supplies one by one to confirm they are not responsible.
  • removed the 3 mid-wall fans to confirm the CPU cooler fan is not responsible. The server is dead quiet which pretty much eliminates coil whine (electronic origin) and PSU fan whine...
I believe the noise is caused by the intake fans spinning at a speed where airflow turbulence causes this whining, or possibly because they are running at a speed where some sort of resonance occurs with the chassis. In order to confirm this, I removed the chassis top cover and sure enough, the noise is much more bearable (can still hear it but its very faint).

Naturally I tried to manually change the fan speeds using web IPMI (ver. 3.52), but I cannot see an option to do so. Using IPMIView under "IPMI device > Fan Speed Mode", I tried changing the setting but the fans are not changing speed (do I need a reboot?).

Next I tried changing the speeds via ipmitool, but got various errors like:

Code:
Unable to send RAW command (channel=0x0 netfn=0x30 lun=0x0 cmd=0x91 rsp=0x82): Unknown (0x82)


After researching a bit, I think these X9 boards (Nuvoton NCT6776F) do not take granular ipmitool commands.... Only fan mode can be set with something like:

Code:
ipmitool raw 0x30 0x45 0x01 0x01


Which similarly to changing the fan mode via IPMIView does nothing....

The mid-wall fans are running at 2400RPM (max 6300RPM). System temps are ALL "acceptable" (ambient temp is 79F/26C)

Code:
=== CPU (4) ===
CPU  0:   38C
CPU  1:   26C
CPU  2:   28C
CPU  3:   29C

=== DRIVES ===
   da0:   31C [2.00TB] ML0221F306AUSD       Hitachi Deskstar 5K3000 (Hitachi HDS5C3020ALA632)
   da1:   34C [2.00TB] S1E1RH1L             Seagate Barracuda 7200.14 (AF) (ST2000DM001-1CH164)
   da2:   31C [2.00TB] 5YD9RJXW             Seagate Barracuda Green (AF) (ST2000DL003-9VT166)
   da3:   32C [2.00TB] ML0220F31B18RD       Hitachi Deskstar 5K3000 (Hitachi HDS5C3020ALA632)
   da4:   31C [2.00TB] ML4220F318UPDK       Hitachi Deskstar 5K3000 (Hitachi HDS5C3020ALA632)
   da5:   37C [3.00TB] PK2234P8JYHX5Y       HGST Deskstar NAS (HGST HDN724030ALE640)
   da6:   35C [3.00TB] PK1234P8K1GMLP       HGST Deskstar NAS (HGST HDN724030ALE640)
   da7:   32C [3.00TB] WD-WCC7K3KL0CV0      Western Digital Red (WDC WD30EFRX-68N32N0)


I wrote to Supermicro but unfortunately since the motherboard and chassis are old and not under warranty, they have been of very little help... I also would like to keep original Supermicro parts if possible and avoid "ghetto style" modifications :cool:

My questions:
  1. Is it possible to manually change the fan speeds on these X9SCL-F boards?
  2. Has anyone had problems like this before?
Server specs:
  • Chassis: Supermicro SC-836 (3U 16 bays)
  • Motherboard: Supermicro X9SCL-F (IPMI 3.52 BIOS 2.3a)
  • CPU: Intel Xeon E2-1220v2
  • RAM: 32GB ECC (4x 8GB Kingston)
  • PSU: 2x redundant PWS-920P-SQ
  • PDU: PDB-PT825-N24
Looking forward to your recommendations and feedback of experience!
You might try plugging in the rear exhaust fans, and then using the 'Standard' or 'Optimal' fan mode.
 

freenas-supero

Contributor
Joined
Jul 27, 2014
Messages
128
No, I'm not kidding... :smile:

Hahha :D Yes I "heard" this technique before. Bone conduction method. Works well but I could not pinpoint the origin of the noise that way although I did not think about probing da5. Why da5?????? Now I'm very curious....

You might try plugging in the rear exhaust fans, and then using the 'Standard' or 'Optimal' fan mode.
The rear exhaust fans are unplugged. They always were as I don't need them (IMO). Are you suggesting to plug them like putting a piece of cardboard to block the air flow going through them?

Also like I said in my OP, I tried changing the fan modes but nothing happens. Using IPMIView I can monitor real time the fan speeds, and changing the fan mode and hitting "Update" button does nothing. Do I need to reboot the server after changing the modes for the changes to be taken into account?
 

rvassar

Guru
Joined
May 2, 2018
Messages
972
Why da5?????? Now I'm very curious....

Simple... It has the highest temp. That and I have some passing familiarity with HGST drives. They can make some odd noises as the bearings start to fade, and I'm assuming (possibly incorrectly) the elevated temp might be a proxy for age. But mostly I picked it just to have a starting point. :smile:
 

Spearfoot

He of the long foot
Moderator
Joined
May 13, 2015
Messages
2,478
The rear exhaust fans are unplugged. They always were as I don't need them (IMO). Are you suggesting to plug them like putting a piece of cardboard to block the air flow going through them?
No, I'm think running them would be a good idea. More fans spinning, but at a lower speed, may get rid of your noise problem and still keep everything cool.

Also like I said in my OP, I tried changing the fan modes but nothing happens. Using IPMIView I can monitor real time the fan speeds, and changing the fan mode and hitting "Update" button does nothing. Do I need to reboot the server after changing the modes for the changes to be taken into account?
If you can't change the system's fan mode, then something is wrong with your BMC/IPMI. I've seen this before...

Your X9SCL-F is probably very similar to my X9DRi-LN4F+... and it sounds like you need to re-configure your BMC.

Shut down the OS on your X9SCL system. Then navigate to its IPMI web interface, under 'Maintenance': you should see options to reset the BMC to factory default, and to do unit and iKVM resets (see attached screenshot). Be sure you know the default BMC credentials -- probably ADMIN:ADMIN on these older systems -- and save a copy of your IPMI configuration. Then reset the IPMI to 'Factory Default'. After the BMC 'reboots', see if you can change the fan mode. If so, you can restore your IPMI configuration and start tinkering with fan modes to find the optimum setting for your system.

Good luck!

ipmi-menu.jpg
 

freenas-supero

Contributor
Joined
Jul 27, 2014
Messages
128
Hmm' kay. So I proceeded to reset IPMI/BMC to factory default, but still no way of changing fan settings within the Web IPMI.

Still the same from IPMIView. The setting does nothing. Is it possible the only way of changing the fan settings is in the BIOS on these older boards ???

I also tried the screwdriver thing. I will report when I have a better idea.. For now, its either the mid-wall fans or one (or many) hard drives on the bays 1 to 4 because the noise seems "stronger" on the left side of the chassis.

Bays 5-8 are empty so that's good.
 

Spearfoot

He of the long foot
Moderator
Joined
May 13, 2015
Messages
2,478
Hmm' kay. So I proceeded to reset IPMI/BMC to factory default, but still no way of changing fan settings within the Web IPMI.

Still the same from IPMIView. The setting does nothing. Is it possible the only way of changing the fan settings is in the BIOS on these older boards ???

I also tried the screwdriver thing. I will report when I have a better idea.. For now, its either the mid-wall fans or one (or many) hard drives on the bays 1 to 4 because the noise seems "stronger" on the left side of the chassis.

Bays 5-8 are empty so that's good.
You're running the latest BMC firmware for your X9SCL-F -- 3.52 -- and the IPMI webpage and IPMIView both let you toggle the fan mode; but doing so has no effect, right?

Odd... And you've reset the BMC, but still no joy?

An alternative is to re-install the BMC firmware, being careful not save any settings.

Other than that, I'm all out of ideas.
 

freenas-supero

Contributor
Joined
Jul 27, 2014
Messages
128
You're running the latest BMC firmware for your X9SCL-F -- 3.52 -- and the IPMI webpage and IPMIView both let you toggle the fan mode; but doing so has no effect, right?
100% Correct.

And you've reset the BMC, but still no joy?
Again, 100% Correct.

I replied to Supermicro to ask how to live change the fan settings on these motherboards without rebooting and going into the BIOS.... Sounds a bit stupid to have to do so on a server motherboard.... They're really not of any help because there is no warranty...
 

freenas-supero

Contributor
Joined
Jul 27, 2014
Messages
128
I'm making ZERO progress....

Supermicro are very vague in their answers. The tech rep keeps telling me that I can change the fan settings on this motherboard while its NOT working and my research shows its not possible on these old X9 boards with C series chipsets.... Apparently, and it appears to be the case, that fan speeds cannot be fine tuned on these boards and ONLY changed via preset modes (FULL, OPTIMAL, etc....).

They recommended to use SMCIPMITool to set/get the fan modes. I did, and set the mode to FULL. I'd normally expect the fans to rev up from 2400RPM to 6300RPM which is the maximum speed of these FAN-0094L4's but they wont budge...

Everything indicates that the only way to control the fan speeds is to either go in BIOS (which works, I've done it before) or use ipmitool (or another similar tool) and chanhge the fan speed mode. Now why its not working on my board, defective BMC chip ?

Code:
wuser@workstation:~/SMCIPMITool_2.25.0_build.210326_bundleJRE_Linux_x64$ java -jar SMCIPMITool.jar XXX.XXX.XXX.XXX ADMIN XXXXXXXXXXXXXX ipmi fan
Current Fan Speed Mode is [ Full Speed ]

Supported Fan modes:
0: Standard Speed
1: Full Speed
2: Optimal Speed


This weekend I will strip the entire server apart. I've had enough.
 

Spearfoot

He of the long foot
Moderator
Joined
May 13, 2015
Messages
2,478
I'm making ZERO progress....

Supermicro are very vague in their answers. The tech rep keeps telling me that I can change the fan settings on this motherboard while its NOT working and my research shows its not possible on these old X9 boards with C series chipsets.... Apparently, and it appears to be the case, that fan speeds cannot be fine tuned on these boards and ONLY changed via preset modes (FULL, OPTIMAL, etc....).

They recommended to use SMCIPMITool to set/get the fan modes. I did, and set the mode to FULL. I'd normally expect the fans to rev up from 2400RPM to 6300RPM which is the maximum speed of these FAN-0094L4's but they wont budge...

Everything indicates that the only way to control the fan speeds is to either go in BIOS (which works, I've done it before) or use ipmitool (or another similar tool) and chanhge the fan speed mode. Now why its not working on my board, defective BMC chip ?

Code:
wuser@workstation:~/SMCIPMITool_2.25.0_build.210326_bundleJRE_Linux_x64$ java -jar SMCIPMITool.jar XXX.XXX.XXX.XXX ADMIN XXXXXXXXXXXXXX ipmi fan
Current Fan Speed Mode is [ Full Speed ]

Supported Fan modes:
0: Standard Speed
1: Full Speed
2: Optimal Speed


This weekend I will strip the entire server apart. I've had enough.
I'm sorry for your troubles.

FreeNAS/TrueNAS comes with ipmitool installed. You should be able to run this command to set the fan mode to 'full':
Code:
ipmitool -I lanplus -H <BMC IP address> -U <BMC Username> -P <BMC password> raw 0x30 0x45 0x01 01
This works for me, on my older X9DRi-LN4F system.

Good luck
 

freenas-supero

Contributor
Joined
Jul 27, 2014
Messages
128
@Spearfoot , no problem man, I will find a way to make this work somehow.

Supermicro has told me that the fan speeds should change when I issue the fan mode change from IPMIView or with SMCIPMItool.... Now they asked me to clear the CMOS by shorting JPT1....

I need to open up the chassis and shutdown freenas. Problem is, I've got a bunch of clients using it as their cloud storage.... Needs a major shutdown but this weekend I will take the time to do it.

The command you suggested is identical to the one I issued directly on freenas via SSH and has no effect.

Like I said to Supermicro, the chassis fans are running at around 2400RPM and wont budge unless I manyally enter BIOS and set the mode to FULL then they will spin at 6300RPM (or around) which is their max speed acc. https://store.supermicro.com/80mm-fan-0094l4.html
 
Top