OpenCorsairLink in a jail to control fans?

[sretalla]

For the first line, couldn't we just use #!/usr/bin/env perl -w for cross-platform compatibility?

If it works, I'm for it.

I hate modifying the main config and don't want it checking the mini-config for more items than needed, so I would still prefer to work with the automatic filter for disks... maybe we could add the filter string to the mini-config (limiting it to that rather than a potentially long list... my primary nas has 24 spinners and 5 SSDs, so that's a long line to manage). Basically I don't want to have to reboot in order to have the port-init script launched again... maybe there's a better way for that to work too that I haven't though of yet.

 

[Mannekino]

Hi @sretalla I'm gonna spend this weekend to try and get the script(s) to work so the fans will automatically adjust speed based on HDD temps. I have a rather odd thing in the results when I manually try to control.

Code:

./OpenCorsairLink.elf --device 0
Dev=0, CorsairLink Device Found: Commander PRO!

Vendor: Corsair
Product: Commander PRO
Firmware: V0.7.199
Temperature 0: 36.66 C
Temperature 1:  0.00 C
Temperature 2:  0.00 C
Temperature 3:  0.00 C
Output 12v: 12.06 V
Output 5v:  4.97 V
Output 3.3v:  3.36 V
Fan 0:  Mode: Auto/Disconnected
        PWM: 50%
        RPM: 786
Fan 1:  Mode: 4-Pin
        PWM: 50%
        RPM: 767
Fan 2:  Mode: Auto/Disconnected
        PWM: 50%
        RPM: 2783
Fan 3:  Mode: Auto/Disconnected
        PWM: 50%
        RPM: 778
Fan 4:  Mode: 4-Pin
        PWM: 50%
        RPM: 0
Fan 5:  Mode: 4-Pin
        PWM: 50%
        RPM: 0

Fans 0, 1, and 3 are exactly the same yet fan 1 says Mode: 4-pin and the others say Mode: Auto/Disconnected

You have any idea why this is the case? Should I worry about that.

 

[sretalla]

You have any idea why this is the case? Should I worry about that.

You might have run one of the versions of OpenCorsairLink that set the mode successfully at some point... the latest version seems unable to do that, but Auto works just fine, so no issue unless setting the fan speeds won't work (I suspect it does though... easy to test).

It seems the more recently bought hardware that I have came from factory set in Auto mode, whereas the older ones were set to 4 pin.

 

[Mannekino]

You might have run one of the versions of OpenCorsairLink that set the mode successfully at some point... the latest version seems unable to do that, but Auto works just fine, so no issue unless setting the fan speeds won't work (I suspect it does though... easy to test).

It seems the more recently bought hardware that I have came from factory set in Auto mode, whereas the older ones were set to 4 pin.

It seems to be working a bit different with the previous compiled binary I was using. I've identified the following behavior:

When I use this to set the PWM percentage it works just fine with the new binary:

Code:

./OpenCorsairLink.elf --device 0 --fan channel=0,mode=0,pwm=100

With the old binary I would use this command to check fan speeds after changing it.

Code:

./OpenCorsairLink.elf --device 0 --fan channel=0

When I use the above command with the new binary it sets the PWM value to 0% for that fan. If I use the command below to check the speeds it keeps the 100% value I set prior and I see all the values also.

Code:

./OpenCorsairLink.elf --device 0

However the above command would not show fan speeds with the old binary.

Hope that makes sense.

Would this changed behavior mess with your script in any way? Should I use the new or old one?

 

[sretalla]

Would this changed behavior mess with your script in any way? Should I use the new or old one?

The new one is all good... I've switched the latest version to assume --device 0 for the checks.

 

[Mannekino]

Alright good to know. I've been at it for a couple of hours now trying to understand it all. I might have some more questions tomorrow :)

Working through the script now line by line to configure everything for my system.

 

[Mannekino]

Hi @sretalla thank you again for all the work you put into this. I've been at it this weekend to get your script going. It's been a bit overwhelming but I got a start but I also have a bunch of questions.

1. Regarding the script mode.

I have script mode set to "ocl" to control my Commander Pro. Are there still IPMI commands being issued in this mode? Because I have an X9 board and the IPMI stuff doesn't really work that well with regard to issuing commands to change fan speeds, or is it only addressing the Commander Pro?

2. My chassis and fan setup

I have a total of four fans hooked up the Commander Pro inside a Fractal Design Node 804 (album). Three Noctua chassis fans (same model) and Noctua CPU fan (connected to PWM header on the motherboard) and small Noctua fan on the HBA.



I have my fans hooked up like this:

• PWM header 1: exhaust HDD (channel=0)
• PWM header 2: intake HDD (channel=1)
• PWM header 3: HBA fan (channel=2)
• PWM header 4: exhaust motherboard (channel=3)

I configured the ocl_zones like below. Would you say this is correct?

Code:

my @ocl_zones = (
    { 'Fan 4' => 3 },  # CPU
    { 'Fan 1'=> 0, 'Fan 2' => 1,}, # HD
);

3. Is my SATA DOM interfering with my fan control?

Here is a list of all my disks (HDD and SDD)

Code:

da3 36C VAHTEG6L WDC
da2 38C VAJ08P2L WDC
da1 36C VDGV0SPD WDC
da0 34C VDGWGP1D WDC
ada2 39C 100C SMC0515D94519B854045 SuperMicro
ada1 S2E5NXAGA01052 SAMSUNG
ada0 S2E5NXAGA01048 SAMSUNG

As you can see my SATA DOM (ada2) is also reporting a temperature and it's generally higher than the drives of my storage pool (da0-3). I assume the script is also taking this temperature into account, is that correct? If so, how can I exclude the SATA DOM temperature from being used in the calculations?

4. The configuration file

As stated above I have four HDDs I would like to be cooled properly. I was thinking of either using the temperature of the warmest drive or the average of the two warmest drives. What would you suggest I do? I've read the script and README.md file but I don't really grasp the concept of these variables:

• our $config_Kp = 8/3;
• our $config_Ki = 0;
• our $config_Kd = 18;

I'm currently testing with this config file.

Code:

our $config_num_disks = 2; # number of warmest disks to use for average temperature
our $config_Ta = 38.0;       # target average temperature for X warmest disks
our $config_Kp = 8/3;
our $config_Ki = 0;
our $config_Kd = 18;
our $config_hd_fan_start = 50; # HD fan duty cycle when script starts

5. Controlling the HBA fan

For now I've set the HBA fan to 50% PWM but I would like to control the speed of this fan also based on the temperature probe I've hooked up to channel 0. As you can see in the image above I have the probe stuck to the heatsink of the HBA using thermal adhesive tape. This would obviously involve customizing the script. I was thinking of doing this properly by forking your repository on Github and then adding another thermal zone called "HBA". Would this be a good approach? I have no idea where to start even because I have very little Github experience and don't know Perl that well, but I would like this to be a learning exercise also and if I get the approach right at least I have a starting point.

I want to have those fancy graphs also, but that's the next step I've disabled influx for now.

Hopefully you can get through this wall of text :) getting the SATA DOM removed from the monitoring is the highest priority right now otherwise I can't really use it.

Thanks.

 

[sretalla]

I have script mode set to "ocl" to control my Commander Pro. Are there still IPMI commands being issued in this mode?

No. I think all the IPMI commands for CPU temp were excluded and with ocl mode IMPI isn't used for fans either.

As you can see my SATA DOM (ada2) is also reporting a temperature and it's generally higher than the drives of my storage pool (da0-3). I assume the script is also taking this temperature into account, is that correct? If so, how can I exclude the SATA DOM temperature from being used in the calculations?

You can play around with line 879:
next if (/SSD|Verbatim|Kingston|Elements|Enclosure|Virtual|KINGSTON/);

Add SuperMicro for example like this:
next if (/SSD|Verbatim|Kingston|Elements|Enclosure|Virtual|KINGSTON|SuperMicro/);

It won't go into the hd_list that way.

I configured the ocl_zones like below. Would you say this is correct?

Zones look to be set OK.

I was thinking of either using the temperature of the warmest drive or the average of the two warmest drives. What would you suggest I do?

I like the idea of averaging disks rather than taking just a single one...
The .ini files from Kevin are there in the repo, take the first 2d one and it should be fine.

I was thinking of doing this properly by forking your repository on Github and then adding another thermal zone called "HBA". Would this be a good approach? I have no idea where to start even because I have very little Github experience and don't know Perl that well, but I would like this to be a learning exercise also and if I get the approach right at least I have a starting point.

Sounds great.

Get yourself an account at github and push the fork button on my repo in the top right and you'll get your own one to play with. If you come to a point where you're happy with the code, you can make a pull request and I can bring it into my script with that.

 

[Mannekino]

You can play around with line 879:
next if (/SSD|Verbatim|Kingston|Elements|Enclosure|Virtual|KINGSTON/);

Add SuperMicro for example like this:
next if (/SSD|Verbatim|Kingston|Elements|Enclosure|Virtual|KINGSTON|SuperMicro/);

It won't go into the hd_list that way.

Thanks, it turned out my SATA DOM already had "SSD" in its description so it wasn't included. However my two Samsung SSDs were included so I added the word "SAMSUNG" to the list and now it's only my four regular HDDs being put in the hd_list. I'm currently running the script inside a tmux window and I've decreased the debug level to 1, seems to be working well.

I like the idea of averaging disks rather than taking just a single one...
The .ini files from Kevin are there in the repo, take the first 2d one and it should be fine.

That's what I'm using now, thanks. Could you point me to some resources explaining those variables, Kp, Ki and Kd?

I do feel that it's changing the duty cycle a bit too often and therefore the disk temperatures are being increased/decreased which causes more duty cycle changes. I'm not doing anything on disks right now and the temperature in my home is stable. Could the variables mentioned above influence this behavior of the script?

Below is one hour of logging. It keep increasing the duty cycle, and then when the disks are cooler it decreases again feeding itself into a loop I feel like.

Code:

PID Fan Controller Log  ---  Target 2 Disk HD Temperature = 38.00 deg C  ---  PID Control Gains: Kp =  2.667, Ki =  0.000, Kd =  18.0
                               Max   Ave  Temp   Fan   Fan  Fan %   CPU    P      I      D      Fan
2021-01-10 da0  da1  da2  da3 Temp  Temp   Err  Mode   RPM Old/New Temp  Corr   Corr   Corr    Duty
14:01:28   34   37   38   37  ^38  37.50 -0.50   n/a  1770  57/55   29  -2.67   0.00    0.00   54.67%
14:03:29   34   36   37   37  ^37  37.00 -1.00   n/a  1704  55/45   28  -5.33   0.00   -4.50   44.83%
14:05:29   34   36   37   36  ^37  36.50 -1.50   n/a  1606  45/32   29  -8.00   0.00   -4.50   32.33%
14:07:28   34   36   38   37  ^38  37.50 -0.50   n/a  1658  32/39   28  -2.67   0.00    9.00   38.67%
14:09:29   34   37   38   37  ^38  37.50 -0.50   n/a  1637  39/36   29  -2.67   0.00    0.00   36.00%
14:11:29   34   37   38   37  ^38  37.50 -0.50   n/a  1609  36/33   29  -2.67   0.00    0.00   33.33%
14:13:29   34   37   38   37  ^38  37.50 -0.50   n/a  1601  33/31   28  -2.67   0.00    0.00   30.67%
14:15:29   35   38   39   38  ^39  38.50  0.50   n/a  1685  31/42   29   2.67   0.00    9.00   42.33%
14:17:30   35   37   39   38  ^39  38.50  0.50   n/a  1714  42/45   28   2.67   0.00    0.00   45.00%
14:19:29   35   38   39   38  ^39  38.50  0.50   n/a  1725  45/48   28   2.67   0.00    0.00   47.67%
14:21:30   35   37   39   38  ^39  38.50  0.50   n/a  1743  48/50   28   2.67   0.00    0.00   50.33%
14:23:29   35   37   38   38  ^38  38.00  0.00   n/a  1710  50/46   31   0.00   0.00   -4.50   45.83%
14:25:30   34   37   38   37  ^38  37.50 -0.50   n/a  1667  46/39   28  -2.67   0.00   -4.50   38.67%
14:27:29   34   37   38   37  ^38  37.50 -0.50   n/a  1633  39/36   27  -2.67   0.00    0.00   36.00%
14:29:30   34   37   39   37  ^39  38.00  0.00   n/a  1668  36/41   30   0.00   0.00    4.50   40.50%
14:31:30   34   37   39   37  ^39  38.00  0.00   n/a  1671  41/41   29   0.00   0.00    0.00   40.50%
14:33:30   35   37   39   38  ^39  38.50  0.50   n/a  1722  41/48   30   2.67   0.00    4.50   47.67%
14:35:30   35   37   39   38  ^39  38.50  0.50   n/a  1743  48/50   29   2.67   0.00    0.00   50.33%
14:37:31   34   37   38   37  ^38  37.50 -0.50   n/a  1660  50/39   28  -2.67   0.00   -9.00   38.67%
14:39:30   34   37   38   37  ^38  37.50 -0.50   n/a  1637  39/36   28  -2.67   0.00    0.00   36.00%
14:41:31   35   37   38   38  ^38  38.00  0.00   n/a  1674  36/41   30   0.00   0.00    4.50   40.50%
14:43:30   35   38   38   37  ^38  38.00  0.00   n/a  1671  41/41   29   0.00   0.00    0.00   40.50%
14:45:31   34   38   39   37  ^39  38.50  0.50   n/a  1728  41/48   29   2.67   0.00    4.50   47.67%
14:47:30   34   37   39   37  ^39  38.00  0.00   n/a  1690  48/43   28   0.00   0.00   -4.50   43.17%
14:49:31   34   37   39   37  ^39  38.00  0.00   n/a  1694  43/43   28   0.00   0.00    0.00   43.17%
14:51:31   35   37   39   37  ^39  38.00  0.00   n/a  1703  43/43   29   0.00   0.00    0.00   43.17%
14:53:30   35   37   39   38  ^39  38.50  0.50   n/a  1743  43/50   29   2.67   0.00    4.50   50.33%
14:55:31   35   37   38   38  ^38  38.00  0.00   n/a  1711  50/46   29   0.00   0.00   -4.50   45.83%
14:57:31   35   37   38   37  ^38  37.50 -0.50   n/a  1664  46/39   28  -2.67   0.00   -4.50   38.67%
14:59:30   35   37   38   37  ^38  37.50 -0.50   n/a  1640  39/36   31  -2.67   0.00    0.00   36.00%

Edit:
@sretalla could you give me a quick rundown how to get the graphs in Grafana?. I've used the Community Plugin first and both Grafana and InfluxDB. I've created the database (I think) and added it as a data source in Grafana and the test went well. Where to go from here I have no idea.

Can I conceptualize InfluxDB as MySQL for example? That's the pretty much the only DB I have experience with. How can I explore what's in it? Are there tables and rows? How does it know where to write what since you didn't create a table?

Edit2:
Are these actual errors? How should I intepret these lines in the debug log?

Code:

2021-01-10 20:09:46: temperature error = -0.5
2021-01-10 20:11:46: temperature error = -0.5
2021-01-10 20:13:46: temperature error = -0.5
2021-01-10 20:15:47: temperature error = -0.5
2021-01-10 20:17:46: temperature error = -0.5
2021-01-10 20:19:46: temperature error = -0.5
2021-01-10 20:21:46: temperature error = 0.5
2021-01-10 20:23:47: temperature error = 0.5
2021-01-10 20:25:47: temperature error = 0.5
2021-01-10 20:27:46: temperature error = -0.5
2021-01-10 20:29:46: temperature error = 0

 

[sretalla]

Could you point me to some resources explaining those variables, Kp, Ki and Kd?

I think this link covers that question.

Script to control fan speed in response to hard drive temperatures

"man cat" explains that "cat -e" will show a "$" at the end of every line. I'm puzzled by them at the start of the lines though. That is not what I see when I use "cat -e" to view the script on my machine. I wonder if some Windows carriage returns got in there somehow when the script was put...

www.truenas.com

how to get the graphs in Grafana?. I've used the Community Plugin first and both Grafana and InfluxDB. I've created the database (I think) and added it as a data source in Grafana and the test went well. Where to go from here I have no idea.

From the Grafana console, Setings | Data Sources then create one for that database.

Then in a new dashboard, create a new Panel and add queries from that data source.

The data sort-of exposes itself if you just click on the fields, but here's a sample of how I set up 2 of the temperature sensors:

And some Fan Speeds:

Can I conceptualize InfluxDB as MySQL for example? That's the pretty much the only DB I have experience with. How can I explore what's in it? Are there tables and rows? How does it know where to write what since you didn't create a table?

It's all a lot simpler than you think... influxdb is designed to just take everything thrown at it and Grafana is great for just grabbing data in (many kinds of log DBs and) influxdb and displaying it with a minimum of fuss and setup.

Let me know if you have trouble getting further with those hints... there are heaps of online resources for influx and grafana together if you just need the basics of those.

Are these actual errors? How should I intepret these lines in the debug log?

Not real errors, just the way that the script is set to log differences from the expected temperature, meaning action needs to be taken on fan speed. These should stop when you go to errorlevel 0 in final running.

 

[Mannekino]

I think this link covers that question.

Script to control fan speed in response to hard drive temperatures

"man cat" explains that "cat -e" will show a "$" at the end of every line. I'm puzzled by them at the start of the lines though. That is not what I see when I use "cat -e" to view the script on my machine. I wonder if some Windows carriage returns got in there somehow when the script was put...

www.truenas.com

Thanks, I'll take a lot at that tomorrow. Gave it a quick glance and seems complicated.

It's all a lot simpler than you think... influxdb is designed to just take everything thrown at it and Grafana is great for just grabbing data in (many kinds of log DBs and) influxdb and displaying it with a minimum of fuss and setup.

Let me know if you have trouble getting further with those hints... there are heaps of online resources for influx and grafana together if you just need the basics of those.

I fiddled around with it a bit and I can get some results. Will try more tomorrow. I also downloaded some Java UI tool for InfluxDB and found out I had some stuff misconfigured.

Turned out my ocl_zones values were wrong, I had:

Code:

my @ocl_zones = (
    { 'Fan 4' => 3 },  # CPU
    { 'Fan 1'=> 0, 'Fan 2' => 1,}, # HD
);

I thought this was to map the PWM channnels to "human readable" Fan numbers, e.g. channel 0 is Fan 1. But the script is using the "Fan 0" to split the various fan values I believe. After I changed it to below I saw the correct values being logged to InfluxDB

Code:

my @ocl_zones = (
    { 'Fan 3' => 3 },  # CPU
    { 'Fan 0'=> 0, 'Fan 1' => 1,}, # HD
);

Which led me to these settings in the script (line 444). What should I configure here?

Code:

my $cpu_fan_header = "Fan 2";   # used for printing to standard output for debugging
my $hd_fan_header  = "Fan 0";   # used for printing to standard output for debugging
my @hd_fan_list = ("Fan 0", "Fan 1");    # used for logging to file

• $cpu_fan_header should probably be set to Fan 3 since that is the fan in my CPU zone
• $hd_fan_header what should I set this to? I have two fans in this zone but the script only allows for a single string to be configured? I have Fan 0 and Fan 1
• @hd_fan_list is correct by default

Final question, are you running Grafana and InfluxDB in a Jail using the FreeBSD packages? InfluxDB is still at 1.8 and hasn't been updated since May 2020. I believe it has a built-in UI now with version 2.

 

[sretalla]

Which led me to these settings in the script (line 444). What should I configure here?

You need to put the same as in the ocl_zones (at least one per zone and in the case of hdd, all the fans you want to graph.

Final question, are you running Grafana and InfluxDB in a Jail using the FreeBSD packages? InfluxDB is still at 1.8 and hasn't been updated since May 2020. I believe it has a built-in UI now with version 2.

I use influx from a jail, but grafana in a docker container elsewhere. I have never needed any UI for influx itself.

 

[Mannekino]

You need to put the same as in the ocl_zones (at least one per zone and in the case of hdd, all the fans you want to graph.

But the variable $hd_fan_header is a string, can I just turn it into a array like this?

Code:

my $cpu_fan_header = "Fan 2";   # used for printing to standard output for debugging
my $hd_fan_header  = ("Fan 0", "Fan 1");   # used for printing to standard output for debugging
my @hd_fan_list = ("Fan 0", "Fan 1");    # used for logging to file

Or do I need to change it to this?

Code:

my $cpu_fan_header = "Fan 2";   # used for printing to standard output for debugging
my $hd_fan_header  = "Fan 0, Fan 1";   # used for printing to standard output for debugging
my @hd_fan_list = ("Fan 0", "Fan 1");    # used for logging to file

 

[sretalla]

But the variable $hd_fan_header is a string, can I just turn it into a array like this?

No, you need to pick one (a master of sorts) for that.

All fans that should be reported on go in hd_fan_list

 

[Mannekino]

Alright, I can see the change in temperatures already in the Reporting section of TrueNAS. As an example one of my drives:

 

[sretalla]

As an example one of my drives:

it's getting hotter? maybe that's on purpose to have quiet fans though...

Looks nice and stable though.

 

[Mannekino]

Yes, the default target temperature in the 2d configuration file is 38°C so it raised the HDD temps in my server by 3-4°C. I had my HDD fans set to a PWM of 60% manually before. Now it goes between 30-55%.

I still need to read about Kp, Ki and Kd values and what role that plays. Maybe I should lower the target temperature to 35°C. Not sure if that is doable in the summer months.

 

[sretalla]

I still need to read about Kp, Ki and Kd values and what role that plays. Maybe I should lower the target temperature to 35°C. Not sure if that is doable in the summer months.

Don't worry too much about the how... those values just control the amount of change in response to the difference from the desired temp. Kevin and Glorious1 have done the work to tune those numbers to arrive at a good setting over time.

I usually set my desired temperature to 38 in winter and 40 in summer to avoid deafening myself with the fans. The whole point of the script is to have the fans as quiet as possible without letting your disks overheat (protecting the disks at the cost of more noise if really needed).

 

[Mannekino]

OK, I think I will lower my target temp to 36°C then. When I had my PWM set to 60% manually I couldn't really hear my server. The Noctua fans I have are pretty silent.

 

[Mannekino]

Hi @sretalla, it's been a couple days now since I started using the script. I lowered the debug level to 1 and I'm still running the script in a tmux window.

I've really enjoyed the proces of getting it to work and I followed your instructions for the Grafana graphs. I managed to create some and what I already had seen in the logging I see now also in the graphs. I think it's feeding itself into a loop of ramping the fans up and down. Do you think there's anyway I can break that cycle and smooth out the curve so it doesn't keep raising and lowering the FanDuty but keeps it at a more stable percentage?

Weirdly enough, if you look at the graphs you can see that it was rather stable today between 10:50 and 13:30, after that it starting the cycle again.