C-states "bug"?

[DrKK]

So, last night in the IRC channel and in the mumble, we talked about a number of interesting (like, literally, interesting) bugs that were cropping up, and one has me fascinated: Bug number 4123.

Basically, this guy brings up two points. The first one is that something totally screwy happens when he tries to use this particular sysctl from the GUI. OK. I am sure they will fix that.

But the second one he brings up is: Why are the C-State-lowest settings defaulting to C1 in the first place (excellent question that I'd like to know the answer to), and why the hell are all the usual C-states not even appearing in the sysctl states available (meh--good question academically, but practically, it's not important).

The answer appears to be important, because with powerd and everything else running, when I go ahead and manually set the cx_lowest to 2:

Code:

[root@freenas] ~# sysctl hw.acpi.cpu.cx_lowest=C2

...sure enough, it shaves a couple watts off the power consumption (no doubt about it!!), and I have noted not a single performance impact, so why would I *NOT* want this to be the default? And I'd like the bug to be fixed (about the CLI sysctl being whacky) so I can make it automatic on my box without having to do a hack.

 

[joeschmuck]

I realize this is a year old thread but @DrKK , did you ever find out more about this? My AMD CPU supports C6 state, not that I need to go that far however I do plan to test how far it goes and to see if it remains stable. I need to dig out my Kill-a-Watt meter first so I can see what kind of savings I am really getting since I also plan to let my hard drives remain spinning. I know I played around with this several years ago but FreeBSD wasn't stable back then for me, hopefully times have changed.

And as you said, a few watts, why wouldn't you want to do it. Not only does it save a tiny bit of power but it generates a little bit less heat too.

 

[DrKK]

I realize this is a year old thread but @DrKK , did you ever find out more about this? My AMD CPU supports C6 state, not that I need to go that far however I do plan to test how far it goes and to see if it remains stable. I need to dig out my Kill-a-Watt meter first so I can see what kind of savings I am really getting since I also plan to let my hard drives remain spinning. I know I played around with this several years ago but FreeBSD wasn't stable back then for me, hopefully times have changed.

And as you said, a few watts, why wouldn't you want to do it. Not only does it save a tiny bit of power but it generates a little bit less heat too.

Well, it's been a year, but let me download to you what I know about various power considerations:

• Just because the CPU goes down to C6 or whatever, doesn't mean you'll be able to set it in FreeBSD. One of the sysctl's will tell you how low FreeBSD can take it:
  Code:

  [root@giskard] ~# sysctl -a | grep cx_
  hw.acpi.cpu.cx_lowest: C2
  dev.cpu.0.cx_supported: C1/1/1 C2/2/117
  dev.cpu.0.cx_lowest: C2
  dev.cpu.0.cx_usage: 19.23% 80.76% last 0us
  dev.cpu.1.cx_supported: C1/1/1
  dev.cpu.1.cx_lowest: C2
  dev.cpu.1.cx_usage: 100.00% last 0us

• I have been using the lowest available cx_lowest for over a year, with no ill effects. Saving of a few watts. Works out to $2/year for me. I have a choice between either receiving $2 for nothing, or NOT receiving $2 for nothing. I choose the former.
  
• Don't bother trying to set a "sysctl" directly in the FreeNAS GUI for this; there is another loader-time setting that over-rides it. This is confusing for many users, for example, bug reports here, here, and even DrKK. So, that means you have to manually set it in the shell after each reboot, or:
• You can set it in Tasks->Init/Shutdown Scripts->Add Task, choosing "command" and "post-init", and then type in your sysctl statement. That works fine.
• The BIOS is important. When I updated my BIOS on the SuperMicro X10SLM, I no longer was able to set/observe each core's C-state separately in sysctl, but rather, I seemed to only be able to see the first one's. I don't know why this is. The other ones were set to C1, or at least, so it reported. So I'm not sure what's going on there, but that behavior in FreeBSD changed with an upgrade to the motherboard BIOS.
• Something in the 9.3 upgrade (I don't know what) caused powerd to use way way way way way way less CPU than it used to. powerd used to be second or third in my CPU-used list, and now it's not even in the top ten.
• Routinely, I set any Supermicro-based FreeNAS for myself, or anyone I build for, to consume the least amount of power, by using a combination of the above sysctl for the C-states, and using powerd. I have noticed no performance impact whatsoever, as the system appears to go straight to max power when demand warrants it.

Anyway, that's about as much as I know, sir.

 

[joeschmuck]

Thank you for the very quick response and all that information. I'm snowed in at home for the day and since I've just finished shovelling out my driveway, now I have something I can do. Interesting about the BIOS but I have read that the BIOS is a big part of the C-States working properly.

 

[DrKK]

It's weird how much clearer things are after you have a year or two of experience, isn't it. Like, my original post doesn't even make any sense, since I appeared to have misunderstood the bug ticket somewhat. lol.

 

[joeschmuck]

Yea but the fact that you posted in the forums helped me because you just shared with me your experience with C-states which it good for me. And you did provide a secret trick to make everything work because honestly, I would have been puzzled at why the systctl didn't work via the GUI. I will try it myself to see my results but I suspect it will fail as you have stated above. This does save me time.

You are a wealth of information my friend.

 

[DrKK]

Well like I said, if you do a post-init command task in the GUI, *THAT* will definitely work, by definition.

 

[cyberjock]

See, and I had a different behavior with powerd when I experimented with it.

https://forums.freenas.org/index.php?threads/powerd-and-how-it-can-impact-server-performance.10739/

 

[joeschmuck]

Odd, I didn't get a notification email that you posted yesterday. Oh well.

Hey, nice posting, lots of great information.

 

[cyberjock]

Just for curiosity's sake I tried turning on powerd and C2 states on my Mini. Here's what I got...

So if you enable powerd and C2 on the FreeNAS Mini, you'll save about 7 watts or so. I didn't test the performance of the server or anything, just checked power usage when I enabled the various settings.

The CPU temp did drop from 46C to 39C. So very slight tangible benefits. Keep in mind the CPU is passively cooled, so CPU temp is higher than active cooled CPUs.

Chassis fan speed stayed at 900 RPM the entire time I was monitoring it.

 

[joeschmuck]

Nice chart, wish I had something like that.

 

[cyberjock]

Nice chart, wish I had something like that.

You can!

 

[joeschmuck]

That is cool, very cool. If I purchased it then I'd be buying as a toy but a very cool toy. Actually, that would come in handy to remotely power on/off my NAS in the basement and then just set the BIOS to turn on when power is applied (I think that is a setting in that BIOS). Thanks, that could make my life just a little easier while adding a little flare to it.

EDIT: Just ordered it. As I mentioned above, it will also allow me to remotely turn on and off the NAS input power and that is a huge benefit. Now if I could only obtain BIOS access remotely without it costing me much, I'd do that. Someday after lighting or a flood takes out my current server, I will buy a much more capable server. I almost can't wait for the lightning strike :D

 

[DrKK]

Why is your Mini using so much more more juice than my X10SLM?

 

[cyberjock]

What's your power usage when idle?

I've got 4 WD Reds plus 2 SSDs and the SATA DOM boot device.

 

[Ericloewe]

Why is your Mini using so much more more juice than my X10SLM?

The ASRock C2X50 boards have a PCI-e switch that is a constant, significant power hog (~10W, by some accounts).

 

[joeschmuck]

So I've done some testing and I used the nice new mFi power device, nice tool and I'll be able to cycle power to the NAS when it's returned to the basement in a few days along with my second small NAS.

So for my system using powerd was useless and actually but adding a single tunable "performance_cx_lowest=C2" dropped my power draw from 85 Watts to 72 Watts. I need to see if there is anything more I can do to my system in order to reduce power consumption, I'd like to drop it another 10 watts if I can. Unfortunately FreeBSD can only drop my CPU down to C2 state, it isn't capable of using the other states (down to C6 for my CPU), but I'm sure there are other items I can tweak.

FreeNAS default (no tweaks), 76 Watts
FreeNAS powerd Enabled, 85 Watts
FreeNAS powerd OFF, performance_cx_lowest=C2 (or Cmax), 72 Watts
FreeNAS powerd ON, performance_cx_lowest=C2, 75 Watts
BIOS turning on VM support, powerd OFF, performance_cx_lowest=C2, 73 Watts

I still have more to do but I'm done for the night.

 

[DrKK]

Wow.

Feel like we should ask a FreeBSD neckbeard about how it happens that powerd adds 10% to your consumption.

 

[joeschmuck]

Very weird thing happened, at 8 hours to the minute my power consumption jumped up from 72 Watts to 84 Watts and it is slowly creeping down (At 80.3 Watts now). Nothing was going on (no backups or use). There was no CPU, Network, or any spike in usage. I don't know why it happened. Guess I'll have to see if this is repeatable. Nothing in the log file either. I enabled powerd again and it jumped right back up to 85 Watts, disabled it and it jumped back to 80.3 Watts. Rebooting drops it back to 72 Watts. Guess I have something to figure out.

I was unable to lower my normal power consumption below 72 Watts via the BIOS settings without having to go through all the stability testing again (underclocking the CPU and other components and lowering RAM voltage) and its not worth the time invested. I think the only other viable option would be to install a SSD and move over all the active files (jails, logs, etc...) and have my hard drives spin down however I like the data being available immediately and not having to wait for the drives to spin up, plus any additional wear and tear on those drives due to all the spinup and down cycles.

Conclusion: I don't have a very efficient MB/CPU combination when the system is idle and FreeBSD/FreeNAS is not capable of dropping the CPU below the C2 state for my CPU.

UPDATE (3/6/2015): After watching my FreeNAS machine power for a week now, the average power consumption is 75 watts. The only time it goes above 75 watts is when it's running a task such as CIFS support and there are periodic very short pulses which I believe are routine FreeNAS background operations. I would like to get my power consumption lower however I don't believe that will be possible with my current system hardware. Who knows, maybe I'll toss this system to my daughter as her main computer and I'll build another one, but since my wife just gave me a very nice laptop (Dell Inspiron 7352 w/Intel i7, 1TB drive, 8GB RAM), I'll have to hold off on any more upgrades for a while although I'm thinking about a 500GB SSD for the laptop.

 

[Fran Aquino]

On my AMD FX 8320E + AMD 760G chipset + 6x Samsung Spinpoint M7 idle power consumption hovers around 100 W with hw.acpi.cpu.cx_lowest=C1
After setting it to C2, idle power went down to 55 W.