Odd issues with schedular/power management?

[aufalien]

Hi,

I've noted 30% diff in performance via fio and iozone between FreeNAS and CentOS6.4 when run on the same hardware (at diff times of course). Linux was the faster one.

My ZFS tunables etc are all the same as is my pool layout.

After spending a good deal of time troubleshooting, I found that these BIOS settings were enabled and causing the issue in FreeNAS;

Enhanced Intel SpeedStep Technology
Intel Turbo Boost Technology

I disabled SpeedStep (Turbo auto disabled and did not allow changes) and FreeNAS not only ran as fast as Linux, but a tad better.

Now check this, upon re enabling SpeedStep and Turbo and then enabling powerd via the FreeNAS UI, the system then ran even a tad better.

So my question is how can I modify powerd to always run hot?

Thanks in advance,

- aurf

** I've another BIOS setting under "Power & Performance" were I've always set it to Performance on my servers.

 

[aufalien]

UPDATE; I found this article;

http://www.ateamsystems.com/blog/Increase-FreeBSD-Performance-With-powerd

Which alludes to 2 settings in rc.conf;

powerd_enable="YES"
powerd_flags="-a hiadaptive"

I looked at rc.conf and don't see powerd enabled so I assume it is done via another method as it is enabled via the UI.

Were would I place the powerd_flags line?

 

[Dusan]

Just enable powerd in the GUI (System->Settings->Advanced) and you are done.
It will start when you save the settings and then also on every reboot (check /var/tmp/rc.conf.freenas, which is dynamically created by /etc/rc.conf.local). The hiadaptive mode is default (http://www.freebsd.org/cgi/man.cgi?query=powerd), so there's no need to specify it.

 

[aufalien]

Ah, very good, thanks Dusan.

With both the BIOS settings and powerd enabled, I'm pushing 26K IOPS were as before it was around 19K (via fio).

- aurf

 

[aufalien]

Not to revive this old thread but how would one make rc.conf changes persistent? I'm testing a few tweaks.

Actually to be more exact, both my fio and iozone tests are showing increases when I disable powerd from the GUI and run it with powerd -a max from a shell.

So I will need to manipulate powerd variables in order to get the desired behavior. Unsure if I want max but perhaps modify polling etc...

 

[Dusan]

While modifying the USB filesystem is possible, it is not recommended. Try this instead:

1. Disable powerd in System->Settings
2. Start it as a Post Init command (http://doc.freenas.org/index.php/Init/Shutdown_Scripts)

 

[aufalien]

Hi D,

I was thinking of this but wanted a more elegant solution. However thinking back, its pretty elegant in itself.

Thanks man.

Dovidjenja,
- aurf

 

[aufalien]

UPDATE;

After disabling powerd via GUI and adding a post init command of

powerd -a hiadaptive -p 25

*The default interval is 250.

My iops via fio jump ~25% which is repeatable. I mild but important tweak.

- aurf

 

[aufalien]

Hey, sorry to keep posting on this old thread, but I need to test this setting;

performance_cpu_freq="HIGH"

However it normally goes into rc.conf.

How would I implement this in FreeNAS?

Sorry and thanks.

I'm not enjoying reviving an old thread, its like making the dead walk :)

 

[cyberjock]

You got your answer before in your own thread in August... http://forums.freenas.org/threads/proper-way-to-perform-rc-conf-and-sysctl-conf-mods.14541/

To quote dlavigne,

rc.conf = a feature request: https://bugs.freenas.org/issues/3051

 

[aufalien]

The odd thing is that I thought it was commited already. Is it not in v9.1.1? I don't see the option.

 

[cyberjock]

Nope. It's "Unscreened". That means not implemented and not even screened for any particular version now or in the future. AKA, "eh.. something that should probably be done, but we have more important demons to slay in the FreeNAS code right now and for the next version".

If you own a TrueNAS box and have a current valid license and you call them they'll probably implement it as a paying customer.

Other than that, sit back and wait it out or pay one of the freelance guys that do code changes to FreeNAS to add the code for you. Hooray for open source code! If you are interested in this, I'd post it here and see if they respond(they might not take too kindly to a PM assuming you can find them). I don't know how involved adding that feature would be, but I probably wouldn't recommend you soliciting for developer resources for this feature unless you plan to offer $100 or more. It will require WebGUI changes as well as python code changes, so not a trivial fix.

 

[Dusan]

All that setting does is that when the computer switches to AC power this will be executed (via devd and /etc/rc.d/power_profile):
sysctl dev.cpu.0.freq=`sysctl -n dev.cpu.0.freq_levels | awk '{ split($0, a, "[/ ]"); print a[1] }'`
i.e. it switches the CPU to the highest frequency. However, it only makes sense if you do not run powerd as powerd would immediately change the frequency back to whatever it thinks is appropriate at the moment.

 

[aufalien]

Hey Cyberjock, many thanks for the reply. I like it very much and will take it under serious advisement.

Hey Dusan, I still need to test as I believe that FreeBSD has scaling/turbo issues with my Intel E5 2400 series CPUs. For example, my CPUs show via powerd -v as not going any higher then 2201 but the CPU, while a 2.2Ghz proc, can boost to 2.7Ghz. Most importantly my testting is showing some oddities. Thirdly, doing a search online shows that I'm not the only one with this concern.

So I need to test test test for myself.

 

[cyberjock]

I'm fairly sure that the turbo is strictly in hardware, so powerd wont list it as a possible option because you can't "set" the CPU to anything above 2201. That sounds completely normal and expected.

Can you describe those oddities in particular? Or did you describe them already above?

Lastly, I believe powerd is lied to and when the CPU is in turbo mode it doesn't actually get told the true CPU speed. I could be wrong though.

Did you read through my thread where I discussed powerd http://forums.freenas.org/threads/powerd-and-how-it-can-impact-server-performance.10739/ ?

 

[Dusan]

So I need to test test test for myself

So, switch off powerd and run the command! That's why I gave you the command. It will switch your CPU freq to "2201".
However, 2201 is the maximum you will ever see and it means that the Intel Turbo Boost is working. ACPI indicates all turbo frequencies as max_frequency+1 (i.e. 2201 in your case). There is no way to see what is the actual frequency.
It's all explained here: https://wiki.freebsd.org/TuningPowerConsumption

ACPI may report extra performance level with frequency 1MHz above the nominal to control Intel Turbo Boost operation. For example, on Core i7-870you may see:
dev.cpu.0.freq_levels: 2934/106000 2933/95000 2800/82000 ...
, where value 2933 means 2.93GHz, but 2934, depending on situation, means 3.2-3.6GHz

 

[cyberjock]

My recommendation would be to disable or enable powerd(whichever you prefer for your situation) and be done with it? If you need more IOPS there's other ways to do that (l2arc, slog, etc.) and considering the time you've spent on this and are likely to continue to spend on it I'd think that it would be easier/cheaper/or both to just add some more hardware. So can you elaborate on why this is so important to you? I'm asking because you sound like a man on a mission and usually those are fun, educational and often find obscure bugs. So I'm wondering if I should jump in on this too...

 

[aufalien]

Wow, that's a great thread, thanks! I did read it a while back which caused me to disable virtualization. However time to get deeper in and reread it.

The guys over in the FreeBSD forum suggested to keep powerd on as it can go turbo and has a better chance to get better performance with it. Hence why I want to test completely with all settings. This is a biz class NAS so I need to perform due diligence so to speak.

As for oddities, during both fio and iozone tests that I run back to back a few times, I get very diff values. My last tests would show lower values that my first tests then visa versa, etc... so on ... ad nauseam.

Nothing real consistent. So I just wanna ensure that its simply par for the course in terms of test suites being no substitute for real world testing. I realize this but still I need to be 100% certain, if only in my mind that I've built the best FreeNAS box possible using the hardware I have.

 

[cyberjock]

You have skype? we can talk for a few mins and I can probably tell you if you've made any of the common newbie errors...I know them all. ;)

 

[aufalien]

Just read your posts, powerd is disabled and via a post init, running as I stated above with;

powerd -a hiadaptive -p 25

Gave me a nice bump.

You know what Cyberjock, you bring up a good point. I may be suffering from analysis-paralysis. Thanks for the post, its like cold water splashed on ones face after a previous nights stooper :)

Dusan, thanks for that info, now it feels clarified.

PS Thanks to both for being online. Cyberjock, I have no Skpe as I try to minimize my eFootprint if you will. But I'll get one setup and let you know.