AMD Ryzen - hwpstate0: set freq failed, err 6

[re0]

Hi FreeNAS community!

I recently built a new "server" (using consumer parts, but that's the FreeNAS spirit) to replace a Pentium system which was lacking the power for VMs. However, I have stumbled across a problem with the CPU frequency being stuck at 3200 MHz constantly (which is the base clock). As far as I know, the all-core turbo frequency is 3400 MHz (versus 1(?) core maximum of 3600 MHz). I cannot confirm whether this problem occurred on the previous system, but what I can confirm is frequency scales fine in Windows 10.

Just to give a bit of a heads up on the system specifications, they are as follows:

• ASRock AB350 Pro4 (BIOS 4.20)
• AMD Ryzen 5 1600
• 16GB ECC RAM (CT4G4WFS824A * 4)
• 2 * 2TB Seagate IronWolf NAS drives (ST2000VN004)
• 2 * 1TB Seagate Barracuda (standard desktop drives)
• EVGA B2 750 PSU (overkill, but in my defence it was pulled from an dismantled system)

I am currently running FreeNAS 11.1-RC3 (updated from 11.0-U4). Part of the reason for updating is because I read the following from the User Guide:

Support has been added for the HBA 9400-81, Intel Skylake and Kaby Lake processors, and Ryzen processors.

Though perhaps I was wrong to make an assumption that this resolved issues relating to Ryzen processor states, and the above actually relates to bug #25987 on iXsystem's Redmine (that is related to hanging, which I have not experienced).

I should note that in the BIOS the following related to power management is enabled:

• Cool'n'Quiet
• Global C-sate control (is actually set to auto at the moment, but "enabled" did not make a difference)

I appreciate it is not a necessarily a comprehensive list, but I cannot think of much more at this stage in time. The only other option I recall having changed is SVM (which is not related to power management and is enabled for virtualisation).

I have executed some commands on the system to provide more information below:

Code:

% sysctl -a | grep cx_
hw.acpi.cpu.cx_lowest: C2
dev.cpu.11.cx_method: C1/hlt
dev.cpu.11.cx_usage_counters: 344990
dev.cpu.11.cx_usage: 100.00% last 2592us
dev.cpu.11.cx_lowest: C2
dev.cpu.11.cx_supported: C1/1/1
dev.cpu.10.cx_method: C1/hlt
dev.cpu.10.cx_usage_counters: 366018
dev.cpu.10.cx_usage: 100.00% last 1658us
dev.cpu.10.cx_lowest: C2
dev.cpu.10.cx_supported: C1/1/1
dev.cpu.9.cx_method: C1/hlt
dev.cpu.9.cx_usage_counters: 345884
dev.cpu.9.cx_usage: 100.00% last 9472us
dev.cpu.9.cx_lowest: C2
dev.cpu.9.cx_supported: C1/1/1
dev.cpu.8.cx_method: C1/hlt
dev.cpu.8.cx_usage_counters: 368543
dev.cpu.8.cx_usage: 100.00% last 720us
dev.cpu.8.cx_lowest: C2
dev.cpu.8.cx_supported: C1/1/1
dev.cpu.7.cx_method: C1/hlt
dev.cpu.7.cx_usage_counters: 345813
dev.cpu.7.cx_usage: 100.00% last 9240us
dev.cpu.7.cx_lowest: C2
dev.cpu.7.cx_supported: C1/1/1
dev.cpu.6.cx_method: C1/hlt
dev.cpu.6.cx_usage_counters: 864803
dev.cpu.6.cx_usage: 100.00% last 4us
dev.cpu.6.cx_lowest: C2
dev.cpu.6.cx_supported: C1/1/1
dev.cpu.5.cx_method: C1/hlt
dev.cpu.5.cx_usage_counters: 334695
dev.cpu.5.cx_usage: 100.00% last 1236us
dev.cpu.5.cx_lowest: C2
dev.cpu.5.cx_supported: C1/1/1
dev.cpu.4.cx_method: C1/hlt
dev.cpu.4.cx_usage_counters: 347619
dev.cpu.4.cx_usage: 100.00% last 3783us
dev.cpu.4.cx_lowest: C2
dev.cpu.4.cx_supported: C1/1/1
dev.cpu.3.cx_method: C1/hlt
dev.cpu.3.cx_usage_counters: 347162
dev.cpu.3.cx_usage: 100.00% last 118441us
dev.cpu.3.cx_lowest: C2
dev.cpu.3.cx_supported: C1/1/1
dev.cpu.2.cx_method: C1/hlt
dev.cpu.2.cx_usage_counters: 352151
dev.cpu.2.cx_usage: 100.00% last 2938us
dev.cpu.2.cx_lowest: C2
dev.cpu.2.cx_supported: C1/1/1
dev.cpu.1.cx_method: C1/hlt
dev.cpu.1.cx_usage_counters: 327927
dev.cpu.1.cx_usage: 100.00% last 2357us
dev.cpu.1.cx_lowest: C2
dev.cpu.1.cx_supported: C1/1/1
dev.cpu.0.cx_method: C1/hlt C2/io
dev.cpu.0.cx_usage_counters: 1676648 687917
dev.cpu.0.cx_usage: 70.90% 29.09% last 895us
dev.cpu.0.cx_lowest: C2
dev.cpu.0.cx_supported: C1/1/1 C2/2/400

CPU frequency information including states:

Code:

% sysctl -a | grep freq
<SNIP>
dev.cpufreq.0.%parent: cpu0
dev.cpufreq.0.%pnpinfo:
dev.cpufreq.0.%location:
dev.cpufreq.0.%driver: cpufreq
dev.cpufreq.0.%desc:
dev.cpufreq.%parent:
dev.hwpstate.0.freq_settings: 3200/3960 2800/2940 1550/1350
dev.cpu.0.freq_levels: 3200/3960 2800/2940 1550/1350
dev.cpu.0.freq: 3200

A bit of a snippit from powerd -v:

Code:

load   0%, current freq 3200 MHz ( 0), wanted freq 2792 MHz
changing clock speed from 3200 MHz to 2800 MHz
powerd: error setting CPU frequency 2800: Device not configured
load  35%, current freq 3200 MHz ( 0), wanted freq 2792 MHz
changing clock speed from 3200 MHz to 2800 MHz
powerd: error setting CPU frequency 2800: Device not configured
load  47%, current freq 3200 MHz ( 0), wanted freq 3499 MHz

I appreciate that many problems encountered are actually due to the underlying FreeBSD system. However, I would certainly be grateful if someone could take a look at my problem and advise what I should do next. For the moment, I have just disabled powerd completely.

 

[nightshade00013]

My guess is that for a while ryzen will be touch and go. Since FreeBSD support is very recently out in the wild there will be issues. This is something you should be filing a bug report on with them so they can get the issues fixed.

One question I have to wonder is that if you had a system up and running with a pentium is why not just grab a xeon instead of a new build. By the time you buy a replacement board and cpu it would likely be cheaper to just get a different cpu.

 

[re0]

Thanks for your response.

My guess is that for a while ryzen will be touch and go. Since FreeBSD support is very recently out in the wild there will be issues.

I had anticipated that. I have read about issues on older AMD hardware especially which have taken time to resolve.

This is something you should be filing a bug report on with them so they can get the issues fixed.

Yes I can and will totally do that. I will report it here when I get a moment.

One question I have to wonder is that if you had a system up and running with a pentium is why not just grab a xeon instead of a new build. By the time you buy a replacement board and cpu it would likely be cheaper to just get a different cpu.

I knew someone would ask this question (or potentially even poke at my build)! I could have just purchased a Xeon (1220 v3 is about £180 new) but I wanted a bit of a project and, ideally, a few more cores and threads, and Intel's CPU positioning is diabolical as there are huge premiums for additional cores (and segmentation of CPUs, removing ECC from all mid-range to high-end desktop CPUs). Furthermore, the old system was pre-built and:

• Motherboard only had 4 * SATA ports
• Case only supported 4 * 3.5" HDDs
• Warranty had expired quite a while ago

So future expansion would need a new case and motherboard or SATA controller card. I am not a sucker for warranties, and I can also admit that quite a few parts in it were pre-owned before even I had them. So really another reason was to build something new and so I researched into Ryzen and ECC support (as this is implemented by the motherboard manufacturer).

 

[joeschmuck]

I read your posting a few times and while it sounded like you said you were doing something, I didn't see anything that you did to change the C State.

I'd recommend that you use a power meter to capture your power draw first, then:
1) Measure the current draw under idle and then a scrub, record these values.
2) Add a tunable "performance_cx_lowest=C2", reboot. Repeat measurements from step 1.
3) Add a tunable "performance_cx_lowest=C1", reboot. Repeat measurements from step 1.

Note: If the above tunable fails to work, try "hw.acpi.cpu.cx_lowest=C1" or "C2".

The C1 level should consume more power than the C2 when the system is idle. If your measurements while in "Auto" are the same as the C2 state, then Auto is working fine.

Why instrument your system to measure current draw, well because you shouldn't trust that a frequency setting means that you are in a power savings mode. Even though I have CPUs which support C6 state, I was unable to use those in FreeNAS/BSD, it just wasn't supported by the OS. The C2 state is a substantial difference.

EDIT: In all of your posting you never mentioned your thread title "hwpstate0: set freq failed, err 6" nor where you received this error message. Please explain in detail where you got this error message.

 

[re0]

Thanks for your reply. I will give this a go to see whether it makes any difference.

I read your posting a few times and while it sounded like you said you were doing something, I didn't see anything that you did to change the C State.

I was unsure of where to start. But I was trying to give as much information as possible to make it easier to understand my issue.

EDIT: In all of your posting you never mentioned your thread title "hwpstate0: set freq failed, err 6" nor where you received this error message. Please explain in detail where you got this error message.

My apologies for omission, I had certainly forgotten to add this into my initial post. I had this spammed in the console with powerd enabled.

 

[joeschmuck]

Have you read this thread? https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=221621

 

[re0]

Sorry for the late reply. I have not really been trying to make progress with this the past few days.

I did have this open in one of my tabs, but only partially read it before bed. I have gone through it since then, and if I am not missing anything then the latest status on this was that there was a patch which "fixes" it in FreeBSD - or by fix it means making it so P0 can change to P2 without having to first go to P1 (which cannot be set) and does not ramp up the wanted frequency in the process of trying to obtain P1.

As far as I can see, from some testing I carried out, this behaviour pretty much mirrors what I am experiencing with the P-States, and I can certainly confirm that frequencies 3200 and 1550 can be set without issue, but 2800 cannot be set manually. But this confirms that this is an issue with FreeBSD itself, and it is something which impacts the Zen architecture.

 

[joeschmuck]

So since this was recently fixed in FreeBAS 11.1, you might want to submit a bug report noting this link and request that the fix also be incorporated. Odds are it will be incorporated in the very near future but if you are willing then you could switch to the nightlies and possibly get it sooner. Again, a bug report should be filed.

 

[re0]

I am currently waiting for an activation link for my account on their Redmine. I have re-requested an activation link by email but it doesn't look to be arriving.

 

[joeschmuck]

I am currently waiting for an activation link for my account on their Redmine. I have re-requested an activation link by email but it doesn't look to be arriving.

Let me know if you don't get it by Monday morning, they may have server work going on this weekend, who knows.

 

[re0]

Let me know if you don't get it by Monday morning, they may have server work going on this weekend, who knows.

I have initially tried to create an account a few days ago and even after re-requesting the activation link around 3-4 times since then I have not received it. I do not know who to chase to get this sorted.

 

[joeschmuck]

I'll contact someone, hopefully it will be resolved shortly.

 

[dlavigne]

I have initially tried to create an account a few days ago and even after re-requesting the activation link around 3-4 times since then I have not received it. I do not know who to chase to get this sorted.

I activated your account so you should be able to create an issue now.

 

[re0]

Thanks for that, dlavigne.

 

[re0]

Oops, looks like I forgot the update this thread!

I have gone ahead and reported the issue on the FreeNAS' Redmine, which can be found under bug #27131.

Thanks for the help, everyone! I would not mind this thread being closed since the problem has been acknowledged and revisions have been made (so a fix is in progress).

I apologise for any ambiguity in the opening post, but it looks like complete understanding has been achieved since then.