So I got bored this morning and decided to experiment with powerd along with some latency issues with my server. Recently migrated my server from Server 2008 R2 to FreeNAS and I have been experimenting with my system.. here's my hardware:
CPU - Intel Xeon E5605 2.13Ghz Quad core (80W thermal) without HT support and BIOS version F12
Motherboard - Gigabyte EX58-UD5
RAM - 12GB of Kingston ECC RAM
Video card - EVGA 512-P3-1310-LR (Nvidia GeForce 210 512MB of RAM)
RAID Cont - Areca 1280ML-24 Port with battery backup
NIC - Intel Server Dual Port(Gb)
Hard Drives - WD20EARS x 18 on RAIDZ3 (23.2TiB total available)
PSU - OCZ Z Series Gold OCZZ850M 850W
Note: I have disabled all unused hardware in the BIOS. This includes onboard NICs, SATA & IDE controller, Firewire, etc.
My server issue (mentioning it here because it is related to this thread): When traversing through directories on my server it often takes 5-10 seconds for a directory to display on my desktops(Windows 7). Even if the directory is empty! I can double click on a new folder and it may load instantly. If I go back to the parent folder and click again it may load instantly or lag(hmmmm!). If I do a 'zpool iostat 1' I can see that the zpool reads about 300k per second every second while the server is lagging then goes idle when the directory is displayed. If I'm trying to stream a video file and traverse directories I can expect my video to freeze until the completion if the directory listing(even if its an empty directory!) which is BS for my hardware. I even tried to force the directory into the ARC by doing a cron job executing 'ls -aR /mnt/tank' every 30 mins. I don't have alot of files either.
So now that I have my FreeNAS server up exactly how I want it I decided it was time to see what kind of power I'm sucking down and hopefully fix my latency problem. This can get annoying if I can expect lag every time I traverse a directory! It's already annoying and I've only been using my server for about 2 weeks.
My UPS has a wattmeter built-in and I've verified that with the server off it reads 000 watts. Booted up my server and after booting is finished I found that powerd is enabled. Good to go. (Hint: Wrong.. but we'll get to that later)
Wattage with system idle - 234w
WOW.. that's a bit high. What's my wattage without the hard drives?
Wattage idle with no hard drives - 99w
What if I disable powerd?
Wattage idle with no hard drives and powerd disabled - 108w
At this point I'd like to mention something about power supplies. They typically have peak efficiency around 50% and have relatively poor efficiency <20% and above 80%. I plan to add some 3TB drives in the next 30 days but I probably should have opted for a 500-600w power supply. Oh well. It was all I had and I wasn't about to buy ANOTHER power supply. I have 2 spares already!
So powerd saves me a whopping 9 watts.
I so hoped for more than that(but 8-12w seems to be pretty typical). But wait.. I've just noticed something. The lag when traversing directories is gone!
Time for some detective work. So what speeds are available to my CPU?
# syctl dev.cpu.0.freq_levels
dev.cpu.0.freq_levels: 2128/80000 1995/66000 1862/55000 1729/45000 1596/37000 1463/30000 1330/25000 1197/20000 1047/17500 897/15000 748/12500 598/10000 448/7500 299/5000 149/2500
WOW. My CPU can go down to 149Mhz. Like I expect to see that, right?
If you run this command you can get the frequency(in Mhz) of your CPU. If you run single CPU this works great, dual CPU you'll have 2 parameters to monitor.
# sysctl dev.cpu.0.freq
dev.cpu0.freq: 149
So my CPU is at 149Mhz. Fantastic. If I disable powerd from the GUI:
# sysctl dev.cpu.0.freq
dev.cpu0.freq: 2128
With the server idle and 1 second intervals while enabling powerd you can see my CPU ramp down:
dev.cpu.0.freq: 2128
dev.cpu.0.freq: 1862
dev.cpu.0.freq: 1729
dev.cpu.0.freq: 1596
dev.cpu.0.freq: 1463
dev.cpu.0.freq: 1330
dev.cpu.0.freq: 1197
dev.cpu.0.freq: 1197
dev.cpu.0.freq: 1047
dev.cpu.0.freq: 1047
dev.cpu.0.freq: 897
dev.cpu.0.freq: 897
dev.cpu.0.freq: 748
dev.cpu.0.freq: 748
dev.cpu.0.freq: 598
dev.cpu.0.freq: 598
dev.cpu.0.freq: 448
dev.cpu.0.freq: 448
dev.cpu.0.freq: 299
dev.cpu.0.freq: 299
dev.cpu.0.freq: 149
So what happens when streaming a blu-ray? I just so happen to have a blu-ray disk on my server...(1 second intervals again)
dev.cpu.0.freq: 149
dev.cpu.0.freq: 149
dev.cpu.0.freq: 149
dev.cpu.0.freq: 149
dev.cpu.0.freq: 299
dev.cpu.0.freq: 149
So my CPU can handle dishing out a blu-ray with 149-299Mhz. Amazing!
So what about a directory listing where the streaming freezes and I have to wait 5-10 second for the directory listing?
dev.cpu.0.freq: 149 <- requested the directory
dev.cpu.0.freq: 149
dev.cpu.0.freq: 149
dev.cpu.0.freq: 149
dev.cpu.0.freq: 299
dev.cpu.0.freq: 149
dev.cpu.0.freq: 149
dev.cpu.0.freq: 149
dev.cpu.0.freq: 149 <- received the directory
WTH? My CPU is idle when I'm sitting at my desk waiting impatiently for a directory listing? This doesn't seem right.
So I go disable powerd, verified CPU frequency is now 2128Mhz and streaming is flawless along with directory listing. I traversed about 15 directories(usually every other directory or so would freeze) and not one freeze or delay of more than 1 second, even directories with 100s of files.
So can we tweak powerd to make it more friendly? After all, this is unacceptable and since I'm bored and this is a learning opportunity why not see about saving those 9watts. Yes, I know if I replace a single incandescent light bulb with a CFL I can save roughly 60-90w, but this is about learning and not about the watts.
With regards to FreeNAS(as well as most other popular OSes) is that the "default" should work for most people. I tend to stick with Intel CPUs, very common hardware, and my hardware far exceeds the recommended specs for the OS. My philosophy is that if you have all 3 of those(of course, AMD CPUs that meet the recommended speeds are acceptable too) and you are being forced to tweak you probably have done something wrong or overlooked something. If you don't have something wrong then its likely a bug. Being that I'm fairly new to FreeBSD I try to avoid jumping to the conclusion that something is a bug when user error is far more. After all, if you aren't sure if what you are doing is right how are you to know if you are doing something wrong?
So I must have something wrong, right? Well, lets take a look at my BIOS settings since its a little hard to get the powerd settings wrong. After all, its a freaking checkbox!
Sure enough, my BIOS has "C3/C6/C7 State Support" and it is disabled. Oops. Let's enable it!
Lets stream a blu-ray and do a directory listing. Sure enough it doesn't freeze. So what happened to CPU freq this time?
dev.cpu.0.freq: 149
dev.cpu.0.freq: 149
dev.cpu.0.freq: 2128 <- opened a directory and received the listing
dev.cpu.0.freq: 1862
dev.cpu.0.freq: 1729
dev.cpu.0.freq: 1596
dev.cpu.0.freq: 1463
dev.cpu.0.freq: 1330
Here's some oddball stuff I'm adding since I did do a little more testing afterwards. I like to play and learn!
1. I disabled the "Virtualization Technology" in my BIOS and saw idle watts drop by 35w. So if you have a FreeNAS server, you should consider disabling this feature as it really adds zero benefit but costs you quite some watts. My CPU temp even went down by 2C!
So what have we learned from all of this:
1. Consider BIOS settings when you have performance issues that can't be identified by insufficient hardware and/or failing hardware.
2. User error with setting up the system.
3. Disable that pesky CPU virtualization feature. It's worthless for FreeNAS and seems to impact CPU power usage and temp.
4. powerd does work exactly as expected, but the results are minimal. If you are having performance issues you may want to try disabling it. You never know when your system is choking on low CPU speed. I've left powerd enabled now that I've fixed my BIOS setting.
5. powerd seems to take about 10-15 seconds to really ramp down CPU frequency all the way. If you are in a business environment I'd probably disable it completely. More than likely the CPU will never really go completely idle and the potential problems don't outweight the benefit. In a business environment you probably aren't too worried about saving 10w. For home use leaving it enabled is probably the better option if you care about saving some watts.
CPU - Intel Xeon E5605 2.13Ghz Quad core (80W thermal) without HT support and BIOS version F12
Motherboard - Gigabyte EX58-UD5
RAM - 12GB of Kingston ECC RAM
Video card - EVGA 512-P3-1310-LR (Nvidia GeForce 210 512MB of RAM)
RAID Cont - Areca 1280ML-24 Port with battery backup
NIC - Intel Server Dual Port(Gb)
Hard Drives - WD20EARS x 18 on RAIDZ3 (23.2TiB total available)
PSU - OCZ Z Series Gold OCZZ850M 850W
Note: I have disabled all unused hardware in the BIOS. This includes onboard NICs, SATA & IDE controller, Firewire, etc.
My server issue (mentioning it here because it is related to this thread): When traversing through directories on my server it often takes 5-10 seconds for a directory to display on my desktops(Windows 7). Even if the directory is empty! I can double click on a new folder and it may load instantly. If I go back to the parent folder and click again it may load instantly or lag(hmmmm!). If I do a 'zpool iostat 1' I can see that the zpool reads about 300k per second every second while the server is lagging then goes idle when the directory is displayed. If I'm trying to stream a video file and traverse directories I can expect my video to freeze until the completion if the directory listing(even if its an empty directory!) which is BS for my hardware. I even tried to force the directory into the ARC by doing a cron job executing 'ls -aR /mnt/tank' every 30 mins. I don't have alot of files either.
So now that I have my FreeNAS server up exactly how I want it I decided it was time to see what kind of power I'm sucking down and hopefully fix my latency problem. This can get annoying if I can expect lag every time I traverse a directory! It's already annoying and I've only been using my server for about 2 weeks.
My UPS has a wattmeter built-in and I've verified that with the server off it reads 000 watts. Booted up my server and after booting is finished I found that powerd is enabled. Good to go. (Hint: Wrong.. but we'll get to that later)
Wattage with system idle - 234w
WOW.. that's a bit high. What's my wattage without the hard drives?
Wattage idle with no hard drives - 99w
What if I disable powerd?
Wattage idle with no hard drives and powerd disabled - 108w
At this point I'd like to mention something about power supplies. They typically have peak efficiency around 50% and have relatively poor efficiency <20% and above 80%. I plan to add some 3TB drives in the next 30 days but I probably should have opted for a 500-600w power supply. Oh well. It was all I had and I wasn't about to buy ANOTHER power supply. I have 2 spares already!
So powerd saves me a whopping 9 watts.
I so hoped for more than that(but 8-12w seems to be pretty typical). But wait.. I've just noticed something. The lag when traversing directories is gone!
Time for some detective work. So what speeds are available to my CPU?
# syctl dev.cpu.0.freq_levels
dev.cpu.0.freq_levels: 2128/80000 1995/66000 1862/55000 1729/45000 1596/37000 1463/30000 1330/25000 1197/20000 1047/17500 897/15000 748/12500 598/10000 448/7500 299/5000 149/2500
WOW. My CPU can go down to 149Mhz. Like I expect to see that, right?
If you run this command you can get the frequency(in Mhz) of your CPU. If you run single CPU this works great, dual CPU you'll have 2 parameters to monitor.
# sysctl dev.cpu.0.freq
dev.cpu0.freq: 149
So my CPU is at 149Mhz. Fantastic. If I disable powerd from the GUI:
# sysctl dev.cpu.0.freq
dev.cpu0.freq: 2128
With the server idle and 1 second intervals while enabling powerd you can see my CPU ramp down:
dev.cpu.0.freq: 2128
dev.cpu.0.freq: 1862
dev.cpu.0.freq: 1729
dev.cpu.0.freq: 1596
dev.cpu.0.freq: 1463
dev.cpu.0.freq: 1330
dev.cpu.0.freq: 1197
dev.cpu.0.freq: 1197
dev.cpu.0.freq: 1047
dev.cpu.0.freq: 1047
dev.cpu.0.freq: 897
dev.cpu.0.freq: 897
dev.cpu.0.freq: 748
dev.cpu.0.freq: 748
dev.cpu.0.freq: 598
dev.cpu.0.freq: 598
dev.cpu.0.freq: 448
dev.cpu.0.freq: 448
dev.cpu.0.freq: 299
dev.cpu.0.freq: 299
dev.cpu.0.freq: 149
So what happens when streaming a blu-ray? I just so happen to have a blu-ray disk on my server...(1 second intervals again)
dev.cpu.0.freq: 149
dev.cpu.0.freq: 149
dev.cpu.0.freq: 149
dev.cpu.0.freq: 149
dev.cpu.0.freq: 299
dev.cpu.0.freq: 149
So my CPU can handle dishing out a blu-ray with 149-299Mhz. Amazing!
So what about a directory listing where the streaming freezes and I have to wait 5-10 second for the directory listing?
dev.cpu.0.freq: 149 <- requested the directory
dev.cpu.0.freq: 149
dev.cpu.0.freq: 149
dev.cpu.0.freq: 149
dev.cpu.0.freq: 299
dev.cpu.0.freq: 149
dev.cpu.0.freq: 149
dev.cpu.0.freq: 149
dev.cpu.0.freq: 149 <- received the directory
WTH? My CPU is idle when I'm sitting at my desk waiting impatiently for a directory listing? This doesn't seem right.
So I go disable powerd, verified CPU frequency is now 2128Mhz and streaming is flawless along with directory listing. I traversed about 15 directories(usually every other directory or so would freeze) and not one freeze or delay of more than 1 second, even directories with 100s of files.
So can we tweak powerd to make it more friendly? After all, this is unacceptable and since I'm bored and this is a learning opportunity why not see about saving those 9watts. Yes, I know if I replace a single incandescent light bulb with a CFL I can save roughly 60-90w, but this is about learning and not about the watts.
With regards to FreeNAS(as well as most other popular OSes) is that the "default" should work for most people. I tend to stick with Intel CPUs, very common hardware, and my hardware far exceeds the recommended specs for the OS. My philosophy is that if you have all 3 of those(of course, AMD CPUs that meet the recommended speeds are acceptable too) and you are being forced to tweak you probably have done something wrong or overlooked something. If you don't have something wrong then its likely a bug. Being that I'm fairly new to FreeBSD I try to avoid jumping to the conclusion that something is a bug when user error is far more. After all, if you aren't sure if what you are doing is right how are you to know if you are doing something wrong?
So I must have something wrong, right? Well, lets take a look at my BIOS settings since its a little hard to get the powerd settings wrong. After all, its a freaking checkbox!
Sure enough, my BIOS has "C3/C6/C7 State Support" and it is disabled. Oops. Let's enable it!
Lets stream a blu-ray and do a directory listing. Sure enough it doesn't freeze. So what happened to CPU freq this time?
dev.cpu.0.freq: 149
dev.cpu.0.freq: 149
dev.cpu.0.freq: 2128 <- opened a directory and received the listing
dev.cpu.0.freq: 1862
dev.cpu.0.freq: 1729
dev.cpu.0.freq: 1596
dev.cpu.0.freq: 1463
dev.cpu.0.freq: 1330
Here's some oddball stuff I'm adding since I did do a little more testing afterwards. I like to play and learn!
1. I disabled the "Virtualization Technology" in my BIOS and saw idle watts drop by 35w. So if you have a FreeNAS server, you should consider disabling this feature as it really adds zero benefit but costs you quite some watts. My CPU temp even went down by 2C!
So what have we learned from all of this:
1. Consider BIOS settings when you have performance issues that can't be identified by insufficient hardware and/or failing hardware.
2. User error with setting up the system.
3. Disable that pesky CPU virtualization feature. It's worthless for FreeNAS and seems to impact CPU power usage and temp.
4. powerd does work exactly as expected, but the results are minimal. If you are having performance issues you may want to try disabling it. You never know when your system is choking on low CPU speed. I've left powerd enabled now that I've fixed my BIOS setting.
5. powerd seems to take about 10-15 seconds to really ramp down CPU frequency all the way. If you are in a business environment I'd probably disable it completely. More than likely the CPU will never really go completely idle and the potential problems don't outweight the benefit. In a business environment you probably aren't too worried about saving 10w. For home use leaving it enabled is probably the better option if you care about saving some watts.