SuperMicro X10SL7-F

[Sir.Robin]

I upgraded to BIOS v1.10. When you enable powerd, do you see the xeon properly scaling down, or is it always at 3300/3100?

So, with powerd disabled:

Code:

NAS01] /# sysctl dev.cpu.0.freq
dev.cpu.0.freq: 3101

With powerd enabled:

Code:

@NAS01] /# sysctl dev.cpu.0.freq
dev.cpu.0.freq: 3100
[warlock@NAS01] /# sysctl dev.cpu.0.freq
dev.cpu.0.freq: 2300
[warlock@NAS01] /# sysctl dev.cpu.0.freq
dev.cpu.0.freq: 1900
[warlock@NAS01] /# sysctl dev.cpu.0.freq
dev.cpu.0.freq: 1800
[warlock@NAS01] /# sysctl dev.cpu.0.freq
dev.cpu.0.freq: 1600
[warlock@NAS01] /# sysctl dev.cpu.0.freq
dev.cpu.0.freq: 1500
[warlock@NAS01] /# sysctl dev.cpu.0.freq
dev.cpu.0.freq: 1100
[warlock@NAS01] /# sysctl dev.cpu.0.freq
dev.cpu.0.freq: 1000
[warlock@NAS01] /# sysctl dev.cpu.0.freq
dev.cpu.0.freq: 875
[warlock@NAS01] /# sysctl dev.cpu.0.freq
dev.cpu.0.freq: 800
[warlock@NAS01] /# sysctl dev.cpu.0.freq
dev.cpu.0.freq: 700
[warlock@NAS01] /# sysctl dev.cpu.0.freq
dev.cpu.0.freq: 500
[warlock@NAS01] /# sysctl dev.cpu.0.freq
dev.cpu.0.freq: 400
[warlock@NAS01] /# sysctl dev.cpu.0.freq
dev.cpu.0.freq: 300
[warlock@NAS01] /# sysctl dev.cpu.0.freq
dev.cpu.0.freq: 200
[warlock@NAS01] /# sysctl dev.cpu.0.freq
dev.cpu.0.freq: 100

I guess throttling works :)

 

[Sir.Robin]

So, i did replacement + installed fresh FreeNAS 9.1.1
- I have also C1, but i don't care about USB3 -> disabled in BIOS
- BIOS updated to R 1.10 (19.7.2013)
- LSI flashed into IT mode
- ZPOOL autoimported without issues, BUT i am missing serial number for one disk

da0 - da5 ... all of them are same WD green model, same size, in one RAIDZ2 pool, all of them connected into LSI controller. da0 is plugged into port0, da1 into port1, ... etc
ada0 ... just UFS disk, plugged into standard SATAIII port.

Any clue why i am missing one serial?

When running freenas on this board directly (bare metal/no hypervisor) I too am missing the da0 serial.
But when running ESXi 5.5 and passthrough, i get all serials

 

[74m]

So AHCI does work with internal ports and FreeNAS 9.1.1 on this mobo :)

Sir.Robin: Very nice. Thank you! :)

Just interest: For what do you virtualize the server?

 

[HolyK]

When running freenas on this board directly (bare metal/no hypervisor) I too am missing the da0 serial.
But when running ESXi 5.5 and passthrough, i get all serials

Well, the info is not in the db at all ...

Code:

sqlite> select * from storage_disk;
1|Disabled|60|||{devicename}da0|1|Disabled|Auto||bay4||1|da0
1|Disabled|60|WD-WMC1XXXXXXXX||{serial}WD-WMC1XXXXXXXX|1|Disabled|Auto||bay3||4|da1
1|Disabled|60|WD-WMC1XXXXXXXX||{serial}WD-WMC1XXXXXXXX|1|Disabled|Auto||bay2||5|da2
1|Disabled|60|WD-WMC1XXXXXXXX||{serial}WD-WMC1XXXXXXXX|1|Disabled|Auto||bay1||6|da3
1|Disabled|60|WD-WMC1XXXXXXXX||{serial}WD-WMC1XXXXXXXX|1|Disabled|Auto||bay6||7|da4
1|Disabled|60|WD-WMC1XXXXXXXX||{serial}WD-WMC1XXXXXXXX|1|Disabled|Auto||bay5||8|da5

 

[Okey Grak]

So, with powerd disabled:

Code:

NAS01] /# sysctl dev.cpu.0.freq
dev.cpu.0.freq: 3101

With powerd enabled:

Code:

@NAS01] /# sysctl dev.cpu.0.freq
dev.cpu.0.freq: 3100
[warlock@NAS01] /# sysctl dev.cpu.0.freq
dev.cpu.0.freq: 2300
[warlock@NAS01] /# sysctl dev.cpu.0.freq
dev.cpu.0.freq: 1900
[warlock@NAS01] /# sysctl dev.cpu.0.freq
dev.cpu.0.freq: 1800
[warlock@NAS01] /# sysctl dev.cpu.0.freq
dev.cpu.0.freq: 1600
[warlock@NAS01] /# sysctl dev.cpu.0.freq
dev.cpu.0.freq: 1500
[warlock@NAS01] /# sysctl dev.cpu.0.freq
dev.cpu.0.freq: 1100
[warlock@NAS01] /# sysctl dev.cpu.0.freq
dev.cpu.0.freq: 1000
[warlock@NAS01] /# sysctl dev.cpu.0.freq
dev.cpu.0.freq: 875
[warlock@NAS01] /# sysctl dev.cpu.0.freq
dev.cpu.0.freq: 800
[warlock@NAS01] /# sysctl dev.cpu.0.freq
dev.cpu.0.freq: 700
[warlock@NAS01] /# sysctl dev.cpu.0.freq
dev.cpu.0.freq: 500
[warlock@NAS01] /# sysctl dev.cpu.0.freq
dev.cpu.0.freq: 400
[warlock@NAS01] /# sysctl dev.cpu.0.freq
dev.cpu.0.freq: 300
[warlock@NAS01] /# sysctl dev.cpu.0.freq
dev.cpu.0.freq: 200
[warlock@NAS01] /# sysctl dev.cpu.0.freq
dev.cpu.0.freq: 100

I guess throttling works :)

Weird. Upon boot with powerd enabled, my throttling does not work. If I disable, then re-enable powerd, the throttling begins to happen. Do you mind seeing if you can reproduce the same behavior?Maybe it is a haswell issue.. If not I might re-install as I've been playing around with powerd to try fixing it, not knowing simple restart would do the trick.

 

[HolyK]

2 Okey Grak: I can confirm your observations. Intel i3-4130

Powerd was enabled before last reboot => throttling does not work

Code:

[root@HolyNAS] /mnt# sysctl dev.cpu.0.freq
dev.cpu.0.freq: 3400
[root@HolyNAS] /mnt# sysctl dev.cpu.0.freq
dev.cpu.0.freq: 3400
[root@HolyNAS] /mnt# sysctl dev.cpu.0.freq
dev.cpu.0.freq: 3400
[root@HolyNAS] /mnt# sysctl dev.cpu.0.freq
dev.cpu.0.freq: 3400

Disable/Enable powerd => will make throttling work

Code:

[root@HolyNAS] /mnt# sysctl dev.cpu.0.freq
dev.cpu.0.freq: 3400
[root@HolyNAS] /mnt# sysctl dev.cpu.0.freq
dev.cpu.0.freq: 2700
[root@HolyNAS] /mnt# sysctl dev.cpu.0.freq
dev.cpu.0.freq: 2200
[root@HolyNAS] /mnt# sysctl dev.cpu.0.freq
dev.cpu.0.freq: 1800
[root@HolyNAS] /mnt# sysctl dev.cpu.0.freq
dev.cpu.0.freq: 1300
[root@HolyNAS] /mnt# sysctl dev.cpu.0.freq
dev.cpu.0.freq: 875
[root@HolyNAS] /mnt# sysctl dev.cpu.0.freq
dev.cpu.0.freq: 400
[root@HolyNAS] /mnt# sysctl dev.cpu.0.freq
dev.cpu.0.freq: 200

 

[raidflex]

I can confirm the same thing with powerd. It will work if you reboot with it disabled and then enable it after the reboot.

 

[Sir.Robin]

Sir.Robin: Very nice. Thank you! :)

Just interest: For what do you virtualize the server?

No problem!

The plan were to be able to use my box for more than just nas. With vsphere i can do a bit lab/testing aswell :)




Sent from my iPhone using Tapatalk - now Free

 

[Okey Grak]

I'm also seeing really weird power behavior with this setup. I'm consuming at 40-45W at the plug, and it makes no difference if powerd is enabled or disabled. dev.cpu.0.freq showing 200mhz or 3300mhz makes no difference in power consumption. I wonder if the CPU is doing proper voltage reduction?

I have only one SSD hooked up to the box, and 32GB of ECC ram, and a platinum rated PSU. Seems quite high for idle.

 

[Okey Grak]

IPMI is reporting a VCORE of 1.83V with or without scaling. I'm wondering if there is a either a BIOS bug or FreeBSD issue not allowing these haswell chips to fully scale down.

 

[jgreco]

That's similar to our Sandy's. powerd isn't that helpful on modern gear, it was meant for older (early- to mid-2000's) gear that didn't idle efficiently but would get fairly efficient if you dropped clock speed.

 

[jonnn]

I'm also seeing really weird power behavior with this setup. I'm consuming at 40-45W at the plug, and it makes no difference if powerd is enabled or disabled. dev.cpu.0.freq showing 200mhz or 3300mhz makes no difference in power consumption. I wonder if the CPU is doing proper voltage reduction?

I have only one SSD hooked up to the box, and 32GB of ECC ram, and a platinum rated PSU. Seems quite high for idle.

Thanks for the info. That is disappointing.

I have been testing FreeNAS/ZFS on this temporary setup:

Cheap ass MSI H61 motherboard with 4x SATA ports
Pentium-Class IVY-Bridge CPU
2GB of random cheap ram
4x 7200 RPM Toshiba drives in RAIDZ2
Terrible 600W cheap ass PSU (probably rated 60+ Wood)

With all the drives spun down, I get about 31-32 W at idle. With no drives (similar to your setup) it's like 27-28W

You do have much more ram, but my experience shows ram doesn't draw much?

Try underclocking the CPU in the BIOS.

I set my max multiplier from 25x to 20x and saw a power drop at idle. Performance wasn't noticeably affected... still get like 550 MB/s on scrubs, 90+MB/s on network transfers and good rebuild speeds.

 

[jonnn]

Thanks for the info. That is disappointing.

I have been testing FreeNAS/ZFS on this temporary setup:

Cheap ass MSI H61 motherboard with 4x SATA ports
Pentium-Class IVY-Bridge CPU
2GB of random cheap ram
4x 7200 RPM Toshiba drives in RAIDZ2
Terrible 600W cheap ass PSU (probably rated 60+ Wood)

With all the drives spun down, I get about 31-32 W at idle. With no drives (similar to your setup) it's like 27-28W

You do have much more ram, but my experience shows ram doesn't draw much?

Try underclocking the CPU in the BIOS.

I set my max multiplier from 25x to 20x and saw a power drop at idle. Performance wasn't noticeably affected... still get like 550 MB/s on scrubs, 90+MB/s on network transfers and good rebuild speeds.

I did forget that this board has an LSI controller integrated. That could account for the difference.

 

[cyberjock]

RAM typically uses a whopping 1-2w per stick, tops. What I find hilarious is that the "low power" DDR3 sticks only save a fraction of a watt... per stick. In fact, when I experimented with them in the lab, I couldn't even actually prove solidly that wattage was lower because CPU usage is not actually constant. Even in DOS 6.22 the power usage fluctuated.

 

[jgreco]

About 8 watts for the 2308 IIRC...

 

[jyavenard]

So, with powerd disabled:

Code:

NAS01] /# sysctl dev.cpu.0.freq
dev.cpu.0.freq: 3101

With powerd enabled:

is powerd disabled by default in FreeNAS?

having said that; i've always been a bit suspicious of userland daemon managing what should be real time (or close to) feature...
IMHO there are better ways to control power usage than throttling the CPU frequencies. That was fine back in the days, but with features like C-state they are more efficient way to do things

not sure how much of FreeNAS let you tweak; but this page may be relevant:
https://wiki.freebsd.org/TuningPowerConsumption

 

[jgreco]

see comment 91

 

[jyavenard]

see comment 91

thanks... I only saw it after I replied and continued on reading other messages.

would be interesting to boot that box under linux and check power usage there. Linux has always been a bit ahead than FreeBSD when it comes to hardware support.

 

[Posttime]

Yeah maybe. But it restricts my options. And SM have been acting like a bunch of assholes about this so I'm disinclined to give them my business, or that of my company, for a bunch of their shiny new x10 boards.

According to SM:

FYI, this is how they suggest how to pick out a C2 stepping board:

The board serial number has year and month manufactured information after​

first two letters.​

​

For example, ZM 137S99999​

​

That means board has been manufactured on July 2013.​

​

If you get a board for September or later, then it should be using C2 chipset.​

​

To be 100% certain, you would have to check with CPU-Z or in BIOS after powering up board though.​

​

A little late to the party on this...however I just picked up a X10slh-f board two weeks ago and it was a ZM 138S9999999 board so I figured I need to RMA it. Contacted tech support and they were not able to tell me for sure if it is C1 or C2. Didn't have my CPU yet so I just RMA'd the board on crossship as I understood "ZM13S"... was the magic number. So the new board I get is...ZM138S9999999. Contact the RMA dept and my contact there HUY (great guy by the way) assures me that te new board is C2 and that it was bench confirmed prior to leaving. My Cpu turns up a couple days later, I test both boards, both are C2.:( $30 down the drain to return the board I thought would be C1 stepping. Not all ZM138 boards are C2 though, however anything being added to supplier stock now is so if your local shop was backordered you should be getting C2 board exclusively on 138 numbers....at least on the X10SLH-F.[/quote]

 

[Sir.Robin]

Yeh... i think you stress too much with the C1 vs C2 thang... it's a server... who gives a flyin' f*** bout sleepmode? :p