FreeNAS + ESXI Lab Build Log

[Chris Moore]

Does anyone know the amperage or total wattage that the X9 series motherboards can provide via their fan headers? I would only need one of these fan headers to run 2 fans and I could make them two redundant(ish) fans.

If I recall correctly, it is 1 amp. The high speed fans that servers often use can draw a lot of power.

Has anyone done a comparison on fan speed and noise when using either software to control or hardware (undervolting)? What are your experiences? I wanted to try the software first, as I could control it remotely with IPMI and without the need to add additional components to the build, but if undervolting with resistant wires works better, I have he parts for that too.

I found the resisters easier to deal with because the fan control through IPMI was a bit difficult to manage. When the fans run fast enough to keep the system cool, they just keep it cooler when not under load.

 

[Chris Moore]

• First, and most importantly, the wife test. After my wife was settled in and my child asleep I had turned on the server and had her give me some more feedback. The great news is that the loud "whine" she heard last time was the alarm from having both PSUs in without both being plugged in. As I won't make that mistake again, this saved me $140 for new PSUs (at least for now). She also stated that the noise level was acceptable, which is also good because that was without any modification and should be the loudest they should be.

• I took a few videos, but the files seem to be too large and the noise cancelling / lack of scientific approach / lack of measurement device makes them less valuable. If anyone would still like to hear/see the footage please just ask.

Sure, I would love to see the video.

 

[Maelos]

If I recall correctly, it is 1 amp. The high speed fans that servers often use can draw a lot of power.

I found the resisters easier to deal with because the fan control through IPMI was a bit difficult to manage. When the fans run fast enough to keep the system cool, they just keep it cooler when not under load.

I know the tribal knowledge is 1 amp, but I have never seen it actually written out. From the fan listing by SuperMicro I can say that they are at least 0.6 amp which should be enough for two Noctua fans. Right now I have 3/4 Noctua fan's hooked up with 3 pin fan extenders. I may purchase 4 pin fan extensions and 4 pin fan splitter (I only have a bunch of 3 pin ones, I think). The chassis scared me a bit this am. I tried to rush to get it into ESXI/remote service by attaching the internal drive bay and SSD, and when in blind boot (no monitor hooked up - rushed) the fans would rev up and down and I never was able to ping the old ESXI IP. I am guessing the boot order between the SSD and the SATA-DOM is off, or some other issue. I am going to read up on the BIOS and should have it racked and working by tomorrow.

I uploaded the videos to my youtube account in a playlist here: https://www.youtube.com/playlist?list=PLOoJL8D_5qVfmAOYnf1Spzj_lCrJh6YjF for viewing. Please accept my apologies for not doing it a bit more scientifically and with better equipment. This may still be something I do prior to adding the undervolting wires. Further sound tests will include the active CPU cooling as I don't plan to undo that step unless needed (reapplying thermal paste).

 

[Chris Moore]

I didn't find anything in the manual about it, but I was skimming over it.
https://www.supermicro.com/manuals/motherboard/C606_602/MNL-1259.pdf

I know for sure that they will handle 1 amp because the 5000 RPM Delta fans that come standard in some of the chassis are rated for 1 amp. I have seen 2.5 amp in some documentation, but that may be board specific. I would not hesitate to put 1 amp load on there. The "FanA" header on my main server is running 3 fans.

 

[Bidule0hm]

The ATX spec for a fan is 2 A max for 1 sec (startup) and 1 A max per fan header. So the MB must be able to provide at least 1 A indefinitely and at least 2 A for at least 1 sec.

http://www.formfactors.org/developer\specs\rev1_2_public.pdf

 

[Maelos]

The ATX spec for a fan is 2 A max for 1 sec (startup) and 1 A max per fan header. So the MB must be able to provide at least 1 A indefinitely and at least 2 A for at least 1 sec.

http://www.formfactors.org/developer\specs\rev1_2_public.pdf

That PDF is saved and the link will be used in my guide. Thank you for that.

 

[Bidule0hm]

You're welcome :)

 

[Maelos]

So I got ESXI 6.5 running on it. It was indeed the wrong boot order - it had the SATA DOM first. I ordered a PWM splitter and a set of 5 PWM extensions from Newegg for cheap then later found out that the Noctua NH-U9DXi4 came with splitters - doh! I looked through the BIOS and made sure the settings all looked good, but I oddly could not find anything about the fans or other "thermal management" as mentioned in the Mobo manual. Below are a few pictures. I know there may be some updating to do of the BIOS.

- Penguin Computing Icebreaker?

-- no Thermal Management option here

I went ahead and used the 4 Noctua 4-pin noise reducers and one 3-pin noise reducer that I bought from Chris's suggestion. Is there any difference between the 3 and 4pin noise reduction/undervolting/resistor wires, especially if I can't even find a way to control the PWNness? That question barely makes sense to me. Why even have a PWM undervolting/resistor wire unless you can somehow turn the undervolting off when PWM is accessible? Is it only controllable by the awesomely retro SuperDoctor application?

Also, what gives with "Penguin Computing, Icrebreaker ###"? Is this board a fraud or just rebranded?

The good thing is, despite all of this, and the rails taking yet another day. I am now able to test the energy usage of the machine while being able to start getting the VMs and FreeNAS going.

The beast is alive!

 

[Chris Moore]

Did you login to the IPMI console? It saves you from needing a local keyboard, mouse and monitor.

 

[Maelos]

I am going to try that today from work when I have the time. I am also going to switch the chassis fans over to the 3 pin resistor wires as Noctua claims their 4 pin resistors only work with their fans. I doubt this is true, but I will give it a shot and see the difference.

 

[Bidule0hm]

but I oddly could not find anything about the fans or other "thermal management" as mentioned in the Mobo manual.

The FAN settings are only accessible in the IPMI interface AFAIK.

 

[Chris Moore]

IPMI is really a local network thing. You would not want to expose it to the Internet. Many would even say that it should be on a separate network so you can control access to it.

Sent from my SAMSUNG-SGH-I537 using Tapatalk

 

[svtkobra7]

Also, what gives with "Penguin Computing, Icrebreaker ###"? Is this board a fraud or just rebranded?

https://www.penguincomputing.com/products/storage/

 

[Chris Moore]

what gives with "Penguin Computing, Icrebreaker ###"?

https://www.penguincomputing.com/products/storage/

So it is the name of the company plus the model name/number, from the retailer that originally sold the unit.
That is a handy thing.

You could probably edit or delete that with the IPMI management commands.
If you wanted to.

 

[Maelos]

The FAN settings are only accessible in the IPMI interface AFAIK.

Will be playing with that today hopefully. Good to know.

IPMI is really a local network thing. You would not want to expose it to the Internet. Many would even say that it should be on a separate network so you can control access to it.

This will be the practice after I learn a bit more about it. I am keeping everything a tad bit easier for maintenance/learning at the moment, but once I have the settings in a good place I will isolate things a bit better. I do not have the IPMI address open to the outside, but when I get on a VM I hope I will be able to reach the internal address I gave it.

https://www.penguincomputing.com/products/storage/

I looked at their website too, but I was sold a SuperMicro board. I just was caught a little off guard by the different make and model. I wonder if they are a company that takes old equipment, fixes it up, and sells it under a different name. I hope I am not losing out on the quality of the SM product line. That is the entire reason I went for the combo deal. I don't mind the change in name as long as the quality underneath is the same. Ahh, just read up on what "whitebox"ing is. That may be the case. No biggie if so, just unexpected.

 

[Chris Moore]

I looked at their website too, but I was sold a SuperMicro board. I just was caught a little off guard by the different make and model.

They (Penguin Computing) bought the Supermicro system when it was new, customized it and sold it as their own product. Kind of like a company that buys a Ford truck body and puts a camper on it, then sells it as a motor home.

 

[svtkobra7]

They (Penguin Computing) bought the Supermicro system when it was new, customized it and sold it as their own product.

I looked at their website too, but I was sold a SuperMicro board. I just was caught a little off guard by the different make and model.

I'm not a morning person ... I was hoping the dots would connect themselves as the caffeine hadn't quite kicked in yet :)
http://www.supermicro.com/wheretobuy/namerica.cfm?rgn=101 (cell C3)

Kind of like a company that buys a Ford truck body and puts a camper on it, then sells it as a motor home.

Its Supermicro ... can't you do a little better than Ford in your analogy? :) [Not that there is anything wrong with Fords]

 

[Maelos]

Looks like the IPMI does not work while using only one Ethernet interface. This is not a shocker, but wanted to try it anyway (using a VM to connect to the hardware below). I am going to try to install the SuperDoctor software on a Window's VM, but am saddened that it only supports SuSe and RedHat Enterprise Linux Distros. I assume it would work on Fedora/CentOS, but I had hoped to use Debian for my server's(') distro. I wanted to wait before using one of the Cisco switches as I only have one Ethernet cable coming into the lab. I only have one H110 for remote outlet control, so the only way I could control both the switch and the server with it is putting it between the surge protector (could do) or UPS (not exactly how its meant to be used).

Good testing so far, much to come. I won't update this post too often until I have something good or issues arise. I am working my way up the layers from hardware, BIOS, IPMI, ESXI (rights, VM setup), then finally to FreeNAS for home storage/backup/media serving use. The rest of the lab and VMs are not as relevant to this community, so I will only keep to the highlights on those.

 

[Ericloewe]

Looks like the IPMI does not work while using only one Ethernet interface.

Sure it does, assuming you're using the correct port.

 

[svtkobra7]

Sure it does, assuming you're using the correct port.

Correctamundo (with one caveat, correct port and correct IP address for the IPMI interface). Edit: OK, technically 3 caveats, the JPB1 jumper needs to be set correctly (2-35 in your manual).



You have three options for which port IPMI uses and you must connect your interfaces accordingly. I thought you had the same motherboard as I so I annotated a picture of the rear of my server, but then I realized yours is a X9DRi-F so I dropped in the diagram from that board's quick reference guide. The design is similar, on both, it is the network interface above the USB ports.

• Dedicated: Always use dedicated IPMI interface.
  
• Shared: Always use LAN1 interface.
  
• Failover: If dectected on IPMI, then use IPMI, otherwise fall back to LAN1.

[ I don't think the manual goes into much detail on this but didn't check ]

So to Eric's point, if you haven't changed the BIOS setting, and you have an Ethernet cable connected to the IPMI interface = it works. Double check that. Also, set it to a static IP on your network.

I couldn't live without IPMI (well, I could probably live, but not happily). To @Chris Moore 's prior comment, I agree it should be on a separate network once you get everything configured and to that end, I'd recommend using the Dedicated setting mentioned above. Some go as far as to disconnect it when not in use; however, I think that may be a little too cautious / overkill. And make sure you change the default admin/admin username/password.

Looks like the IPMI does not work while using only one Ethernet interface.

How do you have it connected and what do your IPMI BIOS settings look like?

Edit: Here is the link to SuperMicro's IPMIView (not sure if you found it or were guided to it yet): ftp://ftp.supermicro.com/utility/IPMIView/