ASRock Rack E3C246D4M-4L motherboard review/fixes

[bcat]

[Note to mods: This post isn't really specific to SCALE, since it's about the hardware itself. But I can't seem to post in the general hardware forum, only the Core and SCALE ones. Please feel free to move this thread as needed.]

Edit 2023-02-04: Added a section about the preinstalled backplate, which can cause some cooler installation annoyance.

I recently built a TrueNAS box using this server mobo, and although the build worked out well in the end (see my signature for full specs), there were a few things I wasn't sure about going into it, as well as some weird issues with MAC addresses that took some time to resolve. I figured I'd post a short summary here so the info is online and searchable in case someone else wants to use this board with TrueNAS (or, I suppose, more generally).

Good: Using IPMI and iGPU simultaneously

Since I run Plex on this machine, I have a Xeon E-2276G installed, and I wanted to ensure I could use the iGPU for transcoding at the same time as using the IPMI KVM. I read about some server boards unconditionally disabling the iGPU when any add-in video card was present, including the BMC's built-in video support. I reached out to ASRock Rack support, who confirmed that with the latest beta BIOS (version 2.41, at the time of posting), both could be used.

Indeed, this works fine. To keep the iGPU from being disabled, you simply have to enable the "IGPU Multi-Monitor" option in the BIOS:

Show : IGPU Multi-Monitor BIOS settings

Good: Using PCIe bifurcation on the x16 slot

I didn't expect any issues here, but just want to explicitly document that PCIe bifurcation works fine on the x16 slot (PCIE6). It allows up to x8/x4/x4 (finest-grained bifurcation supported by the C246 chipset), and I've successfully used three NVMe SSDs on a bifurcation riser card.

Note that if you want to use the mezzanine slot (MEZZ_1), it appears to automatically bifurcate the x16 into x8 (PCIE6) and x8 (MEZZ_1). I'm not sure if it's possible to further bifurcate PCIE6 into x4/x4, as I don't have any of ASRock Rack's proprietary mezzanine cards to test with.

Show : PCIe bifurcation BIOS settings

Confusing: BMC port bonding settings missing by default

By default, the dedicated IPMI interface and the NC-SI interface (shared with LAN2, the first of the i210 LAN ports) are bonded together. For my own sanity, I wanted them to be separate. This is supposed to be configurable in the BMC's network settings, but the Settings > Network Settings > Network Bond Configuration page was entirely missing from the UI by default.

Instead, there are ipmitool commands to enable and disable bonding. (Source)

Disable BMC port bonding:

Code:

$ sudo ipmitool raw 0x32 0x71 0x01 0x00 0x00 0x01 0x64 0x00 0x03 0x01

Enable BMC port bonding:

Code:

$ sudo ipmitool raw 0x32 0x71 0x06 0x00 0x03
$ sudo ipmitool raw 0x32 0x71 0x01 0x01 0x00 0x01 0x64 0x00 0x03 0x01

I can't call this "bad" because ASRock Rack documented the necessary IPMI commands, but it was confusing to a n00b like me. Interestingly, after changing the bonding config via ipmitool, the Network Bond Configuration page in the Web interface magically shows up, so changes after the first can be made directly from the browser.

Confusing: Preinstalled backplate conflicts with some coolers

Although the socket is normal, the motherboard came with a preinstalled backplate stock on with a fairly strong adhesive (similar to other ASRock Rack motherboards of the same era). This potentially conflicts with coolers that don't use similar mounting hardware (e.g., Intel stock coolers that expect their pushpins to go through the mobo). Although the backplate can be (carefully) removed using a heat gun (or hair dryer, in a pinch... I've done that to get stubborn adhesives to let go in the past), I fortunately had a cooler (Deepcool AK400) that mounted with M3 screws of appropriate length that I could use the preinstalled backplate and just ignore the one that came with the cooler.

Bad: BMC MAC addresses duplicated

After disabling IPMI bonding, I noticed a more fundamental problem with the IPMI network setup. The dedicated IPMI interface and the NC-SI interface had the same MAC address. This seemed unexpected to me, as it didn't match another ASRock Rack mobo I've used (which had separate MACs for each IPMI interface). Additionally, the system boot was delayed for over a minute on a "Waiting for BMC MAC Initializing" message (no screenshot, sorry) before reaching the BIOS splash screen.

I reported this to ASRock Rack and they confirmed the two interfaces were supposed to have separate MAC addresses. However, my board was produced back in April 2019 (even though I bought it new from Newegg in October 2022... I guess it was sitting on a shelf for a while), and apparently that batch didn't have NC-SI MAC addresses programmed. Oops.

To their credit, ASRock Rack support was super helpful, and they gave me a new MAC with their vendor prefix to use. (In retrospect, had I known from the start that the interface never had a MAC to begin with, I could've just picked up a locally assigned one, but I didn't know that when I opened the ticket.)

But that left the problem of actually programming the new MAC. ASRock Rack provided me a utility (BMCMAC) to do the programming, but it didn't recognize my BMC. However, I found online that someone had reverse engineered a different ASRock Rack tool to obtain ipmitool commands for reading and writing BMC MAC addresses on the X470D4U, a board contemporary to mine. I tried them out (read first, of course, then write), and they worked fine. Additionally, the minute+ BMC delay on boot went away after making the two MAC addresses distinct.

Here are the commands; YMMV if you want to use them. All I can say is they didn't brick my particular BMC. (Source)

Get BMC eth0 (dedicated) MAC:

Code:

$ sudo ipmitool raw 0x3a 0xa1 0x00

Get BMC eth1 (NC-SI) MAC:

Code:

$ sudo ipmitool raw 0x3a 0xa1 0x01

Set BMC eth0 (dedicated) MAC:

Code:

$ sudo ipmitool raw 0x3a 0xa0 0x00 0xXX 0xXX 0xXX 0xXX 0xXX 0xXX

Set BMC eth1 (NC-SI) MAC:

Code:

$ sudo ipmitool raw 0x3a 0xa0 0x01 0xXX 0xXX 0xXX 0xXX 0xXX 0xXX

Bad: i219 MAC address unset

I'm pretty sure MAC addresses on this mobo are just cursed. Not counting the dedicated IPMI port, the board has four LAN ports. LAN1 has an i219 NIC, and LAN2/3/4 have i210 NICs. There's a sticker on the LAN1 port with a certain MAC address printed (XX:XX:XX:XX:XX:8B), and it seemed pretty clear that the LAN (not BMC) MAC addresses were intended to be sequentially assigned from that. But here are the MAC addresses I actually had:

• LAN1 (i219): 88:88:88:88:87:88 (not XX:XX:XX:XX:XX:8B)
• LAN2 (i210): XX:XX:XX:XX:XX:8C
• LAN3 (i210): XX:XX:XX:XX:XX:8D
• LAN4 (i210): XX:XX:XX:XX:XX:8E

So what's this 88:88:88:88:87:88 nonsense? Turns out it's—you guessed it—the default MAC address for the i219 NIC if one isn't configured in the NIC's EEPROM. (Seems to be a recurring theme here.)

In this case, I knew what the LAN1 MAC address should be, but didn't have the utility to program it. The relevant Intel EEUPDATE utility isn't broadly available, as it's provided primarily for OEMs. I reached out to ASRock Rack about this as well, but I think my question got lost amongst the BMC MAC confusion. (And, to be clear, their support was incredibly patient and helpful, so I've got no complaint there.) I managed to find a copy of EEUPDATE version 5.30 online (originally provided by MSI after they had the same issue on some of their motherboards), and that point, programming the correct MAC was easy. (The tool includes good documentation and runs fine under FreeDOS.)

I'm hesitant to link it here because I found it second-hand from a forum, and, in retrospect, running binaries from untrusted sources that directly manipulate hardware is probably not the most secure of things. But if you have the same i219 MAC address issue and want to get it fixed, I figured this would be enough context to get started. (And maybe you can get a known-good copy of EEUPDATE directly from ASRock.)

Conclusion

Overall, as someone relatively new to server hardware, I'm happy with this motherboard. It does everything I need it to, it's not too old, and it wasn't too expensive or hard to find (vs., say, the Gigabyte MW34-SP0 I really wanted that seems to be unobtainium in the US). But there seems to be some "early batch weirdness" around improperly programmed MAC addresses. Maybe more recently produced batches have solved it, but if not, I wanted to document the fixes since it took a fair bit of digging (about a calendar month of back-and-forth with support, plus various bits of digging online) to get the MAC issues resolved.