A good Ryzen server motherboard at last?

[Ericloewe]

There might finally be an interesting server motherboard for the Ryzen platform: The ASRock Rack X470D4U

It's pretty standard fare: 6 SATA ports from the chipset (an ASMedia part, IIRC) and two more from a separate ASMedia SATA controller. Two Intel I210 NICs; six fan headers; good PCIe connectivity in traditional slots plus two x4 connection via the M.2 connectors x2 and x4 via the first and second M.2 connectors, respectively; IPMI via an ASpeed 2500...

I haven't heard much about the Ryzen platform with FreeNAS 11.2 - if anyone has any experiences, please share.

 

[Jailer]

Looks interesting. With the recent price drops of the Ryzen 2K series this should make for a nice viable option for an AMD build.

Now, who's going to be the guinea pig?

 

[Ericloewe]

I might bite if the few people who have used Ryzen with FreeNAS chime in with their experiences.

 

[Constantin]

two x4 connection via the M.2 connectors

Careful, per AsRock it's only one NVME m.2 slot at x4, the other at x2 (great that they have two, not so great that one is slower than the other, making this ineligible for a mirrored fast SLOG). No onboard SFP+, HBA. Three PCIe slots (two @ x8, the other @ x4) that will likely get filled quick if this is supposed to be used as a file server. Perhaps use the last PCIe slot with a NVME x4 adapter and fill the first two with a HBA and a 10GBe NIC? I suppose the m.2 x2 slot might make a good L2ARC or boot disk location.

Between the cost of acquiring the CPU, HBA, NIC, and this board, I don't quite see the point, unless you already have a spare Ryzen processor and RAM knocking about. The main benefit appears to be a cheap motherboard that you can potentially drop a very fast, inexpensive CPU into. So if you need a powerful CPU this might be interesting and certainly a lower price point than something comparable from Intel. But what is the use case? Blu-Ray ripper? 10 channel Plex transcoder?

I am happy that Intel is getting some competition as that might reduce the tray prices for the mid-/high-tier server CPUs that Intel appears to be earning itself silly on right now. However, much of this board is basic and the limitations (NVME slot speeds, only two PCIe slots at x8, etc.) are a bit of head-scratcher. Was this a workstation Ryzen motherboard design that was updated w/a IPMI BMC? Aren't there better inexpensive motherboard contenders for server applications?

For example, one cost optimized baseline Supermicro Epyc 7000 board (H11SSL-i) is available for USD120 more than the Ryzen board mentioned above, can handle 1TB of RAM (vs. 256MB), features three PCI-E 3.0 x16, another three PCI-E 3.0 x8, a m.2 slot at x4, 16 SATA ports (of which 2 can be used as SATADOMs), etc. Plus, you get SuperMicro tech support vs. AsRock. :D

As usual, SuperMicro offers far more iterations of this board with various options built-in if you want them to be. But with six PCIe @ x8+ slots, who cares? Chris Moore's house heaters (aka 24+bay file servers) would likely be very happy using this board - simply add a HBA or two, perhaps a 10GB NIC, and still have 3 slots left over. For the long run, this seems like a much better server platform than the AsRock board above.

 

[Ericloewe]

Careful, per AsRock it's only one NVME m.2 slot at x4, the other at x2

Good catch. I saw the second one was x4 and didn't look too closely.

Three PCIe slots (two @ x8, the other @ x4) that will likely get filled quick if this is supposed to be used as a file server.

That's very typical of low-end Intel servers as well. The platform only supports 16x lanes from the CPU plus a bunch from the PCH multiplexed onto the equivalent of just x4.

I mean, even for a large NAS, what do you need? SAS3008, 10GbE NIC, maybe an SLOG and maybe an L2ARC? That all still fits neatly with room for a PCIe boot device in the x2 M.2 socket. This thing will go a long way.

 

[Arwen]

Keep in mind AM4 processors can only have 24 PCIe lanes. It's the Epyc that have 128 PCIe lanes. So an AM4 board that has 8+8+4+4+2 lanes is over-subscribed, and using the chipset's PCIe bus expander.

@Constantin, it does appear that you could use both M.2 slots for symetrical speed, mirrored SLOGs. One slot is PCIe 3.0 x 2 lanes and the other slot is PCIe 2.0 x 4 lanes. Thus, about the same speed.

That board is also a bit weak on exposed USB ports. It has 2 on the back panel, and 2 via header. That's it. So if you need 2 for mirroring USB flash drives AND one for your UPS, you have to make sure to bring out the header ports. (Or use a header to USB Type A adapter cable, and have one USB flash drive inside.)


I just bought myself a Ryzen 5 2400G, the fastest / most cores with graphics, for my new ASRock DeskMini A300. It rocks! (Don't pardon the pun :). Still have some quirks to work out. But I like it so far. It will be a desktop & run Linux, though I might see if it will dual boot TrueOS.

Edit: Corrected EPYC PCIe lane count from 64 to 128.

 

[Ericloewe]

the other slot is PCIe 2.0 x 4 lanes

I missed that, too. A bit lame...

 

[Epism]

I might be a guinea pig for this. With the new 3000 CPUs coming out I was looking at shutting down my FreeNAS VM and moving my 2700x into this board (with 32GB ECC RAM) for my new FreeNAS/PMS build. Would reuse my LSI2008 or utilize the onboard with 6 larger drives vs 8 smaller drives.

 

[Constantin]

@Constantin, it does appear that you could use both M.2 slots for symetrical speed, mirrored SLOGs. One slot is PCIe 3.0 x 2 lanes and the other slot is PCIe 2.0 x 4 lanes. Thus, about the same speed.

Oh snap. I should have looked more closely. Thank you!

I'd still opt for the SuperMicro board. The incremental cost is low and the board has far more flexibility down the road than the AsRock. For example, even a Optane P4801x module is PCIe 3.0 x 4, why buy a board which needs to sacrifice a PCIe slot just so the SLOG can do its best work? Optane modules that can take advantage of 3.0x4 speeds should only become more common, not less, with time.

That's why I opted for the X10SDV-7TP4F, it's not the fastest board but it features on-board SFP+, 16-Channel LSI 2116 HBA, 2 SATADOMs, two PCIe 3.0x8 slots when I need them, and a low-power processor. I likely could have saved some money going for the 2-Core version of the board but wanted the extra cores in my back pocket since this is a embedded series. Now, *if* I want to follow best practices, I should probably fill one of those PCie slots with a x8 riser card that bifurcates into two x4 buses and stick another P4801x in there. Maybe fill the other one with a 1TB L2ARC NVME?..

No doubt, there are use case for Ryzen processors. However, I'd opt for the Epyc series unless I had a spare Ryzen processor and RAM burning a hole in my pocket.

 

[Apollo]

I have been running my 1900x Threadripper for quite some time now. Very happy with it.
I miss IPMI though.

 

[arn0z]

I might bite if the few people who have used Ryzen with FreeNAS chime in with their experiences.

I've been using Ryzen 7 2700X for about 6 months and everything works fine. I recently noticed some wrong temp readings in FreeNAS UI and sysctl but that's about it. I also have overclocked RAM with absolutely no issues. I had problems with sticks working nicely together (had to up the voltage a bit). Ran memory tests for about a week and that too passed.

@Apollo Have you had wrong temp readings with your 1900x ?

 

[Apollo]

@Apollo Have you had wrong temp readings with your 1900x ?

Yes, it has always been very spiky type readings.

I made some screenshots in the following post:
https://www.ixsystems.com/community/threads/amd-threadripper-build.70194/

Before doing the dive with the 1900x, I had was able to test the 1950x for short while to evaluate performance under Windows, I was impressed. Then I decided placing an order for my own PC at Amazon when they screwed my order and decided they would cancel it rather than honoring it (I am still ranting about it) and no longer buy from them. When second Gen 2950x came out I purchased one to make it my personal PC from another distributor of course.
When I first put it together, I got high temperature especially at bootup where temperature (displayed on motherboard) would be going near or on the high side of 90 degrees C.
I wasn't sure this was correct and contacted AMD to find out, but they didn't seem very clear about it.
There are several ways temperature can be read from the processor, the package temperature or the various core temperatures.
I then conducted a series of test in order to evaluate effect of temperature and understand its behavior.

Threadripper has a very aggressive profile for power optimization and constantly adjust clock frequency and voltages based on CPU utilization requirements. As a result, you will see spikes of 10-15 degrees C when this happens, and it happens at regular rate.

If you disable dynamic precision boost and set frequency to be fix, like 2.4GHz, then you can achieve low temperatures close to room temperature but then again you waste on its performances.

I then tested the 2950x with running Freenas and I got similar temperature profiles it would seem.
When I got it working fine and performed a series of test, then I decided to buy the 1900x that is currently in my Freenas box right now. I wasn't entirely sure to expect the same support as it is first Gen but so far, except for a few early beta test issues, not specifically related to the 1900x, it has been very good.
Temperatures can be high as seen on the above post.

On my daily work on my PC running 2950x, I do often make use of all the cores and I have also ran benchmarkings tools to stress test the system and temperature can be very high in the upper 90 degrees C. When CPU gets really hot, you can see throddeling of frequency taking place. Because of it, I am not really worried. Overall, it does put less heat than my old i7-920 overclocked, but temperature transients are very significant.

I am also running ECC for the simple reason of data integrity.

Currently, Freenas doesn't report temperature per cores even though it appears to be the case.

 

[arn0z]

@Apollo Interesting. So my CPU temp is on average 95 deg. I made a post on it asking people with Ryzens to provide some readings to compare.

https://www.ixsystems.com/community...cpu-reading-is-100°-celcius.75665/post-526015

I had a look at yours and it seems like under load yours does indeed go really high at the 90s but at least goes back down to the 30s when idle. I’m seeing constant 95 deg readings. It’s absolutely wrong because on boot it’s impossible to go from 30 to 100 in a matter of seconds with no load.

Another difference between your readings and mine is that they’re made from the legacy UI. I haven’t done that and will try and see if the readings differ in the different UIs. I have yet to open a ticket since I have yet to collect some data to submit.

Other than that I’ve had no issues with Ryzen like I said and it’s been awesome. I love supermicro and I think they make great mobos but I’ll never shell out hundreds for Intels Ryzen equivalents. Ryzen fills the need for enthusiast server builds and that’s exactly what I want. Intel is too enterprisey. But that’s another discussion. That’s for you reply! I’ll get back to you after I’ve done some tests with legacy UI temp readings.

 

[Apollo]

@Apollo Interesting. So my CPU temp is on average 95 deg. I made a post on it asking people with Ryzens to provide some readings to compare.

https://www.ixsystems.com/community/threads/ryzen-7-2700x-cpu-reading-is-100°-celcius.75665/post-526015

I had a look at yours and it seems like under load yours does indeed go really high at the 90s but at least goes back down to the 30s when idle. I’m seeing constant 95 deg readings. It’s absolutely wrong because on boot it’s impossible to go from 30 to 100 in a matter of seconds with no load.

Another difference between your readings and mine is that they’re made from the legacy UI. I haven’t done that and will try and see if the readings differ in the different UIs. I have yet to open a ticket since I have yet to collect some data to submit.

Other than that I’ve had no issues with Ryzen like I said and it’s been awesome. I love supermicro and I think they make great mobos but I’ll never shell out hundreds for Intels Ryzen equivalents. Ryzen fills the need for enthusiast server builds and that’s exactly what I want. Intel is too enterprisey. But that’s another discussion. That’s for you reply! I’ll get back to you after I’ve done some tests with legacy UI temp readings.

At boot, my 1900x is pooling above 215W and I suspect BSD is doing some intensive work or simply override CPU thermal management.
When your CPU is borderline 100 degrees C, what is your CPU doing?
You said you topped off voltage to make RAM stable, are you overclocking manually? If so that maybe the problem.

 

[Apollo]

My temps on the 1900x are:

dev.amdtemp.1.core0.sensor0: 61.7C
dev.amdtemp.1.sensor_offset: 0
dev.amdtemp.1.%domain: 1
dev.amdtemp.1.%parent: hostb23
dev.amdtemp.1.%pnpinfo:
dev.amdtemp.1.%location:
dev.amdtemp.1.%driver: amdtemp
dev.amdtemp.1.%desc: AMD CPU On-Die Thermal Sensors
dev.amdtemp.0.core0.sensor0: 67.7C
dev.amdtemp.0.sensor_offset: 0
dev.amdtemp.0.%domain: 0
dev.amdtemp.0.%parent: hostb0
dev.amdtemp.0.%pnpinfo:
dev.amdtemp.0.%location:
dev.amdtemp.0.%driver: amdtemp
dev.amdtemp.0.%desc: AMD CPU On-Die Thermal Sensors
dev.amdtemp.%parent:
dev.cpu.15.temperature: 67.7C
dev.cpu.14.temperature: 67.7C
dev.cpu.13.temperature: 67.7C
dev.cpu.12.temperature: 67.7C
dev.cpu.11.temperature: 67.7C
dev.cpu.10.temperature: 67.7C
dev.cpu.9.temperature: 67.7C
dev.cpu.8.temperature: 67.7C
dev.cpu.7.temperature: 67.7C
dev.cpu.6.temperature: 67.7C
dev.cpu.5.temperature: 67.7C
dev.cpu.4.temperature: 67.7C
dev.cpu.3.temperature: 67.7C
dev.cpu.2.temperature: 67.7C
dev.cpu.1.temperature: 67.7C
dev.cpu.0.temperature: 67.7C
[

You can see temperature per CPU are identical which is wrong.

 

[arn0z]

When your CPU is borderline 100 degrees C, what is your CPU doing?

Nothing. Someone is watching Plex on my server (no transcoding, direct play).

Here is my temps:

Code:

device    amdtemp
device    coretemp
kern.module_path: amdtemp
net.inet6.ip6.use_tempaddr: 0
net.inet6.ip6.temppltime: 86400
net.inet6.ip6.tempvltime: 604800
net.inet6.ip6.prefer_tempaddr: 0
hw.sfxge.restart_attempts: 3
hw.usb.template: 0
dev.amdtemp.0.core0.sensor0: 105.1C
dev.amdtemp.0.sensor_offset: 0
dev.amdtemp.0.%parent: hostb0
dev.amdtemp.0.%pnpinfo:
dev.amdtemp.0.%location:
dev.amdtemp.0.%driver: amdtemp
dev.amdtemp.0.%desc: AMD CPU On-Die Thermal Sensors
dev.amdtemp.%parent:
dev.cpu.15.temperature: 105.1C
dev.cpu.14.temperature: 105.1C
dev.cpu.13.temperature: 105.1C
dev.cpu.12.temperature: 105.1C
dev.cpu.11.temperature: 105.1C
dev.cpu.10.temperature: 105.1C
dev.cpu.9.temperature: 105.1C
dev.cpu.8.temperature: 105.1C
dev.cpu.7.temperature: 105.1C
dev.cpu.6.temperature: 105.1C
dev.cpu.5.temperature: 105.1C
dev.cpu.4.temperature: 105.1C
dev.cpu.3.temperature: 105.1C
dev.cpu.2.temperature: 105.1C
dev.cpu.1.temperature: 105.1C
dev.cpu.0.temperature: 105.1C

 

[arn0z]

You said you topped off voltage to make RAM stable, are you overclocking manually? If so that maybe the problem.

I did add more voltage to SoC which AMD recommends. Its sitting at 1.1V. RAM on the other hand is more heavily overclocked. I am using factory timings to get the 3600MHz and to increase stability I have increased a bunch of RAM voltages manually. I could list them here but it would be of no use I think. Overclocking RAM would only make RAM heat up more I don't think it would affect CPU temp drastically like that. Correct me if I am wrong.

 

[Apollo]

I did add more voltage to SoC which AMD recommends. Its sitting at 1.1V. RAM on the other hand is more heavily overclocked. I am using factory timings to get the 3600MHz and to increase stability I have increased a bunch of RAM voltages manually. I could list them here but it would be of no use I think. Overclocking RAM would only make RAM heat up more I don't think it would affect CPU temp drastically like that. Correct me if I am wrong.

Is 3600MHz the CPU clock or the RAM? If that is the RAM, I would not bother overclocking it.
As you can see, and how I have pointed out, all the cores show the same temperature. This makes impossible seeing the true picture as all the other cores maybe much cooler.
You definitely do not use much CPU. Is your system fairly warm or hot? How fast is the fan blowing?
There is a test you could conduct and that is to run Plex by forcing transcoding. This will force most if not all the CPU to go full throttle.
If you have some 4K video like the ones I have used to stress my system, that would be great.
You just need to get a base line about your system performance and whether or not forcing transcode will actually affect temperature and if so, how.
At first, just run it for like 20 seconds and see how it behaves. For forcing transcoding, the best way is to run it trough the web browser and adjust resolution to something much smaller than the original video.

 

[arn0z]

Is 3600MHz the CPU clock or the RAM? If that is the RAM, I would not bother overclocking it.
As you can see, and how I have pointed out, all the cores show the same temperature. This makes impossible seeing the true picture as all the other cores maybe much cooler.
You definitely do not use much CPU. Is your system fairly warm or hot? How fast is the fan blowing?
There is a test you could conduct and that is to run Plex by forcing transcoding. This will force most if not all the CPU to go full throttle.
If you have some 4K video like the ones I have used to stress my system, that would be great.
You just need to get a base line about your system performance and whether or not forcing transcode will actually affect temperature and if so, how.
At first, just run it for like 20 seconds and see how it behaves. For forcing transcoding, the best way is to run it trough the web browser and adjust resolution to something much smaller than the original video.

3600MHz is the RAM clock. And it is overclocked at 3600MHz from who knows what the minimum is. These sticks are supposed to be overclocked. Anyways, my system is fairly cool. Fan is blowing at low-medium range.

I got some images today after I found a good time to restart the server.

Well.... There's definitely something wrong with the middleware. Look at that -273 deg celcius temp... I mean that's just wrong. My server is not the coldest spot in the universe lol. I'm not even gonna run the plex tests you have suggested (I initially wanted to do that but first wanted to restart to get a nice clean slate and then saw the 0 kelvin anomaly). This to me is a blatant FreeNAS bug. I'm going to report it. If you have any other suggestions, let me know. Thanks for everything.

Other than that, everything runs fine. I have tried transcoding from 4K to 1080p and its easy and fast. That's one transcode however. I have not tried many.

 

[Junglist724]

Looks interesting. With the recent price drops of the Ryzen 2K series this should make for a nice viable option for an AMD build.

Now, who's going to be the guinea pig?

I retired my 2700x + Crosshair VII Hero from use in my main desktop to being used as a proxmox node for a while. Just today I decided to install FreeNAS bare metal to see if it's easier to manage for shared storage. I didn't get any of the idle state soft lockups that I've seen other people complain about, but for some reason my 2950x proxmox node did lockup constantly until I disabled C6 with kernel boot parameters. Makes me think it depends on the motherboard(using the x399 MEG Creation which doesn't give me much control over c-states or psu idle current).

Hardware-wise I think this config is a pretty good platform for FreeNAS:

• 2700X + Crosshair VII Hero
• 2 x 16GB 2666 ECC UDIMMS
• 1TB Mushkin Reactor boot drive
• 4 x 8TB Seagate Exos 7E8 in mirrored vdevs
• 256GB PM981 for L2ARC
• Intel X550-T2 NIC
• Spare M.2 22110 for an Optane 905p when I can afford it
• Spare x8 slot for an HBA card since x470 only has 6 SATA

Once I get that Optane and HBA I'll be using all available cpu pci-e lanes. Of course I'll have to use this for a while to see if it's stable.