AMD EPYC 3251 SuperMicro Server

Kevern

Cadet
Joined
Apr 9, 2017
Messages
7
So, in case any of you are wondering, I built a second FreeNAS box and wanted to try out the EPYC. I mean, I get to say EPYC and it's got 8 cores / 16 threads.

So I bought one here at SuperMicro, through a reseller. It's small enough to fit in a 16" deep network rack.
https://www.supermicro.com/Aplus/system/Embedded/AS-5019D-FTN4.cfm

I put four 5TB Seagate laptop spinning drives into it and an M2. I also bought an internal USB 2.0 header adapter to put inside for the USB drives, and 64GB ECC RDIMMs.

It's all set up. A couple of things. Transfer speed is what you'd expect with ZFS1, it's fine responsive with 105MB/s transfers over standard gigabit Ethernet.

It's fast enough, but initially when I installed Plex into a jail on 11.2, it was awful. My Xeon processor system (4/8, 32GB, 8x4TB-WD Reds, 9211 SAS Adapter) would yawn at transcoding 1080p down to a phone, from a big MKV through Plex on an Ubuntu VM. This thing would crap the bed on trying to fast forward a movie over the web browser. So, I moved the VM out of the Plex jail and into an Ubuntu VM, like my other server, but with 6 cores, 8GB RAM. Works fine.

It's mostly loafing now. What else can I throw at it?

This new server has been EATING USB drives, but I think it has to do with the internal USB hub thing I got. It had power from molex, but internally that's unecessary unless I'm running LED lighting. And I'm not. So I pulled the power, but still left the USB hub. However, I've torched 4 or 5 32GB USB drives. Some were Sandisk Cruzer Fits and now am trying Verbatim Nano USB drives. I've thumped one or two of the latter already too. So now I've taken the top off and have my KVM USB hooked internally to that (on a shelf below my rack) and the USB sticks only in the two exterior USB slots. I installed to one, removed the install USB, then mirrored a second one... so far so good.

So, points to take. The EPYC has run everything pretty well so far. VM's have all loaded up (Windows 10, Ubuntu 19.04) so far... Something to add to your list, if you want.
 

Chris Moore

Hall of Famer
Joined
May 2, 2015
Messages
10,080
What are you using these USB drives for?
So I bought one here at SuperMicro, through a reseller. It's small enough to fit in a 16" deep network rack.
Personally, I think this chassis is too small to be useful. Not enough room for drives, but it might be good for some things.
 

Junglist724

Cadet
Joined
May 6, 2019
Messages
5
I've been thinking about getting one to fill my unused Caselabs S3+pedestal. The case has room for 12 x 3.5" drives worth of 5.25" to 3.5" hotswap bays and possibly room for a x8/x8 bifurcation riser so I can add 10Gbe and an HBA card. Still wish these Supermicro boards made use of the 4 x 10Gbe already built into each Zen die.
 

Kevern

Cadet
Joined
Apr 9, 2017
Messages
7
What are you using these USB drives for?
Boot drives.
Chris Moore said:
Personally, I think this chassis is too small to be useful. Not enough room for drives, but it might be good for some things.
I have 13TB of drive space; mostly just media stuff. There isn't anything earth shattering on it except family pictures for 17 years or so... and I have those also saved elsewhere. It's tiny, has a minor power footprint, allows me to goof off with some VM's...
 

Yorick

Wizard
Joined
Nov 4, 2018
Messages
1,912
Intel 320 40gb as boot drive? Dirt cheap on eBay and will outlast your USB by a factor of - ludicrous.
 

Kevern

Cadet
Joined
Apr 9, 2017
Messages
7
Yorick said:
Intel 320 40gb as boot drive? Dirt cheap on eBay and will outlast your USB by a factor of - ludicrous.
I would love another SATA port for that. I have a spare 32GB SSD (KingDian, I think) that I'd mix in. I like ludicrous mode though.

One of the bigger points of the overall first post was people inquiring about EPYC Chips and FreeNAS. It's up, it's running. I would like to know why Intel is "strongly recommended" on the hardware page. Are they recommending Atom processors? They're Intel? Perhaps a Celeron is recommended, because Intel, over an EPYC 7000 class processor? What is it about Intel?
 

Arwen

MVP
Joined
May 17, 2014
Messages
3,600
@Kevern, in the past, Intel had a bit of monopoly on server CPUs and server boards that support them. (Other vendor's server boards...) AMD did a GOOD job on the AMD64 instruction set, and did have quality server processors, (like the Opterons of 12 years back). AMD even had multi-core processors before Intel, (IIRC). But in recent years, (until about 2 years ago), AMD was lagging noticably in server CPU sales.

This impacted software development from FreeBSD developers, who spent more time adding new CPU features and fixing CPU bugs on Intel products. (As well as the system board features that went with them...) Intel was more common, so those processors got more attention.

That said, Intel sat on butts too long for most of us. AMD has come out with a nice selection of server processors. Perhaps not the quad socket possible with Intel, but at the lower end, better in my opinion. Some of those embedded server processors, with 10Gbps Ethernet look wonderful for things like miniature NASes and firewalls.
 

Yorick

Wizard
Joined
Nov 4, 2018
Messages
1,912
HBA can solve your SATA port drought. If you ever get tired of USB boot.

Sounds like you suspect some bias, @Kevern . I haven’t detected any.

The hardware guide highly recommends ECC, and server boards, which on the Intel side gives you i3, Xeon, and on the very low end Pentium since Kaby - though I’d expect to see i3 as the lowest step to go to. SM server boards are well regarded. Some form of BMC is a lifesaver, and stable SATA ports onboard help.

On the AMD side, tales of a successful EPYC build since November 2018. That’s how recent this is. https://www.ixsystems.com/community...cro-h11ssl-i-do-freenas-11.63854/#post-490815

I think we will see more people building on EPYC. It’s a very strong contender, particularly when more throughput is wanted. Those performance numbers in that thread are juicy.

Embedded support for FreeBSD, and server board support, are a work in progress.

https://www.ixsystems.com/community/threads/amd-epyc-embedded-3000-series-bsd-support.61812/page-2

And on the Ryzen side, my understanding is Asrock are the only ones taking ECC seriously. I don’t know how good their Ryzen server boards are, or whether they have any.

As people gather more experience with EPYC, I’d expect the hardware guide to be updated. Allow there to be a period of gathering some experience with the platform. First successful reported build in November is still quite recent.
 

JJT211

Patron
Joined
Jul 4, 2014
Messages
323
I've been thinking about getting one to fill my unused Caselabs S3+pedestal. The case has room for 12 x 3.5" drives worth of 5.25" to 3.5" hotswap bays and possibly room for a x8/x8 bifurcation riser so I can add 10Gbe and an HBA card. Still wish these Supermicro boards made use of the 4 x 10Gbe already built into each Zen die.

Where have you seen a x8/x8 bifurication riser?

BTW, I just got the same exact SM AMD Epyc 3251 boards. Got all hardware installed, working on ESXi as we speak

EDIT: In case anyone with same board is trying to install FreeNAS over ESXi and is having trouble enabling passthrough for the HBA, go to BIOS and enable IOMMU. Took me all night to figure that one out.
 
Last edited:

R0lly

Cadet
Joined
Jan 27, 2019
Messages
5
So you can confirm that ESXi runs on this board without any issues?
That would be great news! :cool:
I'm tempted to build a lightweight system based on this board or the ASRock Rack version with a Intel® X550-AT2 onboard.
 

JJT211

Patron
Joined
Jul 4, 2014
Messages
323
So you can confirm that ESXi runs on this board without any issues?
That would be great news! :cool:
I'm tempted to build a lightweight system based on this board or the ASRock Rack version with a Intel® X550-AT2 onboard.

Confirmed!!
 

JJT211

Patron
Joined
Jul 4, 2014
Messages
323

JJT211

Patron
Joined
Jul 4, 2014
Messages
323
So you can confirm that ESXi runs on this board without any issues?
That would be great news! :cool:
I'm tempted to build a lightweight system based on this board or the ASRock Rack version with a Intel® X550-AT2 onboard.

I may have spoken too soon. :(

https://www.ixsystems.com/community/threads/m11sdv-8c-ln4f-amd-epyc-3251-crashing.77685/

Although, I may have just gotten a bad one. I think im gonna try and RMA this one while I can. FreeNAS and ESXi were running great up until now. So that tells me there shouldnt be any compatitability issues.
 
Last edited:

JJT211

Patron
Joined
Jul 4, 2014
Messages
323
Update:

So all is good. It was just a bad RAM slot on SM Mobo. RMA'd board, got new one in and it's been running super solid for the last few months. Ive been so impressed with this AMD EPYC, I just ordered the 7281 EPYC 16 core to run in our data center. Look forward to putting that bad boy through the paces
 
Top