AMD Threadripper build.

Apollo

Wizard
Joined
Jun 13, 2013
Messages
1,458
Hi folks,

I couldn't find any resource available regarding AMD Threadripper as a viable solution to a Freenas box.
I just wanted to chime in with my findings so far.

As I have upgraded my personal PC to the latest Threadripper 2950X, I was able to perform some experiment based on existing ZFS pools.
Here are the details:

CPU: AMD Threadripper 2950X
MB: Asrock x399 Taichi
RAM: 8 x 16GB Kingston KVR24E17D8/16 ( Total 128GB)

Volumes:
2 x RED 4TB each in Mirror configuration.
3 x HGST 10TB each in RAIDZ1 configuration (encrypted).

Both volumes are replicated with backup server from my normal server.

I have run scrub on both of them.
I have installed Nextcloud 14 based on Danb35 script and accessed my replicated Nextcloud data with it from the Mirror volume.

I have installed Ubuntu 18 as a virtual machine without any issues.
I was unsuscessful at installing Docker, because it is looking at my RAIDZ1 volume which already contains .behyve, but volume has zfs set to readonly.

Ultimately, I will be moving my current server based on Xeon E3 1241 V3 to the AMD 1900x which is currently on its way.
I will report back when replacement is finalized.
 
Last edited:

Stux

MVP
Joined
Jun 2, 2016
Messages
4,419
The most important thing is long term stability. Ryzens have had stability problems with FreeBSD since release but it looks like they are finally settling down.

Try searching the forum for Ryzen
 

Apollo

Wizard
Joined
Jun 13, 2013
Messages
1,458
The most important thing is long term stability. Ryzens have had stability problems with FreeBSD since release but it looks like they are finally settling down.

Try searching the forum for Ryzen
All I could find about Ryzen is referring to Ryzen 3, 5 and 7. I couldn't find anything relevant to Threadripper. At least not on this forum.
Even the FreeBSD website doesn't seem to provide infos. It is like searching for a needle in a hay stack.
 

Apollo

Wizard
Joined
Jun 13, 2013
Messages
1,458
Last night I got my AMD Threadripper 1900x with Asrock x399 Taichi up and running with Freenas 11-2 Beta.
I was able to import encrypted volumes without any issues. I have successfuly been running concurrent scrubs on the various volumes.
Starting with a fresh install of Freenas and a new clean volume, I was able to install the Docker Host and Linux Virtual machines.
Installing new iocage jails is no sweat either.
Installing Plex plugins work as well, though I haven't figured out how to map a dataset as a Plex library yet.

Everything I have done so far seems to be working nicely.
 

Stux

MVP
Joined
Jun 2, 2016
Messages
4,419
All I could find about Ryzen is referring to Ryzen 3, 5 and 7. I couldn't find anything relevant to Threadripper. At least not on this forum.
Even the FreeBSD website doesn't seem to provide infos. It is like searching for a needle in a hay stack.

At the end of the day, a thread ripper is made out of ryzen dies. What applies to ryzen probably applies to threadripper
 

Apollo

Wizard
Joined
Jun 13, 2013
Messages
1,458
At the end of the day, a thread ripper is made out of ryzen dies. What applies to ryzen probably applies to threadripper
That may be true, but then again, I understand Threadripper comes from EPYC.

So far so good.
I have been able to create a Mountpoint for Plex. Transcoding 4K videos doesn't seem to be too much work for the 1900x. I haven't pushed too far yet but it definitely more responsive than my Xeon E3.

No issues with replication being run to it.

Two annoying behavior though:

1)
When two or more volumes with active iocage, upon boot, it will take a fairly long amount of time to boot. It seems to get stuck at the :
Consoles: EFI console
There is a "/", "-", "\" spinning at low speed. At first I thought it to be AMD issues related, but upon activating only one of the iocage volume everything seems to go back to normal.
This alone would cause "iocage" list to fail or not start.

2)
This one is more problematic but if I request system to shutdown, via CLI or Web interface, Freenas will terminate all its process as intended but will not cause system to power down.
Rebooting doesn't exhibit this behavior and system will reset as expected.

Other than that I believe Threadripper to be a solid platform.
 

newfreenas

Dabbler
Joined
Nov 7, 2018
Messages
30
> Other than that I believe Threadripper to be a solid platform.

been a little over a month since your post, wondering if you still feel that way? :) I would like to build a FreeNAS build

I am looking to build a threadripper build, are you using bhyve at all for multiple VMs?

can you share the hardware you have? (I notice you mention 2950 and 1900 in this thread, seems that you used 1900 for FreeNAS and not 2950? curious to know if there was a specific reason why?)
 

Apollo

Wizard
Joined
Jun 13, 2013
Messages
1,458
> Other than that I believe Threadripper to be a solid platform.

been a little over a month since your post, wondering if you still feel that way? :) I would like to build a FreeNAS build

I am looking to build a threadripper build, are you using bhyve at all for multiple VMs?

can you share the hardware you have? (I notice you mention 2950 and 1900 in this thread, seems that you used 1900 for FreeNAS and not 2950? curious to know if there was a specific reason why?)
I still feel very strong about the Threadripper build.

I bought the 2950 when it came out as my desktop was getting antiquated and one of the Windows 10 update and Intel lack of support against the latest security threat for the CPU retired it officially.
With the 2950 at hand, I was able to put it to the test on Freenas 11.1 and was very pleased with it.
At the same time, my Xeon-E3 was giving me more trouble than ever (most likely caused by Freenas 11-1/11.2 still being in Beta) as it would render my Nextcloud unuseable because the the jail would loose network connection or because some background jobs would be killed due to memory starvation. Possibly a mix of bugs and lack of sufficient amount of RAM.
However, while I still get some warning or critical status from NetData, I no longer get those on a regular basis. Once in a while and mostly because of replication as replication is going to consume as much RAM as needed which kinds of sucks.

I was hopping to upgrade to an 8 core on second Gen Threadripper, but the 1900x was never going to make it to second Gen as a 2900x so I gave up on the idea. The next one in line was the 2920x I believe (12 cores) but was not yet available and pricing would have most likely been a bit steep.
At work we have one Threadripper 1950x which has been running on Ubuntu for over a year now, I believe and is doing some Android builds on a regular basis. It has been a very strong performer and very reliable as it seems.

The 1900x I bought was on sale and I decided to make the jump when I did. I must have got it when it was at its lowest price point I think. I was a bit cautious because the test I performed on the 2950X was no guarrantee to be the same with the first gen do to hardware differences.

As I am writing this post, I decided to run some test in term of throughput from my Nextcloud iocage jail.

The schreenshot attached is a quick test I have been running while writing this post.
The screenshot shows throughput conducted from my Nexctloud jail synching into my Windows 10 2950x NVME drive.
in just 15 minutes later, I was able to sync well over 80GB of photos, quite impressive. It doesn't quite max out the LAN but can easily hit 940Mbit/. I must add, that my Nextcloud connection is being routed through my haproxy running on my pfsense.

Compared to my Xeon E3 and Pentium G4600, Threadripper is a beast.

I am still using the G4600 for servicing my backup volume via replication.

The XEON E3 in comparison is like a dinosaur.

I am hitting 970Mbit/s throughput during replication whereas XEON E3 couldn't do more than 650Mbit/s. I guess encrypted volumes and SSL encryption was too much for it.

I am running on Nextcloud 13.7 and have been trying to migrate to 14 but I am hitting some snags with the upgrade. This is not Threaripper issue.
Because of it, I haven't had a chance to proceed with the move to VM and Rancher OS (Docker container ) yet. I didn't have much time to go through the process either.

I have however performed numerous install in different iocage jail with Danb35 script for Nextcloud and since Freenas 11-2 RC1, I haven't experienced any show stopper. Behavior has been very predictable and installing a full fleidge Nextcloud jail takes a mere 6 minutes or so.
I am running the Plex plugin and it has been stable too.

On the other hand I have a few issues that may or may not be specific to Threadripper.

I did order a LSI 9240-8i (I Flashed ii to IT mode) to be used on my 10 disk volume but under Pre Freenas 11.2-RC1, I would get LSI related errors that would cause my system to freeze.
Because of it, I have been using my old hpt2320 to host my boot volume and 2 of my 10 disk drive ( the 8 remaining disks are connected to the 8 SATA ports) . I know the hpt is not an ideal solution but for now it works without any issues.
I will have to conduct some more test with my 2950x when I get a chance.

For some reason, sending the shutdown command will cause the system to initiate shutdown but the system will not power-off. Initiating a restart doesn't exhibit such issue.
Trying to run concurrent Plex transcoding (playing a few 4K video and 1080p 20Mbit/s+) onto a few Firefox instances on my Windows PC was no real issues until I loose network connection and apparently having Freenas freezing on me. I suspect a Jail plugin crash but I can't say for sure.
I have run iperf and replication and system is very stable, while servicing Plex and Nextcloud. I suspect temperature could be the issue has 1900X would easily peak in the high 80% CPU usage when transcoding videos. The transcoding test was on about 4-6 different 4K and 1080P video being transcode to much lower resolution and bitrate.

Other than that it has been running solid for the last 17 days around the time of Freenas 11.2-RC1 update.

Power consumption isn't too bad on the low 120W and can peak to above 200W when taxing CPU's during transcoding.

One criteria I had in mind when purchasing AMD for this build was the availability of the NVMe interfaces in the event RAM would still be an issue.
With 80GB being completely used I do not experience any perceiveable performance degradation and Freenas Web interface hasn't been this fast and fluid. I can say the same of Nextcloud.
This is litteraly night and day.
I am hoping to play with PCIe NVMe as L2Arc but it is not going to be anytime soon anyway. For now I don't see it to be a requirement.


My ARC Hit ratio is in the high 90%.

Was this a good choice? Well so far I am still very impressed by the performance. Neither one of the Intel platform I have could match it. The G4600 could be a contender but lacks the number of cores and is still can't use more than 64GB of RAM.
Not having IPMI capability is a small sacrifice to pay. This is why I have a small video card installed on the motherboard.




















I have a few areas
 

Attachments

  • 1900x-nextcloud.png
    1900x-nextcloud.png
    1.3 MB · Views: 1,089
Last edited:

newfreenas

Dabbler
Joined
Nov 7, 2018
Messages
30
thank you for the details response, I really appreciate it. I will take the dive... hoping for some discounts on black friday/monday and will try it out then.
 

newfreenas

Dabbler
Joined
Nov 7, 2018
Messages
30
to add one more thing, I am collecting links etc and hoping for some blackfriday discounts... so I have some time before I purchase but... in regards to cooling what are you using?


I am relatively new to this sort of thing, should I be looking at liquid cooling? when I look at your system build I see no mention...also could you share the case you used?

> AMD Threadripper 1900x
> Asrock x399 Taichi
> Kingston 80GB UDIMM ECC DDR4-2400 (5*16GB)
> FreeNAS-11.2.......
> 10 x WD RED 4TB in two Vdevs of RAIDZ2: 2*(3 data +2 redundancy)
 

Apollo

Wizard
Joined
Jun 13, 2013
Messages
1,458
to add one more thing, I am collecting links etc and hoping for some blackfriday discounts... so I have some time before I purchase but... in regards to cooling what are you using?


I am relatively new to this sort of thing, should I be looking at liquid cooling? when I look at your system build I see no mention...also could you share the case you used?
My Threadripper is using the Freezer 33 TR from Artcic.
The case is a Rosewill RSV-L4412.

One issue I have is that the Freezer 33 TR is about 5mm or so too tall and the case will not close properly. Something to watch for.
The official Threadripper Wreath CPU heatsink wasn't available. A bad move from AMD and its partner.

The Freezer 33TR seems to be doing a fine job in general when not being pushed too hard. I had issues at the end of summer but I am not sure if it was due to Freenas 11.x bugs and instability.
If you go with water cooling, I would say to be cautious due to potential leaks that could occur over time.
I used to have a watercooled i7-920 before moving to the 2950X threadripper for my main home PC. I designed the loop myself and has been very solid for the past couple of years.
Currently my 2950X has the Cryorig A40, but my PC never stay on while I am away.
When I run 2950X on all cores and threads, the temperature will rise fairly quickly to the point it will start throttling down the CPU.
I think Cryorig A40 should be adequate to keep the 1900X cooler, has it has less cores. The Rosewill case will not accomodate the radiator so this will make it an unfit solution. I do like this hot swappable HDD tray. This is the reason I got the case in the first place, but the case is really long which is unfortunate.
Some other thought about Cryorig A40 is the extra fan for VRMs, while quite a nice addition, the engineer who designed it never thought about the cooling tubes as being an issue. Currently, they are running in front of the fan which is causing a drop in efficiency. They could have come up with a better solution instead. Also, the length of the cooling tubes can be of a concern depending on your case size and configuration as the tubes are not very long and do not provide much flexibility as a result.

One thing to know about AOI (All In One) watercooling is that they never seem to come with any kind of flow sensor, so you can't really tell if the loop is working. This was a major drawback when I first setup my 2950X Windows 10 PC. I had lots of issues back then because of the Bios.
 

inis

Dabbler
Joined
Aug 9, 2017
Messages
23
Any concerns on cpu cooling? Far as I know no Ryzen or newer AMD has temp sensors on freenas. This kind of worries me since my last freenas box ran for 4yrs straight and I'm on a new Ryzen 7 2700 with no temp monitoring.
 

Apollo

Wizard
Joined
Jun 13, 2013
Messages
1,458
Any concerns on cpu cooling? Far as I know no Ryzen or newer AMD has temp sensors on freenas. This kind of worries me since my last freenas box ran for 4yrs straight and I'm on a new Ryzen 7 2700 with no temp monitoring.
CPU temperature is available on Freenas-11-2-RC1 which I am currently on.
CPU temperature shows up in Netdata as well.
I am not sure which sensors are actually used, per core sensor or Die sensor.
But it would seem Legacy GUI and Netdata are using single core temperature sensor while Angular GUI is looking at the Motherboard temperature it seems(my own observation).

Only concern I have had with CPU was when running Plex to transcode several 1080p and 4K and was causing system to crash, but I don't know if it was temperature related or simply due to Freenas and iocage instability. At the time temperature would ramp-up, power consumption was at the highest around 290W.
On the bellow Sintel 4K screenshot, I just ran a few minutes ago, 5 instances of Sintel 4K in Firefox.
The original file is 4K @ 40.6Mbps.
On screenshots you can see transcoding at 84.2Mbps on 2 Firefox instances and 63.2 for the other 2. The last one I can't say.
At that time CPU what able to hit 100% usage and hit in the high 90 degrees C.
For this run, 1900x hasn't crashed and is on its 27th day without reboot.
Has it is, CPU cooler isn't bad on my typical daily run. However, stressing it like I did would be a huge concern over long period of time.
A beeffier cooler like the official Wreath from AMD would be a better fit I think.
 

Attachments

  • Threadripper-1900x-temperatures.png
    Threadripper-1900x-temperatures.png
    187.5 KB · Views: 1,103
  • Threadripper-1900x-temperatures-Angular.png
    Threadripper-1900x-temperatures-Angular.png
    16.5 KB · Views: 806
  • Threadripper-1900x-temperatures-Netdata.png
    Threadripper-1900x-temperatures-Netdata.png
    579 KB · Views: 710
  • Threadripper-1900x-temperatures-Netdata-1.png
    Threadripper-1900x-temperatures-Netdata-1.png
    493.6 KB · Views: 745
  • Threadripper-1900x-temperatures-transcoding.png
    Threadripper-1900x-temperatures-transcoding.png
    1.2 MB · Views: 1,005
  • Threadripper-1900x-temperatures-transcoding-Sintel-4K-1.jpg
    Threadripper-1900x-temperatures-transcoding-Sintel-4K-1.jpg
    313.9 KB · Views: 1,155

newfreenas

Dabbler
Joined
Nov 7, 2018
Messages
30
My Threadripper is using the Freezer 33 TR from Artcic.
The case is a Rosewill RSV-L4412.

One issue I have is that the Freezer 33 TR is about 5mm or so too tall and the case will not close properly. Something to watch for.

likely an issue for most coolers....with that experience would you recommend or get a different case next time? what would you get? was looking at cube cases but the one you have appears to have top grade ventilation

I am not going with water cooling... albeit wraith ripper does not appear available even now...maybe there was an issue?



lastly, thank you for this invaluable knowledge!
 

Apollo

Wizard
Joined
Jun 13, 2013
Messages
1,458
The case is fine, but I wish it was shorter. I haven't really used the hot swapable tray since I went with replication over LAN.
I can have at most 12 HDD drives on hot swapable tray.
If I didn't have the case already, I might have gone with a smaller forma factor.
As a note, I went with the 4TB RED drives as there was not many HDD solution as we have today. 10 drives is fine and having them as 2 Vdevs on one volume I have a slight performance increase. The newer HGST 10TB NAS drives I am using for backup in 3 RAIDZ-1 seem to have an even better performance as a volume.
I bought enough to have them as a 5 disk RAIDZ-2 as a replacement in the event I would fail transition to Threadripper on the RED drives.
If I was going to build a new system, I would highly recommend the HGST NAS drives. They are 7200RPM as opposed to the RED at 5200 or 5400RPM.
HGST are more responsive as a single drive and run a bit warmer but not like as hot as a desktop drive.
Fewer drives also means higher reliability and easier maintenance.
I believe the 5 drive RAIDZ2 is as good a performer or possibly better as my 10 drive 2 RAIDZ2 Vedv.
With less then 8 drive solution, you no longer need and external RAID card in IT mode.
Another note, at the beginning of this post, I mentioned I had issues with LSI on Threaripper and the outcome was consistent with system disconnecting drives and ending up freezing the system. For the last couple of weeks I have been using my G4600 to run replication over lan with the LSI 9240-8i reflashed in IT mode. I currently have 2 of the LSI and I did stress them by running my replication and going through scrubbing and such.
I had no problem with the LSI whatsoever, however, over the weekend, I was still trying to perform some test with iocage install of Nextcloud and I was moving one SSD over between my G4600 and Xeon E3. G4600 is my replication server, so when I do replication from my Threadripper, I do it over LAN to the G4600. My Xeon E3 is actually being repurposed as a Freenas box and is on a different LAN. It connects to another old PC acting as my pfsense firewall. The intent is to experiment with Freenas iocage without messing up with my main system.
As a result of moving back and forth, and I believe after performing an update to 11.2-RC2 on Xeon E3, and moving the SSD hosting the iocave jail to G4600, upon attaching it I got the very same error message with LSI wen it used to run on Threadripper.
As a result, G4600 would freeze.
It as been very solid when I was running replication on 12 HDD composed of the following:

5 HDD RAIDZ-2 (Encrypted)
3 HDD RAIDZ-1
2 HDD RAIDZ-1 (Encrypted) ( removed 1 drive on purpose, so it is in degraded state, but scrubing and replication are still implemented)
2 HDD Mirror

When I attached the SSD, I removed all the HDD after powering down G4600. I didn't go though the process of detaching them from GUI. They would show as not available in GUI or locked.
When I attached SSD, the first thing I got was MPS0 failure causing system to freeze upon .

This makes me wonder if LSI really has issue with Threadripper or maybe I am just fighting a bug.

By the way, I was finally able to update to Nextcloud 14 and I have been tweaking my MySQL database parameter and I must say I am really, really happy about the outcome. I have also upgraded my internet speed over the week end.
I am now even more impressed by the general condition and performance of the Threadripper.
It is really running at blazing speed.

Do you mind sharing your hardware wishlist?
 

newfreenas

Dabbler
Joined
Nov 7, 2018
Messages
30
my hardware wishlist is still in the making, but to be fair aside from AMD CPU, 16GB sticks of RAM (intend to get 64 to start) and at least 20TB I do not care what it looks like as long as its properly cooled and not extremely loud. I'm happy to take recommendations... I've read most of hardware reference guide and was slightly deterred because of lack of reference to AMD. I liked this chassis --> https://www.amazon.com/dp/B0091IZ1ZG/ but I also liked the cube chassis and I really do not need one that big. I have a lot of analysis paralysis... feel free to help! frankly I almost bought much of what you have until I saw you had an issue with the case closing... I know its a small thing to deal with... but buying new stuff I just want to account for things like that.


also planning to use next cloud... but only because I am familiar with it, and thoughts on owncloud?




I tried to get table formatting in this but I failed... I posted picture below of what the text below should look like.


ID Part Links
1 Case https://www.amazon.com/dp/B00D6GINF4/ref=twister_B07GTKNJHH

https://www.amazon.com/Fractal-Design-Supply-MicroATX-FD-CA-NODE-804-BL/dp/B00K6OVG0I/


2 Drives https://www.amazon.com/Seagate-Barracuda-3-5-Inch-Internal-ST8000DM004/dp/B075WYBQXJ//ref=as_li_ss_tl

3 Mother Board https://www.amazon.com/MSI-ThreadRipper-SLI-Motherboard-X399/dp/B074DHQR9N
https://www.amazon.com/ASRock-Socket-MicroATX-Motherboard-X399M/dp/B079Z8W61Z/

4 CPU 2950X - 16/32 - 900
5 Cooling https://www.amazon.com/Cooler-Master-MAM-D7PN-DWRPS-T1-Wraith-Ripper/dp/B07H25DZ3M/ (discontinued?)

https://www.amazon.com/ARCTIC-Freezer-Threadripper-3-Phase-Motor-Regulation/dp/B075MYVW5J/
6 RAM 16 GB sticks (32gb to start)
7 UPS
8 USB
10 Power Supply
11 Fans

upload_2018-11-20_22-10-56.png
 

CraigD

Patron
Joined
Mar 8, 2016
Messages
343
If you can buy drives designed for NAS operations

Ironwolf and WD Reds are a much better option

Have Fun
 

Apollo

Wizard
Joined
Jun 13, 2013
Messages
1,458
For the drive I would go with the HGST Deskstat NAS.
I cannot say anything about Ironwolf as I have had no experience with them.

What memory do you have in mind?
I use the Kingston ValueRAM 2400 DDR4.
I would leave USB and use SATA SSD instead. Kingston 400 series is good.

For the fan, try to fiind a similar one about 5 mm shorter if possible. The issue is that the case cove will hit the top of the heatsink and potentially danage the heatsink or sould stress the board and SPU.
You will need a low end graphic card with that, as the Threadripper doesn't have video, unless you can manage seting up COM port but I have never tried.

If you wan 20TB of storage space with 8TB drive in RAIDZ2, you will need at least 5 drives. With that in mind, you do not ned to go with the oversized Roswill case.
You might be better off with a small case supporting 6 or more HDD I think.
 

Apollo

Wizard
Joined
Jun 13, 2013
Messages
1,458
also planning to use next cloud... but only because I am familiar with it, and thoughts on owncloud?

View attachment 26658
I have been using Owncloud starting with Owncloud 8 I think, until Nextcloud was forked. I have been using Nextcloud since then and it is getting better all the time.
I am running a few instances in iocage jails and it is working very nicely.
As I have mentionned , I have excellent transfer speed over LAN.
Last night I was able to install Docker Host with Rancher and thanks to the documentation comming with Freenas 11.2-RC2, I was able to procedd with the install of Docker itself and a few containers.
They seem to work fine but I couldn't really do anything with them yet as I am not familiar with the environment.

With Docker, each container will require their own address and port to communicate with the server, you may need a good firewall. I built mine out of a Ryzen 3 2200 to run pfsense.
If you don't have one, you might consider get the hardware to build one or buy one off the shelf.
Maybe someone else can comment on that.
 

Apollo

Wizard
Joined
Jun 13, 2013
Messages
1,458
In the link you provided, I am being redirected to Amazon showing the M version of the board. I would avoid it has it has only 4 DDR4 slots which limit you to 64GB.
I would suggest you go with the X399 (Without the M) has it supports 8 DDR4 slots and more NVME slots.
 
Top