Hello everyone!
I am building a server for a client of mine and I am interested in some feedback from the community. This is my first FreeNAS build, and I have pretty limited experience in file storage, but I enjoy the challenge. As this is a Ryzen build, I really want to documents my challenges and bugs for the community at large.
My client needs a server that can handle basic file storage, windows VMs, PLEX, databases, phones and firewall security(pfsense). I originally considered a DL380 G7, but the power consumption would probably be double that of Ryzen. Everything that I read about ZFS and btrfs is extremely exciting, and FreeNAS eventually won out because of the flexibility and VMs.
Parts List:
I knew it was going to pretty experimental trying to run everything through FreeNAS, so I will try to switch to a type 1 hypervisor (
vSphere Hypervisor)-- I haven't run a VM server before, so should be interesting.. I will assign the 4TB drives to FreeNAS and just use it for what it was designed to do( file storage). I will use the SSD for the databases and unimportant VM on it. I will probably assign a few virtual cores to Plex server as well.
*12 VMware vSphere 6.5a still has the SMT bug.. I think they fixed it for 6.5.1, but the free version of the software has not updated to that version yet.. no big deal SMT->off -> no pink/purple screen. ( might be a few days until a progress further(busy)) * gave me the idea to try FreeNas with SMT off
*13 Virtualization should work now.. ESXI works with SMT off in 6.5.. when I get back I will recheck FreeNAS-- I also might take the cpu cooler off and check the manufacturer day for the other SMT bug
*14 FreeNas VM's/jails are working. All 16 threads. SMT enabled. I am now going to run the kill ryzen script on a live usb to test SMT stuff. At this point though FreeNas is working as intended and looks stable. I will update with the results of my test and determine whether I am going to run ESXI or FreeNas as the head OS.
*15 I don't have a passable ryzen cpu. kill-ryzen.sh got an error in 93 seconds, however it was never a segway error. I have talked to an AMD rep and thought about it a while, I will never use this server to compile anyone on so this synthetic test will probably never result in any damage. Since I am opting to not replace the system cpu, the system will probably never be considered "production" ready. However I am continuing with the project with the hopes of production ready system.
*16 I have been working a lot on testing VM's under bhyvee... It has been very interesting. FreeNas has been mostly stable for its part. There are a few bugs that still need to be address that have forced me to restore to factory default, but I am getting faster at restoring. The UEFI boot loader under bhyvee has been causing me a good deal of problems. I have successfully gotten Ubuntu and FreeBPX(centOs) stable and shutting down correctly.
*17 pfSense has been causing a lot of problems... It does not even want to load the .iso with bhyvee.
*18 I can confirm that 6.5 U1 ESXI does fix the SMT bug... all 16 threads available.
*19 Props to the 11.0-U3 doc editors-- updated for tips* "Best Quality on RealVnc" and grep POPCNT /var/run/dmesg.boot for amd
things that I had to learn from other sources
*20 I have decided to move the firewall to another device(instead of VM)... it was causing a lot of issues and a firewall/router should be a separate device anyway. ESXI could do it, but the FreeNAS had to be VM'd with SATA passthrough and you need to spend a decent amount of money for compatible SATA passthrough cards I think. I could not get my 88SE9215 to work with 6.5
*21 So I have had my fun and now I have to get something stable. To the point:
I have decided to go FreeNAS with 2 VMs (ubuntu and freepbx). I will run an unraid server for mirrored copying and a windows VM. The data will also be copied offsite to another offsite NAS. -- I will do a real post summarizing everything and hopefully giving an idea of the stability, and the bugs that still exist. I hope to put almost everything back on this server once AMD processor is fully supported. I hope to report on the stability in 3 months.
I welcome any feedback and suggestions. Thanks - Wackymole
I am building a server for a client of mine and I am interested in some feedback from the community. This is my first FreeNAS build, and I have pretty limited experience in file storage, but I enjoy the challenge. As this is a Ryzen build, I really want to documents my challenges and bugs for the community at large.
My client needs a server that can handle basic file storage, windows VMs, PLEX, databases, phones and firewall security(pfsense). I originally considered a DL380 G7, but the power consumption would probably be double that of Ryzen. Everything that I read about ZFS and btrfs is extremely exciting, and FreeNAS eventually won out because of the flexibility and VMs.
Parts List:
- Silverstone Technology CS380B Silverstone
- Intel PRO/1000 Pt Dual Port Server Adapter
- Kingston Value Ram 2 Sticks 16 GB KVR21E15D8/16 (32GB ECC)
- AMD RYZEN 1700 8 core
- Rosewill ARC Series 450w
- PNY GeForce GTX 1050
- ASRock AB350 PRO 4
- 2 * 4 TB WD Red ( different sources)
- 32 flash drive (OS)
- 240 GB Sandisk SSD G26
- Assembled Everything--Success UEFI boot screen!
- Checked Settings -SMT-yes, IOMMU -yes
- Correctly put the Ram in slot (a2, b2) dual channel
- Ran memtest86+ for Ram check-passed
- Ran Ubuntu 17.04 -- ECC check through dmesg-- ECC enabled
- Ran FreeNAS 11 -stable -- looks good, stable, user files good, custom volumes
- installed plex plugin-- worked, tried VM- no good-
- Could not get a blank VM to start.. even more troubling can't find assign more than 1 virtual core (only one virtual core supported warning)
- checked settings
- Updated to FreeNAS 11 nightly builds-- still not fixed...
- Given up on running everything through FreeNAS for now... need to add a bug report for Ryzen FreeNAS VMs
- Booted up Vsphere VWare..
- Checked ESXi-- no VM so recheck bios-- SVM (why?) not disabled so I enabled.
- Rechecked FreeNas 11-Release- VM's work perfectly.. all 16 threads with smt and SVM
- - skip down
I knew it was going to pretty experimental trying to run everything through FreeNAS, so I will try to switch to a type 1 hypervisor (
vSphere Hypervisor)-- I haven't run a VM server before, so should be interesting.. I will assign the 4TB drives to FreeNAS and just use it for what it was designed to do( file storage). I will use the SSD for the databases and unimportant VM on it. I will probably assign a few virtual cores to Plex server as well.
*12 VMware vSphere 6.5a still has the SMT bug.. I think they fixed it for 6.5.1, but the free version of the software has not updated to that version yet.. no big deal SMT->off -> no pink/purple screen. ( might be a few days until a progress further(busy)) * gave me the idea to try FreeNas with SMT off
*13 Virtualization should work now.. ESXI works with SMT off in 6.5.. when I get back I will recheck FreeNAS-- I also might take the cpu cooler off and check the manufacturer day for the other SMT bug
*14 FreeNas VM's/jails are working. All 16 threads. SMT enabled. I am now going to run the kill ryzen script on a live usb to test SMT stuff. At this point though FreeNas is working as intended and looks stable. I will update with the results of my test and determine whether I am going to run ESXI or FreeNas as the head OS.
*15 I don't have a passable ryzen cpu. kill-ryzen.sh got an error in 93 seconds, however it was never a segway error. I have talked to an AMD rep and thought about it a while, I will never use this server to compile anyone on so this synthetic test will probably never result in any damage. Since I am opting to not replace the system cpu, the system will probably never be considered "production" ready. However I am continuing with the project with the hopes of production ready system.
*16 I have been working a lot on testing VM's under bhyvee... It has been very interesting. FreeNas has been mostly stable for its part. There are a few bugs that still need to be address that have forced me to restore to factory default, but I am getting faster at restoring. The UEFI boot loader under bhyvee has been causing me a good deal of problems. I have successfully gotten Ubuntu and FreeBPX(centOs) stable and shutting down correctly.
*17 pfSense has been causing a lot of problems... It does not even want to load the .iso with bhyvee.
*18 I can confirm that 6.5 U1 ESXI does fix the SMT bug... all 16 threads available.
*19 Props to the 11.0-U3 doc editors-- updated for tips* "Best Quality on RealVnc" and grep POPCNT /var/run/dmesg.boot for amd
things that I had to learn from other sources
*20 I have decided to move the firewall to another device(instead of VM)... it was causing a lot of issues and a firewall/router should be a separate device anyway. ESXI could do it, but the FreeNAS had to be VM'd with SATA passthrough and you need to spend a decent amount of money for compatible SATA passthrough cards I think. I could not get my 88SE9215 to work with 6.5
*21 So I have had my fun and now I have to get something stable. To the point:
I have decided to go FreeNAS with 2 VMs (ubuntu and freepbx). I will run an unraid server for mirrored copying and a windows VM. The data will also be copied offsite to another offsite NAS. -- I will do a real post summarizing everything and hopefully giving an idea of the stability, and the bugs that still exist. I hope to put almost everything back on this server once AMD processor is fully supported. I hope to report on the stability in 3 months.
I welcome any feedback and suggestions. Thanks - Wackymole
Last edited:
