FreeNAS on Hyper-V vs FreeBSD 9, NAS4Free and FreeBSD 10

Status
Not open for further replies.

renegm

Cadet
Joined
Dec 18, 2015
Messages
6
Hi
This is my first post, my English and my knowledge are not perfect, sorry. I want to hear suggestions for improve performance in my FreeNAS box.
My Hardware configuration:
Supermicro A1SAI-2750F, 32GB, 2x4TB HDD, 240GB SSD. Full list in: http://secure.newegg.com/WishList/PublicWishDetail.aspx?WishListNumber=29059826

Hyper-V 2012 R2.
FreeNAS VM configuration: 8 Cores, 16GB, VHDD 16GB boot on SSD, VHDD 20GB on SDD for other FreeNAS stuff and 2xHDD 4TB (direct attach, not virtualized) in stripe.

Little FAQ
1) Why virtualized? Uuu... short answer: because.
2) Why Hyper-V? I want direct, raw access to physical hard drives. This board does not support VT-d.
3) Why stripe? There are only 2 drives. I want a simple file server carrying not critical data. This server will contains Windows 10 File-History backups and time-line backups for a few machine. This is not a "my house burned down" backup.

Mini how-to or what i did for FreeNAS on Hyper-V.
Hyper-V network legacy drivers sucks. And you need HV-storage driver for Physical HDD access. FreeBSD 10 contains drivers but FreeBSD 9 (FreeNAS kernel) don’t. So, what you need?
1) First step compile drivers: http://www.freshports.org/emulators/hyperv-is/ in whatever FreeBSD 9 machine. (I can share the drivers if someone want it. With a little tip: Do not trust in a random internet guy.)
I put the drivers in an ISO file but you can use another method. (4 little files: hv_vmbus.ko, hv_utils.ko, hv_netvsc.ko, hv_storvsc.ko)
2) Create FreeNAS VM using normal network (no legacy) and install. Reboot, and replace installer media with driver’s media. You don't have network or SCSI drives access yet. Go to shell, mount your HV_drivers CD and copy files to /boot/modules
3) Load drivers:
kldload /boot/modules/hv_vmbus.ko
kldload /boot/modules/hv_utils.ko
kldload /boot/modules/hv_netvsc.ko
kldload /boot/modules/hv_storvsc.ko
4) FreeNAS can see network adapter and SCSI disks now (temporarily, you need to add to loader). Exit to Console Setup and configure Network Interfaces
5) Open your browser and go to FreeNAS web interface. Skip wizard (You can do that later) go to System/Tunables and add drivers tunables (4 tunables one for each one. (DriverName_load means filename without dot and extension, e.g. hv_netvsc_load):
Variable: DriverName_load
Value: YES
Type: Loader
Enabled: Checked
6) Cross your fingers and reboot. Everything must be fine. Enjoy.

My tests
Same virtual machine (I shutdown VM and select the appropriate boot VHDD) Sharing Cifs, ZFS no compression, 1x4GB file to and from my PC, all hardware connected to 1GB Netgear switch)
----------------
NAS4Free and FreeBSD 10 (out of the box, NAS4Free slightly better than FreeBSD 10)
Write 70~74 MBs
Read 85~95 MBs
FreeBSD 9 (out of the box)
Write ~43 MBs (min 25MBs max 46MBs)
Read ~63 MBs
FreeNAS (with Enabled autotune)
Write ~40 MBs (min 20MBs max 42MBs)
Read ~60 MBs stable
----------------
Scrubs run 130~140 MBs in all machines. There is not difference here. I think network is the problem. Iperf return ~500 Mbits/sec on FreeBSD 9 and FreeNAS, and >800 in NAS4Free and FreeBSD 10.

Clearly, Kernel 10 with integrated drivers has an edge over Kernel 9 + module drivers.
NAS4Free is a great product, don't mess with that. But I prefer FreeNAS jails and flexibility over speed and current kernel of NAS4Free. I can wait for version 10 FreeNAS for full speed. But in my domestic undocumented test FreeBSD 9 does better than FreeNAS for a little.
My question is: Who is eating my MBs? I don’t expect full speed but every MBs will be welcome. More Ram?, Any tweak? Other ideas?

Thanks you.
P.S. I don't want to initiate a flame war. And I don’t want to drastically change my configuration either (put more HDD , buy another server, change to VMWare etc.)
 

BERKUT

Explorer
Joined
Sep 22, 2015
Messages
70
Xeon E5504
HDD: 6x3TB (3 vdevs)
RAM: 24 GB
Network Intel VT quad 1G: 4 Gbits. (MTU 9000)
Standart FreeNas settings, with compression on, atime off
disk on windows allocated with 8k sector size.

I'm on node with Hyper-V and inside VM have this speed:
Write 90-100 MBytes/s (with SLOG) (random write different files),
Write 150 MBytes/s (with SLOG) seq big one file
Read ~300-480 MBytes/s
Read ~200-300 MBytes/s if big file


I think you have problem with network, 1 Gb network is too slow...
 
Last edited:

renegm

Cadet
Joined
Dec 18, 2015
Messages
6
Write 150 MBytes/s
I do not expect to see that speed. 300MBs neither (Max 1GB network is 119MBs). 150MB is the maximum for 4TB WD red http://www.wdc.com/wdproducts/library/?id=368&type=8&cn=2879-800002
Throwing another two drives on (4x4TB Raid Z1), will make a big difference I guess (never close of your numbers). My hardware is very similar to FreeNAS mini (different MB). In some future I will upgrade to metal. Maybe FreeNAS 10 will solve these issues but virtualization (with my hardware) is just good enough, but not awesome.
 

BERKUT

Explorer
Joined
Sep 22, 2015
Messages
70
150 MBytes is speed with SLOG, if disable SLOG speed write up to 300-400 MBytes/s.
1 GB NICs is too slow for FreeNas with 32 GB RAM, if you want stable fast speed look on 10 GB or use NIC with min 2 chipsets or also MPIO.

On this forum you can found this formule
If you have powerfull NIC
(vdevs speed MBytes/s) * (5 sec) / 0.125 = minimum RAM need.

If your bottleneck is NIC:
(NIC speed MBytes) * (5 sec) / 0.125 = minimum RAM need.

0.125 is 1/8 of RAM.
 

renegm

Cadet
Joined
Dec 18, 2015
Messages
6
You are right, but I can't afford 10GB right now (and is overkill too in my little home network). I'll do it when I grow up :). 32GB in this box is for virtualization purpose. I assigned 16GB to Freenas by equation 8GB + 1GB per TB. My current configuration is: This box (with FreeNAS permanently, and max 3 others temp on-off virtual machines), my desktop PC, 3 laptops and a few devices (phone, tablets, TV,...).
On other hand, with 10GB network we can see total throughput>150MBs but one single file is limited by disk speed (stripe drives). Intensive use of my LAN is between {PC and G-Box }<->FreeNAS.
In extra tests looks like samba performance is another peace of crap. NFS performance is much better than CIFS. (read and write single file through NFS >80MBs).
So, its everything: Virtualized enviroment (without VT-d) + OldKernel +Single Disks + Samba = Not awesome performance.
 

m0nkey_

MVP
Joined
Oct 27, 2015
Messages
2,739
Your biggest issue is lack of VT-d support on your board/CPU. Even with Hyper-V, you won't get direct access to the drives.
 

renegm

Cadet
Joined
Dec 18, 2015
Messages
6
Even with Hyper-V, you won't get direct access to the drives.
Well, I have direct access to the drives. I dont have device access to the drives. I mean, file system exist on drives directly. Not in some file representing a hard drive. If I decide take this drive and put in another PC or boot from usb/cd whatever they will works. Smart functions (in FreeNAS virtual machine) are gone without VT-d, but still exists in Hyper-V. Easy report of status:
PS C:\> Get-Disk 0 | Get-StorageReliabilityCounter

ObjectId : \\?\scsi#disk&ven_wdc&prod_wd40efrx-68wt0n0#4&2fd46013&0&010000#{53f56307-b6bf-11d0-94f2-00a0
c91efb8b}
PassThroughClass :
PassThroughIds :
PassThroughNamespace :
PassThroughServer :
UniqueId : \\?\scsi#disk&ven_wdc&prod_wd40efrx-68wt0n0#4&2fd46013&0&010000#{53f56307-b6bf-11d0-94f2-00a0
c91efb8b}
DeviceId : 0
FlushLatencyMax : 1531
LoadUnloadCycleCount : 368
LoadUnloadCycleCountMax :
ManufactureDate :
PowerOnHours : 431
ReadErrorsCorrected : 0
ReadErrorsTotal : 0
ReadErrorsUncorrected : 0
ReadLatencyMax : 863
StartStopCycleCount : 10
StartStopCycleCountMax :
Temperature : 31
TemperatureMax :
Wear :
WriteErrorsCorrected :
WriteErrorsTotal :
WriteErrorsUncorrected :
WriteLatencyMax : 827
 

m0nkey_

MVP
Joined
Oct 27, 2015
Messages
2,739
You have a software layer between the FreeNAS VM and the underling hardware. Hyper-V has access to the drives, Your FreeNAS VM does not since you don't have VT-d. What you're doing is presenting the drives a pseudo devices.

Hyper-V network legacy drivers sucks. And you need HV-storage driver for Physical HDD access. FreeBSD 10 contains drivers but FreeBSD 9 (FreeNAS kernel) don’t. So, what you need?

That alone tells me your FreeNAS VM does not have real drive access. You actually had to compile a kernel module to make it work.

While yes, it will work, there is no guarantee that it'll keep working because Hyper-V did not handle the request as expected. Just make sure you have a reliable backup and are not storing anything important.
 

renegm

Cadet
Joined
Dec 18, 2015
Messages
6
While yes, it will work, there is no guarantee that it'll keep working
"guarantee" is a very big word. Whatever you do, you never have guarantee, only odds.
I think the biggest problem with virtualization is the disks files. One error, a little mess with this file, and everything is corrupt. I dont use crypto, so with raw/direct access to disks, one error and something will go wrong until stop, like every metal enviroment. ZFS is demanding and greedy but not fragile or nobody would use it. I have (at work) developers/testings VM with PostgreSQL/FreeBSD. We have had trivial problems with this configuration and insoluble problems with virtual disks.
I have some confidence in the strength of solution. The performance is somewhat disappointing.

But yes, this VM FreeNAS purpose is: share some files, photos, music, video (phone, PCs or TV) (everything exists in others locations) and non critical local backup of PCs. I don't have any pacemaker running in a FreeNAS jail.
 

tvsjr

Guru
Joined
Aug 29, 2015
Messages
959
While you might get this to work, it will never have performance even close to a hardware solution. You will also have no monitoring, SMART testing, etc. of your drives... so you may have drives that fail silently, with your first alert being when the pool won't mount.

In short... this won't end well.
 

renegm

Cadet
Joined
Dec 18, 2015
Messages
6
My first post was 2 years ago. There were several penalties in my configuration caused by the lack of native drivers in FreeNAS. I had an improvement when I upgraded to FreeNAS 11 (native drivers) 6 months ago. But not impressive.
When the FreeNAS 11.1 came out I made several changes. And the result is wonderful.
I had upgraded the Hyper-V to 2016 but FreeNAS 11 and earlier were not compatible with the Gen2 VMs. FreeNAS 11.1 is compatible. This translates into several advantages that I will not comment, but native drivers on VM Gen2 is the best that can be achieved. So I installed the FreeNAS 11.1 in a new VM Gen2.
VM is more or less the same: 4 Cores, 16GB RAM, 2 Virtual Disks 40GB each (one Boot, one for the Jails) and 2 HDD 4TB direct attached.
Test results:
Y is a network drive pointing to a dataset in FreeNAS (compression lz4).
hYZbmEo.png

The result is impressive. The limit on a 1Gb network is 118.7 MB / s. And I have 118.5. This is not improvable.
By comparison, the test executed on a local HDD (1TB mechanical disk, 7200RPM)
COLc3Cr.png

Doing a "real job" Note CrystalDiskMark uses factors of 1000, Windows uses 1024, here the maximum limit would be 113MB/s.
Lvgyxte.png

50oncOk.png

I'm not saying that virtualizing is a good idea and I'm not recommending it. What I am saying is that TODAY there is no performance penalty for virtualizing FreeNAS. FreeNAS 11.1 on a VM Gen2 on Hyper-V 2016 will have the same performance as the FreeNAS 11.1 running on hardware with equivalent resources.
How stable is all this? Well, when I went to upgrade the Hyper-V from 2012 to Hyper-V 2016 I did not remember the admin password. So I would say it is very stable. And I have not had problems of any kind in these 2 years.
And the limit is reached by reading from FreeNAS.
 
Last edited by a moderator:
Status
Not open for further replies.
Top