Hello,
(first, my apologize for my English, it's not my mother tongue)
I'm currently configuring a new build and i'd like to get some feedback around it and using QEMU/KVM has hypervisor.
I read many thread talking about qemu or xen or proxmox but most of them are old or about small build which, from my understanding, don't have the same concern than mine.
The build :
Motherboard : Supermicro x9srl-f (https://www.supermicro.com/products/motherboard/Xeon/C600/X9SRL-F.cfm)
Cpu : Xeon E5-2650L v2
Memory : 8x16GB PC3L-12800R Samsung ( M393B2G70DB0-YK0 )
HBA : 3 x SAS2008 (M1015 flashed IT)
Case : IPC 4244 (not very happy with it, i'll change it for a Norco soon)
HDD : 12 x 8TB WD Red (WD80EFZX). I paid attention to buy drives to different suppliers to get different production batch.
Boot : 8GB usb drive
UPS not yet bought but i'll use one that will be controlled by FreeNAS
I'll also add a gpu later, looking for Vega.
After 3 days of memtest i'm currently burning disks (smartctl -t long before and after and badblocks wvs -b 4096)
HDD Temperature checked, 40°c max in burning, about 32°c idle, so far so good.
Also planned to stress cpu itself but after hdd due to DOA warranty on hdd (easier to manage with seller than manufacturer).
What will be stored :
How data will be served or used :
Few words about zpool :
After considering cost and played with BiduleOhm calculator (https://forums.freenas.org/index.php?threads/zfs-raid-size-and-reliability-calculator.28191/)
I decided to go to a Raidz3 of 12 drives.
Better reliability than 2 Z2 of 6 drives, more usable capacity and both of them aren't best practices for disk count per vdev.
But looks like compression feature mitigate this issue.
I also consider that backup of movies will represent a big part of my usable space and will not move often, mostly being read so acting as nearly "cold data".
Considering that i'll serve data by 1GbE connection, performance of the zpool shouldn't be an issue.
For different reason i'll crypt the pool. Never used that before (except in test with vm or small disks), read many thing about it on this forum and other FreeBSD sites.
Cpu support AES-NI so i don't expect a performance issue.
Resilver time and scrub will be my first test after filling about 50% of the zpool. If i'm not satisfied, or if someone here convince me to change something, i'll see to change that.
Hope that i don't forgot anything useful.
Considering that i'll not use deduplication and NFS only for few user on a 1GbE connection, i don't think that 128GB of memory is necessary.
I'd like to use 64GB for FreeNAS and keep the rest for Hypervisor and few VM.
Tried VMware but i don't like it. I hope to don't see this thread to become a contest between VMware vs world but :
I've a better experience with Qemu/KVM and despite many people warn on this forum about FreeNAS as a VM for using VMware (including Josh Paetzel blog post), i don't found any valuable reason to use it except it's more often used.
I'll not use proxmox or xen solution, i prefer a clean setup in CLI with minimal hypervisor system.
Considering all of this, i hope to get some feedback about kvm user about NIC card used for FreeNAS VM or any other parameter (cpu topology, etc) that could be useful.
Of course, hba will be passthrough to FreeNAS VM, 64GB of memory will be locked for it and nic will be bridged. That's the 3 recommendation i found for FreeNAS as a VM suitable for this setup.
Of course, any comment about the build, his setup of whatever you could say around that is welcome.
Best regards
Scotty
(first, my apologize for my English, it's not my mother tongue)
I'm currently configuring a new build and i'd like to get some feedback around it and using QEMU/KVM has hypervisor.
I read many thread talking about qemu or xen or proxmox but most of them are old or about small build which, from my understanding, don't have the same concern than mine.
The build :
Motherboard : Supermicro x9srl-f (https://www.supermicro.com/products/motherboard/Xeon/C600/X9SRL-F.cfm)
Cpu : Xeon E5-2650L v2
Memory : 8x16GB PC3L-12800R Samsung ( M393B2G70DB0-YK0 )
HBA : 3 x SAS2008 (M1015 flashed IT)
Case : IPC 4244 (not very happy with it, i'll change it for a Norco soon)
HDD : 12 x 8TB WD Red (WD80EFZX). I paid attention to buy drives to different suppliers to get different production batch.
Boot : 8GB usb drive
UPS not yet bought but i'll use one that will be controlled by FreeNAS
I'll also add a gpu later, looking for Vega.
After 3 days of memtest i'm currently burning disks (smartctl -t long before and after and badblocks wvs -b 4096)
HDD Temperature checked, 40°c max in burning, about 32°c idle, so far so good.
Also planned to stress cpu itself but after hdd due to DOA warranty on hdd (easier to manage with seller than manufacturer).
What will be stored :
- Mostly backup of movies, about 80-85% (of the 80% usable space)
- Also camera raw pictures, computers backup, music. Few VM
- oN big directory of small files (less than 1Mb each) or deep directories structure.
- Loosing data isn't a drama, movies are backup of original disc i own but converting films took a lot of time so i'll prefer to avoid such issue.
- Except movies, all other data are backed up often on another drives with another storage solution.
How data will be served or used :
- 1Gb connection (fair enough)
- few clients
- No deduplication
- NFS only (which means no iSCSI, no CIFS, etc)
- no zil or arc
Few words about zpool :
After considering cost and played with BiduleOhm calculator (https://forums.freenas.org/index.php?threads/zfs-raid-size-and-reliability-calculator.28191/)
I decided to go to a Raidz3 of 12 drives.
Better reliability than 2 Z2 of 6 drives, more usable capacity and both of them aren't best practices for disk count per vdev.
But looks like compression feature mitigate this issue.
I also consider that backup of movies will represent a big part of my usable space and will not move often, mostly being read so acting as nearly "cold data".
Considering that i'll serve data by 1GbE connection, performance of the zpool shouldn't be an issue.
For different reason i'll crypt the pool. Never used that before (except in test with vm or small disks), read many thing about it on this forum and other FreeBSD sites.
Cpu support AES-NI so i don't expect a performance issue.
Resilver time and scrub will be my first test after filling about 50% of the zpool. If i'm not satisfied, or if someone here convince me to change something, i'll see to change that.
Hope that i don't forgot anything useful.
Considering that i'll not use deduplication and NFS only for few user on a 1GbE connection, i don't think that 128GB of memory is necessary.
I'd like to use 64GB for FreeNAS and keep the rest for Hypervisor and few VM.
Tried VMware but i don't like it. I hope to don't see this thread to become a contest between VMware vs world but :
- It's not an open software (it's important for me)
- Free edition of VMware don't allow more than 8 vcpu per vm
- It don't allow to boot a VM from a usb drive (just in case, to be able to boot FreeNAS directly, of course i'll backup the config but i prefer this way to switch between Qemu and FreeNAS)
- VM act as a single thread on the hypervisor (which may be a problem if you over allocate cpu)
- Afaik vfio vgpu feature of Qemu/KVM as no equivalent on VMware (i may have forgot this feature if it was the only problem) some info about vfio vgpu here VGPU on KVM
I've a better experience with Qemu/KVM and despite many people warn on this forum about FreeNAS as a VM for using VMware (including Josh Paetzel blog post), i don't found any valuable reason to use it except it's more often used.
I'll not use proxmox or xen solution, i prefer a clean setup in CLI with minimal hypervisor system.
Considering all of this, i hope to get some feedback about kvm user about NIC card used for FreeNAS VM or any other parameter (cpu topology, etc) that could be useful.
Of course, hba will be passthrough to FreeNAS VM, 64GB of memory will be locked for it and nic will be bridged. That's the 3 recommendation i found for FreeNAS as a VM suitable for this setup.
Of course, any comment about the build, his setup of whatever you could say around that is welcome.
Best regards
Scotty
