- Joined
- Nov 25, 2013
- Messages
- 7,776
You can boot ESXi and have a VMFS on the same drive. Store the boot disk for TrueNAS there. Store all other VMDK on the TrueNAS provided storage.
-
Important Announcement for the TrueNAS Community.
The TrueNAS Community has now been moved. This forum will now become READ-ONLY for historical purposes. Please feel free to join us on the new TrueNAS Community Forums
hypervisor + virtualized Truenas
- Thread starter phier
- Start date
@Patrick M. Hausen oh i see, thanks! just question would be what "size" is optimal for ESXi with just one Truenas VM, and all other vmfs will be on truenas so its okay.
- Joined
- Nov 25, 2013
- Messages
- 7,776
The minimum for VMFS on the boot drive with ESXi 7.x is 128G if I remember correctly. It does like to grab a large part of that for an internal scratch area or some such. So I'd recommend using at least 256G.
Well, okay - makes sense so i think i would go for 512GB NVME, but question would be if Transcend is okay or its better to go for something really good as samsung
was thinking about https://www.amazon.de/-/en/dp/B07CXBXHY9/ref=twister_B07NLKR98D?_encoding=UTF8&th=1
this samsung is almost same as Transcend
but that samsung is way much much expensive
esxi spec
docs.vmware.com
ESXi 7.0 requires a boot disk of at least 32 GB of persistent storage such as HDD, SSD, or NVMe. Use USB, SD and non-USB flash media devices only for ESXi boot bank partitions. A boot device must not be shared between ESXi hosts.
was thinking about https://www.amazon.de/-/en/dp/B07CXBXHY9/ref=twister_B07NLKR98D?_encoding=UTF8&th=1
this samsung is almost same as Transcend
but that samsung is way much much expensive
esxi spec
ESXi Hardware Requirements
Make sure that the host meets the minimum hardware configurations supported by ESXi 7.0.
ESXi 7.0 requires a boot disk of at least 32 GB of persistent storage such as HDD, SSD, or NVMe. Use USB, SD and non-USB flash media devices only for ESXi boot bank partitions. A boot device must not be shared between ESXi hosts.
- Joined
- Nov 25, 2013
- Messages
- 7,776
Check out the Samsung 970 EVO Plus - the Plus is important.
I use a 32 G Transcend to boot ESXi and a 256 G EVO Plus for local VM storage. Two more EVO Plus passed through to TrueNAS.
I use a 32 G Transcend to boot ESXi and a 256 G EVO Plus for local VM storage. Two more EVO Plus passed through to TrueNAS.
@Patrick M. Hausen last time you wrote
The minimum for VMFS on the boot drive with ESXi 7.x is 128G if I remember correctly. It does like to grab a large part of that for an internal scratch area or some such. So I'd recommend using at least 256G.
Bit confusing, all i need is to boot ESXi and all VMS images store on truenas drives, in that case no point to have these samsung drives?
edit2:
Or maybe it would be better to have one NVME with ESXi + truenas image; then all other VMs on truenas drives.
The minimum for VMFS on the boot drive with ESXi 7.x is 128G if I remember correctly. It does like to grab a large part of that for an internal scratch area or some such. So I'd recommend using at least 256G.
Bit confusing, all i need is to boot ESXi and all VMS images store on truenas drives, in that case no point to have these samsung drives?
edit2:
Or maybe it would be better to have one NVME with ESXi + truenas image; then all other VMs on truenas drives.
Last edited:
- Joined
- Nov 25, 2013
- Messages
- 7,776
You can boot ESXi from 32 G. You cited that part of the documentation yourself. If you want to have a VMFS for VMs on the same drive the minimum is 128 G. But at least ESXi 7.x will waste at least half of that for internal purposes, so better to boot from a smaller drive and use a second one for datastore1 VMFS. My opinion, of course.
ESXi is a hypervisor, it does not support containers. What exactly do you want to run? That should define the platform to pick.
ESXi is a hypervisor, it does not support containers. What exactly do you want to run? That should define the platform to pick.
@Patrick M. Hausen
not sure if u saw my last edit>
edit2:
Or maybe it would be better to have one NVME with ESXi + truenas image; then all other VMs on truenas drives.
what do u mean by
..use a second one for datastore1 VMFS.
1x 32GB to boot Esxi
1x 128Gb to store Truenas VM
truenas pool to host other VMS + data
is that correct?
well regarding what i want to run... hard to say but i was thinking
1) windows OS (as an desktop - here the issue would be the peripherals - ie via network lets say stream 4k resolution maybe impossible) - so maybe not doable
2) ~2 ubuntu machines just for webserver/ filesharing/ etc
3) roon-server (music app)
4) OSX (if possible to emulate without issue) - not sure about that
5) pfsense
6) plex
3)6) for example would be nice to run as a containers - because you never know ahead how much ram will be required
thanks!
not sure if u saw my last edit>
edit2:
Or maybe it would be better to have one NVME with ESXi + truenas image; then all other VMs on truenas drives.
what do u mean by
..use a second one for datastore1 VMFS.
1x 32GB to boot Esxi
1x 128Gb to store Truenas VM
truenas pool to host other VMS + data
is that correct?
well regarding what i want to run... hard to say but i was thinking
1) windows OS (as an desktop - here the issue would be the peripherals - ie via network lets say stream 4k resolution maybe impossible) - so maybe not doable
2) ~2 ubuntu machines just for webserver/ filesharing/ etc
3) roon-server (music app)
4) OSX (if possible to emulate without issue) - not sure about that
5) pfsense
6) plex
3)6) for example would be nice to run as a containers - because you never know ahead how much ram will be required
thanks!
also i can see for example that proxmox can do VM + Linux Containers (LXC)
or maybe i can run a linux VM on ESXi and inside that VM LXC
www.claudiokuenzler.com
or maybe i can run a linux VM on ESXi and inside that VM LXC
Setting up a LXC host on Ubuntu 18.04 in a VMware ESXi virtual machine environment
Yet another tech blog - made in Switzerland.
- Joined
- Nov 25, 2013
- Messages
- 7,776
If you install ESXi on at least 128 G it will create not only the ESXi boot partitions but also a datastore for VM images. But it will waste a lot of space and you will get only 20-30 G for VMs if I remember correctly.
If you install ESXi on 32 G and use a separate SSD for the VM images, you have more room.
If you run TrueNAS CORE on your hardware and use jails for most of your applications you can utilize up zo 70-80% of your capacity.
If you install ESXi on 32 G and use a separate SSD for the VM images, you have more room.
So can TrueNAS. CORE can do VMs and FreeBSD native containers called jails. SCALE can do VMs and Docker/containerd. And yes, there's Proxmox. Pick one. What do you gain by another level of virtualisation? If you use TrueNAS to provide storage to VMs in ESXi via iSCSI you should use at most 50% of the capacity. See the excellent resources on block storage by @jgreco
If you run TrueNAS CORE on your hardware and use jails for most of your applications you can utilize up zo 70-80% of your capacity.
Well, still not clear whats datastore for VMs? Is it store for VMs images? I thought it will be better to store VMs images to truenas pool?
that's whats written in the spec:
Other options for best performance of an ESXi 7.0 installation are the following:
Yes, but these apps i mentioned - not so many are ported to be able to run in jails which imply issue.
Also you can run bhyve VMs - which also can be an issue in some cases.
Another level? Well as I said - some apps as roon,plex there is no point to run these in VMs, its better to use containers as LXC.
Not sure here,
If you use TrueNAS to provide storage to VMs in ESXi via iSCSI you should use at most 50% of the capacity. See the excellent resources on block storage by @jgreco
If you run TrueNAS CORE on your hardware and use jails for most of your applications you can utilize up zo 70-80% of your capacity.
Why I cant use TrueNAS to provide storage to VMs in ESXi for more then 50%? And in parallel store on TrueNAS other data/media to utilize 70-80% of pool storage?
Did you mean
www.truenas.com
?
Plus still not clear why ppl run TrueNAS virtualized via ESXi, if it supports VMs, containers...
whats the specific requirement to have ESXi, instead of using bhyve inside TrueNAS?
thanks
that's whats written in the spec:
Other options for best performance of an ESXi 7.0 installation are the following:
- A local disk of 128 GB or larger for optimal support of ESX-OSData. The disk contains the boot partition, ESX-OSData volume and a VMFS datastore.
- A device that supports the minimum of 128 terabytes written (TBW).
- A device that delivers at least 100 MB/s of sequential write speed.
Yes, but these apps i mentioned - not so many are ported to be able to run in jails which imply issue.
Also you can run bhyve VMs - which also can be an issue in some cases.
Another level? Well as I said - some apps as roon,plex there is no point to run these in VMs, its better to use containers as LXC.
Not sure here,
If you use TrueNAS to provide storage to VMs in ESXi via iSCSI you should use at most 50% of the capacity. See the excellent resources on block storage by @jgreco
If you run TrueNAS CORE on your hardware and use jails for most of your applications you can utilize up zo 70-80% of your capacity.
Why I cant use TrueNAS to provide storage to VMs in ESXi for more then 50%? And in parallel store on TrueNAS other data/media to utilize 70-80% of pool storage?
Did you mean
Some differences between RAIDZ and mirrors, and why we use mirrors for block storage
ZFS is a complicated, powerful system. Unfortunately, it isn't actually magic, and there's a lot of opportunity for disappointment if you don't understand what's going on. RAIDZ (including Z2, Z3) is good for storing large sequential files...
www.truenas.com
Plus still not clear why ppl run TrueNAS virtualized via ESXi, if it supports VMs, containers...
whats the specific requirement to have ESXi, instead of using bhyve inside TrueNAS?
thanks
- Joined
- Nov 25, 2013
- Messages
- 7,776
I use bhyve in TrueNAS in production. I don't know what other people do. I use ESXi for legacy VMs. bhyve does not support 32 bit guests, legacy BIOS boot and a bunch of other things. I run Windows 10, Windows Server 2016 and Ubuntu 18 and 20 in bhyve in production.
Yes, a datastore is for VMs. ESXi has to put them somewhere, right? And instead of going through all this complex TrueNAS in ESXi and iSCSI stuff I run my VMs from a local SSD with sufficient TBW specs and snapshot them every night to an NFS datastore on a TrueNAS system. A separate TrueNAS system that also runs my bhyve VMs and my jails. Not virtualised.
I can only answer your questions if you ask "can TrueNAS do X?" or "can it run in a hypervisor?"
I cannot tell you why you should run one product or another. I am sure Proxmox is a perfectly capable platform. I don't use it because I work on and actively support FreeBSD.
The link you posted is the article I was referring to. If you serve iSCSI to ESXi that is block storage. jgreco writes:
If you run applications in a jail on CORE or Linux containers on SCALE, that's file storage. It's fundamentally different. You can use a way higher percentage of your capacity.
Yes, a datastore is for VMs. ESXi has to put them somewhere, right? And instead of going through all this complex TrueNAS in ESXi and iSCSI stuff I run my VMs from a local SSD with sufficient TBW specs and snapshot them every night to an NFS datastore on a TrueNAS system. A separate TrueNAS system that also runs my bhyve VMs and my jails. Not virtualised.
I can only answer your questions if you ask "can TrueNAS do X?" or "can it run in a hypervisor?"
I cannot tell you why you should run one product or another. I am sure Proxmox is a perfectly capable platform. I don't use it because I work on and actively support FreeBSD.
The link you posted is the article I was referring to. If you serve iSCSI to ESXi that is block storage. jgreco writes:
If you run applications in a jail on CORE or Linux containers on SCALE, that's file storage. It's fundamentally different. You can use a way higher percentage of your capacity.
This seems to be a more advanced topic than i thought.
If i can ask, how do you access win10 via VNCserver/client?
Regarding the datastore - got your point, but you have to somehow size that SSD beforehand, once you want to add more VMs as you initially designed the SSD capacity - you have to replace SSD for a bigger one or add a new one ... so that's why :complex: route made more sense to me...
Also, it looks like SCALE is not for free.
If i can ask, how do you access win10 via VNCserver/client?
Regarding the datastore - got your point, but you have to somehow size that SSD beforehand, once you want to add more VMs as you initially designed the SSD capacity - you have to replace SSD for a bigger one or add a new one ... so that's why :complex: route made more sense to me...
Also, it looks like SCALE is not for free.
- Joined
- Nov 25, 2013
- Messages
- 7,776
You can replace storage drives with larger ones all the same with or without an extra hypervisor. And SCALE is free just like CORE.
I access Windows systems with Microsoft Remote Desktop. And Linux systems with SSH, of course.
I access Windows systems with Microsoft Remote Desktop. And Linux systems with SSH, of course.
- Joined
- Mar 6, 2014
- Messages
- 9,554
+1 on using RDP to manage windows VMs. Most RDP clients support drag-and-drop. There have been some times I've done ad-hoc MacOS / Windows interop tests by running RDP client on MacOS and dragging / dropping files to a SMB share through Windows via RDP session to Windows client. The MS RDP client generally works _very_ well.
@Patrick M. Hausen so i wanted to prevent neverending drives replacement regarding the size... if there is a pool.
well still not clear whats diff or purpose of CORE vs SCALE
@andos what max resolution you can get via RDP - can u get 4k?
thanks
well still not clear whats diff or purpose of CORE vs SCALE
@andos what max resolution you can get via RDP - can u get 4k?
thanks
- Joined
- Nov 25, 2013
- Messages
- 7,776
Practically any resolution the local display you are using supports with RDP:
CORE: FreeBSD, bhyve VMs, jails. No Docker unless you deploy e.g. an Ubuntu VM. Well tried and tested. Still supported, but not many new features expected.
SCALE: Linux, KVM VMs, Docker via k3s/Helm, GlusterFS. No LXC! Rather new-ish and still in development.
www.truenas.com
Pick one.
HTH,
Patrick
Edit: I saw in another thread (no idea about the fans, sorry) that you are already running FreeNAS 9. TrueNAS CORE is really FreeNAS, 3 major versions later (9 --> 11 --> 12 --> 13). FreeNAS --> TrueNAS CORE was just a renaming. This is the same product. TrueNAS SCALE on the other hand is a completely new implementation based on Debian Linux.
CORE: FreeBSD, bhyve VMs, jails. No Docker unless you deploy e.g. an Ubuntu VM. Well tried and tested. Still supported, but not many new features expected.
SCALE: Linux, KVM VMs, Docker via k3s/Helm, GlusterFS. No LXC! Rather new-ish and still in development.
Compare TrueNAS Editions - Powerful Storage Platform
TrueNAS is the ultimate Open Source storage platform. Choose from CORE, Enterprise, or SCALE to get the best performance and reliability.
www.truenas.com
Pick one.
Virtualising in ESXi won't help you in this regard. And you will be getting less usable space from your block storage pool than with either CORE or SCALE.
HTH,
Patrick
Edit: I saw in another thread (no idea about the fans, sorry) that you are already running FreeNAS 9. TrueNAS CORE is really FreeNAS, 3 major versions later (9 --> 11 --> 12 --> 13). FreeNAS --> TrueNAS CORE was just a renaming. This is the same product. TrueNAS SCALE on the other hand is a completely new implementation based on Debian Linux.
Last edited:
Well,
i am still not sure how to "design" size of the drive for VMs, how can u know forehead hw much GB u will need...
Still thinking to go
CORE barebone
and inside CORE one VM ubuntu and inside Proxmox to run containers for some specific apps. Question would be how bhyve will be able to pass through USB ports / etc to Ubuntu VM.
Also one more point; in case described above... still seems its not a good idea to store these VMs (ubuntu, win) on truenas drive... is that correct? SO in that case i have to mount separate drive in Truenas where i store only VMs data?
thanks
i am still not sure how to "design" size of the drive for VMs, how can u know forehead hw much GB u will need...
Still thinking to go
CORE barebone
and inside CORE one VM ubuntu and inside Proxmox to run containers for some specific apps. Question would be how bhyve will be able to pass through USB ports / etc to Ubuntu VM.
Also one more point; in case described above... still seems its not a good idea to store these VMs (ubuntu, win) on truenas drive... is that correct? SO in that case i have to mount separate drive in Truenas where i store only VMs data?
thanks
