Should I stop using jails?

[Nick2253]

@Nick2253 ... errrr ... Bhyve is a hypervisor. And I am pretty darn serious about my home network. Just ordered a fresh Supermicro 8-core Atom based server with 64 GB of RAM to have more breathing room for VMs.

Obviously bhyve is a hypervisor. I'm not sure what you're getting at, and I don't see where I've said or implied otherwise. I think you have somehow assumed that my comment was specifically directed at you, when it was aimed at providing an additional point of view for the OP who is specifically exploring this exact issue, and is looking for recommendations about what to run.

I could have included bhyve in the list of hypervisors, but I tried to focus on products that are either major players in the virtualization market or ones that I have first-hand experience with.

 

[Chris Moore]

@Nick2253 ... errrr ... Bhyve is a hypervisor. And I am pretty darn serious about my home network. Just ordered a fresh Supermicro 8-core Atom based server with 64 GB of RAM to have more breathing room for VMs.

Patrick

Bhyve is a new hypervisor (only since BSD 10) that is not widely known or widely supported and it is still in development, trying to get to a usable state.
You might not agree with this or like it, but that's what it looks like.

Sent from my SAMSUNG-SGH-I537 using Tapatalk

 

[joeschmuck]

Bhyve is a new hypervisor (only since BSD 10) that is not widely known or widely supported and it is still in development, trying to get to a usable state.
You might not agree with this or like it, but that's what it looks like.

I concur, I know Bhyve is a new player in this market and thus it's not as mature as the other players and recommending this to anyone is a tuff sell in my mind, well except to someone who is an "early adopter". Early adopters jump into a project fully expecting to find problems and expect that anything can and will go wrong. The group of people actually using the software will be considerably smaller so this means your ability to find answers will be limited to that much smaller group. But as for Bhyve being considered a stable product, I think it needs some time to prove itself.

 

[diskdiddler]

They claim.... That iocage jails (plugins? Same thing?) well be more easily maintained and have a larger library eventually.

I don't know if this is true or not, but at someone with a server already near processing capacity, I'd definitely like to hope that I can keep using jails. I can't afford the rancheros cpu overhead for Docker containers unfortunately.

 

[mow4cash]

They claim.... That iocage jails (plugins? Same thing?) well be more easily maintained and have a larger library eventually.

FreeBSD already has a huge library, so plugins while nice to have are not a game changer. Some stuff simply won't run on FreeNAS and that is the bigger issue to fix. Maybe it's me but Ranchero is to hard to get working with storage and permissions. The FreeNAS ranchero documentation is great but leaves out storage and permissions. If FreeNAS easy plugins are used then an easier Docker implementation should also be.

 

[KrisBee]

There was mention of a “new virtualization platform” earlier in this thread, I don’t know what that might be except continued use of bhyve and iocage in FreeNAS.

A few points about bhyve:

Bhyve – flaky vnc with lack of non-us keymap support. Makes vm installs tricky as times. (workaround use ssh and/or install vnc server in installed vm, or use rdp ASAP). These problems have hung around for at least 6 months or more.
Virtio nic is preferred to emulated intel NIC for Windows bhyve guests.
No equivalent of guest tools as per vmware or KVM/qemu.
Question marks over timing issues inside bhyve guests.
Bhyve limited to booting guest via one variant of the UEFI firmware or grub-boot loader. Second CSM variant of UEFI firmware does not work, no upstream fix.
FreeNAS implementation only recently looked at how stopping bhyve vms might corrupt guest filesystems. Now there’s both a stop and power off button on the GUI.
Bhyve on AMD CPUs not using all cores correctly, another problem that’s hung around for months.

About docker:

Implementation of Docker-VM leaves users to grapple with Rancher. The base rancheros is not that resource heavy, but the RancherUI is. Getting the all important rancher-nfs working reliably can be a challenge. Implementation of rancheros is not user upgradeable due to limitation of bhyve grub boot, currently it’s v1.1.3 in FreeNAS, but there have been two further releases of rancheros which is now at 1.2.0 with latest docker version etc.

For small docker app usage a DIY combo of debian netinstall plus latest docker, docker compose and portainer might give users a better experience. It’s stable and NFS mounts are more straightforward and all software is user upgradeable.

It's not surprising that serious vm users have turned to ESXI, KVM /qemu etc.

 

[KrisBee]

@mow4cash Just to add, I 'd like see to an honest appraisal of the strengths and weaknesses of bhyve and a clear(er) indication of what "virutalisation" might look like in 12 months time on FreeNAS.

The move from warden to iocage to jails looks messy and could be painful for some. Is the inclusion of a "docker-vm" just to keep the Corral crowd on board and provide another tick box of the list of FreeNAS features? Or is there any serious intent here, like resurrecting pf9?

Right now, as always, it's a bit of a techno guessing game. Stick or twist? Live with the limitations as they are and are likely to be for several months, or look at a virtualised FreeNAS install in an "all in one" setup using perhaps ESXI, or keep FreeNAS strictly for storage back end and use a 2nd server for apps.

 

[Chris Moore]

There was mention of a “new virtualization platform” earlier in this thread, I don’t know what that might be except continued use of bhyve and iocage in FreeNAS.

The bhyve / iocage solution is what I think of as the "new virtualization platform" and it is still in development, hence the number of faults. There was a time, back around version 9.3.x and before on FreeNAS, where we had a VirtualBox implementation that worked reasonably well. That is what I think of as the, 'old' virtualization platform and it may just be me, but that is what I had in mind. Sorry if I was not clear in my meaning.

 

[diskdiddler]

FreeBSD already has a huge library, so plugins while nice to have are not a game changer. Some stuff simply won't run on FreeNAS and that is the bigger issue to fix. Maybe it's me but Ranchero is to hard to get working with storage and permissions. The FreeNAS ranchero documentation is great but leaves out storage and permissions. If FreeNAS easy plugins are used then an easier Docker implementation should also be.

The quantity of warden plugins is lite,.... rarely updated.

The (current listing) of iocage plugins seems hardly any better?

Where is the huge library you mention?

 

[INCSlayer]

The quantity of warden plugins is lite,.... rarely updated.

The (current listing) of iocage plugins seems hardly any better?

Where is the huge library you mention?

the difference there is that making a warden plugin required a very VERY specific set of software to be able to actually create them making them actually hard to create unless you where very versed in that software.
Whereas the iocage system is vastly simpler to create plugins for of course since iocage is not fully "in" yet people havent really looked into creating new plugins for it.
The old system was very hard to create plugins and hadnt really changed the plugins in ages the new system already basically has plugin parity and hasnt even fully launched yet tbh. To me that says that plugins for iocage should be a lot easier to for people to create once the system is fully supported.

 

[Patrick M. Hausen]

Where is the huge library you mention?

https://www.freshports.org

What's keeping you from creating a plain standard jail and typing e.g.

Code:

pkg install nextcloud

?

Upgrading the jail is as simple as

Code:

iocage upgrade ...
iocage console ...
pkg upgrade

I don't miss the old plugin system at all. Whenever I tried to actually use pre-packaged plugins the system tended to get in my way rather than help. Or to put it differently: the closer the new jails are to stock FreeBSD installations the easier it will be for everyone to manage them.

Patrick

 

[adrianwi]

https://www.freshports.org

What's keeping you from creating a plain standard jail and typing e.g.

Code:

pkg install nextcloud

?

If only it were that simple :) The benefits the plugins had over a standard jail is that someone has done the configuration for you. Whilst the pkg for nextcloud might include some of the dependencies, it won't configure them for you.

That said, I think it's worth learning a little about how jails and FreeBSD work, as I agree that building these yourself is better than relying on others to update the plugins, at least the way things currently are.

 

[mow4cash]

@mow4cash
Right now, as always, it's a bit of a techno guessing game.

One thing I miss about Co**al was the transparency Jordan Hubbard gave us. The project was a mess but I enjoyed the transparency. I understand this may not be possible now but it was appreciated. The only thing we have to go off of is tidbits from tickets we have to search for.

FreeNAS really needs to think about their future moving forward. If you are only going to do storage and do it well why even bother with plugins, jails, virtualization. If you are adding those features you need to decide how to implement them correctly. I feel FreeNAS is in limbo and will ultimately lose users in the future. I'm not sure what TrueNAS users require but I would assume for small business an all in one approach would be a nice selling feature.... the other users would probably be happy if they did away with any features above your basic storage needs. Trying to please both makes for a less desirable product IMO. Modules are a nice idea if features are going to half donkey, you could install FreeNAS storage base and at the command line install additional beta features (let's be honest plugins should be BETA even though they have been around forever)

 

[diskdiddler]

If only it were that simple :) The benefits the plugins had over a standard jail is that someone has done the configuration for you. Whilst the pkg for nextcloud might include some of the dependencies, it won't configure them for you.

That said, I think it's worth learning a little about how jails and FreeBSD work, as I agree that building these yourself is better than relying on others to update the plugins, at least the way things currently are.

The list on Freshports is definitely better than the freenas approved ones, but not perfect. I emailed the guy who maintains QBittorrent and he said he'd patch it up to V4 nearly 4 weeks ago.

So at this point, it seems I should just learn how to create a jail and install stuff direct from Freshports? It's certainly not ideal vs the Warden plugin system which was pretty darn user easy. (although obviously a smaller pool)

 

[Patrick M. Hausen]

So at this point, it seems I should just learn how to create a jail and install stuff direct from Freshports?

Not quite. You simply pkg install inside your jail. Freshports is just a searchable directory of FreeBSD ports/packages.

Patrick