Important announcement regarding FreeNAS Corral

[Ericloewe]

I wanted to vent a little about Corral since I was one of the early adopters and contributors to docker container scripts.

The only reason I was one of the earlier adopters of Freenas Corral was because of the hyperconverged architecture and Docker integration. The cherry on top was the UI...and as bugs were fixed the UI got better and better.

I saw a comment that there were so many bugs in Corral and it would have taken it a year to get up to the stability of 9.x. Well, what do you expect when you are designing a system from scratch. The expectation that a brand new system would go to production without some bugs is unrealistic ... the main reason agile methodology was invented to replace waterfall development... unrealistic expectations.

If performance was the major issue with Corral, performance testing should have been done before the official release.

To me, there has to be more to the story. Looks to me like innovation wasn't given a chance. Jordan Hubbard was a very talented man and he is no longer with iXSystems.. and the development team (middleware and ui) were talented as well. Too bad to see all that great work be a so-called "failure".

If it was a matter of lots of simple bugs, I'm sure Corral would still be in development. Clearly it was something more difficult to deal with than that.

 

[Gcon]

tl;dr - Corral was overly ambitious; rushed and very poorly tested with numerous hare-brained engineering decisions. Like a freakish three-headed baby goat - it had to be taken around behind the back shed and shot (in all three heads - so it would stay dead).

"Corral" was a dumb name IMHO. Phonetically too similar to the Canadian software company Corel. Good riddance - never use that name again (well it's tarnished as badly as United Airlines so I doubt you ever will, but just saying...)

Also FreeNAS started flying the roflcopter when they went from 9.3 to 9.10 - a point release that really wasn't, because it made me redo my entire suite of jails! We all know that 9.10 should have been called 10.0 due to this (and if you didn't then you're a dunderhead), but people cut iXSystems some slack because "10" was being reserved for a wonderful OS that was just around the corner (and 10 is a revered number reserved for wonderful OS'es right?) The irony is that the next version didn't even use the "10" but went with the silly "Corral" name *eyes roll*

You need to get back to basic software engineering procedures. If an upgrade requires me to have to redo large parts of my environment (such as all my jails), then it should be a *major* version bump. This much should be obvious. Feature updates are minor releases. Bug-fixes only a secondary points (like 10.1.2 to 10.1.3) etc. Seriously - it's not hard.

It seems with the next version you are digging an even deeper hole of stupidity to step into by this time not even incrementing the minor release and instead going with "9.10.3". What's the point (pardon the pun) of having the major and minor numbers then? *shakes head* Just do this - call the first FreeNAS that incorporates FreeBSD 11 (and I assume breaks the jails setup as it's a newer jail template) "FreeNAS 11.0" and start incrementing sensibly again from there. Even if it doesn't break jails, it's still a good move. It'll get you passed the nonsensical "9.11" hangups in any case.

Land that roflcopter already and refuel before you crash and burn (although with the dev exodus I suspect you might have done that already).

 

[educatedwarrior]

They definitely should have supported jails and docker containers in parallel. They should have supported the existing infrastructure and built on top of it....Therefore it would have gave those using the old version more time and the option to convert to jails or not.

The stability and performance of 9.X and the innovation of Corral would have made a good combination. Where it goes from here, we'll have to wait and see.

 

[drb]

It seems with the next version you are digging an even deeper hole of stupidity to step into by this time not even incrementing the minor release and instead going with "9.10.3". What's the point (pardon the pun) of having the major and minor numbers then? *shakes head* Just do this - call the first FreeNAS that incorporates FreeBSD 11 (and I assume breaks the jails setup as it's a newer jail template) "FreeNAS 11.0" and start incrementing sensibly again from there. Even if it doesn't break jails, it's still a good move. It'll get you passed the nonsensical "9.11" hangups in any case.

I agree! IX Systems needs to prove that they will not follow the Corral fiasco with another fiasco. IX Systems needs to have a rock solid and stable train if they are to regain people's trust.

The proposed 9.10.3 needs to be a new train, 11.0.0.

I would be willing to bet that there are a number of users who also feel that 9.10.2 broke too many things. Leave 9.10 alone, give it bug fixes and security updates only. Perhaps even remove the changes 9.10.2 introduced to prepare for the proposed 9.10.3 that broke things in 9.10.1. Call it 9.10.4 so there is no confusion.

IX Systems has two diverse objectives they must work towards now if they are to regain user's trust:

1. Have a rock solid stable version that does not break things with point releases to satisfy those not needing byhve and docker, but wanting to maintain their systems and jails as they were earlier in 9.10.

2. Migrate features from Coral to satisfy those users who bought in to the promise of bhyve and docker and the new UI.

The only way to accomplish both objectives is with two trains.

 

[Jailer]

The only way to accomplish both objectives is with two trains.

Did you happen to notice the outcome of their previous attempt at that?

One train, more and thorough testing, a sensible naming convention while maintaining transparent communication with the community. That's the winning combination.

 

[drb]

Did you happen to notice the outcome of their previous attempt at that?

One train, more and thorough testing, a sensible naming convention while maintaining transparent communication with the community. That's the winning combination.

I am not proposing 2 trains forever, but just what they were doing before Corral.

Did the 7 train go EOL the day that 8 train came out? Did the 8 train go EOL the day that the 9 train came out?

There needs to be overlap for people to test and plan a migration strategy.

The problem right now is a lot of Corral users expecting the features of Corral in a stable train NOW. That puts a lot of pressure on IX Systems to fix it now.

FreeBSD 10.3 is supported until April 30, 2018. FreeNAS 9.10 should not be dropped because FreeBSD 11 is out. 9.10.3 proposes just that.

It is time to create FreeNAS 11, keeping with the old naming convention. FreeNAS 11 would then become an upgrade path for users of both 9.10 and Corral.

 

[ajschot]

I understand the point of view, but still FreeNAS Corral had me more to offer then 9.10... so I will stay on Corral until dockers are arrived at 9.10 because the crashplan docker app is so great to backup my nas and that is not possible on the jail version. I can only use it as crashplan host.

Also I was in the beta testing too and this versions are the most stable.
Also I do think there are some old points of view in the 9.10 edition as are on nas4free.
Just have 9.10 nighties on my esxi but I see already a lot of GUI problems in that version. Also I don't think it is faster, I feel Corral is faster only maybe only on new systems...... Also there are not so much jails as there are Docker containers besides that they are all very old and really need updates.... that is much easier with dockers. Also cli is so simple to use. So I will stay on 10.0.0.4 for now I miss more in 9.10 then I missed in 10 from 9.10......

 

[gbooker]

I'm am now very thankful I took the route I did on my FN box.

After having created a docker container for Plex, I figured I should actually use it myself. I didn't want to wait for FN 10 (now called Corral) to be released, so in January, I created an Ubuntu 16.04 VM in FN 9.10 using iohyve. I then proceeded to create a docker-compose file for all of the plugins and processes I previously ran in jails to run in containers. Additionally I used ZFS within the VM to snapshot and replicate the containers' config directories to the backup system. The curious can see more details of my experience here and here. The short of it is that one jail in particular, UniFi, caused more headache and frustration than the entire process of creating a VM from scratch and deploying all my containers within it. I'll never go back to jails now.

Then, after a while of running FN on the metal, no jails, and containers in an Ubuntu VM, I started to question whether I should run ESXi on the metal, passthrough the HBA cards to FN, and run Ubuntu as a peer to FN. I questioned this more and more after Corral with reports I saw of loss CPU efficiency inside bhyve VMs under FN.

When Corral came out I decided I was not in a hurry to upgrade but I'd still conduct some tests. In these tests, I discovered that ACLs were not respected across to the docker containers along with other issues. I reported details in the bug report and a little over a week later I noticed a comment about it being the most "notesy" so that a note could be added to the report. That struck me as odd until I saw Corral's relegation a week later. Now I have to wonder if this report is one of the references to "enterprise-quality file access" missing in 9PFS.

Now, I'm sad to say, I'm revisiting my idea of virtualizing FN, but with an entirely different goal. I'm starting to think that I'll be better off moving over to Linux for my storage needs. My current barrier is that my pool is encrypted and Linux cannot read GELI volumes, so I'd have to zfs send the data to another machine. So by virtualizing FN, buying new drives for a new pool, giving some HBAs to FN and others to Linux, I could then zfs send all my data over to Linux. This is a very very serious consideration now and not one I was considering as strongly before the Corral announcement. I truly miss some aspects of running a Linux server, such as FS change notifications that actually work and also update SMB clients as well. There's also the fact that more and more OpenZFS developers are targeting Linux as their primary dev environment than others. I'd miss the automatic snapshot/replication and smart reports that FN offers, but Linux is getting better at comparable scripts.

So I'll wait until ZFS encryption lands in Ubuntu before I do anything which is likely 6-12 months away with either 17.10 or 18.04. After that, I'm sad to say, I'll likely be leaving FN. The maturity of OpenZFS on Linux has made great strides and this fiasco has just caused me to lose confidence in FN's going forward.

 

[ajschot]

It is time to create FreeNAS 11, keeping with the old naming convention. FreeNAS 11 would then become an upgrade path for users of both 9.10 and Corral.

FreeNAS 11?? That would be another 2 years haha
No I no Corral is not perfect but still it has more future then a copy of nas4free.
I keep on using FreeNAS Corral because it has more to offer then 9.10 for me, and on my E5-2658v4 with 64Gb it is fast enough. I miss a lot of features in 9.10.

 

[drb]

FreeNAS 11?? That would be another 2 years haha

In the past the first number of the FreeNAS version was the BSD version. With the proposed 9.10.3 being built on FreeNAS 11, then 9.10.3 should be named FreeNAS 11.0.0. It would not change the proposed time frame at all. Just change the name of the proposed 9.10.3 to 11.0.0.

Do not change 9.10 from FreeBSD 10 to FreeBSD 11 in a minor point release. Leave 9.10.2 to continue on with FreeBSD 10 and bug fixes and security updates as needed. Just like FreeNAS 8 and 9 did.

 

[Ericloewe]

Did the 7 train go EOL the day that 8 train came out?

7 was EOL before 8 ever existed. It also wasn't iXsystems.

Did the 8 train go EOL the day that the 9 train came out?

Pretty much.

Do not change 9.10 from FreeBSD 10 to FreeBSD 11 in a minor point release.

What's this fixation with "point releases"? It's a name.

 

[drb]

The maturity of OpenZFS on Linux has made great strides and this fiasco has just caused me to lose confidence in FN's going forward.

I too am watching OpenZFS on Linux. I chose FreeNAS based on its stability and ZFS. With Corral and the proposed 9.10.3, I am very concerned about FN's stability going forward.

 

[drb]

What's this fixation with "point releases"? It's a name.

That is exactly it, it is a name. Do you test a new version before you upgrade, or do you just blindly upgrade?

So you test the same for each update, or differently for major upgrades? Does one test differently when the underlying FreeBSD major version changes as when a single security patch or bug fix is applied?

If you do the same complete exhaustive tests on your captive systems for each and every update, and formally plan a migration strategy, then the name really doesn't matter.

However, by using a proper naming convention, users can better plan tests and migration strategies. It also allows for the patching of critical problems as a maintenance release without having everything thrown together in a single update.

Why not just give a single meaningless name to every update? That is what 9.10.3 is, a name that indicates a maintenance upgrade that is really a major version upgrade. How can one apply only bug fixes and security updates contained in 9.10.3 without the FreeBSD base change to 11 or all the minor updates to VM's, etc.? Why bother with the 9.10 part of the name, let's just call it version 3 and the next release version 4? Each release could be as simple as a small bug fix or a big as a complete rewrite. It is just a name. Who cares? If everything and anything is bundled in one upgrade, then users must choose whether to upgrade, test, or live with bugs or security issues. Sounds like a great way to a stable system. NOT.

I think there is a very good reason to use a Major.Minor.Maintenance version numbering system. IX Systems did use one in the past, all I am asking is that they go back to it. Maybe I am the only one, but I think it is better than a one name fits all system.

 

[Jailer]

Do you test a new version before you upgrade, or do you just blindly upgrade?

No, that's what you guys are for. :D

 

[Mlovelace]

I too am watching OpenZFS on Linux. I chose FreeNAS based on its stability and ZFS. With Corral and the proposed 9.10.3, I am very concerned about FN's stability going forward.

ZFS on linux has been considered stable and for production use since Sept. 2014 and shouldn't give you pause if you're considering going that route. It wasn't included as a default module until Ubuntu included it in it's 16.04 release, but I've been using it for over 2 years without any problems at all.

 

[userseven]

I have mixed feelings about the events that unfolded in the last few days. I can't even begin to appreciate how hard it must be step back, examine, avoid any sunken cost fallacy traps and then decide to just drop everything. Ouch!
On the other hand almost everything related to FreeNAS 10, at least in hindsight, feels wrong. While it took too long to develop the beta stage seemed to pass in the blink of an eye. The release version was clearly not ready for prime time and there it lays, the original sin of releasing an unfinished product. Pressure. I was one of the least fortunate that had various problems with it since the beginning. But it sure looked the part... and I loved it. Instantly! FreeNAS was finally no longer stuck in the HTML Class of '98 category. It's such a shame for all this to go to waste. Back to the drawing boards, I guess.

 

[Jacopx]

Is possible to migrate back from Corral to 9.10 without problem?

 

[gbooker]

ZFS on linux has been considered stable and for production use since Sept. 2014 and shouldn't give you pause if you're considering going that route. It wasn't included as a default module until Ubuntu included it in it's 16.04 release, but I've been using it for over 2 years without any problems at all.

Agreed. Prior to using FN, my storage was on ZFS on Linux (ZoL). Building that kernel module was quite annoying and took a long time when ZoL had an update and there were several kernels installed since it had to build for each one. Now, it's a binary so that's significantly easier. Additionally you can install Ubuntu on ZFS. It's come a long way and is certainly worthy of serious consideration.

I'm only waiting because I want encryption at rest and with native ZFS encryption coming, I don't want to migrate the data to Linux with LUKS only to migrate to using the native encryption later.

 

[Jailer]

Is possibile to migrating-back from Corral to 9.10 without problem?

https://forums.freenas.org/index.php?resources/migrating-from-freenas-corral-to-freenas-9-10.33/

 

[Ericloewe]

It helps to read the announcements all the way through.

Mod note:

There's an FAQ about moving from FreeNAS Corral to FreeNAS 9.10.2 in the Resources Section. You can find it at this link.

- Ericloewe