Lost Data in Upgrade to 11.2

[Johev]

Any updates on this? Is the latest version of freenas safe?

 

[Ericloewe]

As stated above, snapshots mitigate this problem.

 

[danb35]

Any updates on this? Is the latest version of freenas safe?

As far as I've seen, iX are no closer to figuring out why this is happening than they were when 11.2 released. Though again, snapshots appear to avoid the problem.

 

[Evertb1]

As far as I've seen, iX are no closer to figuring out why this is happening than they were when 11.2 released. Though again, snapshots appear to avoid the problem.

I am looking in this thread for the first time but reading it makes me glad that I am not an early adapter with updates. Everything I read is very troublesome if you ask me.

A couple of years ago I made the switch from Windows Server to FreeNAS with one goal in my mind: having stable and reliable storage at an (almost) professional level and without the update hassles form Windows/Microsoft. I still think that Windows Home server could have been a fantastic product with some work from MS but it was not to be.

So now I am a satisfied FreeNAS user, but the things I see in this thread makes me worry about the future. Yes I make daily and even hourly snapshots of all my datasets. And yes I have a decent back-up strategy. But I should not be afraid to update to a new release. And at this point in time I would not touch the latest update with a ten foot pole.

 

[danb35]

Everything I read is very troublesome if you ask me.

I agree. I did update to 11.2, and everything's working fine with no lost data. I suspect this is true for the vast majority. But there are some who are losing data, and the problem has persisted despite three updates since the 11.2 release. This bug is a showstopper, but it doesn't appear to have actually stopped the show.

 

[Evertb1]

This bug is a showstopper, but it doesn't appear to have actually stopped the show.

It really puzzles me why it appears to be that this is not nr 1 on the priority list at iX. I have been working as a software developer for more then 25 years. And based on my experience I know one thing for sure: you need to keep the data safe. Often you can live with some bugs, but as soon as a bug jeopardize the data security/integrity it should be "all hands on deck" to resolve that bug. In the meantime the roll out of the suspicious version should be frozen. Even knowing that the problems kan be mitigated by using snapshots.

 

[Heracles]

Hi evertb,

I am pretty sure this bug IS a number 1 priority at IX. It is just that to smash the bug, you need to identify it. As of now, only the symptoms are defined. People are looking for evidence, details, info, whatever will allow them to identify WHY its happening and not only WHAT is happening.

To troubleshoot intermittent bugs is one of the most difficult things to do. When things work, there is no problem. When they do not work at all, there are no big problems. It is when they are working partially that you face the biggest of all problems.

Here, I did a fresh install using 11.2-U1 and because of this thread, I will not try to upgrade to 11.2U2. My testing phases are over and I am now in prod, so can nor risk anything like that.

Good luck IX in your hunt for that one,

 

[cobrakiller58]

I've been watching this since I intend to upgrade soon but i don't see anything in common, what the pattern here?
Versions listed that people upgraded from
11.stable, 11.1u6, 11.1u1, 11.1u5, 11.1u7, 11.1-u6.3
version ungraded to
11.2, 11.2u1, 11.2u2
What services were running? I see one SMB but I'm sure more than one person had smb running when they did the update anything other services?
Usb keys and SSD's both reported here but not all data loss incidents had a boot error during the update.
Did all the updates happen from the GUI?

 

[Ericloewe]

SMB seems to always be on the scene of the crime, but that's like saying that everyone who got sick had had a glass of water in the previous 24 hours. I fully expect the vast majority of installations to be running Samba.

I haven't updated my main server yet, but I will make sure to capture the console and run a syslog server and whatever else I can think of when I do.

 

[Kris Moore]

Hey folks! FYI - We're very actively trying to triage this, and so far cannot come up with a reproduction case. What we need is some more information to see if we can replicate internally. Anybody who's been impacted by this, wanting to know the following information:

1) What services do you have enabled
2) What version did you migrate from and too, specifically
3) Did you have any legacy or new plugins installed?
4) If you did have plugins installed, which ones
5) If you did have plugins installed, did any of them have nullfs mounts setup to expose the pool to inside the plugin?

One of my theories is that this is a bad-behaving plugin when nullfs is used to expose the pool. But I'll need more details to drill down and see if that theory holds up. Please throw this information up on the Ticket (https://redmine.ixsystems.com/issues/64512) or here on the forums if you don't do Redmine.

Thanks!

 

[cobrakiller58]

I'm trying to reproduce the problem myself on a VM but is the update server down it says connection refused

 

[Kris Moore]

I'm trying to reproduce the problem myself on a VM but is the update server down it says connection refused

Hang tight, should be back in an hour or so. We're moving some DNS around here.

 

[cobrakiller58]

roger, was the data loss always GUI updates so far or ISO as well?

 

[Kris Moore]

roger, was the data loss always GUI updates so far or ISO as well?

So far I've only heard of it via GUI updates, no specific ISO upgrade that I'm aware of.

 

[guermantes]

So far I've only heard of it via GUI updates, no specific ISO upgrade that I'm aware of.

Perhaps I am mistaken, but is comment 44 on page 2 not an example of a fresh install and loss of data, rather than an upgrade?

Show : comment #44

To completely clarify, and set an example:

Dataset A has a recursive snapshot.
Dataset B does not have a snapshot.

After a fresh USB, I imported the pool, and upgraded the ZFS pool.

Dataset A via zfs list showed that it had 3TB of data, but when creating a SMB share, it did not show any files.
Dataset B showed no data at all. The dataset was there, but no mb was used.

Then I cloned the dataset A snapshot, and promoted it. Dataset A files showed up now. Dataset B was a flop and gone magically.

 

[Kris Moore]

Perhaps I am mistaken, but is comment 44 on page 2 not an example of a fresh install and loss of data, rather than an upgrade?

Show : comment #44

To completely clarify, and set an example:

Dataset A has a recursive snapshot.
Dataset B does not have a snapshot.

After a fresh USB, I imported the pool, and upgraded the ZFS pool.

Dataset A via zfs list showed that it had 3TB of data, but when creating a SMB share, it did not show any files.
Dataset B showed no data at all. The dataset was there, but no mb was used.

Then I cloned the dataset A snapshot, and promoted it. Dataset A files showed up now. Dataset B was a flop and gone magically.

I stand corrected, yes that does look like fresh install also. Good, helps narrow it down then.

 

[Kris Moore]

Hey folks! Quick update here. We're still digging into this and getting closer. I wanted to ask if anybody here who has experienced this still has their FreeNAS Database intact. If so, would you be willing to confidentially share it with us? We need to replicate the exact scenario that triggered this behavior in order to test some potential fixes. If you are willing, please PM, or reach out to me via email to kris [at] ixsystems.com, or our QA manager, jmaloney [at] ixsystems.com

 

[diskdiddler]

I see someone updated a bug on the tracker, is this now been identified?

 

[SynbiosVyse]

Any updates after using my freenas database?

 

[SweetAndLow]

No update other than a guess and fix that went into U3.