Tried 9.3 - Back to 9.2.1.9

[itw]

Well, that was an experience. Upgrading to 9.3 pretty much broke everything I use FreeNAS for.

AFP backed by OS X Server LDAP. Could no longer authenticate over LDAP. It looked like FreeNAS was talking to the server just fine (resolving unames from uids in file ownership, etc.) but AFP wouldn't authenticate.

I use multipath iSCSI from the globalSAN client on my mac and from XenServer 6.2. It all broke. I managed to get globalSAN connecting again, but using multipath resulted in two drives being mounted.

XenServer would simply not connect. I tried a few things and found a couple threads in the beta forum but they looked like they were for an older version of XenServer. I think the threads were right, something changes with the LUN IDs, but I ran out of time and patience. I wasn't beta-testing, after all.

So I reinstall a 9.2.1.9 USB stick, reboot, upload the backup config, reboot, and go to mount my encrypted RAIDZ2. It wouldn't take the password. Can't decrypt the drive. Now I'm sweating. Copy it out of lastpass again, paste. Fail. Type it in manually. Fail. Uh oh. Uploaded the recovery key and thankfully that freaking worked. Reset the password and everything's working again after repairing the Xen SRs.

I'll wait a while.

 

[diedrichg]

Dayum!

 

[nanda]

Same for me. Freenas 9.3 won't boot after having been upgraded from 9.2.something.

https://bugs.freenas.org/issues/6690

I wrote about this well before 9.3 went official, and plenty of people seem to have the same problem, but nothing was ever done.

9.3 should have been pulled and fixed in beta stage.

 

[joeschmuck]

I would highly recommend that everyone use a new USB key, perform an installation from the ISO, restore the config file, and cross their fingers. If it fails then all you have to do is shutdown and reinstall your previous version of FreeNAS. I'm not saying that would fix the problem but it should mitigate many of the upgrade problems I've seen. Also, do not upgrade your ZFS pool, wait a month minimum and if you are fine with 9.3, then upgrade the pool if you like.

 

[jkh]

AFP backed by OS X Server LDAP. Could no longer authenticate over LDAP. It looked like FreeNAS was talking to the server just fine (resolving unames from uids in file ownership, etc.) but AFP wouldn't authenticate.

Did you read the UPGRADING document that was mentioned in the release notes and make sure that your LDAP server was configured to talk SSL/TLS?

 

[jkh]

I wrote about this well before 9.3 went official, and plenty of people seem to have the same problem, but nothing was ever done.

This problem could never be reproduced (still can't) so there was nothing TO be done. In order to fix a bug, assuming that bug is even in your software, you must first be able to see it. We suspect, but cannot prove, that a number of older motherboards (with older BIOSes that cannot deal with GPT labels) and/or older USB sticks that lock up under ZFS may be the culprit, but like I said, we've never seen it. We've even tried to obtain some of the oldest, crappiest hardware we can find and 9.3 installs just fine on it. Oh well! "Works on all of our machines" :)

 

[itw]

Did you read the UPGRADING document that was mentioned in the release notes and make sure that your LDAP server was configured to talk SSL/TLS?

You mean this:

* Directory Services now based on SSSD (System Security Services Daemon) and connections to LDAP servers done with SSL/TLS.

Yes. I read that. It doesn't read like a mandate and doesn't say that your non-SSL/TLS LDAP will no longer work and will silently fail. And like I said, FreeNAS was talking to the LDAP server just fine.

What about the iSCSI??

 

[jkh]

You mean this:

* Directory Services now based on SSSD (System Security Services Daemon) and connections to LDAP servers done with SSL/TLS.
Yes. I read that. It doesn't read like a mandate.

It's a mandate. It's also standard practice, FWIW. Sending passwords in cleartext over the wire is never a good idea.

What about the iSCSI??

What about it? iSCSI has always had its own authentication mechanism using shared secrets. CHAP away, as always.

 

[itw]

It's a mandate. It's also standard practice, FWIW. Sending passwords in cleartext over the wire is never a good idea.

Thanks for that.

What about it? iSCSI has always had its own authentication mechanism using shared secrets. CHAP away, as always.

Thanks. Nevermind I guess. Upgrading completely broke zero authentication iSCSI from XenServer 6.2.

 

[Mlovelace]

Oh well! "Works on all of our machines" :)

This is why I will continue to pay our Netapp service contracts for out mission critical data systems. The TrueNAS price is right, but the attitude I've seen doesn't give me confidence.

 

[jkh]

Upgrading completely broke zero authentication iSCSI from XenServer 6.2.

I doubt this has anything to do with authentication. There is a XenServer compatibility switch you need to throw to get serial number generation to be Xen compatible. We changed from istgt to CTL in 9.3 and this involved going to a completely different iSCSI stack.

 

[jkh]

This is why I will continue to pay our Netapp service contracts for out mission critical data systems. The TrueNAS price is right, but the attitude I've seen doesn't give me confidence.

I don't see how the comparison is even remotely apt. NetApp doesn't let you install their software on arbitrary hardware. They control all the variables for very good reason, this being one of them. If you want to buy a TrueNAS system or even a FreeNAS Certified system, the hardware will be similarly controlled and your 9.2.1.x -> 9.3 upgrade will work just fine. Even a FreeNAS Mini at $995 a pop has been well tested. What we can't test, however, is "any sort of PC hardware ever created at any time", which is pretty representative of the range of hardware that FreeNAS runs on.

As someone else put it: If you don't like FreeNAS on your [self] chosen hardware, you are entitled to a full refund for the price of the software. ;-)

 

[itw]

I doubt this has anything to do with authentication. There is a XenServer compatibility switch you need to throw to get serial number generation to be Xen compatible. We changed from istgt to CTL in 9.3 and this involved going to a completely different iSCSI stack.

I didn't say it did. Care to enlighten us as to this "compatibility switch?" Searching on XenServer compatibility yields nothing.

 

[Mlovelace]

I didn't say it did. Care to enlighten us as to this "compatibility switch?" Searching on XenServer compatibility yields nothing.

This thread should shed some light on your issue.

https://forums.freenas.org/index.ph...ter-9-2-1-8-to-9-3-upgrade.24787/#post-159078

 

[jkh]

See iSCSI extents UI. Screenshot attached.

 

[itw]

After completely screwing it up the first time I tried again this weekend and this is the status:

AFP and LDAP - exactly as Jordan said, enabling TLS to the LDAP server fixed everything. I had TLS on the server but didn't turn it on for the Directory Service. All I had to do was import my StartCom root as a CA. (It is a little confusing having the LDAP config prompt for a "Certificate" when it really wants a "Certificate Authority") Note that it is REALLY nice to have CIFS using the local user database UNLESS all the LDAP trickery is done. I can now get at my server with CIFS if I have to without disabling LDAP for everything else. This helps me out a lot. A project for another day is getting FreeNAS 9.3 to be an AD domain controller.

iSCSI and Xen - I did have to detach and reattach all my SRs but they came right up with the Xen compatibility checkbox. Multipath too. (Who is wrong here? CTL or Xen?)

iSCSI and globalSAN - I don't know why but this still came up as two drives when I launched the second multipath connection. I deleted everything and rescanned and now my multipath is just that again. Last week I had to copy all the data off my extent (Apple HFS formatted) because having it mounted twice completely screwed it up to the point my Mac would only mount the drive read-only and fsck_hfs couldn't fix it. I don't think I lost anything and I certainly can't blame FreeNAS. Before I tried to get it working again today I unmounted it and took a zfs snapshot. Duh.

So I'm back on 9.3 and everything's working. Thanks and sorry for the tone, Jordan. It was all my fault. It was a bit rocky but NO DATA LOST.

 

[Mlovelace]

I don't see how the comparison is even remotely apt. NetApp doesn't let you install their software on arbitrary hardware. They control all the variables for very good reason, this being one of them. If you want to buy a TrueNAS system or even a FreeNAS Certified system, the hardware will be similarly controlled and your 9.2.1.x -> 9.3 upgrade will work just fine. Even a FreeNAS Mini at $995 a pop has been well tested. What we can't test, however, is "any sort of PC hardware ever created at any time", which is pretty representative of the range of hardware that FreeNAS runs on.

As someone else put it: If you don't like FreeNAS on your [self] chosen hardware, you are entitled to a full refund for the price of the software. ;-)

Fair point. I am getting a Z30 TrueNAS quoted with gold support. :D

 

[itw]

I'm going to bump this original negativity again to highlight the RTFM nature of my original post.

Since pulling my head out I have been consistently updating, have recently mirrored my freenas-boot flash drive, and as a final measure, just ran zpool upgrade main-pool.

It's been rock-solid.

 

[cyberjock]

Thanks for the update.

To answer your question about whether CTL or Xen is "wrong" I personally blame Xen. My understanding is that the switch from istgt to ctl results in changes to how the iSCSI targets are presented to Xen. Xen caches some of this stuff and when it changes due to the switch to ctl Xen poops all over itself. It's nice to cache stuff, but you should at least try to validate that what you cached is correct. It's not like validating things (especially after rebooting Xen) should be that much of an inconvenience for Xen. Note that ESXi has some similar problems and if I remember correctly the problem is rectified by either completely unmounting the iSCSI devices from ESXi or rebooting the ESXi host.