FreeNAS AD member broke after 30 days since update from 10.4 to 11.1u1, currently on 11.1u2

[Ionico]

Good afternoon,

I have spent two days trying to figure out what's going on. It started with a few users, but it escalated quickly to just the domain admin being able to connect if I try to browse to the share from the domain controller itself.

In the first day I managed to get in by IP address after removing the old file servers from the dns entry (saw in a post). Whenever I try to log I get prompted for the username and password and failing with access denied. I get the following message:

Code:

../source3/librpc/crypto/gse.c:649(gse_get_server_auth_token)
  gss_accept_sec_context failed with [ Miscellaneous failure (see text): Failed to find cifs/HOSTNAME.DOMAIN@DOMAIN (kvno 2) in  eytab MEMORY:cifs_srv_keytab (aes256-cts-hmac-sha1-96)]
 ../auth/gensec/spnego.c:411(gensec_spnego_parse_negTokenInit)
  SPNEGO(gse_krb5) NEG_TOKEN_INIT failed: NT_STATUS_LOGON_FAILURE

It didn't work and now I can't even access the share through its IP address (which i could before). I get prompted for a password and says: access denied. There is also this error in log.wb-DOMAIN:

Code:

../source3/winbindd/winbindd_pam.c:1500(winbind_samlogon_retry_loop)
  winbind_samlogon_retry_loop: sam_logon returned ACCESS_DENIED.  Maybe the DC has Restrict NTLM set or the trust account password was changed and we didn't know it. Killing connections to domain DOMAIN

Code:

net ads info -U admin ---> works
net ads info -U anyuser ---> works
wbinfo -u | wc -l
56
wbinfo -g | wc -l
67
wbinfo -t
succeeded
wbinfo --ping-dc
succeeded
wbinfo - 'DOMAIN\user'
user:*:24670:20513:user name:/home/DOMAIN/user:/bin/sh

getfacl /mnt/Cif_Share/
# file: /mnt/Cif_Share/
# owner: root
# group: wheel
			owner@:rwxpDdaARWcCos:fd-----:allow
			group@:rwxpDdaARWcCos:fd-----:allow
		 everyone@:r-x---a-R-c---:fd-----:allow
getfacl /mnt/Cif_Share/Antivirus/
# file: /mnt/Cif_Share/Antivirus/
# owner: serveradmin
# group: domain users
			owner@:rwxpDdaARWcCos:fd-----:allow
			group@:rwxpDdaARWcCos:fd-----:allow
		 everyone@:r-x---a-R-c---:fd-----:allow

Testparm shows no errors

Code:

# Global parameters
[global]
		bind interfaces only = Yes
		dos charset = CP437
		interfaces = 127.0.0.1 192.168.2.249
		realm = DOMAIN
		server string = Office File Server
		workgroup = DOMAIN
		domain master = No
		lm announce = Yes
		local master = No
		preferred master = No
		nsupdate command = /usr/local/bin/samba-nsupdate -g
		client ldap sasl wrapping = plain
		logging = file
		max log size = 51200
		kernel change notify = No
		panic action = /usr/local/libexec/samba/samba-backtrace
		pid directory = /var/run/samba
		disable spoolss = Yes
		load printers = No
		printcap name = /dev/null
		allow trusted domains = No
		map to guest = Bad User
		obey pam restrictions = Yes
		security = ADS
		server role = member server
		deadtime = 15
		hostname lookups = Yes
		max open files = 939359
		template shell = /bin/sh
		winbind cache time = 7200
		winbind enum groups = Yes
		winbind enum users = Yes
		winbind offline logon = Yes
		winbind refresh tickets = Yes
		dns proxy = No
		idmap config ionic-office: range = 20000-90000000
		idmap config ionic-office: backend = rid
		idmap config *: range = 90000001-100000000
		idmap config * : backend = tdb
		store dos attributes = Yes
		strict locking = No
		directory name cache size = 0
		dos filemode = Yes
		acl allow execute always = Yes
		ea support = Yes
		create mask = 0666
		directory mask = 0777


[Antivirus]
		path = "/mnt/Cif_Share/Antivirus"
		veto files = /.snapshot/.windows/.mac/.zfs/
		read only = No
		vfs objects = zfs_space zfsacl streams_xattr aio_pthread
		zfsacl:acesort = dontcare
		nfs4:chown = true
		nfs4:acedup = merge
		nfs4:mode = special

Completely baffled I don't know what else to do.

I reverted back to 11.0-u4 as I had the config file but the issue persists. Deleted all cif shares and recreated just 1 for now, same problem.

 

[Ionico]

also, I changed the Idmap Range Low to 10000 as read somewhere it used to be lower before. The first install was a 9.10 that has been gradually updated. But today I did a reinstall of 11.0-u4 as mentioned above

Hardware:
Supermicro X10SRH-CLN4F
Intel(R) Xeon(R) CPU E5-1620 v4 @ 3.50GHz
32GB ECC
raidz2 4x6TB western digital red

 

[Nick2253]

You need to help us understand a little more about what you are doing. What are your clients? Windows? Windows 10? Linux? Have you updated your client OS recently? Are you seeing the same problem across all clients?

Are you connecting with SMB, NFS, AFP? Are you having permission issues anywhere else on the domain? Is your domain Windows Active Directory? Samba? Something else?

 

[bigphil]

Is your dns domain name actually "DOMAIN" or is it "ionic-office"? you need to make sure that the setting Directory>Active Directory>Domain Name (DNS/Realm-Name) is set to your fqdn, not netbios domain name. It also looks like some parameters are missing from your global SMB config. I suggest you create a backup of the smb4.conf file, stop the service, rename or remove the smb4.conf file, restart the SMB service and re-configure your directory settings and SMB service setttings. You can copy/paste in your old share config [Antivirus] so you dont have to reconfigure that. I'd also reset the FreeNAS computer account in AD, make sure you have a specific user account (i.e. like domain\freenasadmin) that has full control on that computer object and then use that account username and password when joining FreeNAS to AD.

 

[Ionico]

You need to help us understand a little more about what you are doing. What are your clients? Windows? Windows 10? Linux? Have you updated your client OS recently? Are you seeing the same problem across all clients?

Are you connecting with SMB, NFS, AFP? Are you having permission issues anywhere else on the domain? Is your domain Windows Active Directory? Samba? Something else?

Clients are a mixed up of windows 7 pro and windows 10 ltsb (2 are pro) clients. I had a NFS share pointing to a different zfs partition that I was preparing for esxi but it is now completely disabled and the drives removed. There is a ftp share for the calendars in thunderbird that points to the same ip. Connecting using SMB. All windows computers have january roll up updates and by tomorrow they will have february including domain servers.
The permission issues are only occurring on the freenas box.

Is your dns domain name actually "DOMAIN" or is it "ionic-office"? you need to make sure that the setting Directory>Active Directory>Domain Name (DNS/Realm-Name) is set to your fqdn, not netbios domain name. It also looks like some parameters are missing from your global SMB config. I suggest you create a backup of the smb4.conf file, stop the service, rename or remove the smb4.conf file, restart the SMB service and re-configure your directory settings and SMB service setttings. You can copy/paste in your old share config [Antivirus] so you don't have to reconfigure that. I'd also reset the FreeNAS computer account in AD, make sure you have a specific user account (i.e. like domain\freenasadmin) that has full control on that computer object and then use that account username and password when joining FreeNAS to AD.

its ionic-office.

I found I had an issue with windows (2008 R2) replicating the directory server (event ID 1864) which is fixed now (causes netlogon to stop and break...). So I can log back in using IP address but its domain name doesn't work. Going through the users I saw a buggy freenas user lingering which has been removed too.

After getting rid of the domain issues, I have removed the freenas user and the freenas computer and recreated them. I can confirm it is being replicated to the secondary domain controller. DNS seems fine, ntp services have been changed to our local ntp server which syncs to the ntp.org pools. Disabled active directory on the freenas box and rebooted. On startup I joined the domain again but the same issue still occurs. I can access by IP and the user's permissions are kept but I can't use the netbios name. The error in the log is:

Code:

 ../source3/librpc/crypto/gse.c:646(gse_get_server_auth_token)
  gss_accept_sec_context failed with [ Miscellaneous failure (see text): Failed to find cifs/HOSTNAME@DOMAIN(kvno 3) in keytab MEMORY:cifs_srv_keytab (aes256-cts-hmac-sha1-96)]
[2018/02/22 22:28:13.146564,  1] ../auth/gensec/spnego.c:569(gensec_spnego_parse_negTokenInit)
  SPNEGO(gse_krb5) NEG_TOKEN_INIT failed: NT_STATUS_LOGON_FAILURE

Funny enough, the domain admin account can log in from the domain controller at lighting speed. I wonder if the token is still valid.
All the shares are owned by the domain admin account and belong to the domain user group account with read and write access. I followed the how to: https://forums.freenas.org/index.ph...directory-folder-file-user-permissions.20610/
Which has worked for over a year

Hope that helps

 

[bigphil]

I suggest doing the next step I said to do which was to backup the smb4.conf file (/usr/local/etc/smb4.conf), stop the SMB service, rename or remove the smb4.conf file, reboot FreeNAS and run through the setup again. From a Windows box, whats the output of "setspn -L <FreeNAS server name>". you should have at least two entries:
HOST/fqdn
HOST/netbios name

 

[Ionico]

I suggest doing the next step I said to do which was to backup the smb4.conf file (/usr/local/etc/smb4.conf), stop the SMB service, rename or remove the smb4.conf file, reboot FreeNAS and run through the setup again. From a Windows box, whats the output of "setspn -L <FreeNAS server name>". you should have at least two entries:
HOST/fqdn
HOST/netbios name

So I left the server running from 22:30 and left the command fail -f /var/log/samba4/log.smbd running. The error logs stopped at 23:55, the freenas box is fully accessible again! Phewwww that was nuts

On a side note, I believe upgrading the 11.1 was a mistake for this current setup.

Thank you for all the help!!!!