Problems joining AD with 11.3

[Fab Sidoli]

Hi All,

I'm having great difficulty configuring the AD directory service and require some help sorting this out.

I had a working 11.2-U7 box that I upgrade to 11.3 and things broke from there. I have also tried a fresh install of 11.3 in case the cause was a db issue created during the upgrade path.

I was using encryption with SASL wrapping set to seal. I also had the idmap backed set to AD and SFU set for NSS info and IDMAP schema mode.

Under 11.3 my previously working certificates get rejected with one of two errors:

"Certificate matching query does not exist.
Error: Traceback (most recent call last):
File "/usr/local/lib/python3.7/site-packages/middlewared/main.py", line 130, in call_method
io_thread=False)
File "/usr/local/lib/python3.7/site-packages/middlewared/main.py", line 1077, in _call
return await methodobj(*args)
File "/usr/local/lib/python3.7/site-packages/middlewared/service.py", line 302, in update
f'{self._config.namespace}.update', self, self.do_update, [data]
File "/usr/local/lib/python3.7/site-packages/middlewared/main.py", line 1077, in _call
return await methodobj(*args)
File "/usr/local/lib/python3.7/site-packages/middlewared/schema.py", line 960, in nf
return await f(*args, **kwargs)
File "/usr/local/lib/python3.7/site-packages/middlewared/plugins/activedirectory.py", line 856, in do_update
{'prefix': 'ad_'}
File "/usr/local/lib/python3.7/site-packages/middlewared/main.py", line 1127, in call
app=app, pipes=pipes, job_on_progress_cb=job_on_progress_cb, io_thread=True,
File "/usr/local/lib/python3.7/site-packages/middlewared/main.py", line 1091, in _call
return await run_method(methodobj, *args)
File "/usr/local/lib/python3.7/site-packages/middlewared/utils/run_in_thread.py", line 10, in run_in_thread
return await self.loop.run_in_executor(self.run_in_thread_executor, functools.partial(method, *args, **kwargs))
File "/usr/local/lib/python3.7/site-packages/middlewared/utils/io_thread_pool_executor.py", line 25, in run
result = self.fn(*self.args, **self.kwargs)
File "/usr/local/lib/python3.7/site-packages/middlewared/schema.py", line 964, in nf
return f(*args, **kwargs)
File "/usr/local/lib/python3.7/site-packages/middlewared/plugins/datastore.py", line 302, in update
data[name] = field.rel.to.objects.get(pk=data[name]) if data[name] is not None else None
File "/usr/local/lib/python3.7/site-packages/django/db/models/manager.py", line 85, in manager_method
return getattr(self.get_queryset(), name)(*args, **kwargs)
File "/usr/local/lib/python3.7/site-packages/django/db/models/query.py", line 385, in get
self.model._meta.object_name
freenasUI.system.models.DoesNotExist: Certificate matching query does not exist."

or

"Certificate matching query does not exist.
Error: Traceback (most recent call last):
File "/usr/local/lib/python3.7/site-packages/middlewared/main.py", line 130, in call_method
io_thread=False)
File "/usr/local/lib/python3.7/site-packages/middlewared/main.py", line 1077, in _call
return await methodobj(*args)
File "/usr/local/lib/python3.7/site-packages/middlewared/service.py", line 302, in update
f'{self._config.namespace}.update', self, self.do_update, [data]
File "/usr/local/lib/python3.7/site-packages/middlewared/main.py", line 1077, in _call
return await methodobj(*args)
File "/usr/local/lib/python3.7/site-packages/middlewared/schema.py", line 960, in nf
return await f(*args, **kwargs)
File "/usr/local/lib/python3.7/site-packages/middlewared/plugins/activedirectory.py", line 856, in do_update
{'prefix': 'ad_'}
File "/usr/local/lib/python3.7/site-packages/middlewared/main.py", line 1127, in call
app=app, pipes=pipes, job_on_progress_cb=job_on_progress_cb, io_thread=True,
File "/usr/local/lib/python3.7/site-packages/middlewared/main.py", line 1091, in _call
return await run_method(methodobj, *args)
File "/usr/local/lib/python3.7/site-packages/middlewared/utils/run_in_thread.py", line 10, in run_in_thread
return await self.loop.run_in_executor(self.run_in_thread_executor, functools.partial(method, *args, **kwargs))
File "/usr/local/lib/python3.7/site-packages/middlewared/utils/io_thread_pool_executor.py", line 25, in run
result = self.fn(*self.args, **self.kwargs)
File "/usr/local/lib/python3.7/site-packages/middlewared/schema.py", line 964, in nf
return f(*args, **kwargs)
File "/usr/local/lib/python3.7/site-packages/middlewared/plugins/datastore.py", line 302, in update
data[name] = field.rel.to.objects.get(pk=data[name]) if data[name] is not None else None
File "/usr/local/lib/python3.7/site-packages/django/db/models/manager.py", line 85, in manager_method
return getattr(self.get_queryset(), name)(*args, **kwargs)
File "/usr/local/lib/python3.7/site-packages/django/db/models/query.py", line 385, in get
self.model._meta.object_name
freenasUI.system.models.DoesNotExist: Certificate matching query does not exist."

Having put these certs in from scratch I know they do exist.

At the moment I would just like to nuke my AD settings altogether and start again but I'm not sure how to do this. I can't do it through the UI and I can't doing via the CLI.

# net -k ads leave
No realm set, are we joined ?

# net ads join -U domainuser
Host is not configured as a member server.
Invalid configuration. Exiting....
Failed to join domain: This operation is only allowed for the PDC of the domain.

<><>< Some other info that may be useful ><><>

I can do a kinit

# kinit domainuser
domainuser@DOMAINS's Password:
# klist
Credentials cache: FILE:/tmp/krb5cc_0
Principal: domainuser@DOMAIN

Issued Expires Principal
Feb 21 13:16:19 2020 Feb 21 23:16:19 2020 krbtgt/DOMAIN@DOMAIN

# wbinfo -t
checking the trust secret for domain LCN via RPC calls failed
wbcCheckTrustCredentials(LCN): error code was NT_STATUS_NO_SUCH_DOMAIN (0xc00000df)
failed to call wbcCheckTrustCredentials: WBC_ERR_AUTH_ERROR
Could not check secret

# ping domain
PING domain (IP): 56 data bytes
64 bytes from IP: icmp_seq=0 ttl=128 time=0.335 ms
64 bytes from IP: icmp_seq=1 ttl=128 time=0.088 ms

Please let me know what other debugging information would be useful.

 

[anodos]

One change in 11.3 is that user-provided CA certificates (as in uploaded to the UI) are automatically placed in /etc/ssl/truenas_cacerts.pem, which is automatically set as TLS_CACERT in the ldap.conf. The certificate dropdown in 11.3 is for SASL_EXTERNAL binds (certificate-based authentication), which your probably not doing. TL;DR, try binding to AD with SSL selected, but without a certificate populated in the dropdown.

 

[Fab Sidoli]

Hi. Thanks for your reply.

I believe we are doing certificate-based authentication.

Anyway, I've tried to leave the certificate field empty but this is what I get.

[certificate] Not an integer
Error: Traceback (most recent call last):
File "/usr/local/lib/python3.7/site-packages/middlewared/main.py", line 130, in call_method
io_thread=False)
File "/usr/local/lib/python3.7/site-packages/middlewared/main.py", line 1077, in _call
return await methodobj(*args)
File "/usr/local/lib/python3.7/site-packages/middlewared/service.py", line 302, in update
f'{self._config.namespace}.update', self, self.do_update, [data]
File "/usr/local/lib/python3.7/site-packages/middlewared/main.py", line 1077, in _call
return await methodobj(*args)
File "/usr/local/lib/python3.7/site-packages/middlewared/schema.py", line 959, in nf
args, kwargs = clean_and_validate_args(args, kwargs)
File "/usr/local/lib/python3.7/site-packages/middlewared/schema.py", line 917, in clean_and_validate_args
value = attr.clean(args[args_index + i])
File "/usr/local/lib/python3.7/site-packages/middlewared/schema.py", line 594, in clean
data[key] = attr.clean(value)
File "/usr/local/lib/python3.7/site-packages/middlewared/schema.py", line 400, in clean
raise Error(self.name, 'Not an integer')
middlewared.schema.Error: [certificate] Not an integer

 

[Fab Sidoli]

I've gone back to basics and have deleted my certs and have done a bind to AD without SSL or TLS. It seems to be very particular about the idmap backend, schema and nss info. I had these set to SFU and AD before. I've been successful in joining the domain, but I get the following issues where trying to query users.

# wbinfo -t
checking the trust secret for domain LCN via RPC calls succeeded
# wbinfo -u | grep "^fs"
fs
# id fs
id: fs: no such user

Any ideas?

 

[Fab Sidoli]

FYI, I can't join the domain in the UI unless I set the schema to be RFC2307. I get random integer errors. It seems very buggy to me give this is a stable release.

 

[anodos]

Looks like a bug in the GUI. Try midclt call activedirectory.update '{"certificate": null}'. In 11.2 certificate-based auth wasn't possible for LDAP. We were performing simple (non-SASL) binds over tls-encrypted transport.

 

[Fab Sidoli]

# midclt call activedirectory.update '{"certificate": none}'
[activedirectory_update] A dict was expected
Traceback (most recent call last):
File "/usr/local/lib/python3.7/site-packages/middlewared/main.py", line 130, in call_method
io_thread=False)
File "/usr/local/lib/python3.7/site-packages/middlewared/main.py", line 1077, in _call
return await methodobj(*args)
File "/usr/local/lib/python3.7/site-packages/middlewared/service.py", line 302, in update
f'{self._config.namespace}.update', self, self.do_update, [data]
File "/usr/local/lib/python3.7/site-packages/middlewared/main.py", line 1077, in _call
return await methodobj(*args)
File "/usr/local/lib/python3.7/site-packages/middlewared/schema.py", line 959, in nf
args, kwargs = clean_and_validate_args(args, kwargs)
File "/usr/local/lib/python3.7/site-packages/middlewared/schema.py", line 917, in clean_and_validate_args
value = attr.clean(args[args_index + i])
File "/usr/local/lib/python3.7/site-packages/middlewared/schema.py", line 583, in clean
raise Error(self.name, 'A dict was expected')
middlewared.schema.Error: [activedirectory_update] A dict was expected

 

[Fab Sidoli]

# wbinfo -t
could not obtain winbind interface details: WBC_ERR_WINBIND_NOT_AVAILABLE
could not obtain winbind domain name!
checking the trust secret for domain (null) via RPC calls failed
failed to call wbcCheckTrustCredentials: WBC_ERR_WINBIND_NOT_AVAILABLE
Could not check secret

 

[anodos]

Sorry, some wires crossed in my head switching from python to json. midclt call activedirectory.update '{"certificate": null}'
Issue with traceback was fixed in commit a8b3733d and scheduled for U1 release.

 

[Fab Sidoli]

# midclt call activedirectory.update '{"certificate": null}' <-
{"id": 1, "domainname": "DOMAIN", "bindname": "DOMAINUSER", "bindpw": "PASSWORD", "ssl": "OFF", "certificate": null, "validate_certificates": false, "verbose_logging": true, "allow_trusted_doms": true, "use_default_domain": true, "allow_dns_updates": false, "disable_freenas_cache": false, "site": "", "kerberos_realm": 1, "kerberos_principal": "", "createcomputer": "Computers/Servers/Storage", "timeout": 60, "dns_timeout": 30, "idmap_backend": "AD", "nss_info": "SFU", "ldap_sasl_wrapping": "PLAIN", "enable": false, "netbiosname": "quark", "netbiosalias": []}

 

[anodos]

# midclt call activedirectory.update '{"certificate": null}' <-
{"id": 1, "domainname": "DOMAIN", "bindname": "DOMAINUSER", "bindpw": "PASSWORD", "ssl": "OFF", "certificate": null, "validate_certificates": false, "verbose_logging": true, "allow_trusted_doms": true, "use_default_domain": true, "allow_dns_updates": false, "disable_freenas_cache": false, "site": "", "kerberos_realm": 1, "kerberos_principal": "", "createcomputer": "Computers/Servers/Storage", "timeout": 60, "dns_timeout": 30, "idmap_backend": "AD", "nss_info": "SFU", "ldap_sasl_wrapping": "PLAIN", "enable": false, "netbiosname": "quark", "netbiosalias": []}

Now run midclt call activedirectory.update '{"ssl": "ON", "enable": true}'

 

[Fab Sidoli]

# midclt call activedirectory.update '{"ssl": "ON", "enable": true}'
local variable 'saved_bind_error' referenced before assignment
Traceback (most recent call last):
File "/usr/local/lib/python3.7/site-packages/middlewared/main.py", line 130, in call_method
io_thread=False)
File "/usr/local/lib/python3.7/site-packages/middlewared/main.py", line 1077, in _call
return await methodobj(*args)
File "/usr/local/lib/python3.7/site-packages/middlewared/service.py", line 302, in update
f'{self._config.namespace}.update', self, self.do_update, [data]
File "/usr/local/lib/python3.7/site-packages/middlewared/main.py", line 1077, in _call
return await methodobj(*args)
File "/usr/local/lib/python3.7/site-packages/middlewared/schema.py", line 960, in nf
return await f(*args, **kwargs)
File "/usr/local/lib/python3.7/site-packages/middlewared/plugins/activedirectory.py", line 876, in do_update
await self.start()
File "/usr/local/lib/python3.7/site-packages/middlewared/plugins/activedirectory.py", line 941, in start
new_site = await self.middleware.run_in_thread(self.get_site)
File "/usr/local/lib/python3.7/site-packages/middlewared/utils/run_in_thread.py", line 10, in run_in_thread
return await self.loop.run_in_executor(self.run_in_thread_executor, functools.partial(method, *args, **kwargs))
File "/usr/local/lib/python3.7/site-packages/middlewared/utils/io_thread_pool_executor.py", line 25, in run
result = self.fn(*self.args, **self.kwargs)
File "/usr/local/lib/python3.7/site-packages/middlewared/plugins/activedirectory.py", line 1425, in get_site
site = AD_LDAP.locate_site()
File "/usr/local/lib/python3.7/site-packages/middlewared/plugins/activedirectory.py", line 501, in locate_site
self._open()
File "/usr/local/lib/python3.7/site-packages/middlewared/plugins/activedirectory.py", line 324, in _open
elif saved_bind_error:
UnboundLocalError: local variable 'saved_bind_error' referenced before assignment

 

[anodos]

# midclt call activedirectory.update '{"ssl": "ON", "enable": true}'
local variable 'saved_bind_error' referenced before assignment
Traceback (most recent call last):
File "/usr/local/lib/python3.7/site-packages/middlewared/main.py", line 130, in call_method
io_thread=False)
File "/usr/local/lib/python3.7/site-packages/middlewared/main.py", line 1077, in _call
return await methodobj(*args)
File "/usr/local/lib/python3.7/site-packages/middlewared/service.py", line 302, in update
f'{self._config.namespace}.update', self, self.do_update, [data]
File "/usr/local/lib/python3.7/site-packages/middlewared/main.py", line 1077, in _call
return await methodobj(*args)
File "/usr/local/lib/python3.7/site-packages/middlewared/schema.py", line 960, in nf
return await f(*args, **kwargs)
File "/usr/local/lib/python3.7/site-packages/middlewared/plugins/activedirectory.py", line 876, in do_update
await self.start()
File "/usr/local/lib/python3.7/site-packages/middlewared/plugins/activedirectory.py", line 941, in start
new_site = await self.middleware.run_in_thread(self.get_site)
File "/usr/local/lib/python3.7/site-packages/middlewared/utils/run_in_thread.py", line 10, in run_in_thread
return await self.loop.run_in_executor(self.run_in_thread_executor, functools.partial(method, *args, **kwargs))
File "/usr/local/lib/python3.7/site-packages/middlewared/utils/io_thread_pool_executor.py", line 25, in run
result = self.fn(*self.args, **self.kwargs)
File "/usr/local/lib/python3.7/site-packages/middlewared/plugins/activedirectory.py", line 1425, in get_site
site = AD_LDAP.locate_site()
File "/usr/local/lib/python3.7/site-packages/middlewared/plugins/activedirectory.py", line 501, in locate_site
self._open()
File "/usr/local/lib/python3.7/site-packages/middlewared/plugins/activedirectory.py", line 324, in _open
elif saved_bind_error:
UnboundLocalError: local variable 'saved_bind_error' referenced before assignment

Yeah, that has been fixed as well for 11.3-U1. PM me a debug and I'll see what the real issue is (the bind failed but was masked by a different error).

 

[Fab Sidoli]

Do you mean /var/log/debug.log?

 

[Fab Sidoli]

Any idea when U1 is going to be released?

 

[anodos]

Do you mean /var/log/debug.log?

/var/log/middlewared.log should be sufficient

 

[Fab Sidoli]

I've just sent this to you. Thanks.