No LDAP / FreeIPA connection after update to 11.3

xenu

Dabbler
Joined
Nov 12, 2015
Messages
43
After updating to FreeNAS 11.3 (coming from 11.2U7) I cannot bind to my FreeIPA/LDAP server anymore.

When I open the WebGUI configuration page for Directory Services -> LDAP the first message I get is "NSS is not a valid idmap backend". The idmap backend dropdown offers me "LDAP" and "RFC2307". Choosing the latter and hitting save results in the same message "RFC2307 is not a valid idmap backend". Choosing "LDAP" lets me save.
When I check "enabled" at the bottom and try to save I get a different error:
"Simultaneous keytab and password authentication are not permitted."
The bind pw field is empty (as in I did not add anything. Maybe a hidden already saved value?). Keytab field value just shows "keytab". Manually entering something in bind pw field and then removing it to empty the field seems to skip this error. Alternatively selecting the keytab dropdown and chosing the only value ("--") also skips this error. For some reason the keytab dropdown does not show any of the three keytabs I had installed and since then also reinstalled via GUI. They are listed in the "Kerberos Keytabs" config page though.
When I do either of the above (set either bind pw or kerberos principal to empty) and try to enable and save the ldap config I get the following error:
Code:
Error: Traceback (most recent call last):
  File "/usr/local/lib/python3.7/site-packages/middlewared/plugins/ldap.py", line 583, in do_update
    await self.middleware.call('ldap.ldap_validate', new)
  File "/usr/local/lib/python3.7/site-packages/middlewared/main.py", line 1127, in call
    app=app, pipes=pipes, job_on_progress_cb=job_on_progress_cb, io_thread=True,
  File "/usr/local/lib/python3.7/site-packages/middlewared/main.py", line 1077, in _call
    return await methodobj(*args)
  File "/usr/local/lib/python3.7/site-packages/middlewared/plugins/ldap.py", line 487, in ldap_validate
    await self.middleware.call('ldap.validate_credentials', data)
  File "/usr/local/lib/python3.7/site-packages/middlewared/main.py", line 1127, in call
    app=app, pipes=pipes, job_on_progress_cb=job_on_progress_cb, io_thread=True,
  File "/usr/local/lib/python3.7/site-packages/middlewared/main.py", line 1091, in _call
    return await run_method(methodobj, *args)
  File "/usr/local/lib/python3.7/site-packages/middlewared/utils/run_in_thread.py", line 10, in run_in_thread
    return await self.loop.run_in_executor(self.run_in_thread_executor, functools.partial(method, *args, **kwargs))
  File "/usr/local/lib/python3.7/site-packages/middlewared/utils/io_thread_pool_executor.py", line 25, in run
    result = self.fn(*self.args, **self.kwargs)
  File "/usr/local/lib/python3.7/site-packages/middlewared/plugins/ldap.py", line 636, in validate_credentials
    ret = LDAP.validate_credentials()
  File "/usr/local/lib/python3.7/site-packages/middlewared/plugins/ldap.py", line 137, in validate_credentials
    ret = self._open()
  File "/usr/local/lib/python3.7/site-packages/middlewared/plugins/ldap.py", line 197, in _open
    ldap.set_option(ldap.OPT_X_TLS_NEWCTX, 0)
  File "/usr/local/lib/python3.7/site-packages/ldap/functions.py", line 103, in set_option
    return _ldap_function_call(None,_ldap.set_option,option,invalue)
  File "/usr/local/lib/python3.7/site-packages/ldap/functions.py", line 55, in _ldap_function_call
    result = func(*args,**kwargs)
ValueError: option error

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.7/site-packages/middlewared/main.py", line 130, in call_method
    io_thread=False)
  File "/usr/local/lib/python3.7/site-packages/middlewared/main.py", line 1077, in _call
    return await methodobj(*args)
  File "/usr/local/lib/python3.7/site-packages/middlewared/service.py", line 302, in update
    f'{self._config.namespace}.update', self, self.do_update, [data]
  File "/usr/local/lib/python3.7/site-packages/middlewared/main.py", line 1077, in _call
    return await methodobj(*args)
  File "/usr/local/lib/python3.7/site-packages/middlewared/schema.py", line 960, in nf
    return await f(*args, **kwargs)
  File "/usr/local/lib/python3.7/site-packages/middlewared/plugins/ldap.py", line 585, in do_update
    raise ValidationError('ldap_update', str(e))
middlewared.service_exception.ValidationError: [EFAULT] ldap_update: option error


If I set encryption mode to "OFF" instead "SSL" I get the following error:
Code:
Error: Traceback (most recent call last):
  File "/usr/local/lib/python3.7/site-packages/middlewared/plugins/ldap.py", line 583, in do_update
    await self.middleware.call('ldap.ldap_validate', new)
  File "/usr/local/lib/python3.7/site-packages/middlewared/main.py", line 1127, in call
    app=app, pipes=pipes, job_on_progress_cb=job_on_progress_cb, io_thread=True,
  File "/usr/local/lib/python3.7/site-packages/middlewared/main.py", line 1077, in _call
    return await methodobj(*args)
  File "/usr/local/lib/python3.7/site-packages/middlewared/plugins/ldap.py", line 487, in ldap_validate
    await self.middleware.call('ldap.validate_credentials', data)
  File "/usr/local/lib/python3.7/site-packages/middlewared/main.py", line 1127, in call
    app=app, pipes=pipes, job_on_progress_cb=job_on_progress_cb, io_thread=True,
  File "/usr/local/lib/python3.7/site-packages/middlewared/main.py", line 1091, in _call
    return await run_method(methodobj, *args)
  File "/usr/local/lib/python3.7/site-packages/middlewared/utils/run_in_thread.py", line 10, in run_in_thread
    return await self.loop.run_in_executor(self.run_in_thread_executor, functools.partial(method, *args, **kwargs))
  File "/usr/local/lib/python3.7/site-packages/middlewared/utils/io_thread_pool_executor.py", line 25, in run
    result = self.fn(*self.args, **self.kwargs)
  File "/usr/local/lib/python3.7/site-packages/middlewared/plugins/ldap.py", line 636, in validate_credentials
    ret = LDAP.validate_credentials()
  File "/usr/local/lib/python3.7/site-packages/middlewared/plugins/ldap.py", line 137, in validate_credentials
    ret = self._open()
  File "/usr/local/lib/python3.7/site-packages/middlewared/plugins/ldap.py", line 251, in _open
    raise CallError(str(saved_simple_error))
middlewared.service_exception.CallError: [EFAULT] {'desc': "Can't contact LDAP server", 'errno': 2, 'info': 'No such file or directory'}

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.7/site-packages/middlewared/main.py", line 130, in call_method
    io_thread=False)
  File "/usr/local/lib/python3.7/site-packages/middlewared/main.py", line 1077, in _call
    return await methodobj(*args)
  File "/usr/local/lib/python3.7/site-packages/middlewared/service.py", line 302, in update
    f'{self._config.namespace}.update', self, self.do_update, [data]
  File "/usr/local/lib/python3.7/site-packages/middlewared/main.py", line 1077, in _call
    return await methodobj(*args)
  File "/usr/local/lib/python3.7/site-packages/middlewared/schema.py", line 960, in nf
    return await f(*args, **kwargs)
  File "/usr/local/lib/python3.7/site-packages/middlewared/plugins/ldap.py", line 585, in do_update
    raise ValidationError('ldap_update', str(e))
middlewared.service_exception.ValidationError: [EFAULT] ldap_update: [EFAULT] {'desc': "Can't contact LDAP server", 'errno': 2, 'info': 'No such file or directory'}


The Certificate value reads:
"freeipa_root_ca (migrated for ldap at 2020-01-29 16:50:56.391583)". I tried both "validate certificates" checked and unchecked.

Some other things I noticed:
# klist
klist: No ticket file: /tmp/krb5cc_0

There are no *keytab* files in /

Manually copying the host keytab to /etc/krb5.keytab and running
Code:
kinit -k -t /etc/krb5.keytab host/freenas01.ipa.mydomain.de@IPA.MYDOMAIN.DE
works to get a ticket.


/etc/local/nslcd.conf is empty except for a comment in the beginning.
/etc/local/openldap/ldap.conf is empty.
directory /etc/kerberos is empty

# service nslcd onestatus
nslcd: no URIs defined in config
nslcd not running?


Any help to debug this greatly appreciated as my nfs shares use nfsv4 with kerberos for access control which currently does not work.

*EDIT: When manually doing the kinit as mentioned above I get the Kerberos Prinicpal offered in the LDAP config dropdown. For some reason the ix-kinit service does not load the saved keytabs automatically. Also tried manually doing "LD_LIBRARY_PATH=/usr/local/lib /usr/local/bin/midclt call kerberos.start" and "LD_LIBRARY_PATH=/usr/local/lib /usr/local/bin/midclt call kerberos.renew".

**EDIT2: kdc service was not enabled and not running.
 
Last edited:

anodos

Sambassador
iXsystems
Joined
Mar 6, 2014
Messages
9,545
After updating to FreeNAS 11.3 (coming from 11.2U7) I cannot bind to my FreeIPA/LDAP server any more.

When I open the WebGUI configuration page for Directory Services -> LDAP the first message I get is "NSS is not a valid idmap backend". The idmap backend dropdown offers me "LDAP" and "RFC2307". Choosing the latter and hitting save results in the same message "RFC2307 is not a valid idmap backend". Choosing "LDAP" lets me save.
When I check "enabled" at the bottom and try to save I get a different error:
"Simultaneous keytab and password authentication are not permitted."
The bind pw field is empty (as in I did not add anything. Maybe a hidden already saved value?). Keytab field value just shows "keytab". Manually entering something in bind pw field and then removing it to empty the field seems to skip this error. Alternatively selecting the keytab dropdown and chosing the only value ("--") also skips this error. For some reason the keytab dropdown does not show any of the three keytabs I had installed and since then also reinstalled via GUI. They are listed in the "Kerberos Keytabs" config page though.
When I do either of the above (set either bind pw or kerberos principal to empty) and try to enable and save the ldap config I get the following error:
Code:
Error: Traceback (most recent call last):
  File "/usr/local/lib/python3.7/site-packages/middlewared/plugins/ldap.py", line 583, in do_update
    await self.middleware.call('ldap.ldap_validate', new)
  File "/usr/local/lib/python3.7/site-packages/middlewared/main.py", line 1127, in call
    app=app, pipes=pipes, job_on_progress_cb=job_on_progress_cb, io_thread=True,
  File "/usr/local/lib/python3.7/site-packages/middlewared/main.py", line 1077, in _call
    return await methodobj(*args)
  File "/usr/local/lib/python3.7/site-packages/middlewared/plugins/ldap.py", line 487, in ldap_validate
    await self.middleware.call('ldap.validate_credentials', data)
  File "/usr/local/lib/python3.7/site-packages/middlewared/main.py", line 1127, in call
    app=app, pipes=pipes, job_on_progress_cb=job_on_progress_cb, io_thread=True,
  File "/usr/local/lib/python3.7/site-packages/middlewared/main.py", line 1091, in _call
    return await run_method(methodobj, *args)
  File "/usr/local/lib/python3.7/site-packages/middlewared/utils/run_in_thread.py", line 10, in run_in_thread
    return await self.loop.run_in_executor(self.run_in_thread_executor, functools.partial(method, *args, **kwargs))
  File "/usr/local/lib/python3.7/site-packages/middlewared/utils/io_thread_pool_executor.py", line 25, in run
    result = self.fn(*self.args, **self.kwargs)
  File "/usr/local/lib/python3.7/site-packages/middlewared/plugins/ldap.py", line 636, in validate_credentials
    ret = LDAP.validate_credentials()
  File "/usr/local/lib/python3.7/site-packages/middlewared/plugins/ldap.py", line 137, in validate_credentials
    ret = self._open()
  File "/usr/local/lib/python3.7/site-packages/middlewared/plugins/ldap.py", line 197, in _open
    ldap.set_option(ldap.OPT_X_TLS_NEWCTX, 0)
  File "/usr/local/lib/python3.7/site-packages/ldap/functions.py", line 103, in set_option
    return _ldap_function_call(None,_ldap.set_option,option,invalue)
  File "/usr/local/lib/python3.7/site-packages/ldap/functions.py", line 55, in _ldap_function_call
    result = func(*args,**kwargs)
ValueError: option error

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.7/site-packages/middlewared/main.py", line 130, in call_method
    io_thread=False)
  File "/usr/local/lib/python3.7/site-packages/middlewared/main.py", line 1077, in _call
    return await methodobj(*args)
  File "/usr/local/lib/python3.7/site-packages/middlewared/service.py", line 302, in update
    f'{self._config.namespace}.update', self, self.do_update, [data]
  File "/usr/local/lib/python3.7/site-packages/middlewared/main.py", line 1077, in _call
    return await methodobj(*args)
  File "/usr/local/lib/python3.7/site-packages/middlewared/schema.py", line 960, in nf
    return await f(*args, **kwargs)
  File "/usr/local/lib/python3.7/site-packages/middlewared/plugins/ldap.py", line 585, in do_update
    raise ValidationError('ldap_update', str(e))
middlewared.service_exception.ValidationError: [EFAULT] ldap_update: option error


If I set encryption mode to "OFF" instead "SSL" I get the following error:
Code:
Error: Traceback (most recent call last):
  File "/usr/local/lib/python3.7/site-packages/middlewared/plugins/ldap.py", line 583, in do_update
    await self.middleware.call('ldap.ldap_validate', new)
  File "/usr/local/lib/python3.7/site-packages/middlewared/main.py", line 1127, in call
    app=app, pipes=pipes, job_on_progress_cb=job_on_progress_cb, io_thread=True,
  File "/usr/local/lib/python3.7/site-packages/middlewared/main.py", line 1077, in _call
    return await methodobj(*args)
  File "/usr/local/lib/python3.7/site-packages/middlewared/plugins/ldap.py", line 487, in ldap_validate
    await self.middleware.call('ldap.validate_credentials', data)
  File "/usr/local/lib/python3.7/site-packages/middlewared/main.py", line 1127, in call
    app=app, pipes=pipes, job_on_progress_cb=job_on_progress_cb, io_thread=True,
  File "/usr/local/lib/python3.7/site-packages/middlewared/main.py", line 1091, in _call
    return await run_method(methodobj, *args)
  File "/usr/local/lib/python3.7/site-packages/middlewared/utils/run_in_thread.py", line 10, in run_in_thread
    return await self.loop.run_in_executor(self.run_in_thread_executor, functools.partial(method, *args, **kwargs))
  File "/usr/local/lib/python3.7/site-packages/middlewared/utils/io_thread_pool_executor.py", line 25, in run
    result = self.fn(*self.args, **self.kwargs)
  File "/usr/local/lib/python3.7/site-packages/middlewared/plugins/ldap.py", line 636, in validate_credentials
    ret = LDAP.validate_credentials()
  File "/usr/local/lib/python3.7/site-packages/middlewared/plugins/ldap.py", line 137, in validate_credentials
    ret = self._open()
  File "/usr/local/lib/python3.7/site-packages/middlewared/plugins/ldap.py", line 251, in _open
    raise CallError(str(saved_simple_error))
middlewared.service_exception.CallError: [EFAULT] {'desc': "Can't contact LDAP server", 'errno': 2, 'info': 'No such file or directory'}

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.7/site-packages/middlewared/main.py", line 130, in call_method
    io_thread=False)
  File "/usr/local/lib/python3.7/site-packages/middlewared/main.py", line 1077, in _call
    return await methodobj(*args)
  File "/usr/local/lib/python3.7/site-packages/middlewared/service.py", line 302, in update
    f'{self._config.namespace}.update', self, self.do_update, [data]
  File "/usr/local/lib/python3.7/site-packages/middlewared/main.py", line 1077, in _call
    return await methodobj(*args)
  File "/usr/local/lib/python3.7/site-packages/middlewared/schema.py", line 960, in nf
    return await f(*args, **kwargs)
  File "/usr/local/lib/python3.7/site-packages/middlewared/plugins/ldap.py", line 585, in do_update
    raise ValidationError('ldap_update', str(e))
middlewared.service_exception.ValidationError: [EFAULT] ldap_update: [EFAULT] {'desc': "Can't contact LDAP server", 'errno': 2, 'info': 'No such file or directory'}


The Certificate value reads:
"freeipa_root_ca (migrated for ldap at 2020-01-29 16:50:56.391583)". I tried both "validate certificates" checked and unchecked.

Some other things I noticed:
# klist
klist: No ticket file: /tmp/krb5cc_0

There are no *keytab* files in /

Manually copying the host keytab to /etc/krb5.keytab and running
Code:
kinit -k -t /etc/krb5.keytab host/freenas01.ipa.mydomain.de@IPA.MYDOMAIN.DE
works to get a ticket.


/etc/local/nslcd.conf is empty except for a comment in the beginning.
/etc/local/openldap/ldap.conf is empty.
directory /etc/kerberos is empty

# service nslcd onestatus
nslcd: no URIs defined in config
nslcd not running?


Any help to debug this greatly appreciated as my nfs shares use nfsv4 with kerberos for access control which currently does not work.

*EDIT: When manually doing the kinit as mentioned above I get the Kerberos Prinicpal offered in the LDAP config dropdown. For some reason the ix-kinit service does not load the saved keytabs automatically. Also tried manually doing "LD_LIBRARY_PATH=/usr/local/lib /usr/local/bin/midclt call kerberos.start" and "LD_LIBRARY_PATH=/usr/local/lib /usr/local/bin/midclt call kerberos.renew".

**EDIT2: kdc service was not enabled and not running.
Several quick points:
1) I'm working on the ldap option error right now. In 11.3 we switched to forcing certificate validation by default. Try applying all the settings (with cert validation unchecked, and "enable" unchecked). The "option error" is being returned by the underlying LDAP C API, and indicates an unexpected failure to initialize the LDAP connection. The two other users I've seen with this problem appear to be running LDAP on a non-standard port (or have the port number appended to the URI string in the hostname field (foo.com:636)).
2) To generate kerberos keytab and krb5.conf, you will need to run 'midclt call etc.generate kerberos', but you will need LDAP enabled.
3) Certficate dropdown is now only for certficate-based authentication (SASL_EXTERNAL). In 11.2, it allowed you to specify a CACERT. In 11.3, user-provided cacerts are concatenated into a single file and used automatically. This is to make things work more seamlessly.

In short,
a) strip port number from URI string if possible
b) uncheck "validate"
c) choose kerberos principal from dropdown (if you're doing SASL_GSSAPI)
d) uncheck "enable"
e) save
f) check "enable" and save again.
 

xenu

Dabbler
Joined
Nov 12, 2015
Messages
43
Thank you for the quick reply, anodos. Unfortunatly the same error message as the first one in my original post.

a) I do not have a port number added to the URI. I removed the saved value and reentered the host. Also tried using the IP adress.
b) unchecked validate.
c) I do not have a kerberos principal offered in the dropdown. When I copy the keytab manually and do a kinit as mentioned above I get the keytab principal offered in the dropdown. I opted to enter the bind pw instead for now.
d) unchecked enable
e) saved -> this works
f) checked enable and saved again -> option error (see above)
 

xenu

Dabbler
Joined
Nov 12, 2015
Messages
43
Solved - Thanks for the help! For anyone running into this issue:
- fixes keytabs not being added /generated

- fixes principals not being listed - though replace
Code:
fields['1']
with
Code:
fields[1]
 

anodos

Sambassador
iXsystems
Joined
Mar 6, 2014
Messages
9,545
Solved - Thanks for the help! For anyone running into this issue:
- fixes keytabs not being added /generated

- fixes principals not being listed - though replace
Code:
fields['1']
with
Code:
fields[1]
You're missing one commit from that. It's better for the case of the kerberos plugin to just grab the latest from 11.3-stable for right now if you run into issues with kerberos and LDAP in 11.3-RELEASE:
 

BeepDog

Dabbler
Joined
Feb 29, 2020
Messages
12
To crosspost:

I was able to work around this, enough to get my Samba shares working again. It's not ideal, and I got occasional other horrible like ldap.NO_SUCH_OBJECT. But sporatically, and not enough to ruin everything.
 
Top