Hi.
I have something very strange going on that I thought was fixed a couple times now, but keeps recurring.
We have our truenas server bound to our AD -- and it works flawlessly on windows. However, something very strange is happening under linux, using the mount command.
For Example:
Of course, the credentials are 100% correct, and I use them all the time on windows system without error. The entries we have in our fstab to mount these shares fail similarly.
The really odd thing is, that if I use dolphin, and I enter my credentials in the old netbios fasion of DOMAINNAME/USERNAME -- it will likewise fail -- BUT if I enter them as UPN (myname@mydomain.com) it works! Of course, the mount command cannot do the latter, so there is no way to do this in fstab or on the command line.
What is even more strange to me, is if I use wbinfo on the console of Truenas, I can pull my sid just fine, and I can look at all the users. It looks like winbindd is doing its job...
The other weird thing is that IF I reboot the truenas server -- it works! For a little while anyway... I can mount on the command line as I normally would. I can use my fstab entries just fine. It will keep those shares mounted indefinitely. If I come back a few hours later, unmount them and try to remount them, they will fail -- indefinitely (until the truenas server is rebooted again, then I will again have a small window in which they will work). Because our truenas server is heavily utilized, I cannot just reboot it whenever I want. I have to coordinate a schedule to do so... which makes this extra difficult.
I have tried using the "REDBUILD DIRECTORY SERVICE CACHE" button under Directory Services -- but it changes nothing. We restart the winbind daemon, but it does nothing. The only thing that seems to help is to reboot the TrueNas Server. I suspect if I disjoined and rejoind the domain, it would probably work too -- but again, this is a live system, I cannot just do that. Restarting the samba service likewise does not help. It seems nothing short of a reboot will fix it (for a short while anyway).
Again, using windows -- we have absolutely no issues like this. It is just weird, like it loses its connection to the DC or something... but winbind commands still work... or maybe munges the credentials maybe... but why would it work after a reboot?
I'm looking in the event logs on the domain controller, but I'm not seeing anything weird. I look in the logs on the TrueNas server and I don't get a lot, but I do get the following:
In the Directory Services -> Active Directory section of TN I have:
Enable (requires password or Kerberos Principal) --- Checked
Verblose Logging --- Unchecked (though I did turn it on to capture the above)
User Default Domain --- Unchecked (though I have tried toggling this to no avail)
Allow DNS Updates --- Checked
Disable FreeNas Cache --- Checked (though was also tired unchecking)
Restrict PAM --- Unchecked
Site Name: Default-First-Site-Name
Kerberos Realm: MYDOMAIN.COM
Kerberos Principal: MY-TRUENAS$@MYDOMAIN.COM
Computer Account OU: MY\\ OU
AD Timeout: 60
DNS Timeout: 10
Winbind NSS info: RFC2307
Netbios Name: my-truenas
Netbios Alias: my-truenas
I'm not sure where else to look on this.
Thank you for your assistance.
I have something very strange going on that I thought was fixed a couple times now, but keeps recurring.
We have our truenas server bound to our AD -- and it works flawlessly on windows. However, something very strange is happening under linux, using the mount command.
For Example:
Code:
mount -t cifs //my-truenas/images /mnt/misc -o username=myuser,domain=mydomain Password for myuser@//my-truenas/images: *********** mount error(13): Permission denied Refer to the mount.cifs(8) manual page (e.g. man mount.cifs) and kernel log messages (dmesg)
Of course, the credentials are 100% correct, and I use them all the time on windows system without error. The entries we have in our fstab to mount these shares fail similarly.
The really odd thing is, that if I use dolphin, and I enter my credentials in the old netbios fasion of DOMAINNAME/USERNAME -- it will likewise fail -- BUT if I enter them as UPN (myname@mydomain.com) it works! Of course, the mount command cannot do the latter, so there is no way to do this in fstab or on the command line.
What is even more strange to me, is if I use wbinfo on the console of Truenas, I can pull my sid just fine, and I can look at all the users. It looks like winbindd is doing its job...
The other weird thing is that IF I reboot the truenas server -- it works! For a little while anyway... I can mount on the command line as I normally would. I can use my fstab entries just fine. It will keep those shares mounted indefinitely. If I come back a few hours later, unmount them and try to remount them, they will fail -- indefinitely (until the truenas server is rebooted again, then I will again have a small window in which they will work). Because our truenas server is heavily utilized, I cannot just reboot it whenever I want. I have to coordinate a schedule to do so... which makes this extra difficult.
I have tried using the "REDBUILD DIRECTORY SERVICE CACHE" button under Directory Services -- but it changes nothing. We restart the winbind daemon, but it does nothing. The only thing that seems to help is to reboot the TrueNas Server. I suspect if I disjoined and rejoind the domain, it would probably work too -- but again, this is a live system, I cannot just do that. Restarting the samba service likewise does not help. It seems nothing short of a reboot will fix it (for a short while anyway).
Again, using windows -- we have absolutely no issues like this. It is just weird, like it loses its connection to the DC or something... but winbind commands still work... or maybe munges the credentials maybe... but why would it work after a reboot?
I'm looking in the event logs on the domain controller, but I'm not seeing anything weird. I look in the logs on the TrueNas server and I don't get a lot, but I do get the following:
Code:
auth_audit.log
{"timestamp": "2022-03-22T11:50:14.195037-0400", "type": "Authentication", "Authentication": {"version": {"major": 1, "minor": 2}, "eventId": 4625, "logonId": "47062a26f238cd72", "logonType": 3, "status": "NT_STATUS_ACCESS_DENIED", "localAddress": "unix:", "remoteAddress": "unix:", "serviceDescription": "winbind", "authDescription": "NTLM_AUTH, nss_winbind, 77817", "clientDomain": "mydomain.COM", "clientAccount": "myuser", "workstation": "EMI-TRUENAS", "becameAccount": "", "becameDomain": "", "becameSid": null, "mappedAccount": null, "mappedDomain": null, "netlogonComputer": null, "netlogonTrustAccount": null, "netlogonNegotiateFlags": "0x00000000", "netlogonSecureChannelType": 0, "netlogonTrustAccountSid": null, "passwordType": "NTLMv2", "duration": 231529}}
{"timestamp": "2022-03-22T11:50:14.196100-0400", "type": "Authentication", "Authentication": {"version": {"major": 1, "minor": 2}, "eventId": 4625, "logonId": "0", "logonType": 3, "status": "NT_STATUS_ACCESS_DENIED", "localAddress": "ipv4:192.168.168.47:445", "remoteAddress": "ipv4:192.168.198.111:34512", "serviceDescription": "SMB2", "authDescription": null, "clientDomain": "mydomain.COM", "clientAccount": "myuser", "workstation": "", "becameAccount": null, "becameDomain": null, "becameSid": null, "mappedAccount": "myuser", "mappedDomain": "mydomain.COM", "netlogonComputer": null, "netlogonTrustAccount": null, "netlogonNegotiateFlags": "0x00000000", "netlogonSecureChannelType": 0, "netlogonTrustAccountSid": null, "passwordType": "NTLMv2", "duration": 234808}}
log.smbd
[2022/03/22 11:51:20.537290, 2] ../../source3/lib/tallocmsg.c:84(register_msg_pool_usage)
Registered MSG_REQ_POOL_USAGE
[2022/03/22 11:51:21.007308, 2] ../../source3/auth/auth.c:347(auth_check_ntlm_password)
check_ntlm_password: Authentication for user [myuser] -> [myuser] FAILED with error NT_STATUS_ACCESS_DENIED, authoritative=1
[2022/03/22 11:51:21.007552, 2] ../../auth/auth_log.c:653(log_authentication_event_human_readable)
Auth: [SMB2,(null)] user [mydomain.COM]\[myuser] at [Tue, 22 Mar 2022 11:51:21.007508 EDT] with [NTLMv2] status [NT_STATUS_ACCESS_DENIED] workstation [] remote host [ipv4:192.168.198.111:34514] mapped to [mydomain.COM]\[myuser]. local host [ipv4:192.168.168.47:445]
In the Directory Services -> Active Directory section of TN I have:
Enable (requires password or Kerberos Principal) --- Checked
Verblose Logging --- Unchecked (though I did turn it on to capture the above)
User Default Domain --- Unchecked (though I have tried toggling this to no avail)
Allow DNS Updates --- Checked
Disable FreeNas Cache --- Checked (though was also tired unchecking)
Restrict PAM --- Unchecked
Site Name: Default-First-Site-Name
Kerberos Realm: MYDOMAIN.COM
Kerberos Principal: MY-TRUENAS$@MYDOMAIN.COM
Computer Account OU: MY\\ OU
AD Timeout: 60
DNS Timeout: 10
Winbind NSS info: RFC2307
Netbios Name: my-truenas
Netbios Alias: my-truenas
I'm not sure where else to look on this.
Thank you for your assistance.
