UID / GID Active Directory disparity between two FreeNAS machines

[Gordon Thagard]

FreeNAS-11.3-U2.1

I'm not sure how it happened. We have two FreeNAS machines that serve both WIndows and NFS shares. Yes, I know. We got them to play nice by creating a Windows AD user and group, NFS mounting them from a UNIX system and observing the uid/gid for files on the mountpoint. We then created a local user and group with the same uid/gid on each UNIX system and told FreeNAS to NFS mapall the user/group. The uid/gid was the same for both FreeNAS machines and it just worked for many years that we could serve Windows and NFS simultaneously without a hitch.

There were issues switching trains to 11.3 on both systems and it took a lot of finagling to get AD to work at all with 11.3, but we did. Now we have a problem. The uid and gid are different on the two FreeNAS boxes. Is there any way to make one system use the same uid/gid for Active Directory users/groups as the other?

 

[anodos]

FreeNAS-11.3-U2.1

I'm not sure how it happened. We have two FreeNAS machines that serve both WIndows and NFS shares. Yes, I know. We got them to play nice by creating a Windows AD user and group, NFS mounting them from a UNIX system and observing the uid/gid for files on the mountpoint. We then created a local user and group with the same uid/gid on each UNIX system and told FreeNAS to NFS mapall the user/group. The uid/gid was the same for both FreeNAS machines and it just worked for many years that we could serve Windows and NFS simultaneously without a hitch.

There were issues switching trains to 11.3 on both systems and it took a lot of finagling to get AD to work at all with 11.3, but we did. Now we have a problem. The uid and gid are different on the two FreeNAS boxes. Is there any way to make one system use the same uid/gid for Active Directory users/groups as the other?

You need to compare / modify the idmap settings on both servers.

 

[Gordon Thagard]

They're both using RID with a low range of 20000 and a high range of 90000000.

 

[anodos]

They're both using RID with a low range of 20000 and a high range of 90000000.

SID->ID mapping is deterministic for AD users/ groups in this case. It should be 20,000 + RID (last component of SID).

 

[Gordon Thagard]

Deterministic? Really? Then I am at a loss as to how the two machines could idmap to two different uids and gids.

Is there a way to "reset" idmap and start fresh? Both systems hickuped during the change train process as uid and gid are different now than how they have been for the past 3-4 years. If it was deterministic then the idmapping wouldn't change, right?

 

[anodos]

Deterministic? Really? Then I am at a loss as to how the two machines could idmap to two different uids and gids.

Is there a way to "reset" idmap and start fresh? Both systems hickuped during the change train process as uid and gid are different now than how they have been for the past 3-4 years. If it was deterministic then the idmapping wouldn't change, right?

How are they different? Can you post an example? ( id <user> on both systems or getent group <group> if it's a group) The winbindd resolver cache can be flushed, but I'd rather understand how this situation happened in the first place.

 

[Gordon Thagard]

root@freenas1:~ # id "AD\build"
uid=90000002(AD\build) gid=20513(AD\domain users) groups=20513(AD\domain users),21159(AD\fc),21161(AD\toronto),21167(AD\zipper),21127(AD\development),21312(AD\sslvpn),21204(AD\ctibuild),21166(AD\walther),21313(AD\virtual),21320(AD\unixusers),21128(AD\tech),90000035(AD\virtual),90000044(AD\fc),90000036(AD\dev),90000037(AD\tech),90000038(AD\sslvpn),90000039(AD\toronto),90000040(AD\unixusers),90000041(AD\zipper),90000045(AD\walther),90000042(AD\ctibuild),90000002(BUILTIN\users)


root@freenas2[~]# id "AD\build"
uid=21173(AD\build) gid=20513(AD\domain users) groups=20513(AD\domain users),21173(AD\build),21159(AD\fc),21161(AD\toronto),21167(AD\zipper),21127(AD\development),21312(AD\sslvpn),21204(AD\ctibuild),21166(AD\walther),21313(AD\virtual),21320(AD\unixusers),21128(AD\tech),90000016(AD\virtual),90000017(AD\fc),90000007(AD\dev),90000009(AD\tech),90000010(AD\sslvpn),90000018(AD\toronto),90000011(AD\unixusers),90000012(AD\zipper),90000019(AD\walther),90000014(AD\ctibuild),90000022(BUILTIN\users)


From UNIX system's perspective we're talking:
freenas1: uid 90000002 gid 90000042, ie build/ctibuild
freenas2: uid 21173 gid 90000014, ie build/ctibuild

Some obfuscation has been done with the domain name and group names, but that's the full output.

 

[anodos]

FreeNAS 01 has an incorrectly allocated ID for UID for AD01\BUILD. The mixture of the two indicates possibly some problems with smb.conf at time when users were accessing the computers (IDs were allocated using the default "tdb" backend). I haven't seen this particular issue before, and am not sure how you got here. Step forward is to back up /var/db/system/samba4/winbindd_idmap.tdb, and then

Code:

service samba_server onestop
rm /var/db/system/samba4/winbindd_idmap.tdb
midclt call idmap.clear_idmap_cache

This will probably affect permissions on possibly all files written to the server(s) and require re-doing ACLs.

 

[Gordon Thagard]

The first time I did as instructed the UID was the same on both systems but the GID was different. So I removed both systems from the domain, again followed the steps, and then rejoined the domain. That finally fixed it. UID and GUD match on FreeNAS1 and FreeNAS2. I do have to re-apply user and group ownership to all datasets but that's a small price to pay. Thank you you very much for your guidance.

 

[Gordon Thagard]

I spoke too soon. FreeNAS1 serves two shares relevant to both UNIX and Windows dev machines. FreeNAS2 serves 1 share. All shares show up with the same GID, 90000013. One share from FreeNAS1 is showing up with the same UID as on FreeNAS2, 21173, while the other share from FreeNAS1 is showing as UID 90000002. I also updated to the latest level this morning on both systems before starting this exercise as the release notes mentioned AD rejoin improvements and a number of samba bug fixes. Sadly, none of those patches address this very strange case.

I'm am wondering if I should hop back on FreeNAS1 and follow all the steps to clear the idmap_cache again or is there something less drastic I can try? It takes quite a while to re-push all those ACLs for all the datasets and even after the latest update getting FreeNAS to rejoin the domain is work - lotsa fiddling before it plays nice.

root@freenas1:~ # id "AD\build"
id "AD\build"
uid=21173(AD\build) gid=20513(AD\domain users) groups=20513(AD\domain users),21173(AD\build),21159(AD\fc),21161(AD\toronto),21167(AD\zipper),21127(AD\development),21312(AD\sslvpn),21204(AD\ctibuild),21166(AD\walther),21313(AD\virtual),21320(AD\unixusers),21128(AD\tech),90000004(AD\virtual),90000005(AD\fc),90000006(AD\development),90000007(AD\tech),90000008(AD\sslvpn),90000009(AD\toronto),90000010(AD\unixusers),90000011(AD\zipper),90000012(AD\walther),90000013(AD\ctibuild),90000002(BUILTIN\users)


root@freenas2[~]# id "AD\build"
uid=21173(AD\build) gid=20513(AD\domain users) groups=20513(AD\domain users),21173(AD\build),21159(AD\fc),21161(AD\toronto),21167(AD\zipper),21127(AD\development),21312(AD\sslvpn),21204(AD\ctibuild),21166(AD\walther),21313(AD\virtual),21320(AD\unixusers),21128(AD\tech),90000004(AD\virtual),90000005(AD\fc),90000006(AD\development),90000007(AD\tech),90000008(AD\sslvpn),90000009(AD\toronto),90000010(AD\unixusers),90000011(AD\zipper),90000012(AD\walther),90000013(AD\ctibuild)


From UNIX system's perspective we're talking:
freenas1, share1: uid 90000002 gid 90000013, ie build/ctibuild
freenas1, share2: uid 21173 gid 90000013, ie build/ctibuild
freenas2: uid 21173 gid 90000013, ie build/ctibuild

 

[anodos]

I spoke too soon. FreeNAS1 serves two shares relevant to both UNIX and Windows dev machines. FreeNAS2 serves 1 share. All shares show up with the same GID, 90000013. One share from FreeNAS1 is showing up with the same UID as on FreeNAS2, 21173, while the other share from FreeNAS1 is showing as UID 90000002. I also updated to the latest level this morning on both systems before starting this exercise as the release notes mentioned AD rejoin improvements and a number of samba bug fixes. Sadly, none of those patches address this very strange case.

I'm am wondering if I should hop back on FreeNAS1 and follow all the steps to clear the idmap_cache again or is there something less drastic I can try? It takes quite a while to re-push all those ACLs for all the datasets and even after the latest update getting FreeNAS to rejoin the domain is work - lotsa fiddling before it plays nice.

root@freenas1:~ # id "AD\build"
id "AD\build"
uid=21173(AD\build) gid=20513(AD\domain users) groups=20513(AD\domain users),21173(AD\build),21159(AD\fc),21161(AD\toronto),21167(AD\zipper),21127(AD\development),21312(AD\sslvpn),21204(AD\ctibuild),21166(AD\walther),21313(AD\virtual),21320(AD\unixusers),21128(AD\tech),90000004(AD\virtual),90000005(AD\fc),90000006(AD\development),90000007(AD\tech),90000008(AD\sslvpn),90000009(AD\toronto),90000010(AD\unixusers),90000011(AD\zipper),90000012(AD\walther),90000013(AD\ctibuild),90000002(BUILTIN\users)


root@freenas2[~]# id "AD\build"
uid=21173(AD\build) gid=20513(AD\domain users) groups=20513(AD\domain users),21173(AD\build),21159(AD\fc),21161(AD\toronto),21167(AD\zipper),21127(AD\development),21312(AD\sslvpn),21204(AD\ctibuild),21166(AD\walther),21313(AD\virtual),21320(AD\unixusers),21128(AD\tech),90000004(AD\virtual),90000005(AD\fc),90000006(AD\development),90000007(AD\tech),90000008(AD\sslvpn),90000009(AD\toronto),90000010(AD\unixusers),90000011(AD\zipper),90000012(AD\walther),90000013(AD\ctibuild)


From UNIX system's perspective we're talking:
freenas1, share1: uid 90000002 gid 90000013, ie build/ctibuild
freenas1, share2: uid 21173 gid 90000013, ie build/ctibuild
freenas2: uid 21173 gid 90000013, ie build/ctibuild

What are the SID values for AD\build and AD\citbuild? "wbinfo --name-to-sid=AD\\build" What is output of "wbinfo -m"?

 

[Gordon Thagard]

[root@freenas1~]# wbinfo --name-to-sid=AD\\build
S-1-5-21-3813042580-2950360293-480820445-1173 SID_USER (1)
[root@freenas1~]# wbinfo -m
BUILTIN
FREENAS1
AD

root@freenas2[~]# wbinfo --name-to-sid=AD\\build
S-1-5-21-3813042580-2950360293-480820445-1173 SID_USER (1)
[root@freenas2~]# wbinfo -m
BUILTIN
FREENAS2
AD

 

[anodos]

What about AD\\ctibuild SID?

 

[Gordon Thagard]

[root@freenas1~]# wbinfo --name-to-sid=AD\\ctibuild
S-1-5-21-3813042580-2950360293-480820445-1204 SID_DOM_GROUP (2)

root@freenas2[~]# wbinfo --name-to-sid=AD\\ctibuild
S-1-5-21-3813042580-2950360293-480820445-1204 SID_DOM_GROUP (2)

 

[Gordon Thagard]

I did a fresh, no-preserve anything install of freenas1 and freenas2 using FreeNAS-11.3-U3.1, then imported the pools and got everything setup before re-joining the domain. That ultimately solved it. Now the uid is 100001174 on both systems. GID stayed the same at 90000013 . I now have to re-apply all those ACLs and permissions. This has been very painful.

 

[Patrick M. Hausen]

That's why I prefer to nail down the UIDs and GIDs with MSFU and manual entry when I create users in AD. Then use nss_ldap with Samba.
@anodos said that Samba in FreeNAS can fall back on MSFU, too, if I remember correctly.

 

[anodos]

I did a fresh, no-preserve anything install of freenas1 and freenas2 using FreeNAS-11.3-U3.1, then imported the pools and got everything setup before re-joining the domain. That ultimately solved it. Now the uid is 100001174 on both systems. GID stayed the same at 90000013 . I now have to re-apply all those ACLs and permissions. This has been very painful.

This may be a winbindd caching bug (perhaps connection to DC is being overloaded during initial UI cache build). Why don't you try disabling the freenas UI cache in DS->Active Directory try again (clearing out the winbindd cache and winbindd_idmap.tdb file?

 

[Gordon Thagard]

Of course, I spoke too soon, but on the flip side it really is fixed now. My initial testing just using wbinfo, etc indicated that it was finally solved, but it was still not right with all shares having same UID/GID after I finally pushed ACLs and permissions and mounted them, so i toggled off cache, as suggested, and re-ran the steps on both systems. Still not right, so I turned the cache back on, and re-ran the steps. Now, I swear, all three shares have the same UID/GID as seen from an NFS mount. Case closed.

 

[clicks]

I stumpled upon a quite similar problem. I replicated a Freenas 11 box to a Truenas 12 box. All UIDs/GIDs have been copied to the new system as well. But the mapping process between the UID/GID to the appropriate Active Directory SIDs got mixed up, as the initial id ranges on Truenas 12 differ from 11. After adjusting the id ranges and recreating the cache the mapping for the primary domain was right.

But the mappings for a secondary (trusted) domain where still off. While the primary domain uses the deterministic rid backend, the secondary domain uses the "default" backend which is tdb. And it seems like the mapping process in the tdb backend works on a first-come first-serve process, which can vary vastly.

I tried different approaches to bring up the tdb-backend with the same mapping as the source freenas box, without any real success. In the end I copied the winbindd_idmap.tdb (and accompanying files) from the source box to the new server. And on first sight it seems to work. Users and groups seem to be mapped the same way on both systems. Is this a sane way to do this, or will I stumple upon new problems in the future?

 

[anodos]

There's a GUI-driven idmap editor in 12.0. You can adjust them to whatever you want. You can also set explicit mappings for your various trusted domains.

The default range when RID is used for new servers was bumped up to avoid idmapping conflicts potentially with some built-in/system accounts. During upgrade, all previous idmapping settings are preserved.