,

I'm happy that this worked for you.

In the future, here's a tip.  There's a way to know the "in memory" parameters of CentrifyDC (documented in the cheat sheet).

$ (sudo or dzdo) adinfo --sysinfo config

This way, if you suspect that the configuration file is correct, but the behavior of the client is different (e.g. the system has been running for a long time, but the parameters may have been edited and committed). 

$ grep sntp /etc/centrifydc/centrifydc.conf
# adclient.sntp.enabled: true
# The interval between sntp clock updates.  The value is the base 2
# adclient.sntp.poll: 15
$ dzdo adinfo --sysinfo config | grep sntp
adclient.sntp.enabled: false

Notice the mismatch between the in-memory parameter vs. the parameter in the config file.

R.P

Hi,

Login fails with "dual-persona" enabled DoD CAC - XEROX (FOR DOD) using kerberos cac authentication

Configure the printer for CAC Authentication.
Login Name for the CAC User in Active Directory = 2001439475@mil
Login Name being used by the printer when the CAC is inserted = 2001439750170084@mil
From the network traces, we see it is sending the AS-REQ with PIV user name (16 digit name) instead of CAC name (10 digits).

 DoD has enabled “dual-persona” on their CAC cards, when they use those cards in  products, the smart card login is failing. From the network traces, we see it is sending the AS-REQ with PIV user name (16 digit name) instead of CAC name (10 digits). 

Further we analyzed it and observed the credentials cache is updated with Default Principal Name as PIV user name.

/opt/nc/dlms/kerberos/apps/klist -c /tmp/krbtc_1 –f

Output

Ticket cache: FILE:/tmp/krbtc_1

Default principal: 2001439750170084\@mil@CAC12DOM.LAB

Valid starting     Expires            Service principal

09/27/17 11:57:32  09/27/17 21:57:37  krbtgt/CAC12DOM.LAB@CAC12DOM.LAB

        renew until 09/28/17 11:57:32, Flags: FRIA

09/27/17 11:57:38  09/27/17 21:57:37  ldap/cac2.cac12dom.lab@

        renew until 09/28/17 11:57:32, Flags: FRAO

09/27/17 11:57:38  09/27/17 21:57:37          renew until 09/28/17 11:57:32, Flags: FRAO

We get the principal name from a certificate associated with authentication on the card. Since the DoD is using Dual ID cards, odds are we are pulling the first cert that is an authentication cert from the card (in this case PIV based one).

This is found in pkinit_matching or pkinit_identity.

How could we change the order here so it pulls the CAC authentication certificate instead of the PIV certificate in kerberos pkinit module?

How could we swap between CAC authentication certificate and PIV certificate in pkinit module in kerberos?

Regards,

Nishanth

Welcome to the Centrify Community .

Try adding the MAC user to be part of the LPOperator of the LPAdmin group. The KB below will step you through the process. This process is for a GPO to allow this.

https://centrify.force.com/support/Article/KB-2986-How-to-use-the-Map-zone-groups-to-local-group-GP-for-Mac-systems

Hi all,

I use a Centrify in Express Mode in my company, for MAC and Linux too. But I noticed that, sometimes when I login to my machine just open a terminal, type "adinfo" and I see "Centrify: disconnected". Why? 

The network service start after the Centrify service and for this reason Centrify don't come up? 

Any solutions?

Thanks

Alex

We use Centrify Express on several production servers that are a bit dated. On our file server (CentOS 6.3, Centrify DC 5.0.4), Centrify has been working great for years. Yesterday, we rebooted the server to troubleshoot an issue with our backup server, and Centrify never bounced back.

Whenever a user attempts to access a Samba share, Centrify would crash and create crash dump files.

We've been troubleshooting non-stop for about 36 hours now, so describing everything we've done is probably useless, so here's where we are now:

1) We used yum remove CentrifyDC to uninstall Centrify (after doing the adleave command).

2) We used yum update to update the server to CentOS 6.9.

3) We reinstalled Samba and were able to verify that we could connect to local Samba shares with local accounts with Centrify removed.

4) We downloaded Centrify Suite 2017 (Centrify DC 5.4.2) and installed it using instructions.

5) We used the adjoin command to joint to our domain without errors. The adinfo command shows valid info for our domain controller.

5) We are still using stock Samba and our backed-up config for that.

At this point, our symptoms are:

1) We can't login to the server with domain credentials. It accepts the username and password, then reverts back to the login screen (yes, home directories exist with the correct permissions).

2) We can't login to the server with local credentials that are not root, but root is working correctly.

3) We cannot connect to Samba shares without getting "Access denied" errors, though permissions have not been changed.

4) We don't know how to test to ensure that Centrify is working correctly or not.

At this point, we're really uncertain as to how to proceed. Given that Samba shares work fine without Centrify installed, we believe that Samba is configured correctly and functioning. We believe that Centrify is not processing authentication correctly or that we have a misconfiguration somewhere, but don't know where to begin.

Any advice is good advice! Thanks in advance for the help.

,

Welcome to the Centrify forums.

Centrify adclient may go in disconnected mode if:

• A local event prevents the client from reaching a Microsoft Active Directory Domain Controller (DC)
  e.g. A OS X system switches to a network outside the corporate site.
• A network event or intermediate device prevents the client from reaching a Microsoft Active Directory Domain Controller (DC)
  e.g. A firewall or network change is introduced.
• An Active Directory topology change
  E.g. an Active Directory sites and services change prevents this.

Note that like any other Active Directory client, we rely on a healthy Domain Name System (DNS) to be able to resolve Active Directory records.  Nonetheless the client also maintains stats and a cache for DNS.

The client will proactively try to connect to systems:

1. primarily in the current AD site

2. alternatively in other AD sites

3. in case no DC is reachable, previously-logged-in users (or prevalidated users) will sign-in with cached credentials.

The client will also keep a tally of eligible domain controllers and proactively probe them for response times.

The good news is that if you can log in in disconnected mode, means that you're logging in with cached credentials.

There are many utilities available to monitor and diagnose connectivity issues, but the most important think to keep in mind is a basic checklist:

a) Has the system connected before?

b) Has there been a change on the network or topology?

c) Has there been a configuration change?

d) Are AD sites and services ( and subnets) properly maintained?

e) Have the unreachable DCs (perhaps in a DR site or in the DMZ) excluded from the pool of elegible DCs?

f) Have there been any DCs decommissioned incorrectly?

With that in mind, your toolset are:

- adcheck:  performs all the basic checks for compatibility, network and AD connectivity

- adinfo:  when used with the -T switch or the --diag option it can provide you with a lot of interesting capability

- syslog:  events are written when the system has disconnected, reconnected, etc.

For more info, check out the cheat sheet:  https://community.centrify.com/t5/TechBlog/TIPS-A-Centrify-Server-Suite-Cheat-Sheet/ba-p/22568

Note on your comment:

"The network service start after the Centrify service and for this reason 
Centrify don't come up?"

The adclient daemon is up, otherwise you would not have been able to log in with cached credentials.

R.P

,

Welcome to the Centrify community. 

I will give you some rapid-fire comments.  Bottomline, due to the poor lifecycle management of the software, there are "many things going on" :

On our file server (CentOS 6.3, Centrify DC 5.0.4), Centrify has been working '
great for years. Yesterday, we rebooted the server to troubleshoot an issue 
with our backup server, and Centrify never bounced back.

Thanks for the feedback. Unfortunately the " set it and forget it"  approach does not help with authentication software (because of proper IT practices, and most importantly, security practices).  CDC 5.0.4 has been EOL since September 2012, and you are using it in CentOS 6.3 that we started to support officially in 2015.  Surprisingly you were doing well.  

2) We used yum update to update the server to CentOS 6.9.
3) We reinstalled Samba and were able to verify that we could connect to 
local Samba shares with local accounts with Centrify removed.
4) We downloaded Centrify Suite 2017 (Centrify DC 5.4.2) and installed it 
using instructions.

Several comments here.  I'm not sure how critical this server is (and there's nothing we can do about the past), but I think looking at change control before doing these steps could have helped better.  That being said:
- With the release of 5.4.0, many things happened with our software (we split many of the open source utilities), this means that the best bet (which was an upgrade in place) wasn't supported from such an older version.  By starting from scratch, you needed to understand how to reconfigure the system.  This is complicated because of the changes below.

- Samba changes.  Since BadLock, we EoL'd all the Centrify-enhanced Samba software, in favor of the adbindproxy Identity Mapper, this means that the deployment is slightly different.

1) We can't login to the server with domain credentials. 
It accepts the username and password, then reverts back to the login screen 
See below on how to determine this issue. Watch for Samba/CDC clashes depending on 
how you set it up.
(yes, home directories exist with the correct permissions).
That's to be expected.  The files won't be destroyed unless a human being or a program does.
2) We can't login to the server with local credentials that are not root, 
but root is working correctly.
root will work correctly.  We don't mess with it.
3) We cannot connect to Samba shares without getting "Access denied" errors, 
though permissions have not been changed.
You need to make sure adbindproxy is set up and configured (see below).  Most
importantly, remove all Centrify-ehnhanced Samba servers from your environment
since they are EOL'd and susceptible to anything from BadLock on.
4) We don't know how to test to ensure that Centrify is working correctly 
or not.
Use the adinfo or adcheck commands.  The most basic test is to see if the system is
disconnected with adinfo -T; after that you can check to see if the users and groups
are resolvable (adquery user/adquery group).  See the link below.

If I were to approach this, I'd simply start in different parts.

Make sure CDC is working correctly and allows login; once that's done I'd work on the Samba integration.

I'd start by disabling Samba; the system does not need to be double-joined.  The potential (this is an assumption) is that Samba and CDC may be having a conflict on UID/GID resolution.  Let's say that userX tries to log in (the UID/GID data from Centrify is there), but depending how the NSS/PAM stack is laid out, the Samba UID/GID is resolved first, then we have a problem.

Tips:

• Use addebug  (e.g. sudo /usr/share/centrifydc/addebug on) and then do a tail -f to the log file
• Try to log in, and you'll see exactly what's going on.
• The adbindproxy guide is here:  https://docs.centrify.com/en/css/2017.2-html/index.html#page/Samba/Installing_the_adbindproxy_components.4.html

Please let us know of any progress you make. 

R.P

,

Thank you for the comments - they've been very helpful! I realize we made a rookie mistake with the updates, and we got started with the adbindproxy documentation last night. We believe we're very close to resolving this, even with all of the junk that happened in between.

In the Centrify log, with debug on (as you suggested), here's what we see when a domain user logs in:

Nov 08 13:41:33 sshd[32443] DEBUG: -> getpwuid_centrifydc_r  UID=0
Nov 08 13:41:33 sshd[32443] DEBUG: getpwuid: UID 0 is in 'nss.uid.ignore' list
Nov 08 13:41:33 sshd[32443] DEBUG: <- getpwuid_centrifydc_r, result=NSS_NOTFOUND(0)
Nov 08 13:41:33 sshd[32443] DEBUG: -> getgrnam_centrifydc_r  group="tty"
Nov 08 13:41:33 sshd[32443] DEBUG: getgrnam: Group 'tty' is in 'nss.group.ignore' list
Nov 08 13:41:33 sshd[32443] DEBUG: <- getgrnam_centrifydc_r, result=NSS_NOTFOUND(0)
Nov 08 13:41:33 sshd[32443] DEBUG: -> getgrnam_centrifydc_r  group="tty"
Nov 08 13:41:33 sshd[32443] DEBUG: getgrnam: Group 'tty' is in 'nss.group.ignore' list
Nov 08 13:41:33 sshd[32443] DEBUG: <- getgrnam_centrifydc_r, result=NSS_NOTFOUND(0)
Nov 08 13:41:33 sshd[32443] DEBUG: -> getpwuid_centrifydc_r  UID=0
Nov 08 13:41:33 sshd[32443] DEBUG: getpwuid: UID 0 is in 'nss.uid.ignore' list
Nov 08 13:41:33 sshd[32443] DEBUG: <- getpwuid_centrifydc_r, result=NSS_NOTFOUND(0)
Nov 08 13:41:33 sshd[32443] DEBUG: -> pam_sm_setcred
Nov 08 13:41:33 sshd[32443] DEBUG: PAM Options: (none)
Nov 08 13:41:33 sshd[32443] DEBUG: PAM Flags: (none)
Nov 08 13:41:33 sshd[32443] DEBUG: Flag PAM_ESTABLISH_CRED or PAM_REINITIALIZE_CRED or PAM_REFRESH_CRED is not given, ignored!
Nov 08 13:41:33 sshd[32443] DEBUG: <- pam_sm_setcred, result=PAM_IGNORE(25)
Nov 08 13:41:33 sshd[32443] DEBUG: -> pam_sm_setcred
Nov 08 13:41:33 sshd[32443] DEBUG: PAM Options: deny
Nov 08 13:41:33 sshd[32443] DEBUG: PAM Flags: (none)
Nov 08 13:41:33 sshd[32443] DEBUG: Flag PAM_ESTABLISH_CRED or PAM_REINITIALIZE_CRED or PAM_REFRESH_CRED is not given, ignored!
Nov 08 13:41:33 sshd[32443] DEBUG: <- pam_sm_setcred, result=PAM_IGNORE(25)

We've been seeing a lot of errors in Samba regarding UIDs not being found, so I believe you're right about the UID/GID conflict (potentially). Here's the log we see in Samba when trying to connect (there's a lot more data here, but this is what I've found to be relevant):

[2017/11/08 13:44:54.814265,  1] ../source3/auth/token_util.c:430(add_local_groups)
  SID S-1-5-21-3931225680-1871015619-2963001510-1368939 -> getpwuid(1001) failed
[2017/11/08 13:44:54.814304,  3] ../source3/auth/token_util.c:316(create_local_nt_token_from_info3)
  Failed to finalize nt token
[2017/11/08 13:44:54.814337,  1] ../source3/auth/auth_generic.c:127(auth3_generate_session_info_pac)
  Failed to map kerberos pac to server info (NT_STATUS_UNSUCCESSFUL)
[2017/11/08 13:44:54.815371,  3] ../source3/smbd/server_exit.c:249(exit_server_common)
  Server exit (NT_STATUS_CONNECTION_RESET)

Any ideas on what to look at to get CDC working first?

,

Impressive. I'm sorry you had to go through this.

However, it looks like you're close.

I would disable (or remove) samba and just focus on CDC

Is the client connected (adinfo should display)

What's the de-identified output of adinfo -T?

What happens when you do an adquery user or adquery group command?

It would be good to also know the output of the passwd and group stanza on /etc/nsswitch.conf

Some tips on testing raw authentication:

adquery user -A -u test.user

helps you test using just the agent, bypassing PAM, NSS or any app like Login or SSH.  It will prompt you for the password of the user and if you type it correctly, that's an end-to-end test of the client connectivity.

Keep this handy:  https://community.centrify.com/t5/TechBlog/TIPS-A-Centrify-Server-Suite-Cheat-Sheet/ba-p/22568

Most folks find what they need there.

R.P

 Great tips! A lot of these give good output - the connection with the domain seems to be fine. My guess is it's related to NSS or PAM like you mentioned (those have come up a lot in our research), we just don't know what we're looking at.

Here's the output of the commands you suggested.

adinfo -T (unidentified):

Domain Diagnostics:
  Domain: [correct.domain.name]
    DNS query for: _ldap._tcp.[correct.domain.name]
    DNS query for: _gc._tcp.[correct.domain.name]
  Testing Active Directory connectivity:
    Global Catalog: lares.[correct.domain.name]      gc:       3268/tcp - good
    Global Catalog: zeus.[correct.domain.name]
      gc:       3268/tcp - good
    Global Catalog: aphrodite.[correct.domain.name]
      gc:       3268/tcp - good
    Global Catalog: flora.[correct.domain.name]
      gc:       3268/tcp - timeout
      No TCP LDAP response, giving up on flora.[correct.domain.name]    Global Catalog: fauna.[correct.domain.name]     gc:       3268/tcp - timeout
      No TCP LDAP response, giving up on fauna.[correct.domain.name]
    Global Catalog: ares.[correct.domain.name]
      gc:       3268/tcp - good
    Global Catalog: artemis.[correct.domain.name]      gc:       3268/tcp - good
    Global Catalog: helios.[correct.domain.name]
      gc:       3268/tcp - good
    Global Catalog: ceres.[correct.domain.name]      gc:       3268/tcp - good
    Domain Controller: helios.[correct.domain.name]
      ldap:      389/tcp - good
      ldap:      389/udp - good
      smb:       445/tcp - good
      kdc:        88/tcp - good
      kpasswd:   464/tcp - good
      ntp:       123/udp - good
    Domain Controller: artemis.[correct.domain.name]
      ldap:      389/tcp - good
      ldap:      389/udp - good
      smb:       445/tcp - good
      kdc:        88/tcp - good
      kpasswd:   464/tcp - good
      ntp:       123/udp - good
    Domain Controller: flora.[correct.domain.name]
      ldap:      389/tcp - timeout
      No TCP LDAP response, giving up on flora.[correct.domain.name]
    Domain Controller: lares.[correct.domain.name]
      ldap:      389/tcp - good
      ldap:      389/udp - good
      smb:       445/tcp - good
      kdc:        88/tcp - good
      kpasswd:   464/tcp - good
      ntp:       123/udp - good
    Domain Controller: ceres.[correct.domain.name]
      ldap:      389/tcp - good
      ldap:      389/udp - good
      smb:       445/tcp - good
      kdc:        88/tcp - good
      kpasswd:   464/tcp - good
      ntp:       123/udp - good
    Domain Controller: fauna.[correct.domain.name]
      ldap:      389/tcp - timeout
      No TCP LDAP response, giving up on fauna.[correct.domain.name]
    Domain Controller: ares.[correct.domain.name]      ldap:      389/tcp - good
      ldap:      389/udp - good
      smb:       445/tcp - good
      kdc:        88/tcp - good
      kpasswd:   464/tcp - good
      ntp:       123/udp - good
    Domain Controller: zeus.[correct.domain.name]
      ldap:      389/tcp - good
      ldap:      389/udp - good
      smb:       445/tcp - good
      kdc:        88/tcp - good
      kpasswd:   464/tcp - good
      ntp:       123/udp - good
    Domain Controller: aphrodite.[correct.domain.name]
      ldap:      389/tcp - good
      ldap:      389/udp - good
      smb:       445/tcp - good
      kdc:        88/tcp - good
      kpasswd:   464/tcp - good
      ntp:       123/udp - good

Adquery user works fine as well. In fact, running just that command tries to list every user in AD, which is a lot. Here's an example of one (with PHI removed):

[username]:x:819327050:817889793:[Last Name], [First Name] [Initial]:/export/homes/[username]:/bin/bash

NSSwitch.conf (sections you asked for):

passwd: centrifydc files
shadow: centrifydc files
group: centrifydc files

Adquery user -A -u [username] gave a ton of output that is correct, including name, uid, gid, shell, home, dn, sid, userPrincipalName, guid, account info, group memberships, etc. I don't want to copy all of that in here due to the personally identifiable information, but it's working fantastically.

Interestingly enough, however, that command does not prommpt for a password like you mentioned. It just gives output.

So you're onto something here in that NSS and/or PAM is likely where the mixup is happening. Do you know where we can go next?

,

My bad!

It was
adinfo -A -u username

Sorry.

Do me a favor,  as root, switch user (su) to an ad user  (you'll be able to do it), then try to su to another known user (type the correct password).  If you are able to su from the normal account to the other user with the password, then your issue is with OpenSSH or the GUI.

I have to continue working on something else now.  Ciao.

R.P

 Thanks for all of your support! You've been super helpful.

I ran the command adinfo -A -u [username] with my domain credentials, and it DID prompt for a password and said the password is correct. So that works great.

The su is where we broke here. I tried the command "su - [username]" with a domain user, and got the following errors:

su: warning: cannot change directory to /export/homes/[username]: Permission denied
su: /bin/bash: Permission denied

This is basically what we've been seeing across the board since getting this far and where we've been stuck.

We'll continue digging, I think you gave us a lot to read and a lot to work on. I appreciate the support and would, of course, love to hear your feedback when you have time.

Will post updates as we make progress.

Do an adquery user for that user and note the UID/GID.

Now do an ls -ln on that home directory .

If the numbers are different, you have an identity mismatch.  This could be due to something changing in the domain (long shot, but perhaps migrated??? or conflict with another source like local file? most likely human intervention???).

chmoding will solve the issue for that user and we provide a tool called adfixid to do this in bulk.

*** HANDLE WITH CARE ***

https://community.centrify.com/t5/TechBlog/HOWTO-Using-ADFIXID-and-ADRMLOCAL-to-clean-up-the-local-unix/ba-p/21461

 thanks again for pointing us in that direction!

We found that the adquery and the permissions showed the same UID, but the logs I posted above don't - they look like they're showing that the uid=0? Not sure what that means from a log perspective, but it's weird for sure.

Our local Linux guy found that permissions on our server seem to be really messed up. He made a permissions change on / to 0555 and that got SSH somewhat working - it allows the user to login, gives access denied on the home directory, but gives a bash shell nonetheless. It's really strange.

I think there's a good chance that Centrify is no longer our problem (though there are some config changes we can do to clean up some of the services). We're gonna keep digging on this permissions thing.

What are your thoughts on that?

So we conducted another test and we found that the UID=0 is what's actually occurring here, but we're not sure why or how to change that.

Here's what we did to find this out:
We changed permissions to 755 on one user's home directory, and it allowed us to login without any Access Denied errors, but it's dropping us in the /root folder, which is indicative of the UID=0 (which is root).

Any idea why that might be happening? The logs I posted earlier show it occurring with Centrify, but my guess is it has to do with NSS or PAM like you mentioned before.

Also, I ran the adfixid command just out of curiosity, and it picked up 0 errors immediately - didn't really appear to scan at all...

,

No thoughts on my end.  I'm afraid not familiar enough with the history of your system to be useful.

Some tips and a shameless plug:

- Remember that Express only provides 10% of what Centrify does

- Keep track of the software lifecycle (especially on security software);  Express users get support only on the latest releases.  Should you want to enjoy more capability and SLA-based support, just contact us.

//shameless plug start

- Finally, Authentication is only the beginning of having the proper security posture.  Embrace the least privilege principle and secure shared accounts.  Privilege users are not only on systems, but on apps as well and Centrify Identity Platform can help.

//shameless plug end

Cheers.

 Thanks for the info and for all of the help today.

One last question, regarding my double post: Have you seen this incident where UIDs are mapped to 0 automatically? Please see my last post on the previous page as well as my Centrify log to see what I mean.

We're going to keep digging, and I will, of course, update as we progress.

We made some really good progress on this today - we were actually correct with the UID=0 issue. We tweaked some permissions, and that helped, but it didn't solve a thing.

So I added the following code to smb.conf:

idmap config [domain.name]:backend = ad
idmap config [domain.name]:range = 10000-199999999

And we changed the range for the idmap config default to 3000-7999.

Our UIDs are now being pulled correctly and providing basic access, but the GIDs are now where we struggle.

The user is being placed in the primary GID of Domain Users (I used adquery to verify the GID), but the group that we apply permissions to is not Domain Users, so it is not providing access for that reason.

Currently trying to narrow down how we can get it to see more AD groups for access. Will continue to post updates if we figure it out.

OH! I forgot to note:

Permissions settings did solve our issues with Home Directories, but ONLY once we figured out the above UID problem.

Hi,

Glad that I can get any advise from here. I am right now hardening Red Hat 7.4 with the STIG complianced as requested from the US government. I wonder how I can solve this problems as Active directory user with Centrify express. Since I have used centrify, the local user name are not existed on this template.  I had two below failed results and the the items are getting more on when new user logged on. 

Ensure All Files Are Owned by a User

Ensure All Files Are Owned by a Group

The most of violating directories are /proc/ and /home/ directory. If you could give me the solution to get pass above failed result will be much appreciated.

Thanks,

sahn