Re: adclient taking lots of CPU and debug log shows an NSSGetPasswdDataByName for all operations

Is there any way to verify what process or executable the adclient is being called by?

Not without a trace or the log.

Tip:

The application name or PID is typically in the call:

Aug 11 17:00:46 engcen6 adclient[1583]: DEBUG <fd:25 id(36586)> Result: group='dwirth' password='x' GID='1040188499' members=dwirth (returnlen=32)
Aug 11 17:00:46 engcen6 adclient[1583]: DEBUG <fd:25 id(36586)> <- getgrgid_centrifydc_r, result=NSS_SUCCESS(1)

 
Now that you understand what's going on, you will be able to pinpoint it.  Understanding your app is key.

You can do some basic health checks as well to make sure your systems are operating as expected, AD environments are dynamic. 

A) Make sure that unreachable domain controllers are not on the eligible list.
You can use the adinfo -T [domain.name]  (repeat for each trusted domain) or the adinfo -y netstate

$ adinfo -T corp.contoso.com
Domain Diagnostics:
  Domain: corp.contoso.com
    DNS query for: _ldap._tcp.corp.contoso.com
    DNS query for: _gc._tcp.corp.contoso.com
  Testing Active Directory connectivity:
    Global Catalog: dc2.corp.contoso.com
      gc:       3268/tcp - timeout
      No TCP LDAP response, giving up on dc2.corp.contoso.com
    Global Catalog: dc1.corp.contoso.com
      gc:       3268/tcp - good
    Domain Controller: dc2.corp.contoso.com

In this example you can see that a DC is down.  If this is in a DMZ or a DR site that is unreachable by the client, it has to be blocked.  There is a parameter for that called dns.block.

Using adinfo -y netstate you can see another relative view:

$ adinfo -y netstate
System Diagnostic
===============Network State===================
Site Map
 corp.contoso.com=>PreferredSite:CorpHQ, SubnetSite:CorpHQ
 hq.fabrikam.com=>PreferredSite:Fab-Site, SubnetSite:Fab-Site
Domain Map
 corp.contoso.com
        dc:     dc1.corp.contoso.com
        gc:     dc1.corp.contoso.com
        forest: corp.contoso.com
        state:  alive
        swept:  28 mins ago

 hq.fabrikam.com
        dc:     fab-dc1.hq.fabrikam.com
        gc:     fab-dc1.hq.fabrikam.com
        forest: hq.fabrikam.com
        state:  alive
        swept:  28 mins ago

Domain Controllers
 dc1.corp.contoso.com (10.0.0.1)
        pinged: 28 mins ago
        state:  up
        ping:   0.001333 secs
        forest: corp.contoso.com
        nbhost: dc1
        site:   CorpHQ
        flags:  WCTKLG
        Blocked Services:        None

 dc2.corp.contoso.com (10.0.0.2)
        pinged: 28 mins ago
        state:  up
        ping:   0.001060 secs
        forest: corp.contoso.com
        nbhost: dc2
        site:   CorpHQ
        flags:  WCTKLG
        Blocked Services:        None

 fab-dc1.hq.fabrikam.com (10.0.0.50)
        pinged: 28 mins ago
        state:  up
        ping:   0.000720 secs
        forest: hq.fabrikam.com
        nbhost: fab-dc1
        site:   Fab-Site
        flags:  WCTKLG
        Blocked Services:        None

B) Make sure you have a global catalog close by  (e.g. same subnet or fast link)

You can also use the adinfo --diag | grep isGlobalCatalogReady

 Domain Controller: fab-dc1.hq.fabrikam.com:389
    Domain controller type: Windows 2008 R2
    Domain Name:            HQ.FABRIKAM.COM
    isGlobalCatalogReady:   TRUE
    domainFunctionality:           4 = (DS_BEHAVIOR_WIN2008_R2)
    forestFunctionality:           4 = (DS_BEHAVIOR_WIN2008_R2)
    domainControllerFunctionality: 4 = (DS_BEHAVIOR

The reason is because you need to be able to get objects from trusted domains.

C) Assess DNS Health

The health of DNS is very important and we keep a cache.

$ adinfo -y dns
System Diagnostic
=======DNS Servers State==========
 DNS Server Used: 10.0.0.50
 DNS Status: Up

=======DNS Server Info=======
Last Sweep:    Thu Aug 11 16:42:49 2016
Fast Sweeps:   4
Deep Sweeps:   1577
Okay Sweeps:   1581
Failed Sweeps: 0
Cache Hits:    28668
Cache Misses:  31
DNS Flushes:   3

Assessing the size of your environment

There are limits to AutoZone as I explained in the previous post.  You can determine how many user or group objects are being retrieved/cached by the express agent.   These commands:

$ adquery user  | wc -l    <= # of users
$ adquery group | wc -l    <= # of groups

If you are over a few thousand of entries here, your environment is beyond our recommendation for the size of Auto Zone.  You have to work in zoned mode.  Exposing all AD users and all groups is not the optimal thing to do, especially when you're dealing with older UNIX apps.

Find out if you have