Thanks again for the help!
So a few things:
I've verified A, B, and C, as well as checking AD size. We're a very small AD (60 or so objects, totoal).
Also, I'm not seeing any network traffic to either of our domain controllers via a tcpdump. All this activity looks to be within the server. Is it hitting cache?
Your statment about PID was very helpful. I've been digesting the log file and think I get it now.
So the below statment is one getpwuid_centrifydc_r function call to adclient? My comments below are in bold:
Aug 12 11:13:55 tms-oda1-nd1 adclient[814]: DEBUG <fd:25 oracle(32464)> -> getpwuid_centrifydc_r UID=1000 #This is the start on the req by PID 32464 to centrifydc by UID 1000 Aug 12 11:13:55 tms-oda1-nd1 adclient[814]: DEBUG <fd:23 NSSGetPasswdDataByName > base.objecthelper age 423, expire age 3600, cutoff time 0, refresh 5, negative=true, cacheOps 7 Aug 12 11:13:55 tms-oda1-nd1 adclient[814]: DEBUG <fd:23 NSSGetPasswdDataByName > base.adagent findByAttr: Not Found:ReeferWeeklyEngineHrsReport.sh category:user attr=sAMAccountName Aug 12 11:13:55 tms-oda1-nd1 adclient[814]: DEBUG <main> daemon.ipcserver Accepted new lrpc2 client on <fd:24> with flags 0x00000802 Aug 12 11:13:55 tms-oda1-nd1 adclient[814]: DEBUG <fd:23 NSSGetPasswdDataByName > base.bind.cache ADCB::search base , filter (&(objectClass=User)(|(objectCategory=Person)(objectCategory=Computer))(displayName=ReeferWeeklyEngineHrsReport.sh)), attrs 2 (cacheOps=7, GC=0) #adclient binds to the cache looking for requested displayName of class User, Person, or Computer Aug 12 11:13:55 tms-oda1-nd1 adclient[814]: DEBUG <main> util.threadpool Pool size 3/4, busy size 1/20 Aug 12 11:13:55 tms-oda1-nd1 adclient[814]: DEBUG <fd:24 NSSGetPasswdDataByUID > daemon.ipcclient2 executing request 'NSSGetPasswdDataByUID' in thread 140710770633024 Aug 12 11:13:55 tms-oda1-nd1 adclient[814]: DEBUG <fd:24 NSSGetPasswdDataByUID > daemon.ipcclient2 Getting passwd data for uid 1000 Aug 12 11:13:55 tms-oda1-nd1 adclient[814]: DEBUG <fd:23 NSSGetPasswdDataByName > base.objecthelper age 423, expire age 3600, cutoff time 0, refresh 5, negative=true, cacheOps 7 Aug 12 11:13:55 tms-oda1-nd1 adclient[814]: DEBUG <fd:24 NSSGetPasswdDataByUID > base.adagent Find GUID: 72a8607af7bd4d8ab7427d237505eba5 (7) Aug 12 11:13:55 tms-oda1-nd1 adclient[814]: DEBUG <fd:23 NSSGetPasswdDataByName > base.bind.cache ADCB::search base , filter (&(objectClass=User)(|(objectCategory=Person)(objectCategory=Computer))(displayName=ReeferWeeklyEngineHrsReport.sh)), attrs 1e (cacheOps=7, GC=1) #same operation with different attrs Aug 12 11:13:55 tms-oda1-nd1 adclient[814]: DEBUG <fd:23 NSSGetPasswdDataByName > base.objecthelper age 423, expire age 3600, cutoff time 0, refresh 5, negative=true, cacheOps 7 Aug 12 11:13:55 tms-oda1-nd1 adclient[814]: DEBUG <fd:24 NSSGetPasswdDataByUID > base.objecthelper age 422, expire age 3600, cutoff time 0, refresh 5, negative=true, cacheOps 7 Aug 12 11:13:55 tms-oda1-nd1 adclient[814]: DEBUG <fd:23 NSSGetPasswdDataByName > base.adagent findByAttr: Not Found:ReeferWeeklyEngineHrsReport.sh category:user attr=displayName Aug 12 11:13:55 tms-oda1-nd1 adclient[814]: DEBUG <fd:24 NSSGetPasswdDataByUID > util.except (NotFound) : No such unix user with uid=1000 (reference ipcclient2.cpp:945 rc: 0) Aug 12 11:13:55 tms-oda1-nd1 adclient[814]: DEBUG <fd:23 NSSGetPasswdDataByName > base.bind.cache ADCB::search base , filter (&(objectClass=User)(|(objectCategory=Person)(objectCategory=Computer))(cn=ReeferWeeklyEngineHrsReport.sh)), attrs 2 (cacheOps=7, GC=0) #same operation with different attrs Aug 12 11:13:55 tms-oda1-nd1 adclient[814]: DEBUG <fd:24 NSSGetPasswdDataByUID > daemon.ipcclient2 No user data: No such unix user with uid=1000 Aug 12 11:13:55 tms-oda1-nd1 adclient[814]: DEBUG <fd:24 NSSGetPasswdDataByUID > daemon.ipcclient2 request 'NSSGetPasswdDataByUID' complete Aug 12 11:13:55 tms-oda1-nd1 adclient[814]: DEBUG <fd:23 NSSGetPasswdDataByName > base.objecthelper age 423, expire age 3600, cutoff time 0, refresh 5, negative=true, cacheOps 7 Aug 12 11:13:55 tms-oda1-nd1 adclient[814]: DEBUG <main> util.threadpool Pool size 3/4, busy size 1/20 Aug 12 11:13:55 tms-oda1-nd1 adclient[814]: DEBUG <fd:23 NSSGetPasswdDataByName > base.bind.cache ADCB::search base , filter (&(objectClass=User)(|(objectCategory=Person)(objectCategory=Computer))(cn=ReeferWeeklyEngineHrsReport.sh)), attrs 1e (cacheOps=7, GC=1) #same operation with different attrs Aug 12 11:13:55 tms-oda1-nd1 adclient[814]: DEBUG <fd:23 NSSGetPasswdDataByName > base.objecthelper age 423, expire age 3600, cutoff time 0, refresh 5, negative=true, cacheOps 7 Aug 12 11:13:55 tms-oda1-nd1 adclient[814]: DEBUG <fd:23 NSSGetPasswdDataByName > base.adagent findByAttr: Not Found:ReeferWeeklyEngineHrsReport.sh category:user attr=cn Aug 12 11:13:55 tms-oda1-nd1 adclient[814]: DEBUG <fd:23 NSSGetPasswdDataByName > base.adagent findObject: NotFound:ReeferWeeklyEngineHrsReport.sh Category:user Aug 12 11:13:55 tms-oda1-nd1 adclient[814]: DEBUG <fd:23 NSSGetPasswdDataByName > base.objecthelper 'ReeferWeeklyEngineHrsReport.sh' is not a canonical name Aug 12 11:13:55 tms-oda1-nd1 adclient[814]: DEBUG <fd:23 NSSGetPasswdDataByName > util.except (NotFound) : No such unix user 'ReeferWeeklyEngineHrsReport.sh' (reference ipcclient2.cpp:936 rc: 0) Aug 12 11:13:55 tms-oda1-nd1 adclient[814]: DEBUG <fd:23 NSSGetPasswdDataByName > daemon.ipcclient2 No user data: No such unix user 'ReeferWeeklyEngineHrsReport.sh' Aug 12 11:13:55 tms-oda1-nd1 adclient[814]: DEBUG <fd:23 NSSGetPasswdDataByName > daemon.ipcclient2 request 'NSSGetPasswdDataByName' complete Aug 12 11:13:55 tms-oda1-nd1 adclient[814]: DEBUG <fd:25 oracle(32464)> <- getpwuid_centrifydc_r, result=NSS_NOTFOUND(0) #The final return saying that whatever this PID is looking for isn’t found
UID 1000 on this Oracle serveris a non-priviliged user named "grid," so I think something that the grid user is doing, maybe within oracle, is asking for a permissions validation on /var/log/crontab? Maybe we need to change the permissions on /var/log/crontab to allow the grid user access to it? Right now, the rights for that folder are 0700 (owner has all, everyone else has nothing; owner being root).
Am I on the right track here?