Hi RP - thanks for the quick response!
As I am also one of the app's developers, I can say that the app does no AD related activities. Apache runs as a local user, speaks to a mysql database using mysql authentication. The only other resources aside from local files and the database connection are that some pages make a call out to another web service we host using credentials managed outside of AD (web tokens, etc). However, we see the immediate response time falloff even on pages that make only database calls. As we use amazon's RDS service, the DNS query is usually not cached between requests since that allows them to initiate immediate failovers, etc.
So, that being said, when we add firewall rules that block off all network connectivity to all 3 domain controllers, with the exception of DNS, everything still works. When we block DNS access to one specific server, it all goes to heck. I have a hard time believing this isn't DNS related based on the data thus far, so I greatly appeciate your help in tracking down additional data points to bring to light.
You'll see in the diagnostic info below lots of failures as we've tested various scenarios of blocking access to ports and seeing the results.
Thanks for your help!
diagnostic info
adinfo --diag for GCs:
Domain Controller: dc1-us-east-1c.corp.zebit.com:389
Domain controller type: Windows 2012 R2
Domain Name: CORP.ZEBIT.COM
isGlobalCatalogReady: TRUE
domainFunctionality: 4 = (DS_BEHAVIOR_WIN2008_R2)
forestFunctionality: 4 = (DS_BEHAVIOR_WIN2008_R2)
domainControllerFunctionality: 6
Domain Controller: dc1.corp.zebit.com:389
Domain controller type: Windows 2012 R2
Domain Name: CORP.ZEBIT.COM
isGlobalCatalogReady: TRUE
domainFunctionality: 4 = (DS_BEHAVIOR_WIN2008_R2)
forestFunctionality: 4 = (DS_BEHAVIOR_WIN2008_R2)
domainControllerFunctionality: 6
Domain Controller: dc1-us-east-1a.corp.zebit.com:389
Domain controller type: Windows 2012 R2
Domain Name: CORP.ZEBIT.COM
isGlobalCatalogReady: TRUE
domainFunctionality: 4 = (DS_BEHAVIOR_WIN2008_R2)
forestFunctionality: 4 = (DS_BEHAVIOR_WIN2008_R2)
domainControllerFunctionality: 6
adinfo --sysinfo netstate
===============Network State===================
Site Map
corp.zebit.com=>AWS-EAST-US-1a
Domain Map
corp.zebit.com
dc: dc1-us-east-1a.corp.zebit.com
gc: dc1-us-east-1a.corp.zebit.com
forest: corp.zebit.com
state: alive
swept: 52 mins ago
Domain Controllers
dc1-us-east-1a.corp.zebit.com (10.172.32.10)
pinged: 52 mins ago
state: up
ping: 0.000692 secs
forest: corp.zebit.com
nbhost: dc1-us-east-1a
site: AWS-EAST-US-1a
flags: WCTKLG
Blocked Services: None
dc1.corp.zebit.com (10.120.50.10)
pinged: 34 mins ago
state: up
ping: 0.065199 secs
forest: corp.zebit.com
nbhost: dc1
site: La-Jolla-Office
flags: WcTKLG
Blocked Services: None
adinfo --sysinfo dns
System Diagnostic
=======DNS Servers State==========
DNS Server Used: 10.172.32.10
DNS Status: Up
=======DNS Server Info=======
Last Sweep: Wed Jul 27 17:03:22 2016
Fast Sweeps: 5
Deep Sweeps: 1539
Okay Sweeps: 1544
Failed Sweeps: 0
Cache Hits: 23935
Cache Misses: 33
DNS Flushes: 4
=======DNS Server List=======
IP: 10.172.32.10
Status: Alive
udpSuccess: 389
tcpSuccess: 52
udpNoSuchName: 0
tcpNoSuchName: 0
udpTruncations: 0
tcpTruncations: 0
udpIOFailures: 0
tcpIOFailures: 0
udpTimeouts: 0
tcpTimeouts: 0
udpFailures: 0
tcpFailures: 0
udpServerFail: 0
tcpServerFail: 0
lastQueryTime: Wed Jul 27 17:21:40 2016
lastDnsCode: 0
Average Time: 0.000838041 seconds
IP: 10.172.40.10
Status: Alive
udpSuccess: 15
tcpSuccess: 53
udpNoSuchName: 0
tcpNoSuchName: 0
udpTruncations: 0
tcpTruncations: 0
udpIOFailures: 0
tcpIOFailures: 0
udpTimeouts: 0
tcpTimeouts: 0
udpFailures: 0
tcpFailures: 0
udpServerFail: 0
tcpServerFail: 0
lastQueryTime: Wed Jul 27 17:03:22 2016
lastDnsCode: 0
Average Time: 0.00234096 seconds
IP: 10.120.50.10
Status: Alive
udpSuccess: 0
tcpSuccess: 52
udpNoSuchName: 0
tcpNoSuchName: 0
udpTruncations: 0
tcpTruncations: 0
udpIOFailures: 0
tcpIOFailures: 0
udpTimeouts: 0
tcpTimeouts: 0
udpFailures: 0
tcpFailures: 0
udpServerFail: 0
tcpServerFail: 0
lastQueryTime: Wed Jul 27 17:03:22 2016
lastDnsCode: 0
Average Time: 0.129375 seconds
=======DNS Cache contents==========
Hdc1-us-east-1a.corp.zebit.com=>dc1-us-east-1a.corp.zebit.com 10.172.32.10
Hdc1-us-east-1c.corp.zebit.com=>dc1-us-east-1c.corp.zebit.com 10.172.40.10
Hdc1.corp.zebit.com=>dc1.corp.zebit.com 10.120.50.10
Hstg-01-dmz-www-1a-01.corp.zebit.com=>stg-01-dmz-www-1a-01.corp.zebit.com 10.172.65.72
S_gc._tcp.aws-east-us-1a._sites.corp.zebit.com=>dc1-us-east-1a.corp.zebit.com:3268:100:0
S_gc._tcp.corp.zebit.com=>dc1-us-east-1a.corp.zebit.com:3268:100:0 dc1-us-east-1c.corp.zebit.com:3268:100:0 dc1.corp.zebit.com:3268:100:0
S_kerberos._tcp.aws-east-us-1a._sites.corp.zebit.com=>dc1-us-east-1a.corp.zebit.com:88:100:0
S_ldap._tcp.aws-east-us-1a._sites.corp.zebit.com=>dc1-us-east-1a.corp.zebit.com:389:100:0
S_ldap._tcp.corp.zebit.com=>dc1-us-east-1c.corp.zebit.com:389:100:0 dc1-us-east-1a.corp.zebit.com:389:100:0 dc1.corp.zebit.com:389:100:0