Ok, I just found a partial solution. I disabled the Apple smartcard support with
sudo defaults write /Library/Preferences/com.apple.security.smartcard DisabledTokens -array com.apple.CryptoTokenKit.pivtoken
and the applications now see the smartcard, and I was able to unlock it in Keychain Access. Unfortunately, that disables smartcard login on the Mac, which is something we do use.