Mike,
The built-in SSH client on Mac OS X 10.10 has no support for the CAC. You need to use an SSH client (and server) that pass along the x509 certificate. Tectia has offerings for Linux and Windows, with a MacOS client in pre-release.
Hope this helps,
--Chan