Techniques are provided for automated key management for accessing remote devices using single sign-on techniques. One method comprises maintaining a data record identifying target user devices that a given source user device is authorized to access; and initiating storage of a public key of the given source user device in a file of at least one target user device, wherein the given source user device accesses the at least one target user device using a secure remote connection protocol based on the public key of the given source user device stored in the file of the at least one target user device. The data record may further comprise a fingerprint of a key of the at least one target user device, and the method may further comprise comparing a fingerprint of the key returned by the at least one target user device to the fingerprint of the key obtained from the data record.