SSH key management

SSH is probably the most common method of logging into Unix machines over a network connection. It has pretty much replaced the older telnet and rlogin protocols because it encrypts data over the network and has other “goodness”.

However one of the “good” parts of ssh is probably the least well managed; ssh keys.

What is an ssh key?

When attempting to login to a server, ssh will attempt multiple different authenticators. Now we all know (and hate) the standard password mechanism.

$ ssh test2
sweh@test2's password: 
Last login: Sun Sep  4 12:02:13 2016 from test1

A few years ago I described some tests on using Kerberos as an authenticator.

I even use Google Authenticator as a form of two factor authentication.

But the big one, and one of the most powerful, are ssh keys.

An ssh key is a public/private key pair. A user can generate their own key with ssh-keygen and then copy the public part to the remote server.

$ ssh-keygen
Generating public/private rsa key pair.
Enter file in which to save the key (/home/sweh/.ssh/id_rsa):
Enter passphrase (empty for no passphrase):
Enter same passphrase again:
Your identification has been saved in /home/sweh/.ssh/id_rsa.
Your public key has been saved in /home/sweh/.ssh/id_rsa.pub.
The key fingerprint is:
ad:cc:ec:e6:9b:7f:ba:96:32:1d:83:b7:b4:1d:cf:00 sweh@test1.spuddy.org
The key's randomart image is:
+--[ RSA 2048]----+
|                 |
|                 |
|                 |
|         .E      |
|        S...     |
|       +..= o    |
|        =+ B =   |
|       .+.* o o  |
|       o=*++     |
+-----------------+
$ scp ~/.ssh/id_rsa.pub test2:.ssh/authorized_keys
sweh@test2's password:
id_rsa.pub                                    100%  403     0.4KB/s   00:00

We can now use this to authenticate instead:

$ ssh -i ~/.ssh/id_rsa sweh@test2
Enter passphrase for key '/home/sweh/.ssh/id_rsa': 
Last login: Sun Sep  4 12:10:54 2016 from test1

Indeed, if the id_rsa file exists (or one of a few other names) then it will be selected automatically (with some details that don’t matter, here).

$ ssh sweh@test2
Enter passphrase for key '/home/sweh/.ssh/id_rsa': 
Last login: Sun Sep  4 12:14:51 2016 from test1

Why would you do this?

In the “bad old days” when servers were DES based, and so limited to 8 characters, ssh keys could use much longer passphrases. This benefit has been reduced with modern SHA hashing of passwords :-)

But ssh keys still have some advantages. From a user perspective we can run an agent that can remember the unlocked key and present it automatically. This means the user doesn’t have to keep retyping their passwords:

$ eval `ssh-agent`
Agent pid 1904
$ ssh-add ~/.ssh/id_rsa
Enter passphrase for /home/sweh/.ssh/id_rsa: 
Identity added: /home/sweh/.ssh/id_rsa (/home/sweh/.ssh/id_rsa)
$ ssh test2 echo running remotely with no password
running remotely with no password
$ ssh test2
Last login: Sun Sep  4 12:15:01 2016 from test1

$ logout
Connection to test2 closed.
$ ssh test2 echo third login                      
third login

Another, much more powerful configuration, is that restrictions can be placed on the ssh public key. For example, we can say that it can only be used from a specific server and can only be used to run a specific command.

eg on server test2 I modify the authorized_keys file so it starts:

from="10.0.0.158",command="/bin/echo forced command" ssh-rsa AAAAB3Nz....

Now if I try to use the key for a normal login:

$ ssh test2
forced command
Connection to test2 closed.

We can see the command= stuff was forced and I wasn’t able to do anything else. Similarly if I tried to use the key from a different server then it won’t be permitted.

This is extremely powerful for automation purposes; for example a production server could transfer files to DR on an automated basis using an ssh key that has restrictions like this; the key can only do what it is permitted to do, as opposed to having a password based login that can do anything on the remote machine.

So what problems are there with ssh keys?

No enforcement on passphrase complexity

The passphrase set on a key when we create it has no real complexity rules that can be applied. This is a pure client side activity. Indeed it’s possible to create keys without a passphrase (which is good for automation tasks, such as the DR file copy mentioned above) but a potential risk.

No easy distribution

If I want to login to 100 servers then I need to copy that key to all 100 servers. If a new server is built then I need to remember to copy the key there. Some tools such as ssh-copy-id help to do this, but it’s still a mostly manual task.

No expiration

An ssh key is an authenticator and it can be copied between machines. This means that the private key could be copied around and potentially leaked. At this point the the key is exposed to offline attacks against the passphrase (worse, the key could have been generated without a passphrase!).

In an enterprise environment with thousands of users, all it takes is one exposed key to open a potential entry point for an attacker.

We’ve learned that expiring user passwords is bad because it causes people to pick easy to hack passwords.

However ssh keys are strong cryptographic keys, generated by tools, with no human memory requirements. They can theoretically be expired and a new key forced, but there’s nothing “out of the box” that enables this.

Unauthorized granting of access.

An authorized_keys file can have multiple keys permitting access. This is useful, especially for automation tasks; eg key1 can be used from server1 and can copy files to directory1, and key2 can be used from server2 and can copy files to directory2.

But this also allows humans to add their keys to an account. Let’s say you have an oracle account that your database runs under. You don’t allow people to login directly but force them to go via “break glass” process, in order to control and monitor access to this privileged account. Without any strong controls, the first time I login I could add my personl key to oracle’s authorized_keys file. Now I can login as oracle whenever I like, bypassing break-glass.

Similarly, people tend to not share passwords, but if Fred has a login on a server then he could add John’s public key to his authorized_keys file and now John can login to the server as well… without any authorization or oversight. Your central access admin request system has been totally bypassed.

Keys on NFS servers…

A long standing problem with NFS is that it is an effective “trust the client” environment. The client says “user 12345 wants to access file X” and the server has to trust the client when it says you are user 12345. Now this has been changed, to some extent, with NFSv4 and with kerberos but this isn’t as widely deployed as we might wish.

And commonly, user HOME directories are on these NFS servers. So now it becomes possible for one SA to add their key to another user’s authorized_keys file and now login to other servers that they wouldn’t normally be permitted to. All of the previous problems are exposed by having keys stored on NFS.

So why use it?

If ssh keys have all these problems then why use them?

Mostly, as NIST says, ssh keys are extremely powerful; the command and from restrictions allow you to build solutions that are more tightly locked down than other authenticators.

HOST keys

In addition to the ssh keys we’ve been discussing there’s also the concept of a host key. These are a form of ssh key that’s used to identify the host.

First time you login to a machine you’ll be presented with the public key signature of a host, which you can accept. Next time you login the client will verify the key hasn’t changed; you have a level of trust that the server you’re accessing is the server you meant to access:

$ ssh test2
The authenticity of host 'test2 (10.0.0.159)' can't be established.
ECDSA key fingerprint is 32:31:44:a0:9b:61:1b:ff:2f:db:76:ae:a9:a5:36:2b.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added 'test2,10.0.0.159' (ECDSA) to the list of known hosts.
forced command
Connection to test2 closed.

$ ssh test2
forced command
Connection to test2 closed.

This is (mostly) OK, but what happens if a server is rebuilt? Now the key will change. In the modern dynamic world (eg dynamic building/destroying of Amazon EC2 instances) your host keys may change all the time.

Ideally we need a way to manage these host keys so that the client can determine the current host key; this will solve the “trust key presented on first connect” and solve the “key changes on rebuild” problem.

Conclusion

Some of the problems with ssh keys can be mitigated by changing the default config. Instead of having the authorized_keys file in the user’s home directory, the configuration can point to a location that only root can modify.

e.g. /etc/ssh/sshd_config could read

AuthorizedKeysFile /etc/ssh/auth_keys/%u

Each file in that directory should be owned by root:root and have permission 0644 (-rw-r--r--). Now Fred can not grant John access to the system, unauthorized access to oracle is prevented, NFS risks have gone away…

We haven’t solved the whole problem, of course; your SAs can still manage this and bypass your access admin processes.

And we’ve now got a massive “key distribution” problem. Where should John’s keys be copied to? When he leaves, or changes teams, can we remove all these keys? Can we force key rotation when they get to a certain age? Can all this be automated? Maybe even hooked into your access admin tools?

“John is an Oracle DBA; he’s allowed to login to the oracle servers” is a problem your access admin tools should already be solving; can some form of ssh key management be added so that John’s public key is published to those servers automatically?

There’s a number of challenges that such an automation tool needs to deal with:

• Allow a human to have their own unique key (maybe self-generated or system generated), and have that deployed to servers they’re allowed to login to.
• Allow a functional account to have multiple keys with command and from (and other) restrictions applied to them. Auto-generate the keys and auto-deploy the private key to the authorized servers.
• Handle key rotation so that automation processes don’t break (add new public key to target servers, replace private keys, remove old public keys)
• Handle expiration, re-certification, automation, integration…
• Manage the GlobalKnownHostsFile so that clients can be told about server host keys, handle rebuilds/redeploys, dynamic host builds, etc.

SSH keys can be used to bypass your existing controls solutions and so need to be controlled. This is an area that hasn’t seen a lot of focus, but I predict a number of regulatory environments are going to start looking at how keys are managed. You can either forbid keys, totally, or manage them.

I recommend trying to manage them!