fritzhardy.com

SSH Gate

With many servers comes more and more complex systems and the need to manage a web of interdependencies. The following describes a setup I have employed that we have come to see as invaluable.

OpenSSH is an incredibly powerful piece of software, and by default most of its features are barely utilized. Each of the following sections shows how to add more juice to a standard OpenSSH setup, and each gets progressively closer to our goal: effortless, encrypted, authentication and communication between Linux servers.

The Default: Secure Password Authentication

# ssh username@remotebox
username@remotebox's password:

This is the default way ssh handles authentication. A user is asked for his/her password on remotebox, the password is transmitted encrypted over the wire, checked against the remotebox database, and voila: remotebox shell.

PROS: - Very secure, have to re-authenticate with every new connection.

CONS: - Have to re-authenticate with every new connection.; - Unattended cron jobs and scripts are a problem since we need a password.

Better: Shared Keys

SSH also supports authentication using shared keys. We generate a pair of keys, one private and one public. When we attempt to authenticate, remotebox generates a random number and encrypts it using our public key (that is all a public key is good for). This is sent back to us, and if we can successfully decrypt it using our private key, we have proven we are who we say we are.

OpenSSH supports RSA and/or DSA authentication (SSH1 and SSH2) using shared keys. We generate our keys using the ssh-keygen program. We will generate a DSA key, which requires SSH2. Also, make sure NOT to take the easy way out. Do NOT create a key with no passphrase (this will be explained later).

# ssh-keygen -t dsa
Generating public/private dsa key pair.
Enter file in which to save the key (/root/.ssh/id_dsa):
Enter passphrase (empty for no passphrase):
Enter same passphrase again:
Your identification has been saved in /root/.ssh/id_dsa.
Your public key has been saved in /root/.ssh/id_dsa.pub.
The key fingerprint is:
0f:12:24:4a:87:e2:55:f7:52:a0:1a:cc:c9:46:9b:64 root@localbox

Now, when we concatenate the id_dsa.pub to a user's ~/.ssh/authorized_keys2 file on remotebox, (authorized_keys for rsa keys), we will be able to connect to that machine as that user.

PROS: - Still secure, have to re-authenticate with every new connection.

CONS: - Still inconvenient, have to re-authenticate with every new connection.

If we had not entered a passphrase above, our private key would not be encrypted. This is an easy way to avoid having to re-enter a passphrase with every connection, and is one way to setup for unattended cron tasks between machines. But it also means that if our localbox is hacked, the hacker would be able to skate into any remote boxes we had access to. By entering a passphrase, we encrypt the private key so it is safer to store on our disk. If our box is hacked, it is still a significant obstacle to actually use the key to impersonate us and gain access to other hosts having our public key.

Unfortunately, if we do things the "right" way, the result is still largely the same as the default password setup... we are entering a passphrase for every new connection.

SSH-Agent

SSH-Agent is a program included with OpenSSH that allows us to properly use encrypted keys, but also avoid having to re-type the passphrase. It does this by caching decrypted private keys, allowing us a higher level of convenience without sacrificing much security. Here is typical ssh-agent output:

# ssh-agent
SSH_AUTH_SOCK=/tmp/ssh-XXM9mJcq/agent.32461; export SSH_AUTH_SOCK;
SSH_AGENT_PID=32462; export SSH_AGENT_PID;
echo Agent pid 32462;

The output of ssh-agent is a series of bash commands setting up environment variables. If we eval this command, these environment variables are exported to our shell.

# eval `ssh-agent`

Once started (best run from our .bash_profile), we need to run ssh-add to add our private keys to its cache:

# ssh-add ~/.ssh/id_dsa
Need passphrase for /root/.ssh/id_dsa

Now that ssh-agent knows about our key, we can access remote systems without a passphrase!

PROS: - Finally we enter a passphrase only once.

CONS: - New ssh-agent for each login session, meaning we have to re-run ssh-add, etc.; - Cron jobs do not inherit the necessary environment variables and cannot contact ssh-agent.

Answer: Keychain

Enter Keychain to squash these last two cons. Keychain is an intelligent bash front-end to ssh-agent, which allows us to use a single ssh-agent process per system, avoiding re-authentication for every new login. Keychain is available at http://www.gentoo.org/proj/en/keychain/index.xml. Once installed, add the following two lines to your .bash_profile:

/usr/bin/keychain ~/.ssh/id_dsa
source ~/.ssh-agent > /dev/null

The first line is the path to our private key (or many lines for many keys). Read the man page for keychain to see lots of other powerful options. Also, you'll need to make sure ~/.ssh-agent actually exists as later versions install files in ~/.keychain (symlinks will suffice). This is the file that contains the environment variables. Upon first login, keychain prompts us for the passphrases to each of our private keys:

KeyChain 2.3.5; http://www.gentoo.org/projects/keychain
Copyright 2002-2004 Gentoo Technologies, Inc.; Distributed under the GPL

* Found existing ssh-agent at PID 11718
* Adding 1 key(s)...
Enter passphrase for /root/.ssh/id_dsa:
* Identity added: /root/.ssh/id_dsa (/root/.ssh/id_dsa)

PROS: - Secure, encrypted keys.; - Finally, re-authentication is rare.

CONS: - Make sure private keys are on a well-protected box.

Subsequent logins do not ever have to re-enter the passphrase in this setup. We have reached our goal... now let's surpass it.

More Improvements: SSH Forwarding Agents and Cron

In our current situation, we can only login to remote machines using our keys from one source... our SSH Gate:

The trusted host sshgate can only connect to each box, the boxes can not connect to one another in a passwordless fashion. To work around this, we could have ssh-agent running on every box, with copie of our keys everywhere, but that is a security hazard. Instead, we will create forwarding agents. With a single line added to a machine's /etc/ssh/ssh_config (not sshd_config), any machine can become a forwarding agent:

ForwardAgent Yes

As a forwarding agent, all connections originating from that trusted machine sshgate, indirectly or directly, can use sshgate's own ssh-agent process rather than ssh-agent on the remotebox:

From the ssh_config manpage: Agent forwarding should be enabled with caution. Users with the ability to bypass file permissions on the remote host (for the agent’s Unix-domain socket) can access the local agent through the forwarded connection. An attacker cannot obtain key material from the agent, however they can perform operations on the keys that enable them to authenticate using the identities loaded into the agent.

If untrust1 is not a forwarding agent, it means nothing if our connection TO untrust1 was FROM a forwarding agent: the ssh-agent socket file from sshgate (or an intermediary forwarding agent) is dumped into untrust1's /tmp. Make sure the ssh_config file on sshgate (and any intermediary forwarding agents) specifies that connections to untrust1 should not be forwarded with a Host declaration indicating such.

Some implications for agent forwarding:

Keychain only needs to run on one box, also setup as a forwarding agent.
Other trusted boxes receive only the public key, but are also forwarding agents.
Untrusted hosts receive only our public key, and these are not forwarding agents.

PROS: - All of the above goals.; - Minimal exposure of ssh-agent, avoiding possibility, however remote, of decrypted keys being extracted from untrusted hosts.; - Minimal configuration on each host.; - We can bounce from host to host within our trust domain!

CONS: - As always, the Keychain box must be kept secure.

How does cron fit into this? We now have unadulterated access from the Keychain box to run commands over ssh to any other box in our trust domain. One possibility is a central enterprise task scheduler that runs commands on remote machines. The cron task simply needs to be passed the ssh-agent info. Here is an example cron entry:

*/2 * * * * root source /root/.ssh-agent; ssh root@somehost "/some/script/somewhere"

Other possibilities abound! For more info, see the article at http://www-106.ibm.com/developerworks/linux/library/l-keyc.html

Hardening the Trusted Host

We have eliminated all of the cons we were concerned about, but that last one will always remain... the security of the one trusted machine. Some possibilities to address that last con:

Strict iptables rules only allow incoming connections for SSH, and perhaps DNS from specified hosts (no spoofing). Perhaps paranoid iptables rules allowing connections from a finite list of hosts.
Users log into a custom shell that only allows a handful of commands. Root access closely guarded.
SELinux to harden the machine.

These measures are beyond the scope of this document, but should be strongly considered. Some possibilities with SELinux:users have no direct access to their own private keys, no access to some /etc files, etc. Very powerful... and doable. We have run variations on the above theme for years.

No matter what measures you put in place, users will always be the weakest link, and they need to be made well aware of the implications. In many ways, we have created a castle gate. If an intruder sneaks through our gate, he/she is not likely to spend a lot of time trying to sabotage the gate, since it is armored, guarded, heavily fortified, and of comparably little interest. Of far more interest is what lies beyond the gate to which he/she now has full access.