Opened 4 months ago

Last modified 3 months ago

#14864 assigned Bug/Something is broken

change server to server authentication to use ssh keys and a stock known_hosts file

Reported by: Jamie McClelland Owned by: Jamie McClelland
Priority: Medium Component: Tech
Keywords: Cc:
Sensitive: no

Description

Currently, our server to server communications uses monkeysphere, which makes it very convenient to drop in root@server.mayfirst.org to a ~/.monkeysphere/authorized_user_ids file, run monkeysphere-authentication update-users <user> and then have access.

However, due to the abuse of the keyservers, it has become increasingly unreliable to exchange keys that way.

So, the proposed fix is to generate an ssh key pair for each root user on each server and then add that key to the ~/.ssh/authorized_keys file on the target server.

In addition we will use a stock ~/.ssh/known_hosts file for each root user so we don't get tripped up by fingerprint verification.

Change History (4)

comment:1 Changed 4 months ago by Jamie McClelland

Owner: set to Jamie McClelland
Status: newassigned

comment:2 Changed 4 months ago by Jamie McClelland

Summary: change server to server authentication to use ssh keys and a stock know_hosts filechange server to server authentication to use ssh keys and a stock known_hosts file

This project has two parts:

  • use ssh public keys to authenticate servers: this part is done in puppet but needs to be implemented in all servers. See medgar.pp for an example. In short we have to copy the public key to a variable at the top of the file and then add that variable to our various define statements that setup access.
  • use a standard known hosts file: this part I just completed in puppet. The next time we push, it should go out to all servers. It involves these changes:
    • For root users:
      • We now have three new files in /etc/ssh/:
        • known_hosts.backup.servers: hashes of our four backup servers (all servers connect via ssh to backup)
        • known_hosts.monitor.servers: hashes of our monitoring server (all servers connect to our monitor server to copy status information)
        • known_hosts.user.servers: hash of our server holding user logins - hay (some servers - gil and paulo - connect to our user server to copy down valid usernames)
    • For the www-data user on hay:
      • known_hosts.mosh.servers: hashes of all mosh servers
  • Change ~/.ssh/config to include a reference to these know known_hosts files using UserKnownHostsFile (allows us to list multiple files to be consulted)

We will have to maintain all of these files - however, the list of mosh servers is the one that changes the most frequently and that is not all that frequently.

Last edited 4 months ago by Jamie McClelland (previous) (diff)

comment:3 Changed 4 months ago by Jamie McClelland

known_hosts.mosh.servers should be known_hosts.red.servers since hay has to connect to all red servers not just mosh servers (e.g. dns servers, network database servers, list servers).

comment:4 Changed 3 months ago by Jamie McClelland

This task is nearly done.

From now on, new servers should have the contents of their ~/.ssh/id_rsa.pub files added to the puppet manifests/globals.pp file. All existing servers have been added. If a key is present in this hash, then the key will be added to the appropriate ~/.ssh/authorized_keys files.

In addition, from now on, if a server is being used by red (e.g. mosh server) then you will need to run:

helper/generate-known-host-files <server>

This will output content that should be added to: modules/mayfirst/ssh/known_hosts.red.servers so that the www-data user on hay can properly connect over ssh to the given server.

That script also maintains the dns servers, monitor servers and backup server lists - maintaining a know_hosts file for each category of servers.

This is all documented on install_kvm.

These changes have been selectively pushed out to our backup servers, jojobe and hay, but not yet in a signed tag.

At the moment, we have a belt and suspenders approach: both raw ssh keys and monkeysphere user ids are in use. If this goes smoothly over the next week I will remove the ~/.monkeysphere/authorized_user_ids files from our non-provileged logins and also remove the ssh-agent-root.service and this ticket will be complete.

Note: we are still using monkeysphere to authenticate root users.

Please login to add comments to this ticket.

Note: See TracTickets for help on using tickets.