| 1 | = Debug Server to Server connections = |
| 2 | |
| 3 | Our servers often need to ssh into each other to carry out various tasks. Most commonly: |
| 4 | |
| 5 | * Each server has to rsync, over ssh, to it's designated backup servers |
| 6 | * Each server copies data to jojobe, our nagios server, so we can get alerts if anything is amiss |
| 7 | |
| 8 | These connections are configured using the [https://monkeysphere.info monkeysphere], specifically: |
| 9 | |
| 10 | * Each server generates an OpenPGP key and corresponding authentication subkey |
| 11 | * Each server runs a ssh agent via runit (/etc/sv/ssh-agent-root) that keeps the authentication subkey loaded in memory so it can use it to access remote servers |
| 12 | * Each remote server is configured with the User Ids of the root OpenPGP keys that should be able to access it. |
| 13 | |
| 14 | Unfortunately, sometimes things go wrong and servers are not able to connect to each other. |
| 15 | |
| 16 | In these examples I refer to the "connecting" server and the "target" server to distinguish between the to. |
| 17 | |
| 18 | Here are the top causes for the failures, and the remedies: |
| 19 | |
| 20 | * Something when wrong with ssh-agent on the connecting server. Fix: Stop and restart the service, check for existence of socket: |
| 21 | {{{ |
| 22 | sv stop ssh-agent-root |
| 23 | sv start ssh-agent-root |
| 24 | ls -l /root/.ssh-agent-socket |
| 25 | }}} |
| 26 | * The target server does not have the latest version of the connecting server's OpenPGP key. Fix: refresh the key, reload the credentials, and test: |
| 27 | {{{ |
| 28 | monkeysphere-authentication refresh-keys <username> |
| 29 | monkeysphere-authentication update-users <username> |
| 30 | cat /var/lib/monkeysphere/authorized_keys/<username> |
| 31 | }}} |
| 32 | * The connecting server has not published the latest version of it's key. Fix: determine the keyid of the server's secret key, and then publish it: |
| 33 | {{{ |
| 34 | gpg --list-secret-key |
| 35 | gpg --keyserver keys.mayfirst.org --send-key <keyid> |
| 36 | }}} |
| 37 | Then, refresh the key on the target (see above). |
| 38 | * The connecting server's OpenPGP key is expired. Fix: extend it: |
| 39 | {{{ |
| 40 | mf-gpg-extend-root-expiration |
| 41 | }}} |
| 42 | (This will also publish it). Then, refresh the key on the target (see above) |