Opened 11 days ago

Last modified 10 days ago

#14863 assigned Bug/Something is broken

fix monkeysphere certifier usage with may first infrastructure in light of key server abuse

Reported by: Jamie McClelland Owned by: Jamie McClelland
Priority: Medium Component: Tech
Keywords: Cc:
Sensitive: no

Description

As dkg has documented a recent attack on his key has caused problems with the normal operations of gpg and by extension for us, monkeysphere.

In the case of MF infrastructure, the problem was that the hourly monkeysphere cron job pulls in updates from the keyserver network for all keys listed as id certifiers. That includes dkg's key. So, we pulled in dkg's bloated on all of our servers.

When the bloated gpg key is pulled in, simply running gpg --list-keys takes forever to completed.

And, since we have an hourly cron job that runs that command, we end up spinning in a CPU death spiral.

Jaime and I immiedately addressed the problem by disabling the /etc/cron.d/monkeysphere hourly cron job.

This is not a long term solution since it means we will never process revocations or other changes to our list certifier keys.

Change History (14)

comment:1 Changed 11 days ago by Jamie McClelland

Here is a summary of changes to fix it.

We have four certifiers: jamie, jaimev, nat, and dkg.

  • Add a known good export of each certifier's key (without bloat) to our puppet repo
  • Instead of initially populating keys using monkeysphere-authentication add-id-certifier by fingerprint, populate them by providing the path to the known good key file
  • Update /etc/monkeysphere/monkeysphere-authentication.conf file to use keys.openpgp.org as the key server for updates.

Here is the diff in our puppet configurations to fix it:

diff --git a/modules/mayfirst/files/monkeysphere/certifiers/dkg.gpg b/modules/mayfirst/files/monkeysphere/certifiers/dkg.gpg
new file mode 100644
index 00000000..85a1a9b3
Binary files /dev/null and b/modules/mayfirst/files/monkeysphere/certifiers/dkg.gpg differ
diff --git a/modules/mayfirst/files/monkeysphere/certifiers/jaimev.gpg b/modules/mayfirst/files/monkeysphere/certifiers/jaimev.gpg
new file mode 100644
index 00000000..3ca93df7
Binary files /dev/null and b/modules/mayfirst/files/monkeysphere/certifiers/jaimev.gpg differ
diff --git a/modules/mayfirst/files/monkeysphere/certifiers/jamie.gpg b/modules/mayfirst/files/monkeysphere/certifiers/jamie.gpg
new file mode 100644
index 00000000..427e991c
Binary files /dev/null and b/modules/mayfirst/files/monkeysphere/certifiers/jamie.gpg differ
diff --git a/modules/mayfirst/files/monkeysphere/certifiers/nat.gpg b/modules/mayfirst/files/monkeysphere/certifiers/nat.gpg
new file mode 100644
index 00000000..95983f57
Binary files /dev/null and b/modules/mayfirst/files/monkeysphere/certifiers/nat.gpg differ
diff --git a/modules/mayfirst/manifests/m_minimal.pp b/modules/mayfirst/manifests/m_minimal.pp
index 4867b07a..230cf035 100644
--- a/modules/mayfirst/manifests/m_minimal.pp
+++ b/modules/mayfirst/manifests/m_minimal.pp
@@ -277,15 +277,19 @@ class mayfirst::m_minimal (
   
   # Valid certifiers
   m_monkeysphere::add_id_certifier { "dkg":
-	  keyid => "0EE5BE979282D80B9F7540F1CCD2ED94D21739E9" 
+	  keypath => "/etc/puppet/modules/mayfirst/files/monkeysphere/certifiers/dkg.gpg",
+	  keyid => "0EE5BE979282D80B9F7540F1CCD2ED94D21739E9"
   }
   m_monkeysphere::add_id_certifier { "jamie":
+	  keypath => "/etc/puppet/modules/mayfirst/files/monkeysphere/certifiers/jamie.gpg",
 	  keyid => "1F9C30CB3CFC5DA9987FA035A014C05A607B7535"
   }
   m_monkeysphere::add_id_certifier { "nat":
-	  keyid => "0ADA1A85619A5D0E5A414F84FFD913B7CB2D0500" 
+	  keypath => "/etc/puppet/modules/mayfirst/files/monkeysphere/certifiers/nat.gpg",
+	  keyid => "0ADA1A85619A5D0E5A414F84FFD913B7CB2D0500"
   }
   m_monkeysphere::add_id_certifier { "jaimev":
+	  keypath => "/etc/puppet/modules/mayfirst/files/monkeysphere/certifiers/jaimev.gpg",
 	  keyid => "63D2D61FC6B6F50966790B61EF3EAA1428D1FF61"
   }
 
diff --git a/modules/mayfirst/manifests/m_monkeysphere.pp b/modules/mayfirst/manifests/m_monkeysphere.pp
index 8a156d79..76c2ad90 100644
--- a/modules/mayfirst/manifests/m_monkeysphere.pp
+++ b/modules/mayfirst/manifests/m_monkeysphere.pp
@@ -81,8 +81,8 @@ define m_monkeysphere::publish_server_keys {
 }
 
 # add certifiers
-define m_monkeysphere::add_id_certifier( $keyid ) {
-  exec { "monkeysphere-authentication add-id-certifier $keyid":
+define m_monkeysphere::add_id_certifier( $keypath, $keyid ) {
+  exec { "monkeysphere-authentication add-id-certifier $keypath":
 	  environment => "MONKEYSPHERE_PROMPT=false",
 	  require => [ Package["monkeysphere"], File["monkeysphere_authentication_conf"] ],
 	  unless => "/usr/sbin/monkeysphere-authentication list-id-certifiers | grep $keyid > /dev/null"
diff --git a/modules/mayfirst/templates/monkeysphere/monkeysphere-authentication.conf.erb b/modules/mayfirst/templates/monkeysphere/monkeysphere-authentication.conf.erb
index 57da0a2a..ee2bccda 100644
--- a/modules/mayfirst/templates/monkeysphere/monkeysphere-authentication.conf.erb
+++ b/modules/mayfirst/templates/monkeysphere/monkeysphere-authentication.conf.erb
@@ -11,7 +11,8 @@
 #LOG_LEVEL=INFO
 
 # OpenPGP keyserver
-KEYSERVER=pool.sks-keyservers.net
+#KEYSERVER=pool.sks-keyservers.net
+KEYSERVER=keys.opengpg.org
 # User who controls the monkeysphere 'sphere' keyring.
 #MONKEYSPHERE_USER=monkeysphere

comment:2 Changed 11 days ago by Jamie McClelland

The only remaining question is: how do I get rid of the bloated key rings? Unless I purge them before running this code, it won't work. I just tested rm /var/lib/monkeysphere/authentication/{sphere,core}/pubring.gpg - which works. But not sure if there is a less extreme method to use.

comment:3 Changed 11 days ago by Jamie McClelland

Owner: set to Jamie McClelland
Status: newassigned

comment:4 Changed 11 days ago by Daniel Kahn Gillmor

please update the keyserver to be hkps://keys.openpgp.org, not keys.openpgp.org.

comment:5 Changed 11 days ago by Jamie McClelland

Summary: fix monkeysphere usage with may first infrastructure in light of key server abusefix monkeysphere certifier usage with may first infrastructure in light of key server abuse

comment:6 Changed 11 days ago by Daniel Kahn Gillmor

Summary: fix monkeysphere certifier usage with may first infrastructure in light of key server abusefix monkeysphere usage with may first infrastructure in light of key server abuse

my fingerprint (mislabled keyid) is wrong in the m_minimal manifest, even though the file has the correct certificate. can we drop the keyid entries entirely for certifiers?

comment:7 Changed 11 days ago by Daniel Kahn Gillmor

Summary: fix monkeysphere usage with may first infrastructure in light of key server abusefix monkeysphere certifier usage with may first infrastructure in light of key server abuse

sorry, i didn't mean to clobber your summary change.

comment:8 Changed 11 days ago by Daniel Kahn Gillmor

Unless the certifier's certificate was fetched by fingerprint from SKS recently, the core keyring should be fine. please don't rm it!

the problem is likely to be for certificates that are stored in the sphere keyring, and in particular, only for hosts that were set up before stretch, such that they have pubring.gpg and not pubring.kbx in /var/lib/monkeysphere/sphere/.

So one way to clean it on a system that is running gpg 2.1 or later is, as the monkeysphere user (untested, please read and think about it first!):

if [ -e /var/lib/monkeysphere/authentication/sphere/pubring.gpg ]; then
    mv /var/lib/monkeysphere/authentication/sphere/pubring.gpg{,.bak}
    GNUPGHOME=/var/lib/monkeysphere/authentication/sphere gpg --import < /var/lib/monkeysphere/authentication/sphere/pubring.gpg.bak
fi

can you test that on one host and verify that it works?

The new import should use the keybox format (pubring.kbx), and the DoS prevention mechanisms on the keybox format (as broken as they might be) will reject the import of flooded certificates.

This import process might take a little while to run unfortunately, due to GnuPG's inefficient implementation.

comment:9 Changed 11 days ago by Jamie McClelland

Thanks dkg. I ran the commands and they completed without errors.

But, I hit (and overcame) two problems:

  • The keyring.kbx file was owned by root, not monkeysphere. Easy to fix!
  • When I finished, monkeysphere-authentication list-certifiers returned nothing. However, when I re-ran monkeysphere-authentication add-id-certifier for each certifier it listed them ok

comment:10 Changed 11 days ago by Jamie McClelland

Also, I fixed the keyid/keyfpr problem and updated with your correct key. I think we could remove the fpr. The reason it's there is so we can first check if a fpr is already a certififier before re-adding it. So, we could either always import the key or we could search on a different identifier.

comment:11 Changed 11 days ago by Jamie McClelland

And, as a note, the import of the pubring.gpg into pubring.kbx returns exit code 2. With -v it reports (just showing the last few lines):

gpg: key 0xF20691179038E5C6: 54612 signatures not checked due to missing keys
gpg: error writing keyring '/var/lib/monkeysphere/authentication/sphere/pubring.kbx': Provided object is too large
gpg: key 0xF20691179038E5C6: public key "[User ID not found]" imported
gpg: Total number processed: 45
gpg:               imported: 45
gpg:           not imported: 1
gpg: 25 keys processed (0 validity counts cleared)
gpg: no ultimately trusted keys found
monkeysphere@viewsic:~/authentication/sphere$ echo $?
2
monkeysphere@viewsic:~/authentication/sphere$

I'm reading this as being successful for our purposes and will tell puppet to ignore the error code of 2.

comment:12 Changed 11 days ago by Jamie McClelland

And, on jessie servers, we get an error adding dkg as a certifier:

0 woodhull:~# monkeysphere-authentication add-id-certifier /etc/puppet/modules/mayfirst/files/monkeysphere/certifiers/dkg.gpg
gpg: /tmp/monkeysphere.qbOhCuOfSH/trustdb.gpg: trustdb created
There was not exactly one gpg key in the file.
255 woodhull:~#

I'm going to ignore this error. It doesn't happen on stretch and it only affects dkg as a certifier. So, on our very few (and decreasing by the day) jessie servers, dkg will not be properly listed as a certifier. However, his root access won't be affected since his key is signed by other certifiers. And, once our last jessie server is upgraded (should be in a matter of weeks) this will no longer be an issue.

comment:13 Changed 10 days ago by Jamie McClelland

I've signed a tag and this is going out to all servers now.

And... I just noticed that the monkeysphere cron job outputs these errors when run manually:

0 neruda:~# flock -n -E 0 /run/monkeysphere-authentication-update-users.lock /usr/sbin/monkeysphere-authentication update-users
ms: Failure (2) searching keyserver hkps://keys.opengpg.org for user id 'James McClelland <jamie@mayfirst.org>'
ms: Failure (2) searching keyserver hkps://keys.opengpg.org for user id 'Daniel Kahn Gillmor <dkg@fifthhorseman.net>'
ms:     ! sub key could not be translated (not RSA?).
ms: Failure (2) searching keyserver hkps://keys.opengpg.org for user id 'Alfredo Lopez <alfredo@mayfirst.org>'
ms: Failure (2) searching keyserver hkps://keys.opengpg.org for user id 'Ross Glover <ross@ross.mayfirst.org>'
ms: Failure (2) searching keyserver hkps://keys.opengpg.org for user id 'Nat Meysenburg <nat@stealthisemail.com>'
ms: Failure (2) searching keyserver hkps://keys.opengpg.org for user id 'Josue Guillen <josue@mayfirst.org>'
ms: Failure (2) searching keyserver hkps://keys.opengpg.org for user id 'Mallory Knodel <mallory@mayfirst.org>'
ms: Failure (2) searching keyserver hkps://keys.opengpg.org for user id 'Joseph Lacey <joseph@mayfirst.org>'
ms: Failure (2) searching keyserver hkps://keys.opengpg.org for user id 'Enrique Rosas <erq@mayfirst.org>'
ms: Failure (2) searching keyserver hkps://keys.opengpg.org for user id 'Steve Revilak <steve@srevilak.net>'
ms: Failure (2) searching keyserver hkps://keys.opengpg.org for user id 'Jaime Villarreal <jaimev@mayfirst.org>'
ms: Failure (2) searching keyserver hkps://keys.opengpg.org for user id 'James McClelland <jamie@mayfirst.org>'
0 neruda:~#

It seems that it is trying to lookup the keys on keys.openpgp.org via user id instead of fingerprint.

comment:14 in reply to:  13 Changed 10 days ago by Daniel Kahn Gillmor

Replying to Jamie McClelland:

It seems that it is trying to lookup the keys on keys.openpgp.org via user id instead of fingerprint.

yes, monkeysphere certainly looks up by user ID -- it's trying to convert from authorized_user_ids to public keys. It has piggybacked on traditional SKS behavior to also treat this as a certificate refresh.

Please login to add comments to this ticket.

Note: See TracTickets for help on using tickets.