Opened 6 years ago

Closed 7 weeks ago

Last modified 7 weeks ago

#10408 closed Bug/Something is broken (fixed)

red.mayfirst.org x.509 certificate expired

Reported by: Daniel Kahn Gillmor Owned by:
Priority: Medium Component: Tech
Keywords: x.509 hay.mayfirst.org red.mayfirst.org Cc:
Sensitive: no

Description

The x.509 certificate for red.mayfirst.org has expired. This is what is used to connect to the database. It needs to be re-issued and replaced in hay:/etc/mysql/server-cert.pem. I'm attaching the CSR for it here. This needs to be done by the MF/PL CA.

Attachments (3)

red.mayfirst.org.csr (1.7 KB) - added by Daniel Kahn Gillmor 6 years ago.
CSR for red.mayfirst.org
red.mayfirst.org.crt (5.6 KB) - added by Jamie McClelland 6 years ago.
red.mayfirst.org.2.crt (5.6 KB) - added by Jamie McClelland 6 years ago.

Download all attachments as: .zip

Change History (12)

Changed 6 years ago by Daniel Kahn Gillmor

Attachment: red.mayfirst.org.csr added

CSR for red.mayfirst.org

Changed 6 years ago by Jamie McClelland

Attachment: red.mayfirst.org.crt added

comment:1 Changed 6 years ago by Jamie McClelland

crt file is attached.

comment:2 Changed 6 years ago by Daniel Kahn Gillmor

thanks, i've installed it. unfortunately, it seems that some versions of mysql can't handle sha256 certs. Can you try reissuing it with an SHA1 digest?

Changed 6 years ago by Jamie McClelland

Attachment: red.mayfirst.org.2.crt added

comment:3 Changed 6 years ago by Jamie McClelland

sha1 version attached as red.mayfirst.org.2.crt​

comment:4 Changed 6 years ago by Daniel Kahn Gillmor

Resolution: fixed
Status: newclosed

OK, turns out this was not how we were handling this cert at all :/ here's the run-down:

the red.mayfirst.org database is secured with a self-signed x.509 certificate, which is distributed to the nodes via puppet/modules/mayfirst/files/mysql-server/red-cert.pem. This certificate is the one that expired. When it was expired, all attempts to connect failed in this way:

0 june:~# red-node-update 
PHP Warning:  mysqli::real_connect(): (HY000/2026): SSL connection error: ASN: bad other signature confirmation in /usr/local/share/red/node/sbin/red-node-update on line 33
0 june:~# 

I updated the cert directly on hay with:

certtool --generate-self-signed --template server-cert.template --load-privkey server-key.pem --hash=sha1 > server-cert.pem

(--hash=sha1 is to avoid the mysql bug linked above, sigh).

Then i pushed the new self-signed cert to puppet

Last edited 6 years ago by Daniel Kahn Gillmor (previous) (diff)

comment:5 Changed 17 months ago by updater

Sensitive: set

Changed to sensitive as part of leadership decision to make all tickets sensitive.

comment:6 Changed 15 months ago by Jamie McClelland

Sensitive: unset

For future reference (since this ticket seems to contain the best docs possible), we can now use sha256.

Some notes:

  • Before running the command, make a backup of the existing key:
    mv /etc/mysql/server-cert.pem{,.bak}
    
  • Run the command from inside /etc/mysql
    cd /etc/mysql
    
  • Generate a new certificate:
    certtool --generate-self-signed --template server-cert.template --load-privkey server-key.pem --hash=sha256 > server-cert.pem
    
  • This generates a cert inside the server's /etc/mysql directory - to get mysql to re-read, restart mysql
    systemctl restart mysql
    
  • Then copy server-cert.pem ./modules/mayfirst/files/mysql-server/red-cert.pem and sign a new puppet tag

Also, fyi, the contenst of server-cert.template:

# The common name of the certificate owner.
cn = "red.mayfirst.org"
# X.509 Certificate options for certtool
organization = "May First/People Link"


dns_name = "red.mayfirst.org"

# In how many days, counting from today, this certificate will expire.
expiration_days = 400
# _NAME = "red.mayfirst.org"

tls_www_server

# Whether this certificate will be used to sign data (needed
# in TLS DHE ciphersuites).
signing_key

# Whether this certificate will be used to encrypt data (needed
# in TLS RSA ciphersuites). Note that it is preferred to use different
# keys for encryption and signing.
encryption_key
Last edited 15 months ago by Jamie McClelland (previous) (diff)

comment:7 Changed 7 weeks ago by JaimeV

Resolution: fixed
Status: closedassigned

The cert expired yesterday. I thought it was a problem on menchu but after a lot of testing and searching I discovered it was happening everywhere and I eventually found this ticket.

I've followed the steps in the last comment above but when the moshes attempt to connect with the new cert they I am seeing the following error.

ERROR 2026 (HY000): SSL connection error: The certificate is NOT trusted. The certificate issuer is not a CA. The certificate chain violates t

Its true, the new cert wasn't signed by any CA, neither was the previous cert as far as I can tell.Has something changed in the mysql config on hay to prevent connection with a self signed cert with no CA? Should we be creating a CA and signing the new cert with it? This seems to be the standard recommendation.

https://mariadb.com/kb/en/certificate-creation-with-openssl/

comment:8 Changed 7 weeks ago by Jamie McClelland

Can you post how you are testing and generating that error?

I see that the certs have definitely expired with:

openssl x509 -in /etc/mysql/red-cert.pem -noout -text -purpose | less

However, I just successfully modified a record on chavez (with expired cert) and received no errors.

Running red-node-update should trigger a sql SELECT statement to be executed against the server and cause a failure, but I'm not seeing any failure on menchu or chavez.

I'm not sure why... but if you can share the command that triggers that error, it would be great.

comment:9 Changed 7 weeks ago by JaimeV

Resolution: fixed
Status: assignedclosed

I was receiving the error when attempting to connect directly with mysql client but it was my mistake, I thought I'd pulled in the new cert but it hadn't yet. After running puppet again everything seems to be working.

I've gone through the list of servers with items stuck in pending since yesterday, cleared out stale /root/.red-server-cli.lock dirs and run red-node-update again there.

Last edited 7 weeks ago by JaimeV (previous) (diff)

Please login to add comments to this ticket.

Note: See TracTickets for help on using tickets.