Opened 8 years ago

Closed 7 years ago

#4434 closed Bug/Something is broken (fixed)

mail.mayfirst.org does not work with POP using starttls

Reported by: takethestreets Owned by: Jamie McClelland
Priority: Medium Component: Tech
Keywords: POP3 mail.mayfirst.org TLS Cc:
Sensitive: no

Description

I was running some tests on various configurations of mail.mayfirst.org under Windows, and got this error when attempting to use POP:

Sending of password did not succeed. Mail server mail.mayfirst.org responded: USER takethestreets set, mate

When I kept all other settings intact but changed the incoming server to chavez.mayfirst.org, POP mail worked fine.

Attachments (1)

ticket4434.debdiff (2.0 KB) - added by Daniel Kahn Gillmor 7 years ago.
debdiff (difference between debian source package in squeeze) changing SSLv23_method to TLSv1_method

Download all attachments as: .zip

Change History (19)

comment:1 Changed 8 years ago by Jamie McClelland

Thanks Jon.

Confirmed on Icedove 3 (what a pain to create a POP account on Icedove 3 - i had to cancel the wizard before it completed).

I haven't debugged yet...

jamie

comment:2 Changed 8 years ago by Jamie McClelland

Summary: mail.mayfirst.org does not work with POP on Thunderbird 5mail.mayfirst.org does not work with POP using starttls

I'm re-summarizing to be more accurate. mail.mayfirst.org now works with thunderbird using tls, but not with starttls and I suspect that's a server-problem.

jamie

comment:3 Changed 8 years ago by Jamie McClelland

I've made a minor adjustment to our courier pop configuration on chavez (not propagated to other servers) and starttls with POP now seems to work both on chavez and mail.mayfirst.org. Jon - can you try again? I think takethestreets is on chavez.

jamie

p.s. I added:

POP3AUTH_TLS="LOGIN PLAIN"

To /etc/courier/pop3d which allows plain passwords if tls is enabled.

comment:4 Changed 8 years ago by takethestreets

No luck - like the original ticket mentions, I can use STARTTLS with chavez but not mail.mayfirst.org. I get the same error. I tested with Thunderbird 6 on Windows XP.

On a related topic, I see that you've published a new "configure e-mail" wiki page. Back in June I created a new page for Thunderbird setup that a) reflects the new mail.mayfirst.org settings, and b) uses screenshots from modern Thunderbird (TB5 revamped the account creation screens). It's here: https://support.mayfirst.org/wiki/email_setup_thunderbird_beta

I imagine that if you have brought the new e-mail configuration pages live, you may want to replace the existing TB page with that one.

comment:5 Changed 8 years ago by Jamie McClelland

Hey Jon - drat.

I saw your new page - I'm not yet publicizing mail.mayfirst.org because I'd like to work out this issue first. I'm also not sure if we want to promote starttls or strict ssl/tls. When we do move it forward, I'll definitely swap out our old thunderbird page with yours.

jamie

comment:6 Changed 7 years ago by Daniel Kahn Gillmor

I can confirm that POP with STARTTLS doesn't work on julia or chavez, but does work on mail.mayfirst.org:

0 dkg@pip:~$ echo QUIT | openssl s_client -starttls pop3 -quiet -connect mail.mayfirst.org:pop3
depth=1 C = US, O = "GeoTrust, Inc.", CN = RapidSSL CA
verify error:num=20:unable to get local issuer certificate
verify return:0
+OK POP3 perditon ready on gil.mayfirst.org 0002aaa1
+OK QUIT
read:errno=0
0 dkg@pip:~$ echo QUIT | openssl s_client -starttls pop3 -quiet -connect chavez.mayfirst.org:pop3
3074746520:error:14094410:SSL routines:SSL3_READ_BYTES:sslv3 alert handshake failure:s3_pkt.c:1195:SSL alert number 40
3074746520:error:1409E0E5:SSL routines:SSL3_WRITE_BYTES:ssl handshake failure:s3_pkt.c:591:
1 dkg@pip:~$ echo QUIT | openssl s_client -starttls pop3 -quiet -connect julia.mayfirst.org:pop3
3075704984:error:14094410:SSL routines:SSL3_READ_BYTES:sslv3 alert handshake failure:s3_pkt.c:1195:SSL alert number 40
3075704984:error:1409E0E5:SSL routines:SSL3_WRITE_BYTES:ssl handshake failure:s3_pkt.c:591:
1 dkg@pip:~$ 

(edited to include test to julia)

Last edited 7 years ago by Daniel Kahn Gillmor (previous) (diff)

comment:7 Changed 7 years ago by Jamie McClelland

That seems to be the opposite of what takethestreets reported (who reports that pop3 STARTTLS works on chavez, doesn't work on mail.mayfirst.org).

Also, at the moment, using Icedove 8.0, i can pop my mail using STARTTLS directly via Chavez, but mail.mayfirst.org is giving me a login error.

jamie

comment:8 Changed 7 years ago by Daniel Kahn Gillmor

I've got that wrong -- sorry!

It looks like my copy of s_client defaults to requiring SSLv3 (which is pretty sloppy, actually). And chavez and julia refuse to speak anything but TLSv1.

mail.mayfirst.org speaks either TLSv1 or SSLv3 (SSLv3 is older and has more outstanding documented problems than TLSv1).

0 dkg@pip:~$ echo QUIT | openssl s_client -quiet -tls1 -starttls pop3 -connect chavez.mayfirst.org:pop3
depth=1 C = US, O = "GeoTrust, Inc.", CN = RapidSSL CA
verify error:num=20:unable to get local issuer certificate
verify return:0
+OK Hello there.
+OK Better luck next time.
0 dkg@pip:~$ echo QUIT | openssl s_client -quiet -tls1 -starttls pop3 -connect julia.mayfirst.org:pop3
depth=1 C = US, O = "GeoTrust, Inc.", CN = RapidSSL CA
verify error:num=20:unable to get local issuer certificate
verify return:0
+OK Hello there.
+OK Better luck next time.
0 dkg@pip:~$ echo QUIT | openssl s_client -quiet -tls1 -starttls pop3 -connect mail.mayfirst.org:pop3
depth=1 C = US, O = "GeoTrust, Inc.", CN = RapidSSL CA
verify error:num=20:unable to get local issuer certificate
verify return:0
+OK POP3 perditon ready on gil.mayfirst.org 0002aaa1
+OK QUIT
read:errno=0
0 dkg@pip:~$ 

If there are specific clients that are failing to connect, it might be worth investigating what version(s) of TLS and SSL they support.

comment:9 Changed 7 years ago by Jamie McClelland

I'm still plugging away at this.

After reading dkg's comments and doing more debugging on starttls via POP, my theory is that perdition handles incoming client connections successfully using various version of tls/ssl, however, when connecting to the target POP server it uses a particular version that is not compatible with our MOSH servers.

With dkg's help - I took the following steps:

  • Established a connection from my local machine to gil.mayfirst.org:
openssl s_client -connect gil.mayfirst.org:pop3 -starttls pop3
  • On gil, I used ps to determine which pid was in use by the connecting process:
ps -eFH | grep perdittion.pop3:
  • Then, I ran ltrace to connect to the pid:
ltrace -p 5701
  • The following is the relevant part of the ltrace output that documents the failure:
SSL_library_init(0, 0x1c5f310, 0, 0, 0)                  = 1
SSL_load_error_strings(0x7f05d448bcf4, 0, 1, 0, 0)       = 0x7f05d474033a
SSLv23_method(6, 1, 0x7f05d494c020, 300, 0)              = 0x7f05d494b1e0
SSL_CTX_new(0x7f05d494b1e0, 1, 0x7f05d494c020, 300, 0)   = 0x1c8f350
SSL_CTX_set_session_id_context(0x1c8f350, 0x41d085, 9, 0, 1) = 1
SSL_CTX_set_default_passwd_cb(0x1c8f350, 0x41c770, 0, 0x6e6f697469647265, 1) = 1
SSL_CTX_set_default_passwd_cb_userdata(0x1c8f350, 0x7fffaca896c0, 0, 0x6e6f697469647265, 1) = 0
SSL_CTX_load_verify_locations(0x1c8f350, 0, 0x1c5f310, 0x6e6f697469647265, 1) = 1
SSL_CTX_set_verify_depth(0x1c8f350, 10, 0x1c8ad01, 0x1c5f31e, 114) = 1
SSL_CTX_set_verify(0x1c8f350, 5, 0x41c520, 0x1c5f31e, 114) = 1
SSL_new(0x1c8f350, 0x1c8f350, 1, 0x1c5f31e, 114)         = 0x1c8ae00
malloc(16)                                               = 0x01c8d3b0
SSL_set_rfd(0x1c8ae00, 4, 0x7f05d3da7e40, 0x1c8d3a0, 0x1c8d3a0) = 1
SSL_set_wfd(0x1c8ae00, 4, 0, 0x7fffaca8967c, 1)          = 1
malloc(40)                                               = 0x01c8b600
__strdup(0x1c8d450, 40, 0x7f05d3da7e40, 0x1c8b5f0, 0x7f05d3da7eb8) = 0x1c8d4b0
free(0x01c8d400)                                         = <void>
free(0x01c8d450)                                         = <void>
free(0x01c8d420)                                         = <void>
setsockopt(4, 1, 20, 0x7fffaca896a0, 16)                 = 0
setsockopt(4, 1, 20, 0x7fffaca896a0, 16)                 = 0
SSL_set_connect_state(0x1c8ae00, 1, 20, -1, 16)          = 0x7f05d4722ae0
SSL_connect(0x1c8ae00, 1, 20, -1, 16)                    = 0xffffffff
SSL_get_error(0x1c8ae00, 0xffffffff, 1, 1, 366)          = 1
SSL_load_error_strings(0, 0xffffffff, 0xffffffff, 1, 0x423860) = 0x7f05d474033a
ERR_get_error(6, 1, 0x7f05d494c020, 300, 0x423860)       = 0x140770fc
ERR_error_string(0x140770fc, 0, 0x7f05d44a2801, 1, 366)  = 0x7f05d46f5b20
_vanessa_logger_log_prefix(0x1c5ed50, 7, 0x424480, 0x4218cf, 0x7f05d46f5b20) = 97520
ERR_get_error(0x7f05d3da7e40, 0, 0x1d43000, 97360, 0)    = 0
ERR_free_strings(1, 0, 0x7f05d44a2801, 1, 366)           = 0
_vanessa_logger_log_prefix(0x1c5ed50, 7, 0x424480, 0x4218cf, 0x4234de) = 0
SSL_shutdown(0x1c8ae00, 0, -96, -1, 0)                   = 1
SSL_free(0x1c8ae00, 0, 1, -1, 0)                         = 0
free(0x01c8d3b0)                                         = <void>
free(0x01c8d4b0)                                         = <void>
free(0x01c8b600)                                         = <void>

comment:10 Changed 7 years ago by Jamie McClelland

Additionally, to help debug, I used the following line replacement in /etc/xinet.d/perdition in the service pop3 stanza which tells perdition to debug and log all connections:

#server_args                        = -i --ssl_mode tls_all,tls_all_force
server_args                         = -C -d -i --ssl_mode tls_all,tls_all_force

comment:11 Changed 7 years ago by Daniel Kahn Gillmor

Thanks for investigating this, Jamie!

I dug around in the various bits of relevant source to try to understand what is going on.

0 dkg@pip:~$ grep '[123]_method' /usr/include/openssl/ssl.h 
const SSL_METHOD *SSLv2_method(void);		/* SSLv2 */
const SSL_METHOD *SSLv3_method(void);		/* SSLv3 */
const SSL_METHOD *SSLv23_method(void);	/* SSLv3 but can rollback to v2 */
const SSL_METHOD *TLSv1_method(void);		/* TLSv1.0 */
const SSL_METHOD *DTLSv1_method(void);		/* DTLSv1.0 */
0 dkg@pip:~$ 
0 dkg@pip:~/src/perdition/perdition-1.19~rc4$ grep -r '[123]_method' .
./perdition/ssl.c:	ssl_ctx = SSL_CTX_new(SSLv23_method());
0 dkg@pip:~/src/perdition/perdition-1.19~rc4$ 

So it appears that perdition is initializing its TLS connections explicitly as SSLv3 (with a fallback to SSLv2 if OpenSSL supports it and the remote client requests it). Alas, the version of openssl in squeeze appears to still support SSLv2, which is known to be a broken cryptosystem. This setting appears to permit an MITM to cause a protocol downgrade attack. :(

Note that the code used by perdition appears to be just the version string it announces. When acting as a server, if a client connects and insists on TLSv1.0, it has no problem using it.

The right approach should be to default to TLSv1 on client-side connections, presumably by using TLSv1_method() in place of SSLv23_method() in perdition/ssl.c. However, since this is a centralized piece of code that is used by perdition for both its client and server work, it's possible that this will reject clients who try to connect to it using only SSLv3 (as chavez does when you try to connect to it directly).

Even with that risk, i think this is the right thing. We shouldn't be willing to accept fallback to SSLv2 under any circumstance (as either client or server), and all the reasonable tools i can think of have supported TLSv1 for years.

I'm preparing a new perdition package with a simple one-line fix, which i plan to roll out on gil.mayfirst.org in a little bit.

Changed 7 years ago by Daniel Kahn Gillmor

Attachment: ticket4434.debdiff added

debdiff (difference between debian source package in squeeze) changing SSLv23_method to TLSv1_method

comment:12 Changed 7 years ago by Daniel Kahn Gillmor

I've just installed the modified perdition package on gil.mayfirst.org. (the packages are in gil:/root/ticket4434 if anyone wants them).

I also disabled the extra logging for perdition on gil by swapping the lines jamie mentioned in comment:10

I observe a few changes in behavior with my updated package installed:

  • gil appears to succeed in negotiating a connection to the backend server's POP daemon using TLSv1 (that's a fix!).
  • gil now rejects connections from POP clients using SSLv3 (it's fine with clients who offer TLSv1) -- this is the same situation as the POP daemons on chavez and viewsic, for example, so i don't consider it a particularly bad regression.
  • gil now rejects connections from IMAP clients using SSLv3 (again, TLSv1 is OK). This diverges from the behavior of the MOSHes (which accept SSLv3 from direct IMAP clients), so it might be unacceptable -- do we have members who use such clients?

I find it curious that the MOSHes accept SSLv3 and TLSv1 for IMAP, but only TLSv1 for POP3.

Perhaps this is due to this explicit setting:

0 viewsic:~# grep TLS_PROTOCOL /etc/courier/*-ssl
/etc/courier/imapd-ssl:##NAME: TLS_PROTOCOL:0
/etc/courier/imapd-ssl:# TLS_PROTOCOL sets the protocol version.  The possible versions are:
/etc/courier/imapd-ssl:# TLS_PROTOCOL="TLS1_1:TLS1:SSL3"
/etc/courier/imapd-ssl:##NAME: TLS_STARTTLS_PROTOCOL:0
/etc/courier/imapd-ssl:# TLS_STARTTLS_PROTOCOL is used instead of TLS_PROTOCOL for the IMAP STARTTLS
/etc/courier/imapd-ssl:# It takes the same values for OpenSSL/GnuTLS as TLS_PROTOCOL
/etc/courier/pop3d-ssl:##NAME: TLS_PROTOCOL:0
/etc/courier/pop3d-ssl:# TLS_PROTOCOL sets the protocol version.  The possible versions are:
/etc/courier/pop3d-ssl:# TLS_PROTOCOL="TLS1_1:TLS1:SSL3"
/etc/courier/pop3d-ssl:##NAME: TLS_STARTTLS_PROTOCOL:0
/etc/courier/pop3d-ssl:# TLS_STARTTLS_PROTOCOL is used instead of TLS_PROTOCOL for the POP3 STARTTLS
/etc/courier/pop3d-ssl:# It takes the same values for OpenSSL/GnuTLS as TLS_PROTOCOL
/etc/courier/pop3d-ssl:TLS_STARTTLS_PROTOCOL=TLS1
0 viewsic:~# 

If we want to enforce TLSv1 (and reject SSLv3 and earlier) across all of MF/PL IMAP and POP3 services, we could certainly be more consistent about it. :P

I can't find any reference here to any explicit decision-making about what versions to use.


So, what do we want to do? I see a few options for the short term:

  • go with stock debian perdition; tell POP3 users that they can't use mail.mayfirst.org
  • go with stock debian perdition; adjust our pop3d-ssl configs to permit SSLv3 (and SSLv2, i think) access
  • use the simple fix from attachment:ticket4434.debdiff; require POP3 and IMAP users of mail.mayfirst.org to use TLSv1-capable POP clients
  • make the patch slightly more complex by only choosing TLSv1 for perdition's ssl client connections, leaving its listening connections potentially downgradable to SSLv3 or even SSLv2. (maybe we can decline SSLv2 at least via SSL_CTX_set_options after invoking SSL_CTX_new)

For the long term:

  • we need to report this issue upstream
  • we should probably report this issue to debian
  • we might consider making a patch that adds a new configuration option to perdition that can provide configuration options for a user to make these choices without recompiling.
  • we need to establish a policy about what versions of these protocols we want to support, and figure out how to work with members whose tools are decrepit enough to not meet our standards.

OK, i hope the above wasn't too confusing. I'd be happy to clarify if people have questions.

comment:13 Changed 7 years ago by Daniel Kahn Gillmor

Ugh. I've just reverted to the stock debian perdition after observing the logs on gil. It seems like we're getting several regular queries from netblocks belonging to Research in Motion (blackberry) and Google, and those clients don't properly negotiate connections to a TLSv1-only configuration.

I suspect these connections are members pulling their mail into other providers, which would mean that the mail would suddenly stop flowing to their blackberry or gmail accounts. I'd like to avoid that kind of freakout.

I'm now leaning toward this short-term option:

  • make the patch slightly more complex by only choosing TLSv1 for perdition's ssl client connections, leaving its listening connections potentially downgradable to SSLv3 or even SSLv2. (maybe we can decline SSLv2 at least via SSL_CTX_set_options after invoking SSL_CTX_new)

And i'm also leaning toward trying to do some sort of tightening of general server configs:

  • make all MOSHes require TLSv1 for IMAP or POP3 connections (either TLS-wrapping or STARTTLS).

but we probably need to think through the consequences of that first.

comment:14 Changed 7 years ago by Daniel Kahn Gillmor

Keywords: TLS added

comment:15 Changed 7 years ago by Jamie McClelland

Great work dkg! This is really helpful.

I have a slight variation on our proposal for moving forward.

Right now, although we have a few groups, including the Praxis Project, using it, mail.mayfirst.org is still considered experimental and is not widely publicized to our members.

That gives us some freedom to experiment and to make new requirements for it's use that are not currently enforced on our MOSH servers.

I suggest that we try to pin down exactly which users are using sslv2 or sslv3 now. Is it possible to log this information if we use verbose logging?

I started the process of researching what versions of tls are used by blackberry and gmail. I came up empty handed for Blackberry (since it's most likely the providers that control this - Verizon/Sprint etc., this could be tricky).

With gmail I hit another road block. I'm not sure I understand the use-case. The only example I could come up with is: a user with a gmail account configures gmail to fetch mail from our servers and put that mail in their gmail account. So... I logged into my test gmail account and when I looked at the options I discovered that gmail only supports this feature using POP (and when I clicked through the option it even exposed to me that it was using port 110). So, I'm not sure why someone would be using IMAP via a google IP address?

In any event... the idea would be: if we can get the (hopefully) handful of people currently requiring sslv3 on mail.mayfirst.org to fix/upgrade/re-configure their clients, then we could apply your first patch, and then advertise mail.mayfirst.org broadly to MFPL members, along with directions on how to fix your clients if you get a failure.

This way we can use mail.mayfirst.org to transition people to the new settings *and* to fix their mail clients at the same time in the least disruptive way.

How does that sound?

jamie

comment:16 Changed 7 years ago by Jamie McClelland

After discussion on irc, dkg and I agreed that we should make the following changes on our mosh servers:

set POP3_TLS_REQUIRED to 1 and unset TLS_STARTTLS_PROTOCOL=TLS1

That will allow clients that can't speak or don't offer tlsv1 compatible hand shakes to use our servers.

Additional coding changes to perdition and imap would be to examine the SSL_CTX_set_options to say NO_SSLv2 and NO_SSLv3 and see if any clients break with that or at the very least, when perdition acts as a client, it should have NO_SSLv2 and NO_SSLv3 set.

jamie

comment:17 Changed 7 years ago by Jamie McClelland

This change has been made in puppet and pushed to chavez for testing today (and mail.mayfirst.org works on icedove!).

I'm checking in with Philadelphia Fight about the timing for the change on didier (since they have a lot of groups that have been using the same pop clients for years - I want to be sure we're ready in case in breaks anything).

jamie

comment:18 Changed 7 years ago by Jamie McClelland

Resolution: fixed
Status: newclosed

This change is now active on all servers.

Please login to add comments to this ticket.

Note: See TracTickets for help on using tickets.