Opened 2 years ago

Last modified 6 months ago

#11705 assigned Task/To do item

case sensitivity for mayfirst / people link logins across all services is inconsistent

Reported by: https://id.mayfirst.org/jaimev Owned by: https://id.mayfirst.org/jamie
Priority: Medium Component: Tech
Keywords: dovecot, case sensitivity, roundcube Cc: https://id.mayfirst.org/jamie, https://id.mayfirst.org/srevilak
Sensitive: no

Description (last modified by https://id.mayfirst.org/jamie)

Some services require all lower case usernames, some will let you in with all combinations of cases, and some will act different depending on whether you use different cased usernames.

We should have a consistent policy across all services.

Change History (16)

comment:1 Changed 2 years ago by https://id.mayfirst.org/jaimev

  • Cc https://id.mayfirst.org/srevilak added
  • Keywords roundcube added
  • Owner set to https://id.mayfirst.org/jaimev
  • Status changed from new to assigned

I think this issue has come up again and I've finally had a chance to do a little research myself.

It seems that on the dovecot side of things there are ways to condition dovecot to convert all usernames to lowercase for authentication like '%Lu' . http://wiki.dovecot.org/Variables However grepping through our configs I can't find anywhere we are explicity doing that although we are clearly getting that effect.

In Roundcube it also seems possible to make sure the username is converted to lowercase when logging in.

// Forces conversion of logins to lower case.
// 0 - disabled, 1 - only domain part, 2 - domain and local part.
// If users authentication is case-insensitive this must be enabled.
// Note: After enabling it all user records need to be updated, e.g. with query:
//       UPDATE users SET username = LOWER(username);
$config['login_lc'] = false;

I don't have the full picture for the login process for either of these yet though. Can either of you help fill me in?

comment:2 Changed 2 years ago by https://id.mayfirst.org/jamie

Hi Jaime and Steve,

Thanks for circling back to this Jaime.

For summary: roundcube uses one method to login a user and access their email (Dovecot/IMAP) and a different method to relay email (Postfix/SASL). Both are done via mail.mayfirst.org using the same username and password. The problem is that the first login is case insensitive while the second login is case-sensitive. So a user can login successfully with a mixed-cap username and then get an error when they try to send email

A few comments:

  • The control panel auto-lower cases all user account login names when they are created (and updated). So logins in MF/PL are NOT case sensitive. This gives us some flexibility because we can either choose to enforce case sensitivity (enforce all lowercase) or we could choose to not enforce case-sensitivity consistently throughout the process.
  • How do our email clients work now?
    • Roundcube currently has: $config['login_lc'] = false; - so no case changing is happening.
    • I don't know if Horde does any conversion or not BUT - horde does not relay email via mail.mayfirst.org. Horde sends directly, so the problem affecting roundcube does not affect horde. With Horde you can currently login with mixed cap username and both send and receive email without any problems.
    • I don't know about other email clients - Thunderbird, etc. I would expect that most clients do not convert usernames to lower case.
  • How does our dovecot/IMAP login work? User names are not case-sensitive because:
    • On gil/paulo, the imap proxy looks up their username to figure out which IMAP server their account lives on and this check is made via a mysql database and by default, mysql searches are not case-sensitive. According to a old but still accurate post we could change that by editing the SQL statement used in the lookup (currently stored on both gil and paulo in /etc/dovecot/mfpl-sql.conf).
    • On the mosh... The username (I think) should be case sensitive, however, on mosh's we have:
      /etc/dovecot/conf.d./99-mfpl.conf:auth_username_format = %Ln
      
      %Ln will lowercase the login name (%Lu is for the full user including email address). So, the mosh's are lowercasing the username.
    • So, we could enforce case sensitivity by simply modifying the auth_username_format parameter on the MOSH's and remove the L.
  • When relaying email via SASL, case-sensitivity is applied. I've searched high and low and can't figure out a way to get postfix to convert a username to lower case before passing it on to SASL for authentication or how to tell SASL to not be case sensitive.

So, I'm starting to think that enforcing case-sensitivity via Dovecot is not our best option because:

  • It could cause Horde logins to break (since Horde is currently not case sensitive
  • It could cause a lot of desktop/phone clients to break because many of these apps allow users to configure the username for incoming email differently than the one for outgoing email. Some people may not use us for outgoing email, just incoming email.

On the other hand, this particular problem seems to only affect roundcube because roundcube re-uses your username/password to relay email behind the scenes which makes the failure to send email really confusing. And, I think we could fix this problem by auto-lowercases all roundcube logins. (An ideal fix, of course, would be to auto-lowercase the username in SASL so that relaying email will always work regardless of caps.)

What do you think? Geez... this got really long.

comment:3 Changed 2 years ago by https://id.mayfirst.org/jaimev

For some reason I think it is bad to give people the false idea that case doesn't matter with something like a user login. So while I am in favor of strict case being enforced across our systems it does seem that will be much more work than just making roundcube auto lowercase the whatever users enter into the login field.

comment:4 Changed 2 years ago by https://id.mayfirst.org/jamie

Here's a one line to run on each mosh to see if anyone has used upper case for login to IMAP in the last week:

zgrep -E "dovecot: (pop3|imap)-login: Login: user=" /var/log/mail.log* | egrep -o "<.*>," | grep '[[:upper:]]'

We could run a slightly different version on gil/paulo - but there still may be some users logging in directly to the MOSH (and it's on the MOSH that we would make this change). So, I think running it on each MOSH would be better.

If nobody has used an upper case login in the last week, then I would be less concerned with breakage by going to strict case. My main concern is making a change to strict case and then having a lot of people who have been successfully logging in for years suddenly can't login and have no idea why.

Do you want to give that a try on all moshes?

jamie

comment:5 Changed 2 years ago by https://id.mayfirst.org/jaimev

I'm not sure that one liner will work. I am logging in as imap user Example from an imap client and through roundcube and dovecot logs 'example' in gil/paulo/floriberto mail.log. I also tried directing my imap client directly to floriberto and see the same.

comment:6 Changed 2 years ago by https://id.mayfirst.org/jamie

Oh damn. Thanks for testing. I just realized I've been logging in via offlineimap as jmcclellanD since we started working on this and I don't see that anywhere in the logs either. I wonder how we could test without blocking people?

Also, I took a closer look at the dovecot proxy query on gil/paulo, which is:

password_query = SELECT NULL AS password, host, destuser, 'Y' as nopassword, 'Y' AS proxy, 'Y' AS starttls FROM proxy WHERE username = '%u'

In other words, regardless of what username you pass, it will always use the username on file when it connects to the backend.

So, in order to enforce strict case sensitivity we would have to change this to be:

password_query = SELECT NULL AS password, host, '%u' AS destuser, 'Y' as nopassword, 'Y' AS proxy, 'Y' AS starttls FROM proxy WHERE username = '%u'

So we pass back to the backend the same string we are given.

comment:7 Changed 2 years ago by https://id.mayfirst.org/srevilak

Wow, this is an interesting one.

Most "traditional" mail clients (Thunderbird, Mutt) ask you for separate IMAP and SMTP credentials. If you mess one of them up, the problem will be more obvious. Roundcube assumes the same username for both SMTP and IMAP. In general, that seems like a pretty reasonable thing to do, except for corner cases like ticket:11699.

'%u' AS destuser seems like a reasonable way to preserve case, assuming that "%u" is properly escaped. Since "%u" hast to be used in the WHERE clause, I'm hoping that's a safe assumption.

There could be a way to circumvent the need for an SMTP login: have roundcube inject mail to localhost:25 and have postfix relay to mail.mayfirst.org. mail.mayfirst.org would have to accept messages from stallman's IP, and stallman's postfix would only be able to listen on localhost.

That's basically taking the position "if you were able to log in to roundcube, then you've demonstrated who you are, and we won't ask you to prove it again when sending mail".

comment:8 Changed 2 years ago by https://id.mayfirst.org/jamie

The problem that we have is that case-sensitivity of usernames is being inconsistently applied specifically during email authentication (and perhaps in other places too).

And, we want a consistent policy: either we enforce case sensitivity or we don't enforce it.

I think the main argument in favor of enforcing case sensitivity is: we could run into problems down the line if we ever implement a system that must enforce case sensitivity. Then we could have people suddenly get locked out because typing in their mixed case username has always worked and now it suddenly doesn't. (I'm pretty sure we will never re-design our system so that the username joe is intentionally a different username than the username Joe.)

The main argument in favor of not enforcing case sensitivity is that it avoids locking out people now in order to avoid a theoretical problem in the future.

Other arguments that could sway the decision include how hard it would be to change the policy. Here's an incomplete run-down:

  • IMAP: case insensitive, would require config changes on both mail.mayfirst.org and all MOSH'es to change to case sensitive
  • SASL (for relaying email): case sensitive, not sure how to change
  • Roundcube: login is case insensitive, but the same credentials are used to relay email which is case sensitive. Also, after login, your preferences are case sensitive. So the user joe and the user Joe will see the same email, but will have different preferences.
  • Horde: case insensitive, not sure how to change
  • id.mayfirst.org (used by support.mayfirst.org): case insensitive, not sure how to change
  • login service (used by share.mayfirst.org and XMPP server): case insensitive, not sure how to change
  • share.mayfirst.org: case sensitive (if you use mixed case username, your login will succeed, but ownCloud will treat you like a new user. This is bad).
  • ssh: case sensitive, probably a bad idea to change
  • The control panel is case insensitive
Last edited 2 years ago by https://id.mayfirst.org/jamie (previous) (diff)

comment:9 Changed 2 years ago by https://id.mayfirst.org/jamie

  • Owner changed from https://id.mayfirst.org/jaimev to https://id.mayfirst.org/jamie

I will take the next step of trying to research who is using mixed case logins

comment:10 Changed 17 months ago by https://id.mayfirst.org/jaimev

This has come up a few more times since we created this ticket. #12220 , #12460 , #12464 , #12607 , #12614 , #12662

I wouldn't call it urgent but it does produce a lot of frustration for the few members who go through this.

comment:11 Changed 12 months ago by https://id.mayfirst.org/jamie

  • Description modified (diff)
  • Summary changed from case sensitivity for dovecot logins to case sensitivity for mayfirst / people link logins across all services is inconsistent

comment:12 Changed 12 months ago by https://id.mayfirst.org/jamie

See also #13163.

comment:13 Changed 12 months ago by https://id.mayfirst.org/jamie

For nextcloud, we are in a better position to do something since we have control over the authentication (via our own module).

One solution would be to simply reject all login attempts that have a capital in the username with a message explaining that you should only use lower case letters in the username.

Before implementing this policy, we could change all existing usernames with mixed case to lower case.

That could be done via the database with something along the lines of:

UPDATE oc_users_external SET uid = LOWER(uid);

But... we would have to carefully examine all tables in the database to find all the ones that reference the uid field.

Unfortunately, we also have to change the names of folders in the /var/lib/nextcloud/data directory - each folder is named after a username.

It seems we have 33 users with mixed case names:

0 lucius:/var/lib/nextcloud/data# ls | egrep '[A-Z]' | wc -l
33
0 lucius:/var/lib/nextcloud/data#

Not a huge problem to simply rename them to the lower case version unless....

Both exist. Wah.

0 lucius:/var/lib/nextcloud/data# for dir in $(ls | egrep '[A-Z]'); do lc=$(echo $dir | tr '[:upper:]' '[:lower:]') && [ -d "$lc" ] && echo "$lc"; done | wc -l
17
0 lucius:/var/lib/nextcloud/data# 

Also, here is an unresolved owncloud issue: https://github.com/owncloud/core/issues/10102

comment:14 Changed 6 months ago by https://id.mayfirst.org/jaimev

The number of instances of both of these numbers has grown.

0 lucius:/var/lib/nextcloud/data# ls | egrep '[A-Z]' | wc -l
58
0 lucius:/var/lib/nextcloud/data# for dir in $(ls | egrep '[A-Z]'); do lc=$(echo $dir | tr '[:upper:]' '[:lower:]') && [ -d "$lc" ] && echo "$lc"; done  | wc -l
44

One member discovered this behavior and e-mailed us about it and from the looks of it has proven that camel case combinations of the username , Username, userName, UsErname will all create new folders.

If you're just reading from here this bug this means that username authentication is always lowercased such that any lowercase version of any of the above variations of "username" will correctly validate as that username given the correct password. The problem is nextcloud will create a separate folder for each varation. In terms of security only the user with the correct password should be able to authenticate and create these variations, so the main issue is confusion and frustration for fear of data loss if someone does it by mistake and enters into a new folder but no other user should have access to different folders based on the variations of username, unless of course explicitly shared with through the interface itself.

Last edited 6 months ago by https://id.mayfirst.org/jaimev (previous) (diff)

comment:15 Changed 6 months ago by https://id.mayfirst.org/jaimev

I've created users example Example and eXample to run tests with.

A simple rsync between usernames like 0 lucius:/var/lib/nextcloud/data# rsync -av Example/files example does not update the database and allow new files to appear in account example. This post suggest using occ cli to rescan the files folder https://help.nextcloud.com/t/copy-data-via-ssh-into-nextcloud-folders/10687

php occ files:scan -v --path "example/files"

The following allows the new files to be recognized.

Looking at the manual: https://docs.nextcloud.com/server/12/admin_manual/configuration_server/occ_command.html#file-operations-label there is a files:transfer-ownership command for the occ cli that can transfer all files and shares from one user to another.

www-data@lucius:~/nextcloud$ php occ -v files:transfer-ownership eXample example

The above command creates a new folder within user example/files folder named "transferred from eXample on 2018-03-24 02-53-46" also through the interface. This is less clean but safer than the above at is it avoid possible filename collisions and overwrites that rsync might execute if one file was newer than an another depending on rsync arguments.

This ticket suggests the version history of a file may not be copied when using the above method https://github.com/owncloud/core/issues/29540

Unfortunately there are no occ commands to directly copy calendars and contacts between users this way. Calendars are stored as tables in the database. It might be possible to copy/syncronize tables between users ouch.

Another option might be to export to export the calendars of the one user and try to import them into calendars of another. Someone has created a script to export specific calendars on the server https://github.com/BernieO/calcardbackup

There are a few caldav clients , one of these might allow us to sync calendars or import exported calendars. https://linux.die.net/man/1/cadaver https://github.com/pimutils/khal

comment:16 Changed 6 months ago by https://id.mayfirst.org/jaimev

Before we get too far into this we want to get a sense of how many of the users who have managed to created multiple accounts using mixed case are actually using calendars and contacts. The query below will return a count of any future events in all of the calendars created by a specific user.

nextcloud=> SELECT COUNT(*) FROM oc_calendars c JOIN oc_calendarobjects co ON c.id = co.calendarid WHERE principaluri LIKE '%/username' AND firstoccurence > extract(epoch from now());

This will check the previous 30 days.

nextcloud=> SELECT COUNT(*) FROM oc_calendars c JOIN oc_calendarobjects co ON c.id = co.calendarid WHERE principaluri LIKE '%/username' AND firstoccurence > extract(epoch from now()) - (86400 *
 30);

And this query will return a count of all of their contacts.

select * from oc_addressbooks where principaluri like '%/username'

Please login to add comments to this ticket.

Note: See TracTickets for help on using tickets.