Opened 5 years ago

Closed 2 years ago

#7550 closed Task/To do item (fixed)

enable db cache of messages on roundcube for better performance

Reported by: https://id.mayfirst.org/jamie Owned by: https://id.mayfirst.org/srevilak
Priority: Urgent Component: Tech
Keywords: roundcube Cc: support-team@…
Sensitive: no

Description

Hilary has commented on experiencing frustration with error and slowness on roundcube (#7091).

roundcube has the following (not enabled) configuration options:

// Type of IMAP indexes cache. Supported values: 'db', 'apc' and 'memcache'.
$rcmail_config['imap_cache'] = null;

// Enables messages cache. Only 'db' cache is supported.
$rcmail_config['messages_cache'] = false;
// lifetime of message cache
// possible units: s, m, h, d, w
$rcmail_config['message_cache_lifetime'] = '10d';

If we enabled these options, my understanding is that users of roundcube would have their email cached on stallman, which would allow for less traffic to the IMAP server. I think this would dramatically increase performance for roundcube users, particularly ones who use MOSH's that have periodic spikes in disk I/O or other bottlenecks.

The cost of this change would be that a lot of user mail would be stored in one central location. I don't think we should take this decision lightly, however, I think it's worth it and here's why:

  • If we set the cache lifetime to 1d, then it will at most force users to experience a slow initial login, followed by a full day of speedy access, while ensuring that messages are only kept for a short period.
  • A speedy user interface for webmail is crucial for people to use it. And based on my read of this ticket, it doesn't seem like we're getting that experience.

Here is another (less controversial and probably less impactful) changes:

// Backend to use for session storage. Can either be 'db' (default) or 'memcache'
// If set to memcache, a list of servers need to be specified in 'memcache_hosts'
// Make sure the Memcache extension (http://pecl.php.net/package/memcache) version >= 2.0.0 is installed
$rcmail_config['session_storage'] = 'db';

// Use these hosts for accessing memcached
// Define any number of hosts in the form of hostname:port or unix:///path/to/socket.file
$rcmail_config['memcache_hosts'] = null; // e.g. array( 'localhost:11211', '192.168.1.12:11211', 'unix:///var/tmp/memcached.sock' );

If we install memcached on stallman and switched from db to memcached we will probably see a performance improvement as well.

jamie

Change History (21)

comment:1 Changed 5 years ago by https://id.mayfirst.org/jamie

  • Owner set to https://id.mayfirst.org/jamie
  • Status changed from new to assigned

comment:2 follow-up: Changed 5 years ago by https://id.mayfirst.org/dkg

What makes you think the memcache option will be an improvement over the db option for session storage? is it just that the sessions won't need to be written to disk? Are disk writes that much of an issue on stallman? If so, wouldn't the db disk writes involved in creating and maintaining the imap cache be significantly burdensome?

Another approach to reduce disk writes would be to make the psql session table be UNLOGGED (assuming we could upgrade psql to 9.1, which is in wheezy). If we do ultimately decide to go the imap_cache route (which i'm not happy about) then maybe we could make the imap cache table UNLOGGED also?

Has any profiling been done to pin down the cause of the slowness? I see srevilak did a bunch of work on #7091 to try to replicate the issue but hasn't succeeded in reproducing it. I'm leery of making major changes (particularly those which leak member data to disk in new locations) without knowing that we know where the problems are.

If the problems are on the backend IMAP servers themselves, maybe we could start by trying to introduce some optimizations there?

comment:3 Changed 5 years ago by https://id.mayfirst.org/srevilak

(comment moved to #7091)

Last edited 5 years ago by https://id.mayfirst.org/srevilak (previous) (diff)

comment:4 in reply to: ↑ 2 Changed 5 years ago by https://id.mayfirst.org/jamie

Replying to https://id.mayfirst.org/dkg:

What makes you think the memcache option will be an improvement over the db option for session storage? is it just that the sessions won't need to be written to disk? Are disk writes that much of an issue on stallman? If so, wouldn't the db disk writes involved in creating and maintaining the imap cache be significantly burdensome?

I'm not sure whether using memcache instead of the db will improve things signficantly. As far as I know disk contention is not a particular issue on stallman. However, I also know that reading from and writing info to memory rather than disk is always faster, so if we want to speed things up, even just small amount, using non-write-based storage should help with that.

Another approach to reduce disk writes would be to make the psql session table be [http://www.postgresql.org/docs/9.1/static/sql-createtable.html UNLOGGED] (assuming we could upgrade psql to 9.1, which is in wheezy). If we do ultimately decide to go the imap_cache route (which i'm not happy about) then maybe we could make the imap cache table UNLOGGED also?

That seems like an improvement - but it would require some tweaking of roundcube (at least once) to drop and re-create the table (assuming roundcube doesn't create the table as unlogged). However, it still seems to be written to (and read from) disk, so I imagine memcache may be faster.

Has any profiling been done to pin down the cause of the slowness? I see srevilak did a bunch of work on #7091 to try to replicate the issue but hasn't succeeded in reproducing it. I'm leery of making major changes (particularly those which leak member data to disk in new locations) without knowing that we know where the problems are.

srevilak's latest post seems to suggest that the slowness happens when the messages are pulled out of the imap server. Although there are some suggestions on how roundcube could do this more efficiently, it seems like caching messages would in fact resolve that issue.

If the problems are on the backend IMAP servers themselves, maybe we could start by trying to introduce some optimizations there?

I welcome any and all IMAP optimizations! However, I have no new ideas. :(.

comment:5 Changed 5 years ago by https://id.mayfirst.org/jamie

I've just installed the following packages on stallman:

memcached php5-memcache

And I've modified the message cache settings to be:

// Type of IMAP indexes cache. Supported values: 'db', 'apc' and 'memcache'.
$rcmail_config['imap_cache'] = 'memcache';

// Enables messages cache. Only 'db' cache is supported.
$rcmail_config['messages_cache'] = 'db';
// lifetime of message cache
// possible units: s, m, h, d, w
$rcmail_config['message_cache_lifetime'] = '1d';

Still open to feedback.

I have not made any changes to the session storage.

jamie

comment:6 Changed 5 years ago by https://id.mayfirst.org/dkg

  • Summary changed from tweak roundcube for better performance to enable db cache of messages on roundcube for better performance

How does roundcube authenticate to memcache? how is that protected (or not) from the other processes running on stallman? Have you made these changes to roundcube-dev as well as to roundcube? Do we have any profiling metrics that we can use to tell whether this is improving the problems for our users?

i'm changing the subject line because this is pretty far from a "tweak" -- it's a major change in where copies of e-mail are stored on disk :/

comment:7 follow-up: Changed 5 years ago by https://id.mayfirst.org/jamie

There is no authentication to memcache (any local user can access it or even modify it). However, only the IMAP indexes are using memcache, the message cache is stored in the database.

I've just enabled the same options on dev.

I'm open to suggestions on profiling metrics, but it's very difficult and seems to be related both to number of messages on the IMAP server and the particular bottle necks being experienced on the IMAP server at the time of the test. I'm relying heavily on Hilary who has been trying to regularly using Roundcube but has reported it to be too slow at times to be use-able. Hilary is an LC member and staff member and an extremely patient person. If she is reporting problems, I think we can expect them to be fairly widespread. I'd hate to see all the work done to bring roundcube to MF/PL be wasted if people give up on it when it's too slow.

jamie

comment:8 Changed 5 years ago by https://id.mayfirst.org/dkg

In general, when making changes to roundcube, we should make them on roundcube-dev, test them, and then roll them out to roundcube. thanks for keeping them in sync :)

Are you sure that the information cached is IMAP indexes, and not IMAP authentication credential information?

comment:9 in reply to: ↑ 7 Changed 5 years ago by https://id.mayfirst.org/dkg

Replying to https://id.mayfirst.org/jamie:

Hilary is an LC member and staff member and an extremely patient person. If she is reporting problems, I think we can expect them to be fairly widespread. I'd hate to see all the work done to bring roundcube to MF/PL be wasted if people give up on it when it's too slow.

This sentence seems to suggest that i'm pushing back against improving performance of roundcube, or that i'm not believing Hilary's problem statements. neither of these things are the case! I clearly want roundcube to be a functional and effective webmail client for people who want/need to use webmail. And i respect and trust Hilary's assessments.

What i am pushing back on is writing extra copies of people's e-mail to disk in places that we don't otherwise expect to have e-mail written to disk, and on making system-wide changes with data security consequences without being clear on whether they fix things.

I remain frustrated that we haven't been able to replicate the problem reliably in a mailbox that we can experiment more readily with :( (i don't want to treat hilary's personal mailbox as a guinea pig)

comment:10 follow-up: Changed 5 years ago by https://id.mayfirst.org/dkg

It occurs to me that if both roundcube and roundcube.dev use memcache on the same host, whatever they're using memcache for might collide with each other. Is this a risk?

Out of curiosity to see what was getting stored in memcache, i used memcdump from libmemcached-tools to see what's there: i got no output, though.

I also tried straceing the running memcached process to see what was talking to it. I watched the process while i logged into https://roundcube.dev.mayfirst.org/ and saw no communications at all.

digging a bit in /srv/roundcube-dev/config/main.inc.php, i note that memcache_hosts is set to null. Maybe that needs to be set to use the cache? session_storage can also be set to use memcache though i'm not sure that's a great idea given the unauthenticated nature of memcache.

It occurs to me also that we might be able to run separate memcache instances on unix-domain sockets, protect them with filesystem permissions, and direct the different roundcube instances toward custom instances.

comment:11 in reply to: ↑ 10 Changed 5 years ago by https://id.mayfirst.org/jamie

Replying to https://id.mayfirst.org/dkg:

In general, when making changes to roundcube, we should make them on roundcube-dev, test them, and then roll them out to roundcube. thanks for keeping them in sync :)

Yes! You are right. I jumped the gun on this one.

Are you sure that the information cached is IMAP indexes, and not IMAP authentication credential information?

I can't be certain... just going with the name of the configuration variable.

Replying to https://id.mayfirst.org/dkg:

Replying to https://id.mayfirst.org/jamie:

Hilary is an LC member and staff member and an extremely patient person. If she is reporting problems, I think we can expect them to be fairly widespread. I'd hate to see all the work done to bring roundcube to MF/PL be wasted if people give up on it when it's too slow.

This sentence seems to suggest that i'm pushing back against improving performance of roundcube, or that i'm not believing Hilary's problem statements. neither of these things are the case! I clearly want roundcube to be a functional and effective webmail client for people who want/need to use webmail. And i respect and trust Hilary's assessments.

Woops...not meant that way at all. I made those statements in response to the metrics question. I don't have technical/numerical metrics, but I do have in-person, reliable, user experience metrics. I think numeric metrics are generally better for evaluating if a change had an impact because they can be measured more accurately, but in this case I'm hoping Hilary can give us useful feedback.

What i am pushing back on is writing extra copies of people's e-mail to disk in places that we don't otherwise expect to have e-mail written to disk, and on making system-wide changes with data security consequences without being clear on whether they fix things.

Yes - I understand - and am glad you are pushing back. I agree - we shouldn't be taking this risk if it doesn't make an impact, and if there is any way to lesson the risk I'd like to do it. And, if others think we shouldn't be doing this at all - please speak up!

Replying to https://id.mayfirst.org/dkg:

It occurs to me that if both roundcube and roundcube.dev use memcache on the same host, whatever they're using memcache for might collide with each other. Is this a risk?

Hm. Yeah, I think it is. Usually, php applications allow you to set a prefix for each application to avoid such collisions. I've just changed the imap_cache setting to db (first testing on dev, then moving to live).

Out of curiosity to see what was getting stored in memcache, i used memcdump from libmemcached-tools to see what's there: i got no output, though.

I also tried straceing the running memcached process to see what was talking to it. I watched the process while i logged into https://roundcube.dev.mayfirst.org/ and saw no communications at all.

digging a bit in /srv/roundcube-dev/config/main.inc.php, i note that memcache_hosts is set to null. Maybe that needs to be set to use the cache? session_storage can also be set to use memcache though i'm not sure that's a great idea given the unauthenticated nature of memcache.

doh. I think that normally is set to localhost. And yeah, I agree about sessions doing better in the database.

It occurs to me also that we might be able to run separate memcache instances on unix-domain sockets, protect them with filesystem permissions, and direct the different roundcube instances toward custom instances.

That seems reasonable. Also of note - the session table in the roundcube database does not seem to be pruned. Seems like we should be purging that table, perhaps weekly?

jamie

comment:12 Changed 5 years ago by https://id.mayfirst.org/dkg

Sounds to me like the prudent approach is:

  • revert these changes on https://roundcube.mayfirst.org/ for now
  • set up a memcached instance using unix-domain sockets that is only accessible to the roundcube-dev@stallman user -- this should be a pretty easy servicedir to write.
  • put as much as possible into memcache for roundcube-dev, pointing it at the unix-domain socket
  • ask hilary to try https://roundcube.dev.mayfirst.org/ to see if it improves things for her.

If we can get a sense of improvement from that, then roll out these changes to https://roundcube.mayfirst.org/

Then,

  • repeat the process with enabling a message cache.

comment:13 Changed 5 years ago by https://id.mayfirst.org/dkg

we should also probably disable the system-wide memcached instance that is listening on the loopback.

comment:14 Changed 5 years ago by https://id.mayfirst.org/jamie

Ok. Changes reverted and memcache removed. And I've asked hilary to test on the dev instance.

comment:15 Changed 5 years ago by https://id.mayfirst.org/dskallman

Is there an update on this?

Also referencing #6855, #6131 & #5270?

comment:16 Changed 5 years ago by https://id.mayfirst.org/srevilak

During the August f2f, we did some experimentation with roundcube and large mailboxes (as part of #7091), and came to the conclusion that imapd was more likely the culprit than Roundcube.

gdl and I searched the fine web for courier performance tuning suggestions. The most common suggestion we found was "switch to dovecot". Jamie and I discussed the prospect of setting up an vm to test dovecot, but we (more specifically, I) haven't made any progress on that front.

Perhaps this could be a project candidate for this weekends f2f.

comment:17 Changed 5 years ago by https://id.mayfirst.org/jamie

  • Owner changed from https://id.mayfirst.org/jamie to https://id.mayfirst.org/srevilak

That sounds like a good idea - If you don't mind I'm going to re-assign to you. I hope we can get that virtual guest for testing up on saturday.

comment:18 Changed 4 years ago by https://id.mayfirst.org/dskallman

Is there an update?

comment:19 Changed 4 years ago by https://id.mayfirst.org/srevilak

Sorry, no update at the moment.

comment:20 Changed 2 years ago by https://id.mayfirst.org/srevilak

  • Resolution set to fixed
  • Status changed from assigned to feedback

We found out that most of the roundcube performance issues were actually courier performance issues. We addressed this by switching from courier -> dovecot as an IMAP Server (see ticket:8368).

The current roundcube config imap and message caching disabled.

config/defaults.inc.php:$config['imap_cache'] = null;
config/defaults.inc.php:$config['messages_cache'] = false;

We are storing session information in the database

config/defaults.inc.php:$config['session_storage'] = 'db';

The sessions table can get rather large over time. I've discovered that periodically running roundcube/bin/gc.sh helps keep the session table at a reasonable size. We should probably run gc.sh via cron, and that deserves a separate ticket.

I like the idea of not caching message content. This avoids the problem of (say) person A seeing cached copies of person B's messages, because B happened to recycle person A's username.

I'm not sure we have work left to do here, so I'm moving this ticket into feedback state.

comment:21 Changed 2 years ago by automatic

  • Status changed from feedback to closed

No news is good news (we hope)! Given the lack of feedback, we think this ticket can be closed.

Please login to add comments to this ticket.

Note: See TracTickets for help on using tickets.