Opened 5 years ago

Closed 4 years ago

#7888 closed Task/To do item (fixed)

Evaluate dovecot as possible replacement for courier

Reported by: https://id.mayfirst.org/srevilak Owned by: https://id.mayfirst.org/srevilak
Priority: Medium Component: Tech
Keywords: dovecot roundcube imap Cc: https://id.mayfirst.org/jamie
Sensitive: no

Description

Several tickets fit the theme of "poor imap server performance". For example, tickets #7550 and #7091?

We'd like to evaluate dovecot as an imap server, and as a possible replacement for courier. This ticket is for conducting the dovecot evaluation.

Change History (10)

comment:1 Changed 5 years ago by https://id.mayfirst.org/srevilak

  • Owner set to https://id.mayfirst.org/srevilak
  • Status changed from new to assigned

comment:2 Changed 5 years ago by https://id.mayfirst.org/srevilak

Setup new server, herman.mayfirst.org for dovecot testing. See #7878

comment:3 Changed 5 years ago by https://id.mayfirst.org/srevilak

herman begins life with the m_minimal server configuration.

  • created (via hosting order) an MX record for dovecot.srevilak.net => herman.mayfirst.org
  • installed postfix. Configured postfix to accept mail for @dovecot.srevilak.net
  • created (via adduser) a local account on herman.mayfirst.org. During tests, we'll send this user mail, and check mail via IMAP.
  • installed dovecot-imapd, dovecot-pop3d. Put /etc/dovecot under local revision control (git). (I'd like to have local revision control to keep track of what's changed in the dovecot configuration)
  • Have very basic dovecot (imapd) configuration on herman. I can read mail with an imap client. That said, haven't done any tuning or performance measurement (yet).
  • Configured postfix to use dovecot's LDA, like so
    # /etc/postfix/main.cf
    mailbox_command = /usr/lib/dovecot/dovecot-lda -f "$SENDER" -a "$RECIPIENT"
    
    upon delivery, dovecot-lda performs some level of message indexing. Index files live in each user's ~/Maildir. Verified that dovecot-lda modifies index files as new messages are received. Need to learn more about what exactly is being indexed.

To dos:

  • look at dovecots use of key/value databases.
  • Documentation indicates that dovecot supports full text indexing. Should look into this too.

comment:4 Changed 5 years ago by https://id.mayfirst.org/srevilak

Notes on today's dovecot work.

Dovecot LDA doesn't deliver to root

Oct 10 06:34:39 herman dovecot: lda(root): Error: chdir(/root/) failed: \
  Permission denied (euid=65534(nobody) egid=65534(nogroup) \
  missing +x perm: /root, dir owned by 0:0 mode=0700)

According to http://www.dovecot.org/list/dovecot/2012-November/069481.html, postfix won't invoke an LDA as root. They suggest aliasing root to a `mortal user'. Root seems to be aliased on moshes, so I don't think it will be an issue there.

Dovecot indexes

Here's a description of dovecot index files: http://wiki2.dovecot.org/IndexFiles. With indexes, you have a fair bit of message metadata stored in a pair of sequential files.

Here's a sample index record (via doveadm dump)

RECORD: seq=11, uid=393, flags=0x00 
 - ext 3 cache     :       9992 (08270000)
 - cache offset=9992 size=668, prev_offset = 0
    - hdr.From: 12: From: Steve R <steve@example.org>
    - hdr.Message-ID: 15: Message-ID: <20131010212516.GC8112@example.org>
    - hdr.Subject: 14: Subject: Mutt says hi
    - hdr.CC: 
    - hdr.CONTENT-DESCRIPTION: 
    - hdr.CONTENT-TYPE: 17: Content-Type: multipart/signed; micalg=pgp-sha256;
        protocol="application/pgp-signature"; boundary="IpbVkmxF4tDyP/Kb"
    - hdr.DATE: 11: Date: Thu, 10 Oct 2013 17:25:16 -0400
    - hdr.IN-REPLY-TO: 
    - hdr.LINES: 
    - hdr.LIST-POST: 
    - hdr.REFERENCES: 
    - hdr.REPLY-TO: 
    - hdr.TO: 13: To: testdovecot@dovecot.example.org
    - hdr.X-LABEL: 
    - flags: (08000000)
    - mime.parts: (4100000036040000000000004c040000000000004d040000000000006c0400000000000002000000480000004a0400000000000057000000000000005a000000000000000e0000000000000010000000000000000200000040000000c3040000000000006600000000000000690000000000000044030000000000005503000000000000) (pos=0 hdr.p=1078 hdr.v=1100 body.p=1101 body.v=1132 flags=41 (pos=1098 hdr.p=87 hdr.v=90 body.p=14 body.v=16 flags=48)(pos=1219 hdr.p=102 hdr.v=105 body.p=836 body.v=853 flags=40))
    - size.virtual: 2232 (b808000000000000)
    - size.physical: 2179 (8308000000000000)
    - date.received: 1381440319 (3f1b5752)

dovecot-mysql

Dovecot's mysql (and postgres) interface seems to involve using a database to store user records. I don't think we'll need this.

testing

http://www.dovecot.org/list/dovecot/2006-February/011635.html mentions an IMAP stress testing tool. The link is dead, but maybe I can get a copy of imaptest.c from dovecot's source code. We'll see.

I've also written a simple message generator, which generates message by clipping sections out of a .txt file. These are more realistic-looking than generating messages with openssl rand.

TODO

  • try out imaptest.c
  • get a courier installation running on herman, for A/B testing

comment:5 Changed 5 years ago by https://id.mayfirst.org/srevilak

(removing comment. Wrong ticket)

Last edited 5 years ago by https://id.mayfirst.org/srevilak (previous) (diff)

comment:6 Changed 5 years ago by https://id.mayfirst.org/srevilak

Now, to work on a courier installation for herman. I'll attempt to use mfpl/puppet's modules/mayfirst/manifests/courier.pp as a baseline.

# don't delete dovecot
echo dovecot-imapd hold | dpkg --set-selections

# apt-get doesn't want to install courier while dovecot is installed.
# So, try downloading the courier package, and installing them 
# directly with dpkg.  Manually deal with conflict.

apt-get download courier-authdaemon courier-authlib courier-authlib-userdb \
  courier-base courier-ssl expect libgamin0 libglib2.0-0 libltdl7 tcl8.5

apt-get install gamin libgamin0
dpkg -i *.deb

dpkg -i --force-conflicts courier-imap_4.10.0-20120615-1_amd64.deb

Courier runs two services:

/etc/init.d/courier-imap
/etc/init.d/courier-authdaemon

Dovecot has one service

/etc/init.d/dovecot

After install

  • stop courier, start dovecot. Verify that imap access for imap://testdovecot@herman.mayfirst.org/INBOX works
  • stop dovecot, start courier-imap and courier-authdaemon. Verify that imap access works for testdovecot@…

This is good. Now, we're clear to start doing A/B testing.

comment:7 Changed 5 years ago by https://id.mayfirst.org/srevilak

I've committed a few things to mfpl/imap-test.

First, there's mail-gen a program to generate mail.

Second, there's ls-inbox.py, which times how long it takes to produce an index listing of INBOX.

ls-inbox.py uses python's imaplib library, which (unfortunately) does not support starttls. I'm going to do a first set of tests without TLS, just because I want to see a ballpark comparison between dovecot and courier (I'll change the test user's password afterwards). For further testing, I'll want to set up dovecot, courier to listen on port 993 (imaps)

comment:8 Changed 5 years ago by https://id.mayfirst.org/srevilak

Test 1: list inbox

100 iterations of ls-inbox.py. Aside from the testing, there's no additional load on herman.

testdovecot's INBOX contains 10,022 messages, using 136M on disk.

Courier:

$ ./ls-inbox.py testdovecot@herman.mayfirst.org PASSWD 100
  ...
Summary for 100 iterations:
min       2274 ms
max      22832 ms
mean      3355 ms
median    2840 ms

median of 2.84s to get an INBOX listing with courier

dovecot:

$ ./ls-inbox.py testdovecot@herman.mayfirst.org PASSWD 100
  ...
Summary for 100 iterations:
min       2128 ms
max       5138 ms
mean      2810 ms
median    2598 ms

median of 2.598s to get an INBOX listing with dovecot.

During this time, iostat reports little disk I/O from either imap server, which gives me the impression that linux is getting most of it's data from I/O cache; we're not hitting the disk.

Test 2: list inbox, with reduced memory

0 herman:/home/testdovecot# free
             total       used       free     shared    buffers     cached
Mem:       2060504     847628    1212876          0     164552     464960
-/+ buffers/cache:     218116    1842388
Swap:       499708          0     499708
0 herman:/home/testdovecot#

There's 1,842,388 kb free (excluding buffers and cache). I want to generate a condition where INBOX won't fit in memory, so I'll use a little C program to eat up some memory.

0 herman:~# ./consume-memory 1800000
allocated memory: 1800000 kb

0 herman:~# free
             total       used       free     shared    buffers     cached
Mem:       2060504    1988404      72100          0       7084      78840
-/+ buffers/cache:    1902480     158024
Swap:       499708        128     499580

Now, "free" is just slightly larger than the size of Maildir.

Courier:

$ ./ls-inbox.py testdovecot@herman.mayfirst.org PASSWD 100
  ...
Summary for 100 iterations:
min       3698 ms
max       7782 ms
mean      4966 ms
median    4637 ms

There's a significant amount of IO during this test (1200--1300 tps, according to iostat).

Dovecot:

$ ./ls-inbox.py testdovecot@herman.mayfirst.org PASSWD 100
  ...
Summary for 100 iterations:
min       2226 ms
max       8053 ms
mean      3671 ms
median    3365 ms

There's almost no disk IO during this test. I assume that dovecot read it's (~ 6.5mb) index file, and is using that to respond imap fetch requests.

When Maildir doesn't fit in memory, dovcot's response time is 27% lower than courier (according to median times). That's not a bad improvement.

Comments

I'm planning to do one more ls-inbox test: reduced memory, with disk contention. I'll try that tomorrow night.

I've tried accessing imap://testdovecot@herman.mayfirst.org with my preferred MUA (mutt, no header caching), and dovecot feels faster than courier, more than these tests reflect. I don't have an explanation for this.

Support team members: if you're inclined to try accessing imap://testdovecot@herman.mayfirst.org with your MUA, go right ahead. On herman, "passwd testdovecot" to whatever you'd like, and try it out (I can reset the testdovecot's password later). Either stop dovecot and start courier, or stop courer and start dovecot. (comment:6 lists the /etc/init.d scripts for each mail server).

comment:9 Changed 4 years ago by https://id.mayfirst.org/jamie

  • Cc https://id.mayfirst.org/jamie added

comment:10 Changed 4 years ago by https://id.mayfirst.org/srevilak

  • Resolution set to fixed
  • Status changed from assigned to closed

I believe the dovecot evaluation is done, so I'm resolving this ticket.

Please login to add comments to this ticket.

Note: See TracTickets for help on using tickets.