Opened 11 years ago

Closed 11 years ago

Last modified 6 years ago

#652 closed Bug/Something is broken (fixed)

pipermail archive of PGP signed messages broken

Reported by: Matt Corks Owned by: Daniel Kahn Gillmor
Priority: Medium Component: Tech
Keywords: mailman Cc: matt@…
Sensitive: no


in some cases, when sending PGP signed messages to the closed list openflows-general, only the signature itself is archived, not the email content.


worse, the downloadable version of the archives for Feb 2008 for that list also just consist of a single PGP signature:

however, the archives for another randomly selected month, June 2007, seem to be fine. PGP signed messages are properly visible both the downloadable mbox file and the regular archive view.

i didn't investigate long enough to see if those signed messages were sent with different headers.

i realize pipermail 0.09 is not a fully mature or feature-rich archive manager, and i'm not sure how well supported it is, but if signed messages could at least be made available via downloadable mbox files, it would be appreciated.


-- mvc

Attachments (1)

personalize_on.patch (552 bytes) - added by Jamie McClelland 11 years ago.
mailman patch to turn on personalization by default

Download all attachments as: .zip

Change History (18)

comment:1 Changed 11 years ago by Jamie McClelland

Does this sounds like the bug report dkg made against the debian mailman package?

If so - dkg - should we apply your patch? Is it likely to make it into etch?

comment:2 Changed 11 years ago by Matt Corks

Cc: matt@… added

The message I referred to above meets the criteria listed by dkg in that bug report, so that's likely it.

Given that this isn't a security issue, my understanding is that this likely won't appear in etch.

comment:3 Changed 11 years ago by Daniel Kahn Gillmor

Yes, i think this is likely the same issue. The Right Thing (if someone has time to do this and properly test) would be to patch mailman, and then rebuild the old archives from the mbox stores.

comment:4 in reply to:  3 Changed 11 years ago by Matt Corks

Can I ask if someone has time to apply this patch? Either that, or convince people to stop signing their messages :)

comment:5 Changed 11 years ago by Daniel Kahn Gillmor

Keywords: mailman added

I've applied that patch to the etch packages and rebuilt them.

I've posted the debian packaging information, including a .deb, which should have the following hash:

6439c88b2025324d888e49fd01dffa6a89366fa4  mailman_2.1.9-7.1_i386.deb

I'd be willing to install the package, on leslie and assata if folks think that'd be the right thing to do. Feel free to assign it to me if you want me to take this on.

I imagine that what i'd do would be:

  • find a list with some verified bad messages in its archive on leslie (i'm thinking of a certain small, recent list that mvcorks and i are both members of)
  • install the upgrade on leslie, make sure mailman restarted
  • send a signed message to the particular list, and see if the archive stores it properly.
  • if successful, install the upgrade on assata
  • check on old messages for the suspect list, if old messages are visible, great! we're done. otherwise...
  • use bin/arch --wipe as suggested by msapiro on a smallish list, making sure that the numbering for archives for that list aren't broken. Document this step.

Then, if there are other list archives (other than the guinea pig list i select), repairing the archive of any particular list could be handled as a separate ticket. How does this sound?

comment:6 Changed 11 years ago by Daniel Kahn Gillmor

sorry: to be precise, the hash of the .deb listed above is an sha1sum.

comment:7 Changed 11 years ago by Jamie McClelland

Thanks Daniel - that's a great plan and thanks for taking this on.

I just have one request - can you add the patch that I'm about to upload to this ticket? It's a very simple one liner that's pretty important (and that I've applied by hand every time we upgrade mailman - ug). If it's a pain after you already built the debs let me know and I can apply it after you do the install.

It turns personalization on by default for new lists - this setting allows us to, by default, enable lists to print the email address a user is subscribed as in the footer of every message and turn on verp messaging so we can more easily unsubscribe aoler's who say we're spamming them.

Changed 11 years ago by Jamie McClelland

Attachment: personalize_on.patch added

mailman patch to turn on personalization by default

comment:8 Changed 11 years ago by Daniel Kahn Gillmor

OK, i've rebuilt the packages to include the personalize_on.patch as well. I'm applying the new package to leslie as i write this, and will followup with the appropriate testing.

comment:9 Changed 11 years ago by Daniel Kahn Gillmor

urg. upgrading mailman doesn't want to complete while there are files in /var/lib/mailman/qfiles/. It's unclear to me how to cleanly handle all those files without causing problems. There are hundreds of messages to deal with:

0 leslie:~# find /var/lib/mailman/qfiles/ -type f | tr '/' ' ' | awk '{ print $5 }' | sort | uniq -c
    586 bounces
      3 commands
      4 in
     45 retry
      8 shunt
0 leslie:~# 

I'm open to suggestions about what to do here. I'm reluctant to keep trying for fear of breaking the MF/PL list infrastructure.

comment:10 Changed 11 years ago by Jamie McClelland

As far as I can tell - these messages are all garbage. They should be kept so we can determine why they are collecting and fix the problem, but they don't hold critical data. I would suggest shutting down mailman, moving them to a new directory, and then upgrading mailman.

comment:11 Changed 11 years ago by Daniel Kahn Gillmor

Owner: changed from Jamie McClelland to Daniel Kahn Gillmor
Status: newassigned

OK, i'll try that.

comment:12 Changed 11 years ago by Daniel Kahn Gillmor

So the commands i ran were:

cd /var/lib/mailman
/etc/init.d/mailman stop && find qfiles | cpio -p -m /root/ticket652/ && find qfiles -type f -delete && /etc/init.d/mailman start && dpkg --install ~/mailman-packaging/mailman_2.1.9-7.2_i386.deb 

and it seemed to work fine. we've got the old files backed up in leslie:/root/ticket652/qfiles if you want to peruse them, jamie.

now i'm on to testing the archives themselves.

Last edited 6 years ago by Daniel Kahn Gillmor (previous) (diff)

comment:13 Changed 11 years ago by Daniel Kahn Gillmor

OK, the difference between a message sent before the upgrade and a message sent after the upgrade shows that archiving of new messages is working correctly.

The commands from the above comment should have been:

mkdir /root/ticket652
chmod 0700 /root/ticket652
cd /var/lib/mailman
/etc/init.d/mailman stop && \
find qfiles | cpio -p -m /root/ticket652/ && \
find qfiles -type f -delete && \
/etc/init.d/mailman start && \
dpkg --install ~/mailman-packaging/mailman_2.1.9-7.2_i386.deb

comment:14 Changed 11 years ago by Daniel Kahn Gillmor

Keywords: added
Resolution: fixed
Status: assignedclosed

OK, after the upgrade, i wiped the test list, and the old mails were properly re-archived:

cp -a /var/lib/mailman/archives/private/$LISTNAME /root/ticket652/ && \
/etc/init.d/mailman stop && \
/usr/lib/mailman/bin/arch --wipe $LISTNAME && \
/etc/init.d/mailman start

One interesting thing is that (at least with the smaller test list), the re-archiving step did not re-create the compressed versions of the archive files (e.g. 2008-April.txt.gz). I'm not sure why.

It also changed a few things in the emitted html other than repairing the broken messages. In particular, it added a few extra newlines, and changed all the "Archived on:" lines to today's date.

Anyway, i think this is completed now. The changes i made to the mailman packaging are in one big diff.

if there are other lists that want their archives rebuilt, those should be opened as a separate ticket.

comment:15 Changed 11 years ago by Matt Corks

Resolution: fixed
Status: closedreopened

Thanks very much for dealing with that, dkg!

Can I ask you to regenerate the archives for openflows-general as well, the list I opened this ticket about?

comment:16 Changed 11 years ago by Daniel Kahn Gillmor

I just rebuilt the archives for openflows-general as well. Sorry i let this slide.

The top link in the description is working now, afaict.

and the text archive is intact now (though the second link doesn't work because the gzip-compressed version wasn't re-generated for some reason).

If this is close enough for you, matt, please close the ticket. If you want me to look into why the gzip-compressed version of the text archive isn't available, let me know (or better yet, send me pointers to how to re-generate it!)

comment:17 Changed 11 years ago by Matt Corks

Resolution: fixed
Status: reopenedclosed

Thanks very much, dkg.

I think the compressed archives are generated by a monthly cron script, since the silcnet archives are now compressed:

So, I anticipate the openflows-general archives will be compressed at the end of the month.

That said, as long as the archives are available in some format, it's not important to me whether they're compressed. So I'm happy to mark this as closed.

Please login to add comments to this ticket.

Note: See TracTickets for help on using tickets.