Opened 4 years ago

Last modified 12 months ago

#5374 assigned Feature/Enhancement Request

Syncing CiviCRM to Mailman

Reported by: Owned by:
Priority: Medium Component: Tech
Keywords: mailman, CiviCRM Cc:
Sensitive: no


Hi there,

I'm currently developing a Civi site for a group that's a potential MFPL member. One of the concerns is that they have discussion lists, and our spec calls for using Mailman integration with Civi as per this doc:

Now, the script to pull the data to a CSV can run on the MOSH server, and sync_members is part of the Debian mailman package. However, normal users can't run sync_members since it's on Leslie, and of course one needs to transfer the CSV file from the MOSH server to Leslie.

Would MFPL be open to having a cronjob on Leslie that transfers the CSV via rsync or some other SSH-tunnelled protocol, then runs sync_members on a nightly basis? This would allow members to use their Civi profiles to add/remove themselves from various discussion groups.

Change History (10)

comment:1 Changed 4 years ago by

  • Owner set to
  • Status changed from new to assigned

comment:2 Changed 4 years ago by

Sorry for the slow response. I just got to this ticket and am mulling it over. It's a fairly significant customization - and to my knowledge, we haven't had any other requests for it, so I'm hesitant to make the exception.

Also, if we did make this exception, I would prefer to make in a way accessible to all members and with minimal admin intervention, but that will require some kind of security to prevent users from modifying other member's email lists, which would add yet more complexities.

On the other hand, the only other method I can think of would be for the member to get their own dedicated virtual server, and essentially that would mean maintain another mailman installation, which is not ideal.

Of course, if mailman 3.0 would ever get released, this wouldn't be a problem...


comment:3 Changed 4 years ago by

Hi Jamie,

Thanks for thinking this over.

I agree that Mailman 3.0 is the ultimate answer. Since that's not an option, I'll make one more pitch for this approach.

To speak to the demand issue, I've had three different groups ask me about Mailman/CiviCRM integration in the last year. Two of them are potential MFPL members (the National Lawyer's Guild and the National Police Accountability Project), and the NLG in particular has expressed strong interest in being a member. I know that this is a mandatory requirement for the NLG; I'm not sure whether it's a dealbreaker for NPAP or not.

For the customization - I'm not sure if I explained the approach I was thinking, just want to make sure we're on the same page. I propose just a cronjob that says, "wget && sync_members memberlist.txt listname". I wanted to be clear that I didn't expect you to run PHP or call the Civi API from Leslie.

Agreed that in an ideal world this could be done with minimal admin intervention, though I suspect that the amount of admin time necessary to handle this manually for all requests would add up to less time than it would take to, say, code this into Red. I see this as comparable to requesting a LimeSurvey account, for instance.

Anyway, I understand the concerns - and I'm definitely looking for alternatives! Unfortunately, the alternatives all require a patch to Mailman (e.g., or, as you suggested, running a separate instance of Mailman. So please let me know whether this is an option, and if there's anything I could to do help you with that.

comment:4 Changed 4 years ago by

That seems like a good approach.

In fact, this request is not that different from providing mailman rss feeds.

I think this is do-able. Do you want to setup a list and a file of subscribers to sync for testing?

I think on leslie, we'd want:

  • a bash script to be executed by cron (e.g. mf-sync-remote-members)
  • a configuration file that contains the lists to be synchronized (e.g. /etc/mf-sync-remote-members.conf). This would be root-owned and the only file admins would have to edit to add a new URL/list pair. It would contain one line per list, e.g.: nlg-members-list:
  • the script would iterate over all lines in the file, pull down the list and run the sync script

Anything I'm missing?

Some questions/thoughts...

  • Do we allow non-https transport? Most civi sites will probably have https already. If leslie is tricked into downloading the wrong file, the damage should only be limited to the list for that member.
  • What if the member deletes the list? The script could be smart enough to detect that the list doesn't exist. One remote corner case would be that someone else creates a list with the same name... that's where adding this to red would be useful, to clean up after a list is remoted. But that could come later.

comment:5 follow-up: Changed 4 years ago by

Hi Jamie,

This sounds awesome. To respond:

  • What about using SCP/SFTP instead of HTTP over SSL? Since all Civi installations will be on MFPL servers, we can simply use the Monkeysphere and not worry about whether a member has SSL.
  • Maybe if the list isn't found, the script can generate an e-mail and/or Nagios warning? That way that list/file pair can be removed from the script.

comment:6 in reply to: ↑ 5 Changed 4 years ago by

  • What about using SCP/SFTP instead of HTTP over SSL? Since all Civi installations will be on MFPL servers, we can simply use the Monkeysphere and not worry about whether a member has SSL.

That seems like a good solution. We would probably want to run the script on leslie as the user "list" (which owns the various mailman stuff). We could generate a monkeysphere user id for list@…. Then each MFPL member would be responsible for identifying a sftp enabled user account and adding "list@…" to that accounts ~/.monkeysphere/authorized_user_ids file. Then, the leslie's configuration file would look more like:

  • Maybe if the list isn't found, the script can generate an e-mail and/or Nagios warning? That way that list/file pair can be removed from the script.

Also seems like a good work-around.


comment:7 Changed 17 months ago by

Hi everyone,

To revisit this ticket - we never followed through because the client (the NLG!) decided against this - but now another MFPL member (CPR) has the same need. And surprise surprise - 3 years later, Mailman 3.0 is STILL not released.

Can we implement this plan as we discussed in the last comment on this ticket? If we could do this some time in October, that would be awesome.

comment:8 Changed 17 months ago by

Hey takethestreets - I did a bit more research and came across this project:

It seems to meet the need without any code installed on the mailman server. It also seems like it provides a lot of flexibility. The downside is that you would need to learn how to create a basic django application in order to take advantage of it.

What do you think?

comment:9 Changed 12 months ago by

Just a quick update to this ticket. The mailman-api package available in jessie, may be the best solution for this problem,

I haven't played with the package at all, but I'm guessing the civicrm extension could be modified to interact with the api, which should allow for native mailman interaction instead of some tacked on additional web application.

comment:10 Changed 12 months ago by

That looks really promising. It seems to depend on python greater than 2.7.5-5~ - which means it will be hard to install on a wheezy machine. However, once we upgrade to jessie getting it on our mailman machines should be quite simple.

Let me know if try it out and think it will work.

Please login to add comments to this ticket.

Note: See TracTickets for help on using tickets.