Opened 2 years ago

Last modified 2 years ago

#5374 assigned Feature/Enhancement Request

Syncing CiviCRM to Mailman

Reported by: https://id.mayfirst.org/takethestreets Owned by: https://id.mayfirst.org/jamie
Priority: Medium Component: Tech
Keywords: mailman, CiviCRM Cc:
Sensitive: no

Description

Hi there,

I'm currently developing a Civi site for a group that's a potential MFPL member. One of the concerns is that they have discussion lists, and our spec calls for using Mailman integration with Civi as per this doc: http://wiki.civicrm.org/confluence/display/CRMDOC41/Mailman+synchronization

Now, the script to pull the data to a CSV can run on the MOSH server, and sync_members is part of the Debian mailman package. However, normal users can't run sync_members since it's on Leslie, and of course one needs to transfer the CSV file from the MOSH server to Leslie.

Would MFPL be open to having a cronjob on Leslie that transfers the CSV via rsync or some other SSH-tunnelled protocol, then runs sync_members on a nightly basis? This would allow members to use their Civi profiles to add/remove themselves from various discussion groups.

Change History (6)

comment:1 Changed 2 years ago by https://id.mayfirst.org/ross

  • Owner set to https://id.mayfirst.org/jamie
  • Status changed from new to assigned

comment:2 Changed 2 years ago by https://id.mayfirst.org/jamie

Sorry for the slow response. I just got to this ticket and am mulling it over. It's a fairly significant customization - and to my knowledge, we haven't had any other requests for it, so I'm hesitant to make the exception.

Also, if we did make this exception, I would prefer to make in a way accessible to all members and with minimal admin intervention, but that will require some kind of security to prevent users from modifying other member's email lists, which would add yet more complexities.

On the other hand, the only other method I can think of would be for the member to get their own dedicated virtual server, and essentially that would mean maintain another mailman installation, which is not ideal.

Of course, if mailman 3.0 would ever get released, this wouldn't be a problem...

jamie

comment:3 Changed 2 years ago by https://id.mayfirst.org/takethestreets

Hi Jamie,

Thanks for thinking this over.

I agree that Mailman 3.0 is the ultimate answer. Since that's not an option, I'll make one more pitch for this approach.

To speak to the demand issue, I've had three different groups ask me about Mailman/CiviCRM integration in the last year. Two of them are potential MFPL members (the National Lawyer's Guild and the National Police Accountability Project), and the NLG in particular has expressed strong interest in being a member. I know that this is a mandatory requirement for the NLG; I'm not sure whether it's a dealbreaker for NPAP or not.

For the customization - I'm not sure if I explained the approach I was thinking, just want to make sure we're on the same page. I propose just a cronjob that says, "wget https://nlg.org/memberlist.txt && sync_members memberlist.txt listname". I wanted to be clear that I didn't expect you to run PHP or call the Civi API from Leslie.

Agreed that in an ideal world this could be done with minimal admin intervention, though I suspect that the amount of admin time necessary to handle this manually for all requests would add up to less time than it would take to, say, code this into Red. I see this as comparable to requesting a LimeSurvey account, for instance.

Anyway, I understand the concerns - and I'm definitely looking for alternatives! Unfortunately, the alternatives all require a patch to Mailman (e.g. http://sourceforge.net/tracker/?func=detail&atid=300103&aid=1605292&group_id=103), or, as you suggested, running a separate instance of Mailman. So please let me know whether this is an option, and if there's anything I could to do help you with that.

comment:4 Changed 2 years ago by https://id.mayfirst.org/jamie

That seems like a good approach.

In fact, this request is not that different from providing mailman rss feeds.

I think this is do-able. Do you want to setup a list and a file of subscribers to sync for testing?

I think on leslie, we'd want:

  • a bash script to be executed by cron (e.g. mf-sync-remote-members)
  • a configuration file that contains the lists to be synchronized (e.g. /etc/mf-sync-remote-members.conf). This would be root-owned and the only file admins would have to edit to add a new URL/list pair. It would contain one line per list, e.g.: nlg-members-list:https://nlg.org/memberlist.txt.
  • the script would iterate over all lines in the file, pull down the list and run the sync script

Anything I'm missing?

Some questions/thoughts...

  • Do we allow non-https transport? Most civi sites will probably have https already. If leslie is tricked into downloading the wrong file, the damage should only be limited to the list for that member.
  • What if the member deletes the list? The script could be smart enough to detect that the list doesn't exist. One remote corner case would be that someone else creates a list with the same name... that's where adding this to red would be useful, to clean up after a list is remoted. But that could come later.

comment:5 follow-up: Changed 2 years ago by https://id.mayfirst.org/takethestreets

Hi Jamie,

This sounds awesome. To respond:

  • What about using SCP/SFTP instead of HTTP over SSL? Since all Civi installations will be on MFPL servers, we can simply use the Monkeysphere and not worry about whether a member has SSL.
  • Maybe if the list isn't found, the script can generate an e-mail and/or Nagios warning? That way that list/file pair can be removed from the script.

comment:6 in reply to: ↑ 5 Changed 2 years ago by https://id.mayfirst.org/jamie

  • What about using SCP/SFTP instead of HTTP over SSL? Since all Civi installations will be on MFPL servers, we can simply use the Monkeysphere and not worry about whether a member has SSL.

That seems like a good solution. We would probably want to run the script on leslie as the user "list" (which owns the various mailman stuff). We could generate a monkeysphere user id for list@…. Then each MFPL member would be responsible for identifying a sftp enabled user account and adding "list@…" to that accounts ~/.monkeysphere/authorized_user_ids file. Then, the leslie's configuration file would look more like:

nlg-members-list|nlg-list@stone.mayfirst.org:path/to/members.txt
  • Maybe if the list isn't found, the script can generate an e-mail and/or Nagios warning? That way that list/file pair can be removed from the script.

Also seems like a good work-around.

jamie

Please login to add comments to this ticket.

Note: See TracTickets for help on using tickets.