Opened 5 weeks ago

Last modified 14 hours ago

#13558 assigned Bug/Something is broken

Ossie and Rose are on CBL

Reported by: https://id.mayfirst.org/jamila Owned by: https://id.mayfirst.org/jaimev
Priority: High Component: Tech
Keywords: Cc: https://id.mayfirst.org/jamie
Sensitive: no

Description

Palante Tech's monitoring system caught that zen.spamhaus.org is reporting four of our clients's sites as on a blacklist, I traced it to the hosts ossie.mayfirst.org and rose.mayfirst.org.

https://www.abuseat.org/lookup.cgi?ip=209.51.163.16 https://www.abuseat.org/lookup.cgi?ip=209.51.180.30

https://mxtoolbox.com/SuperTool.aspx?action=mx%3acommonbound.org&run=toolpage#

this happened previously in #11799

Change History (19)

comment:1 Changed 5 weeks ago by https://id.mayfirst.org/jaimev

Sorry just getting to this now. I'll start looking for any sites that may be sending spam on rose and ossie.

comment:2 Changed 5 weeks ago by https://id.mayfirst.org/jamila

Thank you!

Fwiw, I monitor this for all Palante clients using our Icinga install.

I downloaded this plugin to /usr/lib/nagios/plugins https://exchange.nagios.org/directory/Plugins/Email-and-Groupware/check_bl/details

and then I check this command for every host using hostgroups and services.

# 'check_domain_bl' command definition
define command{
        command_name    check_domain_bl
        command_line    /usr/lib/nagios/plugins/check_bl -H $HOSTNAME$ -B zen.spamhaus.org bl.spamcop.net dnsbl.ahbl.org dnsbl.njabl.org dnsbl.sorbs.net virbl.dnsbl.bit.nl rbl.efnet.org phishing.rbl.msrbl.net 0spam.fusionzero.com list.dsbl.org multihop.dsbl.org unconfirmed.dsbl.org will-spam-for-food.eu.org blacklist.spambag.org blackholes.brainerd.net blackholes.uceb.org spamsources.dnsbl.info map.spam-rbl.com ns1.unsubscore.com psbl.surriel.com l2.spews.dnsbl.sorbs.net bl.csma.biz sbl.csma.biz dynablock.njabl.org no-more-funn.moensted.dk  ubl.unsubscore.com dnsbl-1.uceprotect.net dnsbl-2.uceprotect.net dnsbl-3.uceprotect.net spamguard.leadmon.net opm.blitzed.org bl.spamcannibal.org rbl.schulte.org dnsbl.ahbl.org virbl.dnsbl.bit.nl combined.rbl.msrbl.net
        }

Let me know if you want help getting that set up for your own servers.

comment:3 Changed 5 weeks ago by https://id.mayfirst.org/jaimev

  • Owner set to https://id.mayfirst.org/jaimev
  • Status changed from new to assigned

I also think we should be doing this and I would love to go over those details with you. I just discoved we have a ticket open for this already #5736. I'm going to point to this ticket from there to follow up with later.

For now let's keep this ticket focused on resolving the current listing.

comment:4 Changed 5 weeks ago by https://id.mayfirst.org/jamila

Sounds good! I have cc'd myself to that ticket, so I can help with that once folks are ready. Looks like that ticket was created by Jon Goldberg when he and I set up that monitoring for Palante clients 6 years ago.

comment:5 Changed 5 weeks ago by https://id.mayfirst.org/jaimev

Ok, I've been at this for hours and after a lot of false leads I've found a couple of compromised user accounts on ossie and another on rose. Hopefully these were the source of the blacklisting. rose suffers from a general problem of having many adresses forwarded to other servers putting us in the middle.

Tomorrow I'll check the mail queue again.

comment:6 Changed 5 weeks ago by https://id.mayfirst.org/jamila

Thank you Jaime! My monitoring is still saying they're listed, let me know if there is any other details I can give that might help.

comment:7 Changed 5 weeks ago by https://id.mayfirst.org/jaimev

  • Cc https://id.mayfirst.org/jamie added

Unsure of whether we'd solved the root problem I haven't tried to have them delisted.

Looking at the details on the CBL page it seems they haven't seen any activity n the past 24 hours but the type of activity they point to is bot activity from a compromised site. If that is the case then I have not yet identified the source. The kind of behaviour they describe is quite difficult to catch as we do not keep logs of outgoing tcp activity.

comment:8 Changed 5 weeks ago by https://id.mayfirst.org/jaimev

Ok, CBL hasn't listed any new detections since Sunday for both ip's I've requested they both be removed.

Last edited 4 weeks ago by https://id.mayfirst.org/jaimev (previous) (diff)

comment:9 Changed 4 weeks ago by https://id.mayfirst.org/jaimev

  • Resolution set to fixed
  • Status changed from assigned to closed

Both appear to have been delisted now. I've opened separate tickets to follow up on the suspected sources of spam from these servers

comment:10 Changed 4 weeks ago by https://id.mayfirst.org/jamila

My monitoring concurs. Thank you!

comment:11 Changed 9 days ago by https://id.mayfirst.org/jamila

  • Resolution fixed deleted
  • Status changed from closed to assigned

I'm afraid Ossie is on the CBL again, my monitoring notified me. https://mxtoolbox.com/SuperTool.aspx?action=blacklist%3a209.51.163.16&run=toolpage

comment:12 Changed 8 days ago by https://id.mayfirst.org/jaimev

Thanks jamila. Trying to track this one down now.

comment:13 Changed 8 days ago by https://id.mayfirst.org/jamila

comment:14 Changed 7 days ago by https://id.mayfirst.org/jaimev

I've been fishing in the logs all day and I cannot seem to track down where this is coming from on ossie or rose. Very frustrating.

comment:15 Changed 4 days ago by https://id.mayfirst.org/jamila

Ossie isn't anymore but Rose still is.

comment:16 Changed 19 hours ago by https://id.mayfirst.org/jamila

Spoke too soon, Ossie is back on the list.

comment:17 Changed 16 hours ago by https://id.mayfirst.org/jaimev

  • Priority changed from Medium to High

I am bumping up the priority for this as this ticket at it has been ongoing without solution. I have been able to find the source of these problems.

Whatever process is doing this runs quickly and disappears without leaving any traces we've been able to identify.

The abuseat.org site offers a timestamp for the last time they identify malware from ossie attempting to communicate with a control server. Currently I am going though each site with any activity in the logs around that time and searching for signs of compromise. This is a time consuming process and may not actually catch the real offender.

They also offer a perl script https://www.abuseat.org/shtracer.pl that appears to just use ss and lsof to to track any outgoing requests to the known sinkhole. This seems similar to shell scripts we've attempted to fashion ourselves https://support.mayfirst.org/ticket/13225#comment:17. Their code looks safe enough to try though. I think I could create a separate user for this purpose and leave this running in a screen session.

comment:18 Changed 15 hours ago by https://id.mayfirst.org/jamie

I agree - that sounds like the right path to take and the perl script seems legit and useful. The script may need to be run as root though to capture all the info. But you should be able test that out first.

comment:19 Changed 14 hours ago by https://id.mayfirst.org/jaimev

Ok, tested with another ip before assigning to the abuseat.org sinkhole ip and currently running in a screen session on ossie. http://www.abuseat.org/lookup.cgi?ip=209.51.180.30 doesn't list any activity for rose since April 16th, so it seems less likely we'll catch anything there now. I've gone ahead and requested removal od rose.

Last edited 14 hours ago by https://id.mayfirst.org/jaimev (previous) (diff)

Please login to add comments to this ticket.

Note: See TracTickets for help on using tickets.