Opened 5 weeks ago

Last modified 14 hours ago

#13558 assigned Bug/Something is broken

Ossie and Rose are on CBL

Reported by: Owned by:
Priority: High Component: Tech
Keywords: Cc:
Sensitive: no


Palante Tech's monitoring system caught that is reporting four of our clients's sites as on a blacklist, I traced it to the hosts and

this happened previously in #11799

Change History (19)

comment:1 Changed 5 weeks ago by

Sorry just getting to this now. I'll start looking for any sites that may be sending spam on rose and ossie.

comment:2 Changed 5 weeks ago by

Thank you!

Fwiw, I monitor this for all Palante clients using our Icinga install.

I downloaded this plugin to /usr/lib/nagios/plugins

and then I check this command for every host using hostgroups and services.

# 'check_domain_bl' command definition
define command{
        command_name    check_domain_bl
        command_line    /usr/lib/nagios/plugins/check_bl -H $HOSTNAME$ -B

Let me know if you want help getting that set up for your own servers.

comment:3 Changed 5 weeks ago by

  • Owner set to
  • Status changed from new to assigned

I also think we should be doing this and I would love to go over those details with you. I just discoved we have a ticket open for this already #5736. I'm going to point to this ticket from there to follow up with later.

For now let's keep this ticket focused on resolving the current listing.

comment:4 Changed 5 weeks ago by

Sounds good! I have cc'd myself to that ticket, so I can help with that once folks are ready. Looks like that ticket was created by Jon Goldberg when he and I set up that monitoring for Palante clients 6 years ago.

comment:5 Changed 5 weeks ago by

Ok, I've been at this for hours and after a lot of false leads I've found a couple of compromised user accounts on ossie and another on rose. Hopefully these were the source of the blacklisting. rose suffers from a general problem of having many adresses forwarded to other servers putting us in the middle.

Tomorrow I'll check the mail queue again.

comment:6 Changed 5 weeks ago by

Thank you Jaime! My monitoring is still saying they're listed, let me know if there is any other details I can give that might help.

comment:7 Changed 5 weeks ago by

  • Cc added

Unsure of whether we'd solved the root problem I haven't tried to have them delisted.

Looking at the details on the CBL page it seems they haven't seen any activity n the past 24 hours but the type of activity they point to is bot activity from a compromised site. If that is the case then I have not yet identified the source. The kind of behaviour they describe is quite difficult to catch as we do not keep logs of outgoing tcp activity.

comment:8 Changed 5 weeks ago by

Ok, CBL hasn't listed any new detections since Sunday for both ip's I've requested they both be removed.

Last edited 4 weeks ago by (previous) (diff)

comment:9 Changed 4 weeks ago by

  • Resolution set to fixed
  • Status changed from assigned to closed

Both appear to have been delisted now. I've opened separate tickets to follow up on the suspected sources of spam from these servers

comment:10 Changed 4 weeks ago by

My monitoring concurs. Thank you!

comment:11 Changed 9 days ago by

  • Resolution fixed deleted
  • Status changed from closed to assigned

I'm afraid Ossie is on the CBL again, my monitoring notified me.

comment:12 Changed 8 days ago by

Thanks jamila. Trying to track this one down now.

comment:13 Changed 8 days ago by

comment:14 Changed 7 days ago by

I've been fishing in the logs all day and I cannot seem to track down where this is coming from on ossie or rose. Very frustrating.

comment:15 Changed 4 days ago by

Ossie isn't anymore but Rose still is.

comment:16 Changed 19 hours ago by

Spoke too soon, Ossie is back on the list.

comment:17 Changed 16 hours ago by

  • Priority changed from Medium to High

I am bumping up the priority for this as this ticket at it has been ongoing without solution. I have been able to find the source of these problems.

Whatever process is doing this runs quickly and disappears without leaving any traces we've been able to identify.

The site offers a timestamp for the last time they identify malware from ossie attempting to communicate with a control server. Currently I am going though each site with any activity in the logs around that time and searching for signs of compromise. This is a time consuming process and may not actually catch the real offender.

They also offer a perl script that appears to just use ss and lsof to to track any outgoing requests to the known sinkhole. This seems similar to shell scripts we've attempted to fashion ourselves Their code looks safe enough to try though. I think I could create a separate user for this purpose and leave this running in a screen session.

comment:18 Changed 15 hours ago by

I agree - that sounds like the right path to take and the perl script seems legit and useful. The script may need to be run as root though to capture all the info. But you should be able test that out first.

comment:19 Changed 14 hours ago by

Ok, tested with another ip before assigning to the sinkhole ip and currently running in a screen session on ossie. doesn't list any activity for rose since April 16th, so it seems less likely we'll catch anything there now. I've gone ahead and requested removal od rose.

Last edited 14 hours ago by (previous) (diff)

Please login to add comments to this ticket.

Note: See TracTickets for help on using tickets.