Opened 6 months ago

Last modified 5 months ago

#13558 assigned Bug/Something is broken

Ossie and Rose are on CBL

Reported by: https://id.mayfirst.org/jamila Owned by: https://id.mayfirst.org/jaimev
Priority: High Component: Tech
Keywords: Cc: https://id.mayfirst.org/jamie, https://id.mayfirst.org/endisolation
Sensitive: no

Description

Palante Tech's monitoring system caught that zen.spamhaus.org is reporting four of our clients's sites as on a blacklist, I traced it to the hosts ossie.mayfirst.org and rose.mayfirst.org.

https://www.abuseat.org/lookup.cgi?ip=209.51.163.16 https://www.abuseat.org/lookup.cgi?ip=209.51.180.30

https://mxtoolbox.com/SuperTool.aspx?action=mx%3acommonbound.org&run=toolpage#

this happened previously in #11799

Change History (32)

comment:1 Changed 6 months ago by https://id.mayfirst.org/jaimev

Sorry just getting to this now. I'll start looking for any sites that may be sending spam on rose and ossie.

comment:2 Changed 6 months ago by https://id.mayfirst.org/jamila

Thank you!

Fwiw, I monitor this for all Palante clients using our Icinga install.

I downloaded this plugin to /usr/lib/nagios/plugins https://exchange.nagios.org/directory/Plugins/Email-and-Groupware/check_bl/details

and then I check this command for every host using hostgroups and services.

# 'check_domain_bl' command definition
define command{
        command_name    check_domain_bl
        command_line    /usr/lib/nagios/plugins/check_bl -H $HOSTNAME$ -B zen.spamhaus.org bl.spamcop.net dnsbl.ahbl.org dnsbl.njabl.org dnsbl.sorbs.net virbl.dnsbl.bit.nl rbl.efnet.org phishing.rbl.msrbl.net 0spam.fusionzero.com list.dsbl.org multihop.dsbl.org unconfirmed.dsbl.org will-spam-for-food.eu.org blacklist.spambag.org blackholes.brainerd.net blackholes.uceb.org spamsources.dnsbl.info map.spam-rbl.com ns1.unsubscore.com psbl.surriel.com l2.spews.dnsbl.sorbs.net bl.csma.biz sbl.csma.biz dynablock.njabl.org no-more-funn.moensted.dk  ubl.unsubscore.com dnsbl-1.uceprotect.net dnsbl-2.uceprotect.net dnsbl-3.uceprotect.net spamguard.leadmon.net opm.blitzed.org bl.spamcannibal.org rbl.schulte.org dnsbl.ahbl.org virbl.dnsbl.bit.nl combined.rbl.msrbl.net
        }

Let me know if you want help getting that set up for your own servers.

comment:3 Changed 6 months ago by https://id.mayfirst.org/jaimev

  • Owner set to https://id.mayfirst.org/jaimev
  • Status changed from new to assigned

I also think we should be doing this and I would love to go over those details with you. I just discoved we have a ticket open for this already #5736. I'm going to point to this ticket from there to follow up with later.

For now let's keep this ticket focused on resolving the current listing.

comment:4 Changed 6 months ago by https://id.mayfirst.org/jamila

Sounds good! I have cc'd myself to that ticket, so I can help with that once folks are ready. Looks like that ticket was created by Jon Goldberg when he and I set up that monitoring for Palante clients 6 years ago.

comment:5 Changed 6 months ago by https://id.mayfirst.org/jaimev

Ok, I've been at this for hours and after a lot of false leads I've found a couple of compromised user accounts on ossie and another on rose. Hopefully these were the source of the blacklisting. rose suffers from a general problem of having many adresses forwarded to other servers putting us in the middle.

Tomorrow I'll check the mail queue again.

comment:6 Changed 6 months ago by https://id.mayfirst.org/jamila

Thank you Jaime! My monitoring is still saying they're listed, let me know if there is any other details I can give that might help.

comment:7 Changed 6 months ago by https://id.mayfirst.org/jaimev

  • Cc https://id.mayfirst.org/jamie added

Unsure of whether we'd solved the root problem I haven't tried to have them delisted.

Looking at the details on the CBL page it seems they haven't seen any activity n the past 24 hours but the type of activity they point to is bot activity from a compromised site. If that is the case then I have not yet identified the source. The kind of behaviour they describe is quite difficult to catch as we do not keep logs of outgoing tcp activity.

comment:8 Changed 6 months ago by https://id.mayfirst.org/jaimev

Ok, CBL hasn't listed any new detections since Sunday for both ip's I've requested they both be removed.

Last edited 6 months ago by https://id.mayfirst.org/jaimev (previous) (diff)

comment:9 Changed 6 months ago by https://id.mayfirst.org/jaimev

  • Resolution set to fixed
  • Status changed from assigned to closed

Both appear to have been delisted now. I've opened separate tickets to follow up on the suspected sources of spam from these servers

comment:10 Changed 6 months ago by https://id.mayfirst.org/jamila

My monitoring concurs. Thank you!

comment:11 Changed 5 months ago by https://id.mayfirst.org/jamila

  • Resolution fixed deleted
  • Status changed from closed to assigned

I'm afraid Ossie is on the CBL again, my monitoring notified me. https://mxtoolbox.com/SuperTool.aspx?action=blacklist%3a209.51.163.16&run=toolpage

comment:12 Changed 5 months ago by https://id.mayfirst.org/jaimev

Thanks jamila. Trying to track this one down now.

comment:13 Changed 5 months ago by https://id.mayfirst.org/jamila

comment:14 Changed 5 months ago by https://id.mayfirst.org/jaimev

I've been fishing in the logs all day and I cannot seem to track down where this is coming from on ossie or rose. Very frustrating.

comment:15 Changed 5 months ago by https://id.mayfirst.org/jamila

Ossie isn't anymore but Rose still is.

comment:16 Changed 5 months ago by https://id.mayfirst.org/jamila

Spoke too soon, Ossie is back on the list.

comment:17 Changed 5 months ago by https://id.mayfirst.org/jaimev

  • Priority changed from Medium to High

I am bumping up the priority for this as this ticket at it has been ongoing without solution. I have been able to find the source of these problems.

Whatever process is doing this runs quickly and disappears without leaving any traces we've been able to identify.

The abuseat.org site offers a timestamp for the last time they identify malware from ossie attempting to communicate with a control server. Currently I am going though each site with any activity in the logs around that time and searching for signs of compromise. This is a time consuming process and may not actually catch the real offender.

They also offer a perl script https://www.abuseat.org/shtracer.pl that appears to just use ss and lsof to to track any outgoing requests to the known sinkhole. This seems similar to shell scripts we've attempted to fashion ourselves https://support.mayfirst.org/ticket/13225#comment:17. Their code looks safe enough to try though. I think I could create a separate user for this purpose and leave this running in a screen session.

comment:18 Changed 5 months ago by https://id.mayfirst.org/jamie

I agree - that sounds like the right path to take and the perl script seems legit and useful. The script may need to be run as root though to capture all the info. But you should be able test that out first.

comment:19 Changed 5 months ago by https://id.mayfirst.org/jaimev

Ok, tested with another ip before assigning to the abuseat.org sinkhole ip and currently running in a screen session on ossie. http://www.abuseat.org/lookup.cgi?ip=209.51.180.30 doesn't list any activity for rose since April 16th, so it seems less likely we'll catch anything there now. I've gone ahead and requested removal od rose.

Last edited 5 months ago by https://id.mayfirst.org/jaimev (previous) (diff)

comment:20 Changed 5 months ago by https://id.mayfirst.org/jaimev

shtracer.pl hasn't caught anything and there have been new detections on the abuseat.org site in the past 24hrs. The sinkhole ip appears different though, I am updating shtracer.pl script with the new sinkhole ip and running again.

comment:21 Changed 5 months ago by https://id.mayfirst.org/jaimev

  • Cc https://id.mayfirst.org/endisolation added

Ok, now we have several hits. all of them appear to be coming form the endisolation user running the endisolation.org site. I am copying them here.

Check the shtracer.log file to see the output.

0 ossie:~/tickets/13558# head shtracer.log 
(2018-04-24 12:16:36) HIT tcp    ESTAB      0      284      209.51.163.16:51144     192.42.116.41:http     users:(("php5-fpm",pid=2281,fd=7))
   COMMAND   PID         USER   FD   TYPE             DEVICE SIZE/OFF     NODE NAME
   php5-fpm 2281 endisolation  cwd    DIR              253,0     4096        2 /
   php5-fpm 2281 endisolation  rtd    DIR              253,0     4096        2 /
   php5-fpm 2281 endisolation  txt    REG              253,0  9134872     9897 /usr/sbin/php5-fpm
   php5-fpm 2281 endisolation  mem    REG              253,0    22952    49171 /lib/x86_64-linux-gnu/libnss_dns-2.19.so
   php5-fpm 2281 endisolation  DEL    REG                0,5          31981237 /dev/zero
   php5-fpm 2281 endisolation  DEL    REG                0,5          31981333 /dev/zero
   php5-fpm 2281 endisolation  mem    REG              253,0    43592    49181 /lib/x86_64-linux-gnu/libnss_nis-2.19.so
   php5-fpm 2281 endisolation  mem    REG              253,0    31632    49169 /lib/x86_64-linux-gnu/libnss_compat-2.19.so

The endisolation.org hosting order on ossie contains two wordpress installations one in /home/members/civic/sites/endisolation.org/web and another within /home/members/civic/sites/endisolation.org/web/blog

I parsed out the timestamps of the shtracer.log and grepped against the web log for the site and got the following

0 ossie:~/tickets/13558# while read -r DATE; do TIMESTAMP=$(date -d"$DATE" +%d/%b/%Y:%H:%M); grep "$TIMESTAMP" /home/members/civic/sites/endisolation.org/logs/web.log | awk '{print $7" "$9" "$11}'; done < <(grep -o -E "2018[^)]+" shtracer.log | sort -h | uniq ) | sort -h | uniq -c | sort -h
      2 /archives/1100 302 "-"
      2 /blog/archives/1100 200 "-"
      2 /blog/wp-content/plugins/bbpress/bbp-theme-compat/css/bbpress.css?ver=2.1.2 200 "http://www.endisolation.org/blog/archives/605"
      2 /blog/wp-content/plugins/buddypress/bp-core/css/admin-bar.css?ver=1.6.1 200 "http://www.endisolation.org/blog/archives/605"
      2 /blog/wp-content/plugins/mini-twitter-feed/jquery.minitwitter.css?ver=4.4.15 200 "http://www.endisolation.org/blog/archives/605"
      2 /blog/wp-content/plugins/mini-twitter-feed/jquery.minitwitter.js?ver=4.4.15 200 "http://www.endisolation.org/blog/archives/605"
      2 /blog/wp-content/plugins/sdac-translate/css/sdac-translate.css?ver=4.4.15 200 "http://www.endisolation.org/blog/archives/605"
      2 /blog/wp-content/themes/twentyeleven/style.css 200 "http://www.endisolation.org/blog/archives/605"
      2 /blog/wp-content/uploads/2018/01/cropped-Let_Us_Visit-1.jpg 200 "http://www.endisolation.org/blog/archives/605"
      2 /blog/wp-includes/css/admin-bar.min.css?ver=4.4.15 200 "http://www.endisolation.org/blog/archives/605"
      2 /blog/wp-includes/css/dashicons.min.css?ver=4.4.15 200 "http://www.endisolation.org/blog/archives/605"
      2 /blog/wp-includes/js/admin-bar.min.js?ver=4.4.15 200 "http://www.endisolation.org/blog/archives/605"
      2 /blog/wp-includes/js/comment-reply.min.js?ver=4.4.15 200 "http://www.endisolation.org/blog/archives/605"
      2 /blog/wp-includes/js/wp-embed.min.js?ver=4.4.15 200 "http://www.endisolation.org/blog/archives/605"
      2 /blog/wp-includes/js/wp-emoji-release.min.js?ver=4.4.15 200 "http://www.endisolation.org/blog/archives/605"
      2 /data/admin/ver.txt 302 "-"
      2 /data/admin/ver.txt 404 "-"
      2 /dedecms/data/admin/ver.txt 302 "-"
      2 /dedecms/data/admin/ver.txt 404 "-"
      4 /apple-touch-icon-120x120.png 404 "-"
      4 /apple-touch-icon-120x120-precomposed.png 404 "-"
      4 /blog/archives/605 200 "-"
      4 /blog/archives/605 200 "https://www.bing.com/"
     12 /apple-touch-icon.png 404 "-"
     12 /apple-touch-icon-precomposed.png 404 "-"
     16 /favicon.ico 200 "-"

We will probably have to disable the endisolation.org site to stop this. I will open a separate ticket for that.

comment:22 Changed 5 months ago by https://id.mayfirst.org/jamie

Nice work!!

comment:23 Changed 5 months ago by https://id.mayfirst.org/jaimev

I wasn't able to trigger another instance of the sinkhole connection by visiting any of the above urls on endisolation.org or by forcing all of the pending cron jobs to run so I'm still not sure yet how this thing works.

comment:24 Changed 5 months ago by https://id.mayfirst.org/jaimev

I've run our mf-best-effort-wordpress-fix-compromise on the wordpress sites in question. See #13652.

I am still running shtracer.pl just in case we've missed something.

I also created a logging rule for iptables last night and followed up on the previous experiment in https://support.mayfirst.org/ticket/13225#comment:17

0 ossie:~/tickets/13558# iptables -A OUTPUT -d 192.42.116.41/32 -j LOG --log-prefix "[BADCONNECTION ]"
0 ossie:~/tickets/13558# dmesg -w | awk '/BADCONNECTION/ {system("/root/tickets/13558/track_outgoing_connections.sh")}'
endisol+  5395 14.5  0.6 503168 54236 ?        S    02:56   0:00 php-fpm: pool endisolation.org
endisol+  5395 14.5  0.6 503168 54252 ?        S    02:56   0:00 php-fpm: pool endisolation.org
endisol+  5395 14.5  0.6 503168 54748 ?        S    02:56   0:00 php-fpm: pool endisolation.org
endisol+  5395 14.5  0.6 503168 54748 ?        S    02:56   0:00 php-fpm: pool endisolation.org
endisol+  5395 15.0  0.6 503168 54748 ?        R    02:56   0:00 php-fpm: pool endisolation.org
endisol+ 10285 36.5  0.6 503276 54228 ?        S    04:16   0:00 php-fpm: pool endisolation.org
endisol+ 10285 24.3  0.6 503276 54744 ?        S    04:16   0:00 php-fpm: pool endisolation.org
endisol+ 10285 24.3  0.6 503276 54744 ?        S    04:16   0:00 php-fpm: pool endisolation.org
endisol+ 10285 24.3  0.6 503276 54744 ?        S    04:16   0:00 php-fpm: pool endisolation.org
endisol+ 14227 15.2  0.6 503832 54508 ?        S    10:44   0:00 php-fpm: pool endisolation.org
endisol+ 14227 15.2  0.6 503832 54508 ?        S    10:44   0:00 php-fpm: pool endisolation.org
endisol+ 14227 15.2  0.6 503832 54508 ?        S    10:44   0:00 php-fpm: pool endisolation.org

The above method works although currently provides less detail. It might be something we can build on to trigger our own alarms in the future.

comment:25 Changed 5 months ago by https://id.mayfirst.org/jaimev

Here's an idea. If we can track outgoing connections on a range of ports and identify the initiating process and user efficiently, could we create an alert when the frequency of those connections exceeds a set threshold for that port range?

For example, is the pattern of a single users outgoing connections on ports 80 and 443 for things like site upgrades or cron jobs distinguishable from something like the above. That might be tough, the connections we're tracking above are hours apart and at varying intervals.

comment:26 Changed 5 months ago by https://id.mayfirst.org/jamie

I think that's an excellent idea. I'd suggest we start with:

  • A one liner or mf- style bash script that outputs to standard out on any match against outgoing connections to port 80 or port 443 or an arbitrary IP address
  • Next, add a configuration file (/etc/default/network-monitor) so we can add and remove arbitrary IP addresses via a file
  • Next, add a systemd configuration file to launch it. This is nice because it means the output will be available via journalctl
  • Next we examine the output periodically to look for patterns we can use to identify abuse

comment:27 Changed 5 months ago by https://id.mayfirst.org/jamie

Jaime and I completed most of these steps - documented here: track outgoing connections.

comment:28 Changed 5 months ago by https://id.mayfirst.org/jamila

Thank you for all this work on mitigation! Ossie isn't on the lists right now, but Rose still is.

comment:29 Changed 5 months ago by https://id.mayfirst.org/jaimev

Ok, thanks for the update jamila. I'm running our new mf-ip-track-outgoing-connections service on Rose now. I'll check back in a few hours and see if we caught something.

comment:30 Changed 5 months ago by https://id.mayfirst.org/jaimev

I've identified the source of the malware activity . It is an outdated wordpress site on rose. I've sent the member an e-mail about it and hopefully will get a response soon.

comment:31 Changed 5 months ago by https://id.mayfirst.org/jamila

Hi all, Rose is still on the CBL. Is there still malware?

comment:32 Changed 5 months ago by https://id.mayfirst.org/jaimev

I disabled the affected site and it looks like abuseat.org has detected any recent connections to its sinkhole servers since then. I've requested removal of rose's ip now.

Please login to add comments to this ticket.

Note: See TracTickets for help on using tickets.