Opened 5 months ago

Closed 4 months ago

#13571 closed Bug/Something is broken (fixed)

Help with Site

Reported by: https://id.mayfirst.org/gvshp Owned by: https://id.mayfirst.org/jaimev
Priority: High Component: Tech
Keywords: wordpress compromise Cc: smoskowitz@…, rogermanningnyc@…, https://id.mayfirst.org/jamie
Sensitive: no

Description

Hello:

One of our sites, http://gvshp.org/lpc/ seems to be infected with malware. Is there anything you can recommend we do here? Also, we would like to secure https for all our site. Please advise.

thanks, Sam

Change History (21)

comment:1 Changed 5 months ago by https://id.mayfirst.org/gvshp

We are also hoping you could see what part, the files or the database, is affected. Thanks.

comment:2 Changed 5 months ago by https://id.mayfirst.org/jaimev

  • Keywords wordpress compromise added
  • Owner set to https://id.mayfirst.org/jaimev
  • Status changed from new to assigned

Using wp-cli I was able to verify the same.

0 gvshp@chavez:~/gvshp.org/web/lpc$ wp core verify-checksums
Warning: File doesn't exist: index.php
Warning: File doesn't verify against checksum: wp-settings.php
Warning: File doesn't verify against checksum: wp-load.php
Warning: File doesn't exist: wp-admin/index.php
Warning: File should not exist: wp-includes/index.php
Warning: File should not exist: wp-sql.php
Warning: File should not exist: wp-xml.php
Warning: File should not exist: wp-admin/index__.php
Warning: File should not exist: wp-admin/index.html
Warning: File should not exist: wp-admin/wp-content/plugins/wordpress-support/5308fcf6531d6928afb924b82eab9468.css
Error: WordPress installation doesn't verify against checksums.

The above core wordpress files do show signs of malware. We should check the rest of the files , the database and the user list for other signs of compromise. We have a script that helps us automate parts of this and will attempt to reinstall a clean wordpress and restore your site database at files. However if any anomalies are found in the wordpress theme it will leave you with the default wordpress theme and you'll have to reinstall your theme after auditing it manually. Would you like me to attempt to run our wordpress recovery script on your site?

For security reasons each hosting order site runs as its own user but I see you have at least 3 wordpress installs running under the same web folder. We don't recommend this practice as it allows any compromised site to overwrite the others. For now the entire lpc folder which is a wordpress install in itself should be moved to your home folder to stop malware activity from writing to other files. Other files and folders should also be inspected.

comment:3 follow-up: Changed 5 months ago by https://id.mayfirst.org/gvshp

Thanks. Would you be able to move the other wordpress sites away from the compromised one? We have the custom theme files so yes, please run the script that would check the files for compromise and then attempt to reinstall and restore the files. We can manually fix afterward and update the missing files. The consultant working on the site also noted:

1) Site file and MySql backups: /home/members/gvshp/sites/gvshp.org/users/gvshp/baks_compressed_and_sql 2) At the end of last week a temporary 'index.html' file was placed in the WordPress root and admin directories and the respective 'index.php' files re-named to 'index.php'. 3) In November 2017 vulnerable 'TimThumb' functionality which was removed from the outdated theme.

comment:4 in reply to: ↑ 3 Changed 5 months ago by https://id.mayfirst.org/jaimev

Replying to https://id.mayfirst.org/gvshp:

Thanks. Would you be able to move the other wordpress sites away from the compromised one?

My suggestion would be to create separate hosting orders for each wordpress site using subdomains of your original domain.

So http://gvshp.org/lpc/ could become http://lpc.gvshp.org

http://www.gvshp.org/blog could become http://blog.gvshp.org

http://www.gvshp.org/buildingblocks/wp/ could become http://buildingblocks.gvshp.org

You could then add extra redirects or rewrites to the web configuration for gvshp.org to ensure that urls to the old paths resolve to the correct site.

comment:5 Changed 5 months ago by https://id.mayfirst.org/gvshp

Thanks. My concern with the subdomains is we have so many links to the lpc sie and blog that it would be nearly impossible to go back and change all or most of the links or do redirects to all of them.

comment:6 Changed 5 months ago by https://id.mayfirst.org/jaimev

Are the majority of those links within the site itself? We could do a text based find and replace across all files and the wordpress databases. I don't think you need to create redirects for all possible links, just those that initiate with a pattern so that they are substituted with the new pattern.

comment:7 Changed 5 months ago by https://id.mayfirst.org/gvshp

Yes, most of the links are within the site. Can you Please run the script that would check the files for compromise and then attempt to reinstall and restore the files? We want to try and get that site back and up as soon as possible. Thanks.

comment:8 Changed 5 months ago by https://id.mayfirst.org/jaimev

Ok, I will try this now.

comment:9 Changed 5 months ago by https://id.mayfirst.org/gvshp

Thanks, please let me know when complete.

comment:10 Changed 5 months ago by https://id.mayfirst.org/jaimev

I've run the script earlier. Your theme and the plugins a-to-z-category-listing, and share-this could not be restored from repositories. I've tried to scan the database backup for signs of any javascript that might be inserted in your posts. I do get some results.

0 gvshp@chavez:~$ zgrep -o -E '<script.................................................................................................................................................................................................................' gvshp.org/backups/gvshp_lpc_wp.sql.gz 
<script charset=\\\"utf-8\\\" type=\\\"text/javascript\\\">var switchTo5x=true;</script>\r\n<script charset=\\\"utf-8\\\" type=\\\"text/javascript\\\" src=\\\"http://w.sharethis.com/button/buttons.js\\\"></script>\r\
<script charset=\\\"utf-8\\\" type=\\\"text/javascript\\\">stLight.options({\\\"publisher\\\":\\\"wp.19a79590-783b-4f54-9d76-0098abb6f50e\\\"});var st_type=\\\"wordpress4.4.2\\\";</script>\r\n','yes'),(204,'st_sent',
<script type=\"text/javascript\">var x1qfbpb51iafc7a;(function(d, t) {\nvar s = d.createElement(t), options = {\n\'userName\':\'gvshp\',\n\'formHash\':\'x1qfbpb51iafc7a\',\n\'autoResize\':true,\n\'height\':\'623\',\n

But I have been unable to locate that code in the current db. It may have belonged to one of the plugins that was not restored. I'm not sure.

comment:11 follow-up: Changed 5 months ago by https://id.mayfirst.org/gvshp

  • Cc rogermanningnyc@… added

hi,

The gvshp web 'consultant' mentioned above here.

I see that the web configuration has a last modified date of 2018-03-28 16:03:53 Was there anything changed at that time?

Meanwhile, there are 2 lines relating to the gvshp.org/lpc directory: Redirect /lpc http://www.gvshp.org/_gvshp/preservation/lpcpage.htm Redirect /lpc/ http://www.gvshp.org/_gvshp/preservation/lpcpage.htm

Trying to visit that location in a browser caused the lpcpage.htm file's modified date/time to be that of the the attempted view.

Thoughts?

thanks, rm

comment:12 in reply to: ↑ 11 Changed 5 months ago by https://id.mayfirst.org/jaimev

I see that the web configuration has a last modified date of 2018-03-28 16:03:53 Was there anything changed at that time?

Hi, yes I disabled and then quickly re-enabled the web configuration at that time just trying to get a sense if all of the wp sites were in use there. With so many redirects I wasn't sure what was going on.

Meanwhile, there are 2 lines relating to the gvshp.org/lpc directory: Redirect /lpc http://www.gvshp.org/_gvshp/preservation/lpcpage.htm Redirect /lpc/ http://www.gvshp.org/_gvshp/preservation/lpcpage.htm

I didn't actually modify anything in the web configuration so all of those redirects were there before.

Trying to visit that location in a browser caused the lpcpage.htm file's modified date/time to be that of the the attempted view.

I'm not really sure of the history of this site or how it was setup before.

comment:13 Changed 5 months ago by https://id.mayfirst.org/gvshp

Thanks.

When you scanned the lpc WordPress site files, that included the 'wp-content/uploads' where the media files are right? How likely is it that image files etc may be infected?

Is is possible for anyone with shell access to run the scanning script(s) you use?

r

comment:14 Changed 5 months ago by https://id.mayfirst.org/jaimev

I'm assuming that a file that is actually an image file is a less probable source of compromise.

I scanned the uploads folder for any files that didn't terminate with the default list of extensions accepted by wordpress uploads and for any files that actually contained text regardless of extension. With ssh access you can do the same using the find command like this.

find "uploads" -type f ! \( -iname '*.png' -o -iname '*.gif' -o -iname '*.jpg' -o -iname '*.jpeg'  -o -iname '*.svg' -o -iname '*.tiff' -o -iname '*.mov' -o -iname '*.mp3' -o -iname '*.ogg' -o -iname '*.pdf' -o -iname '*.ico' \) -print
find "uploads" -type f -exec grep -Iq . {} \; -and -print

I found some results but they didn't look suspicous.

comment:15 follow-up: Changed 5 months ago by https://id.mayfirst.org/gvshp

hi MF,

We set up a new hosting order, lpc.gvshp.org, hosted on erica. We moved the WordPress installation you cleaned there, configured for https, and are rebuilding the theme. We deleted the files from the old location (gvshp.org/lpc/) and got it unblocked by google.

Questions:

1) Not seeing the mysql automatic backup 'backups' directory in the 'lpc.gvshp.org' directory

2) To be safe, it'd be great if you could you re-scan the files and database now at lpc.gvshp.org for infection etc.

I ran the 2 scripts you included above on lpc.gvshp.org/web/wp-content/uploads.

The first script found the following infected files which have been deleted. They were text files that required scrolling way down to see the hack code: uploads/2013/12/index.php, uploads/2013/08/index.php, uploads/index.php

The second script found the following image files that were actually HTML files with .jpg extension apparently created during a WordPress xml content file import. The HTML was a WordPress error message page saying the file couldn't be found etc. They didn't seem to contain any bad code: uploads/2010/03/designation-report.jpg, uploads/2010/02/24.jpg, uploads/2010/02/241.jpg, uploads/2010/02/25.jpg, uploads/2010/02/23.jpg

I also ran a few 'find web -type f -mtime -[various days]' searches dated after your WordPress re-install but didn't find anything suspicious.

Thanks! rm

Last edited 5 months ago by https://id.mayfirst.org/gvshp (previous) (diff)

comment:16 in reply to: ↑ 15 Changed 5 months ago by https://id.mayfirst.org/jaimev

  • Cc https://id.mayfirst.org/jamie added

I see what you mean. There do not appear to be any backups directory created on erica as of yet. This may have something to do with the upgrade from mysql to mariadb on erica. Copying jamie here to get his input.

I will rescan the new site again. Glad you caught those files above.

comment:17 Changed 5 months ago by https://id.mayfirst.org/jaimev

I have created a new ticket #13596 to track the missing backups directory issue

comment:18 Changed 5 months ago by https://id.mayfirst.org/jamie

  • Resolution set to fixed
  • Status changed from assigned to feedback

The problem is resolved - and you should see your database backups in the proper directory starting tomorrow morning.

If you need a copy of a backup today - please re-open the ticket and let us know. (All backups went into a system-wide backup directory that is not accessible to members, but admins can still retrieve them for you.)

comment:19 Changed 5 months ago by https://id.mayfirst.org/gvshp

  • Resolution fixed deleted
  • Status changed from feedback to assigned

Thanks for resolving the database backup directory issue.

Were you able to re-scan our site files and database now located at lpc.gvshp.org on erica?

comment:20 Changed 5 months ago by https://id.mayfirst.org/jaimev

So our wordpress restore script does a few things you can also do form the command line.

The first is to verify the checksums of the core wordpress install.

0 lpcgvshp@erica:~/lpc.gvshp.org/web$ wp core verify-checksums
Success: WordPress installation verifies against checksums.

Checking the uploads folder as you've already done returns a few files worth checking. These don't seem to contain anything harmful but if you don't need them I would delete them.

0 lpcgvshp@erica:~/lpc.gvshp.org/web/wp-content$ find "uploads" -type f ! \( -iname '*.png' -o -iname '*.gif' -o -iname '*.jpg' -o -iname '*.jpeg'  -o -iname '*.svg' -o -iname '*.tiff' -o -iname '*.mov' -o -iname '*.mp3' -o -iname '*.ogg' -o -iname '*.pdf' -o -iname '*.ico' \) -print
uploads/wpcf7_captcha/.htaccess
0 lpcgvshp@erica:~/lpc.gvshp.org/web/wp-content$ find "uploads" -type f -exec grep -Iq . {} \; -and -print
uploads/2010/03/designation-report.jpg
uploads/2010/02/24.jpg
uploads/2010/02/241.jpg
uploads/2010/02/25.jpg
uploads/2010/02/23.jpg
uploads/wpcf7_captcha/.htaccess

We also have the mf-find-suspicious-files script available to be run by users which is based on this perl script http://cbl.abuseat.org/findbot.pl This will not catch all compromises and will also return false positives. I ran the script on your themes folder and didn't turn up anything and also on your plugins folder where it flagged a few files that don't really look suspicious to me but I can't be sure.

0 lpcgvshp@erica:~/lpc.gvshp.org/web/wp-content$ mf-find-suspicious-files themes/
0 lpcgvshp@erica:~/lpc.gvshp.org/web/wp-content$ mf-find-suspicious-files plugins
plugins/google-analyticator/google-api-php-client/src/service/Google_Utils.php: Suspicious(base64_decode):    return base64_decode($b64);
plugins/backwpup/inc/class-destination-msazure.php: Suspicious(passthru):                       fpassthru( $blob->g
plugins/backwpup/inc/class-encryption-fallback.php: Suspicious(base64_decode): crypted = base64_decode( $no_pref
plugins/backwpup/inc/class-encryption-mcrypt.php: Suspicious(base64_decode): crypted = base64_decode( $no_pref
plugins/backwpup/inc/class-encryption-openssl.php: Suspicious(base64_decode): crypted = base64_decode( $no_pref
plugins/backwpup/inc/class-install.php: Suspicious(base64_decode): word' ] = base64_decode( $cfg[ 'h
plugins/backwpup/vendor/SwiftMailer/classes/Swift/Plugins/PopBeforeSmtpPlugin.php: Suspicious(fsockopen): $socket = fsockopen(
plugins/backwpup/vendor/SwiftMailer/classes/Swift/Transport/EsmtpTransport.php: Suspicious(EHLO): o perform EHLO instead *
plugins/backwpup/vendor/SwiftMailer/classes/Swift/Transport/AbstractSmtpTransport.php: Suspicious(MAIL FROM):  Send the MAIL FROM command *
plugins/backwpup/vendor/SwiftMailer/classes/Swift/Transport/EsmtpHandler.php: Suspicious(EHLO): which the EHLO greeting 
plugins/backwpup/vendor/SwiftMailer/classes/Swift/Transport/Esmtp/AuthHandler.php: Suspicious(EHLO): which the EHLO greeting 
plugins/backwpup/vendor/SwiftMailer/classes/Swift/Transport/Esmtp/Auth/NTLMAuthenticator.php: Suspicious(base64_decode): esponse = base64_decode(substr(tr
plugins/backwpup/vendor/SwiftMailer/classes/Swift/Transport/Esmtp/Auth/CramMd5Authenticator.php: Suspicious(base64_decode): allenge = base64_decode(substr($c
plugins/backwpup/vendor/WindowsAzure/Common/Internal/Authentication/SharedKeyAuthScheme.php: Suspicious(base64_decode): ignature, base64_decode($this->ac
plugins/backwpup/vendor/WindowsAzure/Common/Internal/Authentication/TableSharedKeyLiteAuthScheme.php: Suspicious(base64_decode): ignature, base64_decode($this->ac
plugins/backwpup/vendor/WindowsAzure/Common/Internal/StorageServiceSettings.php: Suspicious(base64_decode):        // base64_decode will retu
plugins/backwpup/vendor/WindowsAzure/Common/Internal/Resources.php: Suspicious(base64_decode): he check 'base64_decode(<user_acc
plugins/backwpup/vendor/WindowsAzure/Blob/Models/ListBlobBlocksResult.php: Suspicious(base64_decode):  $entries[base64_decode($value['N

comment:21 Changed 4 months ago by https://id.mayfirst.org/gvshp

  • Resolution set to fixed
  • Status changed from assigned to closed

Thanks for the scan and scripts. I got the same results with the 'uploads' directory and those files checked out ok. The the plugins referred to in the perl script results should be ok, but we've had them deactivated anyway and plan to replace them.

thanks again rm

Please login to add comments to this ticket.

Note: See TracTickets for help on using tickets.