Opened 5 months ago

Last modified 4 months ago

#13632 assigned Bug/Something is broken

SSH login details

Reported by: https://id.mayfirst.org/workersliberty Owned by: https://id.mayfirst.org/jaimev
Priority: Urgent Component: Tech
Keywords: Cc: https://id.mayfirst.org/jamie
Sensitive: no

Description

Hi

We are currently fixing the site and updating the security patch. Is it possible to get the current SSH login details to complete the task.

Many thanks

Stephen

Change History (15)

comment:1 Changed 5 months ago by https://id.mayfirst.org/workersliberty

  • Priority changed from High to Urgent

comment:2 Changed 5 months ago by https://id.mayfirst.org/jaimev

  • Owner set to https://id.mayfirst.org/jaimev
  • Status changed from new to assigned

Hi, I've just re-enabled ssh access for the workersliberty user. You can see the list of users with ssh access here.

https://members.mayfirst.org/cp/index.php?area=hosting_order&service_id=3&hosting_order_id=563

ssh access is password based however you can edit the user to add an ssh key. There appear to be a few keys added to the ~/.ssh/authorized_keys file already so you may want to check that these are all valid and under your control.

Let me know if you need any additional help.

comment:3 Changed 5 months ago by https://id.mayfirst.org/workersliberty

Thanks. Hopefully this should be enough to get it resolved.

comment:4 Changed 5 months ago by https://id.mayfirst.org/jaimev

I'm seeing some files that look suspicious and at least two drupal install within the same web directory which is a security risk. Also seeing a mix of different file ownership. It looks like a cleanup of the web directory might be necessary before trying to enable the site again.

comment:5 Changed 5 months ago by https://id.mayfirst.org/workersliberty

Thanks Jaime

Yes our team is going through it all and trying to stop this from ever reocurring. Can we get ROOT access as well so that they can create the virtual host domain file?

Thanks Stephen

comment:6 Changed 5 months ago by https://id.mayfirst.org/workersliberty

Can we also have the FTP Host, Username and password - We have details for sftp but these are not the same.

comment:7 Changed 5 months ago by https://id.mayfirst.org/jaimev

Hi Stephen, sorry we don't allow root access for members on our shared servers however you can edit a web configuration that will be automatically inserted into your virtual host domain file through the control panel in the web configuration tab.

https://members.mayfirst.org/cp/index.php?area=hosting_order&hosting_order_id=563&service_id=7

Also there is no ftp acess, but you should be able to access your files through an sftp connection. Make sure all files inside the web directory are owned by the workersliberty user as this is the user assigned to own and execute your files in the web configuration tab.

comment:8 Changed 5 months ago by https://id.mayfirst.org/jaimev

  • Cc https://id.mayfirst.org/jamie added

Somehow the workersliberty has been injecting spam into the mail queue again even though the web configuration is disabled.

Checking the process tree if it looks like old processes were still open.

workers+ 20426  0.0  1.0 491148 82236 ?        S    Apr15   0:01  \_ php-fpm: pool workersliberty.org
workers+ 20784  0.0  0.0      0     0 ?        Z    Apr15   0:00  |   \_ [sh] <defunct>
workers+ 20489  0.0  0.9 488496 81700 ?        S    Apr15   0:04  \_ php-fpm: pool workersliberty.org
workers+ 21567  0.0  0.0      0     0 ?        Z    Apr15   0:00  |   \_ [sh] <defunct>
workers+ 24400  0.0  0.9 488100 79996 ?        S    Apr15   0:01  \_ php-fpm: pool workersliberty.org
workers+ 24974  0.0  0.0      0     0 ?        Z    Apr15   0:00      \_ [sh] <defunct>
workers+ 20788  0.0  0.0  25132  6200 ?        S    Apr15   0:33 /usr/sbin/sshd
workers+ 21571  0.0  0.0  25132  6108 ?        S    Apr15   2:27 /usr/sbin/sshd
workers+ 24980  0.0  0.0  25128  6088 ?        S    Apr15   0:34 /usr/local/apache/bin/httpd -DSSL
workers+ 21090  0.0  0.0  25132  6224 ?        S    Apr15   1:39 /usr/sbin/sshd
workers+ 21099  0.0  0.0  25132  6144 ?        S    Apr15   1:38 /usr/sbin/sshd
workers+ 21311  0.0  0.0  25132  6216 ?        S    Apr15   0:58 /usr/sbin/sshd
workers+ 21318  0.0  0.0  25132  5752 ?        S    Apr15   2:22 /usr/sbin/sshd

I've restarted php5-fpm and they have disappeared. I think we should change the workersliberty password and check on the .ssh/authorized_keys file.

comment:9 Changed 5 months ago by https://id.mayfirst.org/workersliberty

From our developers:

Today we managed to get the old site live and cleaned it up, but it has gone offline again with an error.

Can you ask MayFirst to check this issue and put the site back as we have found no issues from our side. Can you also find out from them as to why this has happened?

The virus

Can you ask

comment:10 Changed 5 months ago by https://id.mayfirst.org/jaimev

Referencing ticket ##13627 This has happened because your site was not updated with an important drupal security update that was announced widely on drupal lists. ​https://www.drupal.org/sa-core-2018-002

I can see that the version your developers have restored has still not been updated it is at version 8.3.5 and must be at least version 8.5.1 to include the security update.. It will likely be immediately hacked again. I am disabling the web configuration again until this can be done.

It is very important that you apply the security updates to your site. Let us know if you want us to attempt to do this for you.

comment:11 Changed 5 months ago by https://id.mayfirst.org/workersliberty

From the developers:

They took the site down without giving us time to install our upgraded version.

If you can tell mayfirst that we will be installing the upgraded version tomorrow morning and to allow us access to do this.

comment:12 Changed 5 months ago by https://id.mayfirst.org/jamie

  • Resolution set to fixed
  • Status changed from assigned to closed

Looks like you figured out how to enable the site and have ugpraded - thank you! I'm going to close this ticket but feel free to let us know if there are any other problems.

comment:13 Changed 5 months ago by https://id.mayfirst.org/jaimev

  • Resolution fixed deleted
  • Status changed from closed to assigned

There is a new security update available for your site now.

https://www.drupal.org/sa-core-2018-004

comment:14 Changed 4 months ago by https://id.mayfirst.org/jamie

I'm not sure if it's related to the compromise or not, but I noticed that a workersliberty PHP process was consuming a huge amount of RAM on the shared server albizu:

1 albizu:~# ps -e -o rss,cmd | sort -n | tail
54768 php-fpm: pool workersliberty.org                                        
79340 /usr/bin/perl -T /usr/sbin/spampd --pid=/var/run/spampd/spampd.pid --tagall --port=10027 --host=127.0.0.1 --relayport=10025 --relayhost=127.0.0.1 --children=3 --logsock=unix --set-envelope-from --maxsize=2048 --user=spampd --group=spampd
85260 php-fpm: pool workersliberty.org                                        
89580 /usr/bin/perl -T /usr/sbin/spampd --pid=/var/run/spampd/spampd.pid --tagall --port=10027 --host=127.0.0.1 --relayport=10025 --relayhost=127.0.0.1 --children=3 --logsock=unix --set-envelope-from --maxsize=2048 --user=spampd --group=spampd
89760 /usr/bin/perl -T /usr/sbin/spampd --pid=/var/run/spampd/spampd.pid --tagall --port=10027 --host=127.0.0.1 --relayport=10025 --relayhost=127.0.0.1 --children=3 --logsock=unix --set-envelope-from --maxsize=2048 --user=spampd --group=spampd
90560 /usr/bin/perl -T /usr/sbin/spampd --pid=/var/run/spampd/spampd.pid --tagall --port=10027 --host=127.0.0.1 --relayport=10025 --relayhost=127.0.0.1 --children=3 --logsock=unix --set-envelope-from --maxsize=2048 --user=spampd --group=spampd
108084 php-fpm: pool workersliberty.org                                        
422644 /usr/sbin/mysqld --basedir=/usr --datadir=/var/lib/mysql --plugin-dir=/usr/lib/mysql/plugin --user=mysql --log-error=/var/log/mysql/error.log --open-files-limit=65536 --pid-file=/var/run/mysqld/mysqld.pid --socket=/var/run/mysqld/mysqld.sock --port=3306
647992 /usr/sbin/clamd --foreground=true
5348096 php-fpm: pool workersliberty.org                                        
0 albizu:~# 

This shows workersliberty PHP process consuming 5 GB of RAM (that's more than half the available RAM for the entire server).

I didn't notice anything obviously amiss with the process:

0 albizu:~# ls -l /proc/30532/cwd
lrwxrwxrwx 1 workersliberty workersliberty 0 May 21 09:31 /proc/30532/cwd -> /home/members/workersliberty/sites/workersliberty.org/web
0 albizu:~# ls -l /proc/30532/exe
lrwxrwxrwx 1 workersliberty workersliberty 0 May 21 09:09 /proc/30532/exe -> /usr/sbin/php5-fpm
0 albizu:~# cat /proc/30532/environ 
0 albizu:~# 

However, I did notice that workersliberty is still not upgraded to the most recent (security) release.

I see a lot of differences between the installed version of the stock Drupal 8 version - but they all seem like developer customizations, not compromises.

We really need this site upgraded - can you please check in with your developers about upgrading it? If we upgrade it for you it seems like you may lose some of your customizations.

(also I restarted php5-fpm to regain the memory consumed by the single process.)

comment:15 Changed 4 months ago by https://id.mayfirst.org/jamie

I noticed the memory ballooned again. This should not be allowed :(.

So, I added the following to /etc/php5/fpm/pool.d/workersliberty.org.conf:

php_admin_value[memory_limit] = 512M

And restarted php5-fpm. I will test to see if this keeps the memory usage down (see post here).

Please login to add comments to this ticket.

Note: See TracTickets for help on using tickets.