Opened 5 years ago

Last modified 4 years ago

#8625 assigned Bug/Something is broken

CIMAC

Reported by: cimacnoticias@… Owned by: https://id.mayfirst.org/ross
Priority: Urgent Component: Tech
Keywords: mx25.mayfirst.org sluggishness Cc: cesar.marlop@…, erq@…
Sensitive: no

Description

Buenas tardes!

A quien corresponda, la página cimacnoticias.com.mx el día de hoy ha tenido una serie de problemas desde la mañana, ya que en vez de aparecer la página aparece ISP CPOMEGA ERROR 503! SERVICE UNAVAILABLE.

Por lo que les pido que chequen que pasa, por favor!

Estare atenta a su respuesta!

Reciban un saludo cordial de todo el equipo de CIMAC

Change History (19)

comment:1 Changed 5 years ago by https://id.mayfirst.org/erq

  • Cc cesar.marlop@… coordinacioncimac@… added
  • Owner set to https://id.mayfirst.org/erq
  • Status changed from new to assigned

hola, el servidor mx25.mayfirst.org mostraba una sobrecarga en el uso de CPU principalmente provocada por procesos php-cgi. Detuve el servidor Apache y eliminé los procesos extras relacionados. Luego reinicié el servidor Apache. También he reiniciado el servidor mysql.

La carga ahora es normal y el sitio se muestra adecuadamente.

Sin embargo, por la respuesta que ofreció el servidor me parece que la razón del problema puede ser un incremento en el número de visitantes al sitio o bien un ataque de denegación de servicio. El equipo monitoreará el servidor durante el día.

Saludos Enrique

comment:2 Changed 5 years ago by https://id.mayfirst.org/erq

  • Owner changed from https://id.mayfirst.org/erq to https://id.mayfirst.org/ross

yesterday at night, CIMAC reported on #8674 one of their websites to be down. Today I found the mx25.mayfirst.org server with a load average over 100.

I followed the same process as described before in this ticket:

  1. stopped Apache server
  2. killed all php-cgi processes
  3. started Apache server
  4. restarted mysql server

Now the websites are back on line, the remaining questions are:

  • is there a configuration of server or of drupal installations that could be wrong or adjusted to avoid this situation?
  • are those excessive processes legit internet visitors or is it an DDoS attack?
  • in any case what we can do to help CIMAC

I'm reassigning to Ross hoping you can help us. thanks Enrique

comment:3 Changed 5 years ago by https://id.mayfirst.org/erq

  • Priority changed from Medium to High

comment:4 Changed 5 years ago by https://id.mayfirst.org/ross

  • Keywords mx25.mayfirst.org sluggishness added

Enrique,

The question of tuning a server is a complicated one since we don't know exactly what happened or what the state of the server was when CIMAC went down. It would help if we can catch more information when the problem happens. In the future if you could capture the output of vmstat 1 that might be good. Until we know where the bottleneck is, it's pretty hard to answer the question.

mx25 does have a relatively small amount of RAM at 2G. You might want to allocate another gig or two to the server which would certainly help if the problem was disk i/o.

~/ross

comment:5 Changed 5 years ago by https://id.mayfirst.org/cimac

Hola!

Desde hoy en la mañana hemos tenido problemas de acceso al sitio. Situación similar a la de días anteriores. Comentan que les avisáramos cuando este el problema para que pudieran ver a detalle el problema.

Nos pueden apoyar y decirnos lo que sucede.

De antemano gracias y en espera de sus comentarios

comment:6 Changed 5 years ago by https://id.mayfirst.org/cimac

  • Cc erq@… added; coordinacioncimac@… removed
  • Priority changed from High to Urgent

comment:7 Changed 5 years ago by https://id.mayfirst.org/ross

Hola,

No estoy seguro de entender lo que está pasando aquí. Me aparecen mensajes de error ispcpomega en su sitio. ISPCPOMEGA parece que hay un sistema de gestión que normalmente no usamos así que no está claro cómo se está gestionando. ¿Tiene usted alguna idea de este sistema?

~/ross

comment:8 Changed 5 years ago by https://id.mayfirst.org/erq

hi ross, {{{vmstat 1}} offers this output http://paste.debian.net/88590/ and top this other one http://paste.debian.net/88591/

I believe we need more RAM, just as you mentioned, there is 81.7%wa and almost all RAM is in use. Could you help us allocating 2GB more?

comment:9 Changed 5 years ago by https://id.mayfirst.org/ross

Erq,

Could you please explain ispcpomega?

~/ross

comment:10 Changed 5 years ago by https://id.mayfirst.org/erq

ISPomega is the control panel installed in this server. And it's login url can be found here http://mx25.laneta.apc.org/ I think i don't have the user name and password, but I'm sure CIMAC does. Do you need it?

comment:11 follow-up: Changed 5 years ago by https://id.mayfirst.org/ross

The problem is that I'm getting errors from the ispomega service and not from our apache server. Even with the current wait times that you mention, the site should still load, I believe. Since other sites load on the server http://bocadepolen.org/web/ <-- for example.

However, there is also a problem with spam on gaspar under the cesar user. I think this is a user account you use, so we really need to track down that problem as well. I think it's related to wordpress.

comment:12 follow-up: Changed 5 years ago by https://id.mayfirst.org/ross

Enrique,

So it looks like something is wrong with cimac generally, but more specifically with the virtual-user vu2001. I don't know what this virtual user is supposed to be doing, but currently it runs over 100 processes which seems super excessive.

~/ross

comment:13 in reply to: ↑ 11 Changed 5 years ago by https://id.mayfirst.org/erq

Replying to https://id.mayfirst.org/ross:

The problem is that I'm getting errors from the ispomega service and not from our apache server. Even with the current wait times that you mention, the site should still load, I believe. Since other sites load on the server http://bocadepolen.org/web/ <-- for example.

Hi ross, I think http://bocadepolen.org/web is hosted on gaspar.mayfirst.org, while cimacnoticias.com.mx is hosted on mx25.mayfirst.org, that is two different virtual servers.

However, there is also a problem with spam on gaspar under the cesar user. I think this is a user account you use, so we really need to track down that problem as well. I think it's related to wordpress.

In the other hand, I think user 'cesar' is not related to http://cimacnoticias.com.mx/ actually mx25.mayfirst.org is a VPS under CIMAC membership.

So I think this are not related problems.

comment:14 in reply to: ↑ 12 Changed 5 years ago by https://id.mayfirst.org/erq

Hi ross, I found those lots of users 'vu2001', I stopped ispcp_daemon and apache and killed all php-cgi processes, then started them back /etc/init.d/ispcp_daemon start and /etc/init.d/ispcp_daemon start

Now server is again overloaded with more than 10% on load average and only 50MB left free of RAM. I think we need to expand 2GB more of RAM for mx25. I dont know this process, could you help CIMAC on this?

comment:15 Changed 5 years ago by https://id.mayfirst.org/ross

Enrique,

We can give more RAM, but it's not clear that this resolves the problem. How do we know that the vu2001 user wont just start eating up the remaining resources?

Also, doing this requires a server reboot. I can do this tonight, but I would like to have some certainty that this will help.

~/ross

comment:16 Changed 5 years ago by https://id.mayfirst.org/erq

Hola Cesar, Ross y yo hemos encontrado en nuestro análisis de la bitácora de visitas del servidor web que entre 16/Mar/2014 06:26 y 19/Mar/2014 23:25 existe una cantidad extraordinaria de solicitudes hechas desde algunas direcciones IP:

    506 159.16.250.34
    524 65.52.110.184
    530 189.137.153.170
    543 65.52.110.240
    557 50.16.42.155
    573 66.249.85.86
    613 66.249.65.187
    621 66.249.65.219
    631 66.249.65.155
    641 66.249.82.42
    666 157.56.229.245
    673 157.55.34.177
    677 66.249.88.86
    799 209.85.238.122
    819 157.55.32.77
    849 65.55.219.95
   1046 65.55.219.91
   1463 178.255.215.89
   1503 208.115.111.66
   1613 208.115.113.83
   1633 100.43.83.137
   1645 66.249.65.153
   1653 66.249.65.185
   1701 66.249.65.217
   2109 95.108.158.238
   3796 200.27.66.50
   5522 201.137.109.101
   9274 5.10.83.24

Plasmaré un análisis de dichas IPs mañana para ver qué conclusiones podemos derivar de ahí. Por otro lado, Ross ampliará esta noche la cantidad de memoria RAM en el servidor para ofrecer una respuesta emergente y contener el problema que se ha presentado recientemente con cimacnoticias.com.mx Saludos

comment:17 Changed 5 years ago by https://id.mayfirst.org/cimac

Hola Enrique y Ross.

Gracias por la atención y por el aumento en memoria RAM mientras se resuelve el problema. Estamos pedientes del análisis y de las conclusiones que deriven de su diagnóstico.

Saludos.

comment:18 Changed 5 years ago by https://id.mayfirst.org/erq

Hola, ahora el registro incluye un par de días más, va de la fecha 16/Mar/2014 06:26 al 21/Mar/2014 21:13 Por lo que, como pueden ver los números han crecido. Será necesario revisar los datos de cada dirección IP. Lo haré en breve. Enrique

    833 66.249.65.155
    843 189.143.77.69
    874 65.55.219.95
    918 189.194.74.35
    943 50.16.42.155
    997 66.249.88.86
   1174 66.249.64.128
   1193 209.85.238.122
   1193 66.249.64.118
   1235 66.249.64.123
   1357 65.55.219.91
   1392 66.249.85.86
   1561 159.16.250.34
   1619 65.52.110.184
   2105 157.55.32.77
   2109 95.108.158.238
   2282 208.115.111.66
   2364 178.255.215.89
   2370 189.137.48.184
   2551 208.115.113.83
   2725 100.43.83.137
   2907 66.249.65.153
   2923 66.249.65.185
   3020 66.249.65.217
   4119 5.10.83.89
   5713 200.27.66.50
   6084 201.137.109.101
   9650 5.10.83.24

comment:19 Changed 4 years ago by https://id.mayfirst.org/cimac

Hola Compañeros!

Espero se encuentren bien. Queremos saber si tienen algún diagnóstico respecto a la cantidad extraordinaria de solicitudes hechas desde algunas direcciones IP a nuestro sitio.

Lo preguntamos porque hoy hemos tenido dificultades para el acceso al sitio y no quisiéramos tener problemas para actualizar nuestro sitio.

De antemano muchas gracias por las atenciones.

Please login to add comments to this ticket.

Note: See TracTickets for help on using tickets.