Opened 6 years ago

Closed 6 years ago

#7039 closed Bug/Something is broken (fixed)

Sites on Mandela not loading -- mandela review and tuning

Reported by: https://id.mayfirst.org/wedivestweb Owned by: https://id.mayfirst.org/dkg
Priority: Urgent Component: Tech
Keywords: mandela.mayfirst.org kvm rsyslog grub swap Cc:
Sensitive: no

Description

2 sites I have on Mandela won't load (adalahny.org and wespac.org). Adalah did load once after a long wait but won't re-load. I also was not able to login to this support site w/ either user name. ds

Change History (9)

comment:1 Changed 6 years ago by https://id.mayfirst.org/wedivestweb

Same issue as #7038 which was opened 3 hours ago.

comment:2 Changed 6 years ago by https://id.mayfirst.org/dkg

  • Keywords mandela.mayfirst.org kvm added

Yep, it looks like there was an out-of-memory condition on mandela relatively recently. I suspect we should increase the amount of RAM available to mandela. ken (mandela's host) has 32GiB of RAM, and that RAM is allocated among its 10 guests. Mandela currently has 3GiB allocated, plus 1GiB of swap, but ken only has 23 GiB explicitly allocated. i think we should double mandela's ram to 6GiB and remove the swap.

comment:3 Changed 6 years ago by https://id.mayfirst.org/dkg

  • Owner set to https://id.mayfirst.org/dkg
  • Status changed from new to assigned

this is going to require a reboot of mandela, which will happen shortly. I am running some upgrades on mandela before doing the reboot.

comment:4 Changed 6 years ago by https://id.mayfirst.org/dkg

rebooting now. there's a filesystem check. and it looks like mandela's bootloader isn't properly configured for the serial console either. more to come...

comment:5 Changed 6 years ago by https://id.mayfirst.org/dkg

  • Keywords rsyslog grub added
  • Summary changed from Sites on Mandela not loading to Sites on Mandela not loading -- mandela review and tuning

mandela is up.

I added the following two lines to the end of mandela:/etc/default/grub:

GRUB_TERMINAL=serial
GRUB_SERIAL_COMMAND='serial --speed=112500'

and re-ran update-grub, so that should fix the bootloader.

I also noticed the following messages during boot:

Starting SpamAssassin Mail Filter Daemon: Mar 30 12:11:26.610 [1411] warn: logger: failed to add syslog method: logger: syslog initialization failed
spamd.

and sure enough, syslog isn't running on mandela. i'll post more about that here as i figure it out.

comment:6 Changed 6 years ago by https://id.mayfirst.org/dkg

It looks like there is no initscript for rsyslog:

1 mandela:~# dpkg -S '/etc/init.d/*sys*log*'
rsyslog: /etc/init.d/rsyslog
sysklogd: /etc/init.d/sysklogd
0 mandela:~# dpkg -S /etc/init.d/*sys*log*
dpkg: /etc/init.d/rsyslog.dpkg-dist not found.
sysklogd: /etc/init.d/sysklogd
1 mandela:~# dpkg -l '*sys*log*'
Desired=Unknown/Install/Remove/Purge/Hold
| Status=Not/Inst/Conf-files/Unpacked/halF-conf/Half-inst/trig-aWait/Trig-pend
|/ Err?=(none)/Reinst-required (Status,Err: uppercase=bad)
||/ Name           Version        Description
+++-==============-==============-============================================
ii  libparse-syslo 1.10-1         Perl module for parsing syslog entries
un  libsys-syslog- <none>         (no description available)
un  libsyslog-ruby <none>         (no description available)
un  libsyslog-ruby <none>         (no description available)
ii  libunix-syslog 1.1-2          Perl interface to the UNIX syslog(3) calls
ii  rsyslog        4.6.4-2        enhanced multi-threaded syslogd
un  rsyslog-doc    <none>         (no description available)
un  rsyslog-gnutls <none>         (no description available)
un  rsyslog-gssapi <none>         (no description available)
un  rsyslog-mysql  <none>         (no description available)
un  rsyslog-pgsql  <none>         (no description available)
un  rsyslog-relp   <none>         (no description available)
rc  sysklogd       1.5-6          System Logging Daemon
un  syslog-summary <none>         (no description available)
un  syslogd        <none>         (no description available)
un  system-log-dae <none>         (no description available)
0 mandela:~# 

I'm going to try to remove DebianPackage:sysklogd and reinstall DebianPackage:rsyslog to see if we can resolve this.

comment:7 Changed 6 years ago by https://id.mayfirst.org/dkg

I checked the sha1sum of mandela:/etc/init.d/rsyslog.dpkg-dist against the sha1sum of /etc/init.d/rsyslog on another squeeze server, and they matched. so i set up the initscript again:

mv /etc/init.d/rsyslog{.dpkg-dist,}
update-rc.d rsyslog defaults
insserv

and then tried to start the service:

0 mandela:/etc/init.d# service rsyslog start
Starting enhanced syslogd: rsyslogd rsyslogd: [origin software="rsyslogd" swVersion="4.6.4" x-pid="11810" x-info="http://www.rsyslog.com"] (re)start
 rsyslogd-2013:CONFIG ERROR: could not interpret master config file '/etc/rsyslog.conf'. [try http://www.rsyslog.com/e/2013 ]
 rsyslogd:EMERGENCY CONFIGURATION ACTIVATED - fix rsyslog config file!
.
0 mandela:/etc/init.d# 

It turned out that /etc/rsyslog.conf did not exist, but /etc/rsyslog.conf.dpkg-dist does exist, and it has the same sha1sum as /etc/rsyslog.conf on other squeeze systems. The same was also true for /etc/default/rsyslog, maddeningly.

So i did:

0 mandela:/etc/init.d# mv /etc/default/rsyslog{.dpkg-dist,}
0 mandela:/etc/init.d# mv /etc/rsyslog.conf{.dpkg-dist,}
0 mandela:/etc/init.d# service rsyslog restart
Stopping enhanced syslogd: rsyslogd already stopped.
Starting enhanced syslogd: rsyslogd.
0 mandela:/etc/init.d# 

I also ran freepuppet-run to make sure that our local configurations were up-to-date. I'm now doing a sweep for other packages that might be in the same situation, and then may want to do another restart just to make sure.

comment:8 Changed 6 years ago by https://id.mayfirst.org/dkg

I just purged several old packages on mandela that we no longer needed (including mysql-server-5.0) and cleaned up some dcc-client cruft (see #5825). I'm now going to restart mandela one more time to make sure that the boot process works as expected.

comment:9 Changed 6 years ago by https://id.mayfirst.org/dkg

  • Keywords swap added
  • Resolution set to fixed
  • Status changed from assigned to closed

I just did the reboot. i still didn't see the grub bootloader, but maybe i just wasn't paying attention.

mandela is back and running cleanly now, *with* rsyslog, with 6GiB of RAM, and no swap.

Please login to add comments to this ticket.

Note: See TracTickets for help on using tickets.