Opened 6 years ago

Closed 6 years ago

#7039 closed Bug/Something is broken (fixed)

Sites on Mandela not loading -- mandela review and tuning

Reported by: Daniel Strum Owned by: Daniel Kahn Gillmor
Priority: Urgent Component: Tech
Keywords: kvm rsyslog grub swap Cc:
Sensitive: no


2 sites I have on Mandela won't load ( and Adalah did load once after a long wait but won't re-load. I also was not able to login to this support site w/ either user name. ds

Change History (9)

comment:1 Changed 6 years ago by Daniel Strum

Same issue as #7038 which was opened 3 hours ago.

comment:2 Changed 6 years ago by Daniel Kahn Gillmor

Keywords: kvm added

Yep, it looks like there was an out-of-memory condition on mandela relatively recently. I suspect we should increase the amount of RAM available to mandela. ken (mandela's host) has 32GiB of RAM, and that RAM is allocated among its 10 guests. Mandela currently has 3GiB allocated, plus 1GiB of swap, but ken only has 23 GiB explicitly allocated. i think we should double mandela's ram to 6GiB and remove the swap.

comment:3 Changed 6 years ago by Daniel Kahn Gillmor

Owner: set to Daniel Kahn Gillmor
Status: newassigned

this is going to require a reboot of mandela, which will happen shortly. I am running some upgrades on mandela before doing the reboot.

comment:4 Changed 6 years ago by Daniel Kahn Gillmor

rebooting now. there's a filesystem check. and it looks like mandela's bootloader isn't properly configured for the serial console either. more to come...

comment:5 Changed 6 years ago by Daniel Kahn Gillmor

Keywords: rsyslog grub added
Summary: Sites on Mandela not loadingSites on Mandela not loading -- mandela review and tuning

mandela is up.

I added the following two lines to the end of mandela:/etc/default/grub:

GRUB_SERIAL_COMMAND='serial --speed=112500'

and re-ran update-grub, so that should fix the bootloader.

I also noticed the following messages during boot:

Starting SpamAssassin Mail Filter Daemon: Mar 30 12:11:26.610 [1411] warn: logger: failed to add syslog method: logger: syslog initialization failed

and sure enough, syslog isn't running on mandela. i'll post more about that here as i figure it out.

comment:6 Changed 6 years ago by Daniel Kahn Gillmor

It looks like there is no initscript for rsyslog:

1 mandela:~# dpkg -S '/etc/init.d/*sys*log*'
rsyslog: /etc/init.d/rsyslog
sysklogd: /etc/init.d/sysklogd
0 mandela:~# dpkg -S /etc/init.d/*sys*log*
dpkg: /etc/init.d/rsyslog.dpkg-dist not found.
sysklogd: /etc/init.d/sysklogd
1 mandela:~# dpkg -l '*sys*log*'
| Status=Not/Inst/Conf-files/Unpacked/halF-conf/Half-inst/trig-aWait/Trig-pend
|/ Err?=(none)/Reinst-required (Status,Err: uppercase=bad)
||/ Name           Version        Description
ii  libparse-syslo 1.10-1         Perl module for parsing syslog entries
un  libsys-syslog- <none>         (no description available)
un  libsyslog-ruby <none>         (no description available)
un  libsyslog-ruby <none>         (no description available)
ii  libunix-syslog 1.1-2          Perl interface to the UNIX syslog(3) calls
ii  rsyslog        4.6.4-2        enhanced multi-threaded syslogd
un  rsyslog-doc    <none>         (no description available)
un  rsyslog-gnutls <none>         (no description available)
un  rsyslog-gssapi <none>         (no description available)
un  rsyslog-mysql  <none>         (no description available)
un  rsyslog-pgsql  <none>         (no description available)
un  rsyslog-relp   <none>         (no description available)
rc  sysklogd       1.5-6          System Logging Daemon
un  syslog-summary <none>         (no description available)
un  syslogd        <none>         (no description available)
un  system-log-dae <none>         (no description available)
0 mandela:~# 

I'm going to try to remove DebianPackage:sysklogd and reinstall DebianPackage:rsyslog to see if we can resolve this.

comment:7 Changed 6 years ago by Daniel Kahn Gillmor

I checked the sha1sum of mandela:/etc/init.d/rsyslog.dpkg-dist against the sha1sum of /etc/init.d/rsyslog on another squeeze server, and they matched. so i set up the initscript again:

mv /etc/init.d/rsyslog{.dpkg-dist,}
update-rc.d rsyslog defaults

and then tried to start the service:

0 mandela:/etc/init.d# service rsyslog start
Starting enhanced syslogd: rsyslogd rsyslogd: [origin software="rsyslogd" swVersion="4.6.4" x-pid="11810" x-info=""] (re)start
 rsyslogd-2013:CONFIG ERROR: could not interpret master config file '/etc/rsyslog.conf'. [try ]
 rsyslogd:EMERGENCY CONFIGURATION ACTIVATED - fix rsyslog config file!
0 mandela:/etc/init.d# 

It turned out that /etc/rsyslog.conf did not exist, but /etc/rsyslog.conf.dpkg-dist does exist, and it has the same sha1sum as /etc/rsyslog.conf on other squeeze systems. The same was also true for /etc/default/rsyslog, maddeningly.

So i did:

0 mandela:/etc/init.d# mv /etc/default/rsyslog{.dpkg-dist,}
0 mandela:/etc/init.d# mv /etc/rsyslog.conf{.dpkg-dist,}
0 mandela:/etc/init.d# service rsyslog restart
Stopping enhanced syslogd: rsyslogd already stopped.
Starting enhanced syslogd: rsyslogd.
0 mandela:/etc/init.d# 

I also ran freepuppet-run to make sure that our local configurations were up-to-date. I'm now doing a sweep for other packages that might be in the same situation, and then may want to do another restart just to make sure.

comment:8 Changed 6 years ago by Daniel Kahn Gillmor

I just purged several old packages on mandela that we no longer needed (including mysql-server-5.0) and cleaned up some dcc-client cruft (see #5825). I'm now going to restart mandela one more time to make sure that the boot process works as expected.

comment:9 Changed 6 years ago by Daniel Kahn Gillmor

Keywords: swap added
Resolution: fixed
Status: assignedclosed

I just did the reboot. i still didn't see the grub bootloader, but maybe i just wasn't paying attention.

mandela is back and running cleanly now, *with* rsyslog, with 6GiB of RAM, and no swap.

Please login to add comments to this ticket.

Note: See TracTickets for help on using tickets.