Jessie Stretch Upgrade Page

Please track progress and notes on ticket #13853.

Upgrade

These are the general steps if you are NOT upgrading a MOSH (see below for MOSH's), but please read this full page before starting!

rm /etc/apt/sources.list.d/*
rm /etc/apt/preferences.d/*
sed -i "s/jessie/stretch/" /etc/apt/sources.list
apt update
apt dist-upgrade
gpg --change-passphrase root@$(hostname).mayfirst.org
freepuppet-run
freepuppet-run # again
apt dist-upgrade # again
shutdown -h now # or shutdown -r now for physical hosts

Change to puppet .pp file

Each .pp file has the following lines:

  class { "mayfirst::m_unattended_upgrades":
    uu_origin_patterns => [ "a=stable", "n=jessie-backports" ],
    uu_upgrade_email => $parent
  }

Please remove the n=jessie-backports part so it simply reads:

  class { "mayfirst::m_unattended_upgrades":
    uu_origin_patterns => [ "a=stable" ],
    uu_upgrade_email => $parent
  }

MOSH's

Upgrading a mosh? Great. It's simple now.

First, remove "jessie-backports" as described above, commit and push the latest puppet changes to the mosh.

Then, run: mf-dist-upgrade-mosh

It will warn you about problems and try to run the entire upgrade with minimal intervention.

See below for more details about what is happening.

Purge ntp on virtual servers

This is not stricly a stretch upgrade task, but while upgradingvirtual guests, please take the time to purge ntp (and ensure that /etc/ntp.conf). With ntp gone, freepuppet will configure systemd-timesyncd.

Don't purge ntp on physical servers.

mysql/mariadb

This upgrade will automatically transition the server from mysql to mariadb.

If you have the setting log_slow_queries set in /etc/mysql (which works with mysql) it will cause mariadb to fail to start.

Check with: grep -r log_slow_queries /etc/mysql and remove that setting if you find it.

Also, if you have any mysql users that created a password using the old format you are really screwed. You should fix those first or those usernames will stop working.

You can look for them with:

mysql -e "select * from user where char_length(Password) = 16;"  mysql

Reset passwords with (replace $USER and $NEWPASS):

SET PASSWORD FOR '$USER'@'localhost' = password('$NEWPASS');

MOSH only

You make get dependency problems on the initial apt dist-upgrade requiring you to apt remove python-apt lynx-cur first.

You will be prompted for the MySQL host. It is: localhost

Corrupted MySQL proc table

After upgrading, try:

mysql mysql
SHOW PROCEDURE STATUS;

If you get an error about mysql.proc being corrupted, run the following from the root bash shell:

mysql_upgrade

Errors with fail2ban

After the initial upgrade, dpkg reports an error restarting fail2ban.

Next run freepuppet-run which will also fill the screen with errors.

When complete re-run freepuppet-run and all should be swell again.

PHP and Moshes

When upgrading to stretch, php5-fpm will remain and php7.0-fpm will be added. Both will exist at the same time.

Normally, the proper step would be to carefully transition the php configuration from /etc/php5/fpm/pool.d/site.conf to /etc/php/7.0/fpm/pool.d/sites.conf and then purge PHP5. However, on moshes, we are keeping PHP5 until December 31, 2018 to help with the transition of member web sites.

Additionally, red will handle the transition by auto-converting all sites to the most recent version of PHP available, unless the following comment is present in the web configuration:

# mfplphpversion: 5

After upgrading a mosh, you must run /usr/local/share/red/node/sbin/red-regenerate-web-config

If that comment is present, then the site will remain on PHP5 after red-regenerate-web-config is run. Otherwise the site will switch to using PHP7.0.

Configuration files

When prompted, always take the package maintainer's version. Our goal is to make the most minimal changes possible to package maintainers versions.

The following configuration files are handled in puppet (with the stretch base version). If you take a package maintainer's version of a configuration file that is not in this list, please open a ticket to ensure we don't need to add it (don't worry, the original configuration file will be backed up).

Config files safely in puppet:

• /etc/ssh/sshd_config
• /etc/ssh/ssh_config
• /etc/systemd/journald.conf
• /etc/systemd/timesyncd.conf
• /etc/skel/.bashrc
• /etc/login.defs
• /etc/ssl/openssl.cnf
• /etc/puppet/hiera.yaml
• /etc/apt/apt.conf.d/50unattended-upgrades
• /etc/apache2/envvars
• /etc/ImageMagick-6/policy.xml
• /etc/cron.daily/spamassassin
• /etc/spamassassin/local.cf

Root gpg key and monkeysphere

All of our servers can easily ssh between themselves through the monkeysphere. For the monkeysphere to work, the root user on each machine has to maintain an ssh-agent.

Prior to stretch, we could not load a monkeysphere key into ssh-agent that didn't have a password (so we set the password to 'monkeys' for all keys). With stretch, you can't load a key into ssh-agent in an automated way that has a password. Sigh. Also, you cannot automate the change of a gpg key password.

As a result: upon upgrading sites to stretch, you must manually remove the password for the root user's gpg key:

gpg --change-passphrase root@$(hostname).mayfirst.org

The current password is: monkeys. Leave the password empty when prompted to change it..

If you don't, you will get this warning when you run puppet:

remote: Error: /bin/false returned 1 instead of one of [0]
remote: Error: /Stage[main]/Mayfirst::M_minimal/M_gpg::Private_key[root]/Exec[admin:please-manually-change-passphrase-from-monkeys-to-empty-for-root-user-on-this-host]/returns: change from notrun to 0 failed: /bin/false returned 1 instead of one of [0]

Silence

All the noisy output we use to see on a successful run has been eliminated. It's a bit jarring, but I think we'll get used to it:

0 septima:~# freepuppet-run 
Notice: Compiled catalog for septima.mayfirst.org in environment production in 1.72 seconds
Info: Applying configuration version '1508332341'
Notice: Applied catalog in 8.13 seconds
0 septima:~#

Good bye syslog

With stretch, syslog has been purged. That means all services and programs that depend on it have been modified (fail2ban, many of the mf-scripts, etc).

Predictable Network Interfaces Names

With Stretch, network interfaces get ​predictable interface names.

To keep our tried and true eth0, puppet will start adding a udev rule for stretch machines. (You would think systemd-network would allow us to simply add a .link file to /etc/systemd/network - but ​alas this doesn't seem to work without also adding a udev rule - so why bother with the .link file?.)

IMPORTANT: This udev rule is added automatically and should work fine for most non-physical servers (it takes the mac address from puppet's factor and names it eth0), but physical servers, servers running docker or any server with unusual network devices need a change to their .pp file or their networking will get screwed up since facter has no idea which mac address should be used for the real network device. For these cases change:

class { "mayfirst::m_interface": }

To:

class { "mayfirst::m_interface": 
  mac => "03:AB:blah blah mac address"
}

If you want to add additional network interfaces (not new IP addresses, but an actual new network device), you need to make the above change for the eth0 device and then add define statements:

mayfirst::m_interface::name { "03:AB:blah blah mac address":
  iface_name => "eth1"
}

KVM Manager

On physical hosts, kvm-manager will start using systemd instead of runit to manage guests. See the kvm-manager page for details.

To transition a physical host, follow these steps:

• Upgrade the server to stretch
• Run freepuppet-run
• Run mf-convert-kvm-manager-to-systemd (this will simply ensure that the kvm guest environment variables will be used by systemd)
• Add each kvm user to the kvm group (adduser chavez kvm)
• Use sv to stop all existing guests
• Use update-services --remove to ensure all guests won't restart via sv on reboot. /etc/service should be empty.
• Reboot
• One by one, start guests via systemd: systemctl start km@<guest>.
• If everything goes smoothly, enable them at boot time (systemctl enable km@<guest>)

Status

Physical Servers

• barron
• cleaver
• clr
• florence
• linda
• malaka
• medgar
• negri
• vilma
• wiwa

Specialty Servers

• allende
• assata
• banks
• cesar
• cleveland
• dee
• dolores
• fuller
• galeano
• gil
• gamiz
• hay
• iz
• jojobe
• kennedy
• kramer
• leslie
• lucius
• mcchesney
• minnie
• morales
• moses
• paul
• paulo
• peltier
• robideau
• rustin
• sankara
• septima
• stallman
• toussaint
• zimmermann

Member adminstered servers

• achebe
• acholonu
• algernon
• anna
• augusto
• baubo
• borges
• che
• deadeasy
• clyde
• dorvalina
• gabito
• gabo
• franz
• goofball
• juana
• ignatz
• kwame
• llosa
• marti
• neruda
• ngugi
• pianeta
• saadawi
• severo
• shaw
• simone
• sojourner
• yser

MOSH servers

• abernathy
• albizu
• amilcar
• annapurna
• annette
• baldwin
• banksy
• barghouti
• belen
• berrigan
• bety
• biko
• binh
• boggs
• brown
• buffy
• caceres
• cero
• chavez
• chelsea
• clara
• claudette
• colin
• daza
• debs
• didier
• dorothy
• douglass
• eagle
• ekpo
• elizabeth
• ella
• emma
• erica
• floriberto
• foucault
• gaspar
• ginsberg
• goldman
• graciela
• hammer
• hashmi
• herman
• howard
• ivan
• jacobs
• jones
• juanita
• julia
• june
• kahlo
• kerr
• kinoy
• larkin
• lewis
• magon
• malcolm
• mandela
• marisela
• marx
• menchu
• molina
• mumia
• nanny
• octavia
• ossie
• parsi
• pauline
• peery
• peri
• pietri
• proudhon
• ranciere
• randolph
• resnick
• rivera
• rodolpho
• roe
• roscoe
• rose
• rushdie
• sarah
• slaapbeen
• smith
• stokely
• stone
• stoney
• susana
• tresca
• vandana
• viewsic
• voltairine
• wolf
• woodhull
• yippie