Jessie Stretch Upgrade Page

Configuration files

When prompted, always take the package maintainer's version. Our goal is to make the most minimal changes possible to package maintainers versions.

The following configuration files are handled in puppet (with the stretch base version). If you take a package maintainer's version of a configuration file that is not in this list, please open a ticket to ensure we don't need to add it (don't worry, the original configuration file will be backed up).

Config files safely in puppet (or we want package maintainers version)

• /etc/ssh/sshd_config
• /etc/ssh/ssh_config
• /etc/systemd/journald.conf
• /etc/systemd/timesyncd.conf
• /etc/skel/.bashrc
• /etc/login.defs
• /etc/ssl/openssl.cnf
• /etc/puppet/hiera.yaml
• /etc/apt/apt.conf.d/50unattended-upgrades

Change to puppet .pp file

Each .pp file has the following lines:

  class { "mayfirst::m_unattended_upgrades":
    uu_origin_patterns => [ "a=stable", "n=jessie-backports" ],
    uu_upgrade_email => $parent
  }

Please remove the n=jessie-backports part so it simply reads:

  class { "mayfirst::m_unattended_upgrades":
    uu_origin_patterns => [ "a=stable" ],
    uu_upgrade_email => $parent
  }

Purge ntp on virtual servers

This is not stricly a stretch upgrade task, but while upgradingvirtual guests, please take the time to purge ntp (and ensure that /etc/ntp.conf). With ntp gone, freepuppet will configure systemd-timesyncd.

Don't purge ntp on physical servers.

Root gpg key and monkeysphere

All of our servers can easily ssh between themselves through the monkeysphere. For the monkeysphere to work, the root user on each machine has to maintain an ssh-agent.

Prior to stretch, we could not load a monkeysphere key into ssh-agent that didn't have a password (so we set the password to 'monkeys' for all keys). With stretch, you can't load a key into ssh-agent in an automated way that has a password. Sigh. Also, you cannot automate the change of a gpg key password.

As a result: upon upgrading sites to stretch, you must manually remove the password for the root user's gpg key:

gpg --change-passphrase root@$(hostname).mayfirst.org

The current password is: monkeys. Leave the password empty when prompted to change it..

If you don't, you will get this warning when you run puppet:

remote: Error: /bin/false returned 1 instead of one of [0]
remote: Error: /Stage[main]/Mayfirst::M_minimal/M_gpg::Private_key[root]/Exec[admin:please-manually-change-passphrase-from-monkeys-to-empty-for-root-user-on-this-host]/returns: change from notrun to 0 failed: /bin/false returned 1 instead of one of [0]

Silence

All the noisy output we use to see on a successful run has been eliminated. It's a bit jarring, but I think we'll get used to it:

0 septima:~# freepuppet-run 
Notice: Compiled catalog for septima.mayfirst.org in environment production in 1.72 seconds
Info: Applying configuration version '1508332341'
Notice: Applied catalog in 8.13 seconds
0 septima:~#

Good bye syslog

With stretch, syslog has been purged. That means all services and programs that depend on it have been modified (fail2ban, many of the mf-scripts, etc).

Predictable Network Interfaces Names

With Stretch, network interfaces get ​predictable interface names.

To keep our tried and true eth0, puppet will start adding a udev rule for stretch machines. (You would think systemd-network would allow us to simply add a .link file to /etc/systemd/network - but ​alas this doesn't seem to work without also adding a udev rule - so why bother with the .link file?.)

IMPORTANT: This udev rule is added automatically and should work fine for most non-physical servers (it takes the mac address from puppet's factor and names it eth0), but physical servers, servers running docker or any server with unusual network devices need a change to their .pp file or their networking will get screwed up since facter has no idea which mac address should be used for the real network device. For these cases change:

class { "mayfirst::m_interface": }

To:

class { "mayfirst::m_interface": 
  mac => "03:AB:blah blah mac address"
}

If you want to add additional network interfaces (not new IP addresses, but an actual new network device), you need to make the above change for the eth0 device and then add define statements:

mayfirst::m_interface::name { "03:AB:blah blah mac address":
  iface_name => "eth1"
}

KVM Manager

On physical hosts, kvm-manager will start using systemd instead of runit to manage guests.

So, instead of commands like sv stop <guest> and sv start <guest>, in stretch we will use systemctl start km@<guest> and systemctl stop km@<guest> and systemctl disable km@<guest>. Note - we are using a systemd template file (/etc/systemd/system/km@.service) - that's why we have an @ sign in the service name - the part between the @ sign and the period is the name of the guest.

To transition a physical host, follow these steps:

• Upgrade the server to stretch
• Run freepuppet-run
• Run mf-convert-kvm-manager-to-systemd (this will simply ensure that the kvm guest environment variables will be used by systemd)
• Use sv to stop all existing guests
• Use update-services --remove to ensure all guests won't restart via sv on reboot. /etc/service should be empty.
• Reboot
• One by one, start guests via systemd: systemctl start km@<guest>.
• If everything goes smoothly, enable them at boot time (systemctl enable km@<guest>)