[[PageOutline]] = Jessie Stretch Upgrade Page = == Change to puppet .pp file == Each .pp file has the following lines: {{{ class { "mayfirst::m_unattended_upgrades": uu_origin_patterns => [ "a=stable", "n=jessie-backports" ], uu_upgrade_email => $parent } }}} Please remove the n=jessie-backports part so it simply reads: {{{ class { "mayfirst::m_unattended_upgrades": uu_origin_patterns => [ "a=stable" ], uu_upgrade_email => $parent } }}} == Purge ntp on virtual servers == This is not stricly a stretch upgrade task, but while upgrading''virtual guests'', please take the time to purge `ntp` (and ensure that /etc/ntp.conf). With ntp gone, freepuppet will configure systemd-timesyncd. Don't purge ntp on physical servers. == Upgrade == {{{ rm /etc/apt/source.lists.d/* sed -i "s/jessie/stretch/" /etc/apt/source.list apt update apt dist-upgrade freepuppet-run freepuppet-run # again apt dist-upgrade # again shutdown -h now # or shutdown -r now for physical hosts }}} == MOSH only == You make get dependency problems on the initial `apt dist-upgrade` requiring you to `apt remove python-apt lynx-cur` first. You will be prompted for the MySQL host. It is: localhost === Corrupted MySQL proc table === After upgrading, try: {{{ mysql mysql SHOW PROCEDURE STATUS; }}} If you get an error about mysql.proc being corrupted, run the following from the root bash shell: {{{ mysql_upgrade }}} === Errors with fail2ban === After the initial upgrade, dpkg reports an error restarting fail2ban. Next run `freepuppet-run` which will also fill the screen with errors. When complete re-run freepuppet-run and all should be swell again. === PHP and Moshes === When upgrading to stretch, php5-fpm will remain and php7.0-fpm will be added. Both will exist at the same time. Normally, the proper step would be to carefully transition the php configuration from /etc/php5/fpm/pool.d/site.conf to /etc/php/7.0/fpm/pool.d/sites.conf and then purge PHP5. However, on moshes, we are keeping PHP5 until December 31, 2018 to help with the transition of member web sites. Additionally, red will handle the transition by auto-converting all sites to the most recent version of PHP available, ''unless'' the following comment is present in the web configuration: {{{ # mfplphpversion: 5 }}} After upgrading a mosh, you must run `/usr/local/share/red/node/sbin/red-regenerate-web-config` If that comment is present, then the site will remain on PHP5 after `red-regenerate-web-config` is run. Otherwise the site will switch to using PHP7.0. == Configuration files == When prompted, always take the package maintainer's version. Our goal is to make the most minimal changes possible to package maintainers versions. The following configuration files are handled in puppet (with the stretch base version). If you take a package maintainer's version of a configuration file that is not in this list, please open a ticket to ensure we don't need to add it (don't worry, the original configuration file will be backed up). Config files safely in puppet (or we want package maintainers version) * /etc/ssh/sshd_config * /etc/ssh/ssh_config * /etc/systemd/journald.conf * /etc/systemd/timesyncd.conf * /etc/skel/.bashrc * /etc/login.defs * /etc/ssl/openssl.cnf * /etc/puppet/hiera.yaml * /etc/apt/apt.conf.d/50unattended-upgrades * /etc/apache2/envvars == Root gpg key and monkeysphere == All of our servers can easily ssh between themselves through the monkeysphere. For the monkeysphere to work, the root user on each machine has to maintain an ssh-agent. Prior to stretch, we could not load a monkeysphere key into ssh-agent that didn't have a password (so we set the password to 'monkeys' for all keys). With stretch, you can't load a key into ssh-agent in an automated way that ''has'' a password. Sigh. Also, you cannot automate the change of a gpg key password. As a result: upon upgrading sites to stretch, you must manually remove the password for the root user's gpg key: {{{ gpg --change-passphrase root@$(hostname).mayfirst.org }}} The current password is: monkeys. Leave the password empty when prompted to change it.. If you don't, you will get this warning when you run puppet: {{{ remote: Error: /bin/false returned 1 instead of one of [0] remote: Error: /Stage[main]/Mayfirst::M_minimal/M_gpg::Private_key[root]/Exec[admin:please-manually-change-passphrase-from-monkeys-to-empty-for-root-user-on-this-host]/returns: change from notrun to 0 failed: /bin/false returned 1 instead of one of [0] }}} == Silence == All the noisy output we use to see on a successful run has been eliminated. It's a bit jarring, but I think we'll get used to it: {{{ 0 septima:~# freepuppet-run Notice: Compiled catalog for septima.mayfirst.org in environment production in 1.72 seconds Info: Applying configuration version '1508332341' Notice: Applied catalog in 8.13 seconds 0 septima:~# }}} == Good bye syslog == With stretch, `syslog` has been purged. That means all services and programs that depend on it have been modified (fail2ban, many of the mf-scripts, etc). == Predictable Network Interfaces Names == With Stretch, network interfaces get [https://www.freedesktop.org/wiki/Software/systemd/PredictableNetworkInterfaceNames/ predictable interface names]. To keep our tried and true eth0, puppet will start adding a udev rule for stretch machines. (You would think systemd-network would allow us to simply add a .link file to /etc/systemd/network - but [https://serverfault.com/questions/837454/interface-will-not-rename-under-systemd alas this doesn't seem to work without also adding a udev rule - so why bother with the .link file?].) IMPORTANT: This udev rule is added automatically and should work fine for most non-physical servers (it takes the mac address from puppet's factor and names it eth0), but physical servers, servers running docker or any server with unusual network devices need a change to their .pp file or their networking will get screwed up since facter has no idea which mac address should be used for the real network device. For these cases change: {{{ class { "mayfirst::m_interface": } }}} To: {{{ class { "mayfirst::m_interface": mac => "03:AB:blah blah mac address" } }}} If you want to add additional network interfaces (not new IP addresses, but an actual new network device), you need to make the above change for the eth0 device and then add define statements: {{{ mayfirst::m_interface::name { "03:AB:blah blah mac address": iface_name => "eth1" } }}} == KVM Manager == On physical hosts, kvm-manager will start using systemd instead of runit to manage guests. So, instead of commands like `sv stop ` and `sv start `, in stretch we will use `systemctl start km@` and `systemctl stop km@` and `systemctl disable km@`. Note - we are using a systemd template file (/etc/systemd/system/km@.service) - that's why we have an @ sign in the service name - the part between the @ sign and the period is the name of the guest. To transition a physical host, follow these steps: * Upgrade the server to stretch * Run `freepuppet-run` * Run `mf-convert-kvm-manager-to-systemd` (this will simply ensure that the kvm guest environment variables will be used by systemd) * Use `sv` to stop all existing guests * Use `update-services --remove` to ensure all guests won't restart via sv on reboot. /etc/service should be empty. * Reboot * One by one, start guests via systemd: `systemctl start km@`. * If everything goes smoothly, enable them at boot time (`systemctl enable km@`)