Admin Training Manual

This page outlines the information needed to adminster May First/People Link servers.

1. Politics and organization
   1. Review the statement of unity, member agreement, and intentionality statement
   2. Familiarize with current political campaigns of the organization and brief history of past campaigns (see ​MAGNet and ​US Social Forum, ​Allied Media project...).
   3. Politics of free software
   4. Mexican Coop and Media Jumpstart: legal structures
   5. Introduction to leadership committee and ​membership meeting process, as well as commissions, work teams, volunteers and staff
   6. The support team
2. Identity: Many aspects of MF/PL system administration require a login which can be re-used in many places.
   1. Your OpenPGP key ensures that all members can communicate via private and authenticated email.
   2. ​Monkeysphere: converting your OpenPGP key into an ssh-enabled key allows us to grant you ssh access to servers easily and with a convenient method to revoke access if your key is compromised.
   3. May First/People Link accounts via the ​members control panel
      1. Create a membership: Creating your identity under your own membership allows you to continue with your identity even if you no longer provide system admin support
      2. Pick a user account to login to the control panel: this user account can be granted admin access - so you can access all accounts in the control panel. This password is the most sensitive - it should only be used for logging into the control panel. You might pick a username with a -cp suffix to it, like jamie-cp.
      3. Pick a user account as your public identity: via OpenID, you can re-use a single user account when logging into support.mayfirst.org or im.mayfirst.org and other services. Be sure to pick a good user account name and don't change it - since it will be public.
3. Secrets - MF/PL strives to be transparent and public, however, certain information is restricted
   1. ​Control panel - by adding your chosen user account to a red_admin_access table in the control panel database, you will be able to view and edit all aspects of all memberships and their services.
   2. By adding your monkeysphere user id to our puppet configuration you can be added to the list of people with root on all servers. We have a set of guidelines for people with root access, an ssh security policy as well as a draft policy on granting root access
   3. You may also have your OpenPGP key added to our keyringer configuration, which will allow you to decrypt our password file, which contains disk encryption passphrases.
   4. You will also need write access to our git repository.
4. Communication
   1. Once you join, you will be added to our CiviCRM outreach database automatically
   2. Join the ​support-team email list.
   3. Join the IRC Chat
   4. Install mumble and connect to our mumber server
   5. Ensure your browser works with ​live and ​mexcla.
5. The control panel
6. ​Ticket system
   1. Review our FAQ
   2. Tips on answering tickets
   3. Find unassigned tickets
   4. How to create a wiki page
   5. How to translate wiki pages
7. Infrastructure Overview
   1. Physical layout: where are the servers? Where are the data centers? Nearly all servers are hosted in either Telehouse or XO (about 5 - 8 physical servers in each location), both in Manhattan. See contact information for main providers
   2. Virtualization: almost all servers are KVM guests.
   3. We have three types of guests
      1. MOSH: This is an acronym that doesn't spell anything. It refers to guests that provide web and email hosting for most May First/People Members. These guests are connected to our control panel so members can easily add/modify/remove services.
      2. Dedicated MOSHes. These are just like regular MOSHes except they are dedicated to a single member. The dedication allows them to run mod_php instead of running php via fcgid and suexec, which is necessary on a shared machine for security reasons. mod_php runs much faster
      3. Single purpose: we have a number of guests that just provide one or a few dedicated services, such as our freeswitch server, DNS servers, etc.
   4. Puppet: our system for managing servers and services
      1. ​http://servers.mayfirst.org - web front end for following servers and assigning servers to support team
8. Monitoring
   1. ​Checking our Nagios Monitoring server
   2. Cacti - our traffic analyzer
   3. Our piwik installation - monitors our web site traffic
9. Here comes trouble - using our status notification system (​https://status.mayfirst.org)
10. How to install a new KVM guest
    1. How to allocate a new IP address
11. Using Shared Varnish faq/shared-varnish-server
12. Accessing console of our servers
    1. accessing_console_on_virtual_guest
    2. Accessing console on a physical machine
       1. Telehouse
       2. XO
       3. Web architects (jojobe)
13. Upgrading core version of Drupal
14. Renewing x509 certificates
15. Extending Hard disks
    1. Extending a logical volume
    2. Extending a disk for a guest
16. Changing resources allocated to guests via kvm manager files
17. Debugging common problems
    1. email and email list problems
    2. Debugging compromised web sites
    3. DNS problems
18. Suggested tools
    1. Ross has automation scripts for creating KVM guests.
    2. ​clusterssh - for connecting to multiple servers simultaneously, helpful for doing upgrades.
    3. ​irssi - with screen for perpetual irc connection.
    4. ​nagstamon - desktop client for nagios monitors.
    5. mf-go scripts - Helps for ssh defaults with auto-complete for all servers and starts a screen session identified by your username, also connects to consoles with tab completion of the physical host. You can get the repo with (some assembly required):

       git://lair.fifthhorseman.net/~rossg/mf-go