Version 1 (modified by Jamie McClelland, 9 years ago) (diff)


Here Comes Trouble

Here Comes Trouble (htc) is a package for helping the support team with communications during an outage. It facilitates communicating service advisory messages to MF/PL members.

htc consists of the following components:

  • messages git repo - this is a git repository with three subdirectories:
    • templates - pre-written and translated templates for common service advisories
    • queue - messages queued for translation or approval
    • published - messages published, a directory for each year with a directory for each message in the corresponding year
  • trouble - a bash script that simplified the creation of new messages. trouble prompts you through the process of selecting a template, creating a new message with next available number, gpg signing the message, commiting and publishing
  • post-commit hooks
    • rss-publish - generates a new RSS feed based on the most recent 25 messages
    • html-publish - generates browseable html files for the complete set of messages
    • email-publish - based on the RSS feed, emails new messages to service-advisories email list
    • status-publish - based on the RSS feed, publishes new messages on various configured status-supported systems

htc is designed to require only a workstation with the appropriate (and reasonably up-to-date) software installed and Internet access between the workstation and (our server hosted in the UK).


The primary client component is a command line program called trouble.

trouble is a helper script that helps you edit a message based on existing templates, gpg sign the message, commit and push the message.


The message subcommand helps you create a service-advisory.

trouble message [--template <template-name>] [--search <server-search-string>] 
trouble message --resume
  • --template - specify the name of a pre-existing dual language templates
  • --search - if freepupet helper is in your $PATH, this string will be passed to freepuppet-helper ls:<server-search-string>, and the Servers Affected line in the service advisory will be pre-populated with the results
  • --resume - resume an queued message

The dns subcommand helps you manipulate DNS records commonly needed during an outage. It can either manipulate them via the red API (if Telehouse is available) or manipulate them directly on jobjoe (our secondary DNS server) if the red API is not available.

trouble dns [--disable-mail-relay <gil|paulo>] [--redirect-mayfirst-website]

  • --disable-mail-relay - disable the DNS record for the specified mail relay server
  • --redirect-mayfirst-website - replace the DNS record for {www.,} to point to jojobe, where it redirects to