Understanding why a MOSH or a Physical Server is under heavy load

When you find a MOSH or physical server is under heavy load it is often due to a single web site or user getting slammed.

Here are some tricks to find that user.

Resource Hog

The mf-resource-hog script will output the top CPU users by minute going back between 1 and 2 days.

You can modify it's behavior in a number of ways:

• mf-resource-hog cpu: outputs CPU usage. This is the defalt
• mf-resource-hog disk-read: outputs disk read in kB per user.
• mf-resource-hog disk-write: outputs disk writes in kB per user.
• mf-resource-hog tps: outputs transactions per second per block device (only useful on physical servers).

The second argument specifies an interval:

• mf-resource-hog cpu minute: outputs usage on a per-minute basis. This is the default.
• mf-resource-hog cpu hour: outputs usage on a per-hour basis.
• mf-resource-hog cpu day: outputs usage on a per-day basis.
• mf-resource-hog cpu month: outputs usage on a per-month basis.

If you want real-time continuous output, you can run the collector script:

mf-resource-hog-collector [cpu|disk-read|disk-write]

Or, you can check out man pidstat to read up on the underlying tool used to generate this data.

How does it work?

There are three scripts in all:

mf-resource-hog-collector

When run from a cron job (not a terminal), mf-resource-hog-collector runs pidstat or pidstat -d depending on whether cpu or disk-read/disk-write is passed. It runs for 60 seconds, then totals the usage statistics for each user in the output and writes this output to a file in /var/log/mfpl/resource-hog/minute.

When run from a terminal, it outputs the data to standard out instead (and limits output to the top 4 users).

This script runs every minute.

mf-resource-hog-consolidator

Once an hour, mf-resource-hog-consolidator runs. Every hour it averages the totals from /var/log/mfpl/resource-hog/minute and places them in /var/log/mfpl/resource-hog/hour.

At midnight, it averages the totals from /var/log/mfpl/resource-hog/hour and places them in /var/log/mfpl/resource-hog/day. It also deletes old files that have already been averaged.

On the first of the month, it averages the totals from /var/log/mfpl/resource-hog/day and places them in /var/log/mfpl/resource-hog/month.

mf-resource-hog

The mf-resource-hog script is the main user script. It simply displays the output of head -n4 /var/log/mfpl/resource-hog/[interval]/[resource].

sysstat

Another useful tool enabled on all MOSH'es is sysstat. It can provide a recent history of resource usage on the server. This information helps you determine if a resource constraint has only been happening recently, or has been on-going for some time.

It collects data via a cron job that runs sa1. To view the data, run:

sar

To see the data from yesterday:

sar -1

The sysstat commands won't break down resource usage on a per-user basis.