Version 1 (modified by 5 years ago) (diff) | ,
---|
Understanding why a MOSH is under heavy load
When you find a MOSH under heavy load it is often due to a single web site or user getting slammed.
Here are some tricks to find that user.
Resource Hog
The mf-resource-hog
script will output the top CPU users by minute going back between 1 and 2 days.
You can modify it's behavior in a number of ways:
mf-resource-hog cpu
: outputs CPU usage. This is the defaltmf-resource-hog disk-read
: outputs disk read in kB per user.mf-resource-hog disk-write
: outputs disk writes in kB per user.
The second argument specifies an interval:
mf-resource-hog cpu minute
: outputs usage on a per-minute basis. This is the default.mf-resource-hog cpu hour
: outputs usage on a per-hour basis.mf-resource-hog cpu day
: outputs usage on a per-day basis.mf-resource-hog cpu month
: outputs usage on a per-month basis.
If you want real-time continuous output, you can run the collector script:
mf-resource-hog-collector [cpu|disk-read|disk-write]
Or, you can check out man pidstat
to read up on the underlying tool used to generate this data.
How does it work?
There are three scripts in all:
mf-resource-hog-collector
When run from a cron job (not a terminal), mf-resource-hog-collector
runs pidstat
or pidstat -d
depending on whether cpu or disk-read/disk-write is passed. It runs for 60 seconds, then totals the usage statistics for each user in the output and writes this output to a file in /var/log/mfpl/resource-hog/minute.
When run from a terminal, it outputs the data to standard out instead (and limits output to the top 4 users).
This script runs every minute.
mf-resource-hog-consolidator
Once an hour, mf-resource-hog-consolidator
runs. Every hour it averages the totals from /var/log/mfpl/resource-hog/minute and places them in /var/log/mfpl/resource-hog/hour.
At midnight, it averages the totals from /var/log/mfpl/resource-hog/hour and places them in /var/log/mfpl/resource-hog/day. It also deletes old files that have already been averaged.
On the first of the month, it averages the totals from /var/log/mfpl/resource-hog/day and places them in /var/log/mfpl/resource-hog/month.
mf-resource-hog
The mf-resource-hog
script is the main user script. It simply displays the output of head -n4 /var/log/mfpl/resource-hog/[interval]/[resource]
.
sysstat
Another useful tool enabled on all MOSH'es is sysstat
. It can provide a recent history of resource usage on the server. This information helps you determine if a resource constraint has only been happening recently, or has been on-going for some time.
It collects data via a cron job that runs sa1
. To view the data, run:
sar
To see the data from yesterday:
sar -1
The sysstat
commands won't break down resource usage on a per-user basis.