Changes between Initial Version and Version 1 of resource-hog

Timestamp:

May 6, 2016, 2:50:21 AM (10 years ago)

Author:

Jamie McClelland

Comment:

--

resource-hog

@@ +1 @@
+= Understanding why a MOSH is under heavy load =
+
+When you find a MOSH under heavy load it is often due to a single web site or user getting slammed.
+
+Here are some tricks to find that user.
+
+== Resource Hog ==
+
+The `mf-resource-hog` script will output the top CPU users by minute going back between 1 and 2 days. 
+
+You can modify it's behavior in a number of ways:
+
+ * `mf-resource-hog cpu`: outputs CPU usage. This is the defalt
+ * `mf-resource-hog disk-read`: outputs disk read in kB per user.
+ * `mf-resource-hog disk-write`: outputs disk writes in kB per user.
+
+The second argument specifies an interval:
+
+ * `mf-resource-hog cpu minute`: outputs usage on a per-minute basis. This is the default. 
+ * `mf-resource-hog cpu hour`: outputs usage on a per-hour basis. 
+ * `mf-resource-hog cpu day`: outputs usage on a per-day basis. 
+ * `mf-resource-hog cpu month`: outputs usage on a per-month basis. 
+
+If you want real-time continuous output, you can run the collector script:
+
+`mf-resource-hog-collector [cpu|disk-read|disk-write]`
+
+Or, you can check out `man pidstat` to read up on the underlying tool used to generate this data.
+
+=== How does it work? ===
+
+There are three scripts in all:
+
+==== mf-resource-hog-collector ====
+
+When run from a cron job (''not'' a terminal), `mf-resource-hog-collector` runs `pidstat` or `pidstat -d` depending on whether cpu or disk-read/disk-write is passed. It runs for 60 seconds, then totals the usage statistics for each user in the output and writes this output to a file in /var/log/mfpl/resource-hog/minute. 
+
+When run from a terminal, it outputs the data to standard out instead (and limits output to the top 4 users).
+
+This script runs every minute.
+
+==== mf-resource-hog-consolidator ====
+
+Once an hour, `mf-resource-hog-consolidator` runs. Every hour it averages the totals from /var/log/mfpl/resource-hog/minute and places them in /var/log/mfpl/resource-hog/hour.
+
+At midnight, it averages the totals from /var/log/mfpl/resource-hog/hour and places them in /var/log/mfpl/resource-hog/day. It also deletes old files that have already been averaged.
+
+On the first of the month, it averages the totals from /var/log/mfpl/resource-hog/day and places them in /var/log/mfpl/resource-hog/month.
+
+==== mf-resource-hog ====
+
+The `mf-resource-hog` script is the main user script. It simply displays the output of `head -n4 /var/log/mfpl/resource-hog/[interval]/[resource]`. 
+
+== sysstat ==
+
+Another useful tool enabled on all MOSH'es is `sysstat`. It can provide a recent history of resource usage on the server. This information helps you determine if a resource constraint has only been happening recently, or has been on-going for some time.
+
+It collects data via a cron job that runs `sa1`. To view the data, run:
+
+{{{
+sar
+}}}
+
+To see the data from yesterday:
+
+{{{
+sar -1
+}}}
+
+The `sysstat` commands won't break down resource usage on a per-user basis.