Capturing Per-Process Metrics with Percona Monitoring and Management (PMM)

In this blog post, I will show you how to use Percona Monitoring and Management (PMM) to capture per-process metrics in five minutes or less.

While Percona Monitoring and Management (PMM) captures a lot of host metrics, it currently falls short providing per-process information, such as which particular process uses a lot of CPU resources, causes Disk IO or consumes a lot of memory.

In our database performance optimization and troubleshooting practice, this information has proven quite useful in many cases: batch jobs taking much more resources than developers would estimate and misconfigured Percona XtraBackup or Percona Toolkit are among the most common offenders.

Per-process metrics information can also be very helpful when troubleshooting database software memory leaks or memory fragmentation.

You don’t know which processes cause you problems at the outset, so it is important to capture information about all of the processes (or specifically exclude the processes you do not want to capture information about) rather than capture information about selected few.

While capturing such helpful information is not available in PMM out of the box (yet), you can easily achieve it using PMM’s External Exporter support and the excellent Prometheus Process Exporter by Nick Cabatoff.

These instructions are for Debian/Ubuntu  Linux Distributions but they should work with RedHat/CentOS based versions as well – just use RPM package instead of DEB

1: Download the process exporter packages from GitHub:

2: Install the package

(Note: the file will be different depending on the platform and current release.)

3: Run the Exporter

4: Register Exporter with Percona Monitoring and Management

Assuming the current node is already monitored by PMM you just need one command:

This captures process metrics every 10 seconds (adjust interval if desired).

Important note: due to some internal limitations, you need to use a different service name (“processes-my-host”)  for each host. I suggest just adding the hostname to the descriptive name “processes” for simplicity.

5: Get Matching Dashboard from Grafana.com

While you can browse the data captured by the Advanced Data Exploration Dashboard, it is not any fun. I created a PMM-style dashboard and published it on Grafana.com. I based it on Nick’s original dashboard.

To add this dashboard to your PMM Server, click Dashboard Search on your PMM Server.

From there, click on “Import Dashboard”. Use 6033 as the Grafana.com Dashboard ID.

6: You’re done!

You should have data flowing, and you should be able to see the data on the graphs.

In this example, I have pt-query-digest (shown as Perl) parsing the log file and pushing MySQL Server away from memory.

Note, as you likely have many processes on the system, the graphs are designed to show only the top processes. All running processes, however, are available in the drop-down if you want to access the history for a specific process.

Let us know what you think. We are looking at how to integrate this functionality directly into Percona Monitoring and Management!

Related