Probably everyone responsible for managing database servers at some point realizes that it would be helpful to have a tool capable of extracting data directly from raw database files. Something completely offline, without the need to have a running database server, and that would work with data directories, files, or binary backups (storage volume snapshots, XtraBackup, etc).

There are a couple of use cases for that I could think of:

• Crashed/corrupted data recovery – if the database service refuses to start or enters a crash-restart loop and there are no good backups (PITR!) available.
• Backup (additional) verification and data files health check.
• Extraction of individual databases or tables from a full backup.
• Quick provisioning of staging/QA instances directly from live production.
• Recovery of accidentally deleted rows.
• Forensics (i.e., quick seek for data across many old backups without the need to reload them).
• Import/migrate data from very old/legacy database versions.

Backups!

Here, let me drop an obvious yet often neglected statement: nothing will replace good backups! Your database infrastructure should always be planned with the assumption that your data may get damaged or lost at any time. Visit here if you want to check if your current backup policy is up to date: https://www.percona.com/blog/mysql-backup-and-recovery-best-practices/

Now, would an offline tool with the ability to read SQL data from files be another backup method? NO! Modern ACID- and MVCC-compliant data engines are too complex in terms of data durability and consistency. So, reading directly from database files does not guarantee the dump would be consistent. Dedicated backup tools like XtraBackup engage additional steps to ensure consistency, like following the redo log and using backup locks.

Corruption and Crash Recovery

Typically, when MySQL / MariaDB server detects severely corrupted data, it may need to perform an emergency shutdown with the intention of preventing further damage. A lot depends on the scope of the damage. Some minor issues with secondary indexes could often be fixed by re-creating the index, but clustered key corruption often leads to an engine crash whenever the corrupted page is accessed. You may find yourself in a situation similar to these reports:

https://bugs.mysql.com/bug.php?id=111610
https://bugs.mysql.com/bug.php?id=113706

Again, with good backups, this should be no problem. But a bad surprise may force you to try to read all healthy rows in the hope of restoring as much as possible. The task will be, in most cases, very tedious – an attempt to read a row sitting in a corrupted page would crash the server, so you have to wait for a restart before you can try reading the next row.

This was one of the reasons for me to try to come up with a tool fully capable of extracting rows without needing to run MySQL. Otherwise, a challenging project was affordable using vibe-coding, where the agent could build data decoding by reading the relevant engine’s source code. So, let me introduce a new open-source project: innodump! Up to this point, I have implemented all three most popular MySQL and MariaDB engines: InnoDB, MyISAM, and MyRocks (RocksDB). The tool can read and dump the whole data directory, or specific tables, either from a common or general tablespace or from a per-table file. It decodes both the table schema and rows into SQL files or a TSV structure compatible with MySQL Shell dump utilities. The latter allows you to load the dump into another instance very efficiently.

There is an option relevant from the corruption point of view:

--skip-corrupted – in case a corrupted data page is encountered, it prints a comment and skips to the next page, effectively reading as much data as possible.

Example run:

Data Verification

--examine – this option makes the tool read every table and decode every row without storing them; it produces a report of how many rows can be found and how many of them are corrupted.

Note: the tool scans only the clustered (primary) index, so it will not find possible secondary index corruptions; it is expected to only ensure which rows can be restored.

If you are looking for a way to scan InnoDB files for potential corruption, you should always start with the official innochecksum. With innodump’s examination, the main goal is to show what’s actually readable.

Staging/dev/QA Data Provisioned the Easy and Unintrusive Way?

Quite often, a DBA may be tasked with setting up a non-production MySQL or MariaDB instance having a recent data snapshot from production. And typically, this kind of data copy for staging purposes does not have to be point-in-time consistent. And note that using this tool on a live instance or on an unprepared backup will mean various data consistency issues, like dirty pages may not be on disk yet, pages can be mid-write, it can produce rows that are missing, stale, or wrongly flagged, and even whole tables can be missing if created recently enough inside a common space.

Now, if the last regular available backup is up to date enough, it may still be too costly to restore the full backup while only a handful of databases are needed. Here, innodump can dump from specific database folders or files:

The exported folder can use the format compatible with fast MySQL Shell import:

What if no recent backup is available but there is a need to fetch staging data quickly from production? Bearing in mind that reading from a live datadir has the caveats above, which is fine for a throwaway staging copy, the tool can do it gently, as it allows you to limit the IO throughput during exporting. Moreover, it can bypass the OS page cache via the O_DIRECT flag, so as not to trash the server’s cache with garbage and avoid causing additional memory pressure (the kernel might decide to swap out some least recently used user space memory to make room for page cache!). Therefore, to make the export from production with as little impact as possible, I recommend using --io-limit (RATE bytes per second) and --direct-io options, plus being gentle on the CPU side, running the tool with nice.

Example run results without these options used:

And with these safety settings (time longer bc io limit, and page cache is skipped bc direct io):

Recover Deleted Rows?

InnoDB does not immediately purge or overwrite deleted rows in the table spaces. The record is first only delete-marked and stays in the page. If no new writes have actually reclaimed the space yet, such rows usually can be decoded. So, if done quickly, there is a good chance you can restore those rows, and innodump makes it easy via the --deleted-only option!

I hope you may find the tool handy for the cases mentioned above as well as possibly others, like extracting data from some ancient/legacy database versions, where the regular backup tools don’t even work.

Note: it is not an officially supported Percona software, but rather an experimental personal project.

The tool can be found on GitHub, and all feedback is welcome!

https://github.com/PrzemekMalkowski/innodump

The article was created by a human.

J.K. Rowling published the beloved children’s book series Harry Potter between 1997 and 2007. Then she made herself public enemy number one in 2020 when she wrote an essay condemning the transgender movement, particularly where it takes away the rights and safety of women and girls. It was a brave stand in the face of COVID-era craziness. Rowling was among the few rich and famous voices who pointed out what a farce this ideology made of women’s rights. We were at the height of cancel culture, and the trans agenda seemed to be infiltrating every institution.

Six years later, Rowling has penned a new essay in which she revisits her own thoughts and brings some more observations to the public’s attention.

Her convictions on transgenderism’s interference in women’s rights remain consistent. Rowling believes that if one group’s ostensible rights supersede another group’s actual rights, that’s neither just nor liberating. It’s authoritarian and turns the “oppressed” into the oppressor. As National Review contributor Raleigh Adams articulates, this is exactly what Brazilian political theorist Paulo Freire predicted — because at the end of the day, the lust for power eclipses the crusade for equal rights under the law.

Rowling addresses the current climate of transgender activism:

For a decade or more, trans activists and acolytes in positions of influence have insisted and behaved as though women’s rights were out-dated and even meaningless. They created a climate of fear and intimidation whereby a single misplaced pronoun could see a woman disciplined.

That climate persists. Gender ideology remains so entrenched among swathes of the management and political class that women are still having to fight for their rights in law. These have included nurses protesting at having to change their clothes in front of heterosexual, cross-dressing men; rape survivors stigmatised for protesting at the presence of males in their support groups or crisis centres; women investigated and persecuted at work for even questioning the approved gender mantras; parents whose daughters are having to share primary school toilets with boys as they approach puberty; and incarcerated women, already with a high likelihood of sexual violence in their pasts, forced to share intimate facilities with trans identified males.

Rowling still receives death threats, and the activists love to target anything having to do with her, even if it’s a gala performance.

While the climate hasn’t changed, the people pushing it have. Rowling is absolutely right: The transgenderism crusade of luxury beliefs and elitism is cultivated in the minds of philosophy and gender studies professors and spoon-fed in college classrooms. This is not a grassroots movement in which a group is fighting against government institutions for equal rights. This movement was manufactured by those with no accountability to the larger community, and instead of forming from the bottom up, it infiltrates from the top down with draconian precision.

Rowling also points out that the transgenderism belief system is mainly upheld by people who don’t have to deal with the consequences of imposing their beliefs on the culture. She writes:

Trans activism doesn’t impact all women negatively. In fact, the more privileged you are, the less it tends to affect you, and the more it might benefit you. Aside from an avalanche of death, rape and torture threats, it hasn’t noticeably affected my life. Nor does it hurt wealthy women who repost the vogueish mantras while sitting in dressing rooms guarded by private security guards. It doesn’t hurt women in the upper echelons of media, publishing and academia, who don’t need a changing room at work and who feel a snobbish disdain for working class women who don’t want to strip off in front of heterosexual males who’ve ‘identified’ into the women’s changing room. Trans activism doesn’t hurt headline-hungry female politicians whose attitude to women’s rights is ‘whatever is good for me personally’. On the contrary, trans activism has been good for those women. It’s fashionable, and keeps them popular with their progressive tribe, and costs them, personally, absolutely nothing.

Rowling is British, but this is exactly the case here in the States and exactly what I have been pointing out with individuals like former U.S. women’s soccer star Megan Rapinoe: Their beliefs cost them nothing, while the rug is being pulled out from under those who are trying to succeed just like the activists did.

While promoting transgender ideology to the detriment of all women is certainly bad enough, what’s worse is that it has become ensconced in some government institutions. This is especially true for Rowling.

A native of Edinburgh, Scotland, Rowling who has used her money and influence to start a women’s-only rape crisis center called Biera’s House (the only other center of its kind in the city is run by a gender-confused man). She also started J.K. Rowling’s Women’s Fund, which helps women take action against companies and government organizations that have taken away their rights.

Through her fund, Rowling has noticed a startling trend: After the 2025 Supreme Court ruling in Scotland that clarified that “woman” refers to the immutable biological fact of sex, most companies ceased their top-down enforcement of the trans agenda in the workplace. However, government workplaces are entrenched, and because they have access to taxpayer money, Rowling says “they continue to fight the idea that women have, or should have, the rights the Supreme Court upheld.” In other words, the battle is not over.

In America, there is a sense that the battle has been won because of the results of the 2024 election. While transgenderism isn’t currently the main front in the culture war, it’s part of the Left’s omnicause. “Free Palestine,” abortion, socialism, defund the police, abolish ICE, and gender ideology are all being pushed simultaneously.

The silver lining is that each of these issues contradicts one other; the only thing holding them together is hatred of the West. Eventually, the coalition will eat itself. In the meantime, it would be best for our country if that implosion occurs after the Left’s omnicause gets defeated again at the ballot box on November 3.

Plex shipped a security update in August that closed a chain of bugs letting anyone who can reach an exposed Plex Media Server run code on it. In one detailed public account, a hobbyist who later stood up a vulnerable server as a honeypot says attackers used that same chain on Saturday to drop a crypto miner and residential proxy software that lets other people route their traffic through a victim’s home internet connection, essentially using it as a VPN without the victim’s knowledge.

The bugs were originally found by a security researcher who goes by ZM online. In a technical write-up published Thursday, ZM described an unauthenticated remote code execution chain he calls Plex2Shell. In short, an attacker with no real login can trick a server into handing over its owner’s admin token, then plant a malicious file that runs the next time Plex transcodes a video. The chain strings together four vulnerabilities tracked as CVE-2026-96652, CVE-2026-96654, CVE-2026-96655, and CVE-2026-96656. Affected servers are those running versions before 1.43.3.10861.

ZM reported the issue to Plex on July 27th. Plex acknowledged it and paid a $500 bug bounty on August 4th, the same day the fixed build became publicly available to download. On September 1st, Plex posted a public advisory and, according to BleepingComputer, emailed users still on older versions. “We recommend all server owners and Desktop users update to the latest version as soon as possible,” Plex wrote. The company also said it had requested CVEs, which were assigned on September 23.

“Unauthenticated RCE chain in Plex,” ZM posted alongside the writeup and proof-of-concept code. “Patch your servers!” The $500 payout for such a critical exploit was immediately criticized, with multiple X users asking whether ZM had forgotten some zeros on the payout amount.

At the same time, an anonymous blogger who runs boredhackerblog had been waiting for those details to be revealed. “As an infosec person and a Plex user, I was really curious about what this vuln was and if I could see in-the-wild exploitation,” he wrote. He spun up a honeypot on an old Plex build. “As of this morning (Oct 17th), I finally saw in the wild exploitation using exploit code from the authors github,” they reported Saturday. The first payload was a crypto miner. Later the same day, they also saw a tool called myawesomeproxy, which prints a notice that running it means agreeing to “the potential external use of your internet traffic.” In other words, the attackers were turning the box into a residential proxy, the kind that other hackers rent to look like ordinary traffic while also masking their own IP addresses.

Kraken Chief Security Officer Nick Percoco cited the honeypot attack in an X post on Sunday. “If you run Plex at home, update it today,” he wrote, adding, “The payload sits in Plex’s Codecs folder and runs again every time something transcodes, so killing the miner doesn’t fix it.”

Plenty of people still have not patched their Plex servers. Shadowserver data on hosts tagged vulnerable-plex shows nearly 49,000 still exposed as of Saturday, including about 25,100 in North America and about 17,600 in Europe.

Not the First Plex Exploit

Plex has had its fair share of security incidents over the years. In September 2025, Plex told users to reset their passwords after an unauthorized party accessed emails, usernames, and hashed passwords. A month earlier, after a separate bug bounty report, Plex emailed users and told them to update to version 1.42.1. Weeks later, with many servers still vulnerable, it went further. “As a follow-up to the original report and in an abundance of caution due to a population of servers still on an affected Plex Media Server version (versions 1.41.7.x to 1.42.0.x), we have made adjustments for access to affected servers,” Plex wrote. “Specifically, other users to which the server owner has granted access to the server will not be able to connect to it while the server is running an affected version.” That flaw is now tracked as CVE-2025-34158.

Plex did not apply a similar access restriction, or otherwise require the update, after Plex2Shell became public. Its September 1st notice recommended an update and pointed Windows and Mac users at an optional automatic-update setting. That setting defaults to “Ask me,” not to auto-install, and Plex’s own support docs say it only applies to Windows and macOS. Linux, Docker, and NAS installs depend on package managers, container restarts, or manual downloads. Plex’s terms of service say the company can require an update “as a means to protect the Plex Solution from unauthorized use, content, or data,” and that declining one may cut off access. As of Sunday afternoon Eastern time, Plex had also not issued any new public statement about the exploit or the attacks that followed it.

A previous LastPass hack also shows that the stakes involved with this kind of vulnerability in Plex server software can be quite high. In 2023, LastPass said attackers got into a senior DevOps engineer’s home computer “by targeting the DevOps engineer’s home computer and exploiting a vulnerable third-party media software package, which enabled remote code execution capability and allowed the threat actor to implant keylogger malware,” according to Ars Technica. Ars reported that a person briefed on a private LastPass report identified the package as Plex; however, LastPass and Plex have not publicly confirmed that link in matching detail. Either way, it is the nightmare version of what a vulnerable residential Plex server can become when a remote code execution exploit is discovered.

pg_stat_monitor started as a fork of pg_stat_statements, built to be a more advanced replacement for it. It provides more detailed information about query execution and groups it in configurable time buckets, so you can look at what happened in a particular time window. It also powers the Query Analytics feature in Percona Monitoring and Management (PMM). However, over time other work took priority at Percona, and the extension stopped getting the attention it needed. The original developers moved on to other things, and the reasoning behind several design decisions went with them. For a couple of years it stayed in maintenance mode, we were fixing reported bugs, and never stepped back to look at deep architectural issues. All that led to a degraded user experience and stability issues.

2.4.0 is our attempt to turn that around. It is the biggest release in years: about 250 commits since 2.3.2. We restructured the code and fixed bugs across the whole extension, and the C code got about 20% smaller. Among the fixes are several buffer overflows, a use-after-free, a lock ordering problem and a few statistics that were simply wrong. We also backported the pg_stat_statements regression tests. The most important fixes, though, are about memory management, and that is what this post is about. For everything else, see the release notes.

The symptoms

Well, memory leaked under certain workloads. The end was always the same: the process eventually got killed by the OOM killer.

Not every workload was affected, it depended on how it was run. Queries sent over the simple protocol were fine. Named prepared statements leaked memory on every execution. Statements running through SPI inside a background worker leaked the same way. In both cases nothing reclaimed the memory, so growth stopped only when the session or the worker did.

Another problematic case is when statements produce warnings. On PostgreSQL 18 and earlier, each warning caused a small memory leak, which could accumulate over time if many warnings were generated.

How PostgreSQL manages memory

Server code does not call malloc() and free() directly. Instead it uses memory contexts, also known as memory arenas outside PostgreSQL. Every allocation happens within a context, and freeing the context releases all allocated memory at once. Contexts form a tree structure, so freeing a parent context frees every child context as well. This is what lets PostgreSQL manage memory efficiently and survive an error thrown in the middle of a query.

So when you write an extension, you should keep in mind which context your allocations belong to and when that context will be freed.

Four contexts matter for our topic:

• TopMemoryContext is the root context for the backend. It lives as long as the backend does and is only destroyed when the backend exits.
• TopTransactionContext is reset at every commit and every abort, which deletes all of its child contexts.
• MessageContext is reset after each client protocol message.
• ErrorContext is reserved for error reporting. On PostgreSQL 18 and earlier, it is only reset during error recovery.

An extension can create its own context or use one of the existing ones. Choosing a context with the wrong lifetime causes problems: your data either vanishes too early or stays around longer than needed. The latter looks exactly like a leak, even though every byte is still accounted for.

How pg_stat_monitor managed its memory

The original design

pg_stat_monitor collects per-statement statistics in backend-local memory first, and dumps them into the shared time buckets when the statement finishes and we have the full set of data. That local memory lived in a context called PgsmMemoryContext, and it was created as a child of TopMemoryContext. TopMemoryContext lives as long as the backend does. So instead of relying on context lifetime, pg_stat_monitor used a persistent context, and for cleanup it used a reset callback assigned to another context: MessageContext. MessageContext is reset after every client protocol message, and a reset invokes the callback, which cleaned out PgsmMemoryContext without destroying it.

One detail matters for everything below. A reset callback in PostgreSQL is one-shot: it is popped off the context’s callback list before it is called, so it fires once and is gone. To keep the cleanup working, pg_stat_monitor had to register a fresh callback for every message. It did this from the post parse analyze hook, one of the first steps in the statement lifecycle. So callback setup depends not on the statement finishing, but on the next message going through parse analysis. This design has two weak points, and we hit both of them.

The callback is not registered on every execution path

Background workers have no MessageContext. So a statement running through SPI inside a background worker never arms the reset callback at all. Nothing resets PgsmMemoryContext, and because its parent is TopMemoryContext, nothing else is going to. It keeps growing for the life of the worker.

In the extended query protocol, the client sends a named prepared statement as separate messages: Parse once, then Bind and Execute for every run. Parse analysis happens only during Parse, so that is the only message that arms the callback. When MessageContext is reset after Parse, the callback fires and nothing re-arms it. MessageContext is still reset after every Bind and Execute, but no callback is attached to it anymore, so PgsmMemoryContext keeps growing for the life of the session. This is the pattern database drivers use for prepared statements. Note that the SQL-level PREPARE and EXECUTE commands are not affected, because each EXECUTE statement goes through parse analysis itself and re-arms the callback.

The error path allocated into a context nobody resets

pg_stat_monitor records the warnings and errors a statement produces, so it hooks into the code PostgreSQL uses to emit log messages. That hook runs in the middle of error reporting, and error reporting has a memory context of its own, ErrorContext. PostgreSQL keeps that one reserved so it always has somewhere to build a message, even when the backend is short on memory. It is cleaned out when the server unwinds from an error. A WARNING does not unwind anything: the statement simply carries on. PostgreSQL 19 added ErrorContext resets after reporting a warning, so this leak only affects PostgreSQL 18 and earlier.

Our hook copied the query text wherever it happened to be running, so that copy landed in ErrorContext instead of in pg_stat_monitor’s own context. Nothing was ever going to reclaim it. The entry for the statement itself went to PgsmMemoryContext, so it stayed there until the next MessageContext reset cleaned it up, if that ever happened.

Both problems come from the same place. We inherited the design from pg_stat_statements, which we forked years ago, and we never revisited it as pg_stat_monitor grew code paths that upstream does not have. The test suite did not help either. We had nothing that ran a background worker in a loop, nothing that held a session open across thousands of Bind and Execute cycles, and nothing that measured backend memory at all. None of this is exotic.

What we changed

Both problems have the same shape. The memory sat in a context that never dies, and the cleanup depended on an event that does not always happen. Rather than trying to make the cleanup callback fire on every path, we changed where the memory lives.

Every path that can execute a statement runs inside a transaction. Background worker SPI, named prepared statements, ordinary queries. A background worker may not have a MessageContext, but it always has a transaction.

The change is:

• PgsmMemoryContext is no longer a child of TopMemoryContext. It is created as a child of TopTransactionContext, so it is automatically deleted at every commit and every abort. There is no callback to register and nothing to re-arm, so there is no execution path left that can skip the cleanup.
• There is a special handling of subtransactions, but in any case the memory is tied to the top-level transaction and will be cleaned up when that transaction ends.
• The error and utility statement paths no longer allocate an entry at all. They use a local variable that goes away by itself when the function returns, which removes the ErrorContext allocations completely.

This covers all three cases from the beginning of the post. A background worker always runs its statements inside a transaction, so its memory is now freed when that transaction ends, the same as in a regular backend. Named prepared statements no longer depend on parse analysis to clean up after them, every Execute is followed by a commit that frees the memory. Warnings no longer allocate anything that outlives the hook.

What’s next

2.4.0 was about stabilizing the code base we already have. We reworked memory management, removed code we no longer need, and built the test coverage that should have been there from the start. We did not change how the extension is structured. That is the plan for the next major release: we want to revisit the architecture of pg_stat_monitor, starting with how it uses shared memory and how it behaves when that memory fills up. ∎

HARAN — Following a night of wine and music, Jacob reportedly began to assure his new bride Rachel that he hadn’t gotten that drunk at the wedding last night, only to discover he had, in fact, gotten that drunk.

According to new information, Jacob had not married Rachel the night before as previously intended, but he had instead gotten totally smashed and then married her sister Leah.

Leah, who had not overindulged the night before, was reportedly offended when her new husband kept saying, "Oh no, oh no, oh no! This can’t be happening! Where’s Rachel? WHERE’S RACHEL?"

Laban, the father of Leah and Rachel, was suspected of intentionally bullying Jacob into taking shots of tequila as part of a scheme to marry off his eldest daughter first.

"Yeah, he worked seven years for Rachel, but have you seen Leah?" Laban said by way of defense. "I had to get her a husband somehow."

Rachel later said of the matter, "It’s fine."

At publishing time, Jacob agreed to work seven more years for Rachel if Laban would promise not to get him drunk and marry him to one of his camels.

Noah’s Ark has definitely been found and we highlight 9 incredible artifacts that prove it.

Building a Database DBA Lab with Docker, PostgreSQL, MySQL, and Percona Monitoring and Management

How I built a local database playground, learned a few lessons the hard way, and created a practical command reference along the way.

Introduction

One of the best ways to learn database administration is to get your hands dirty.

Documentation is important, but there’s something different about building your own environment, breaking it, troubleshooting it, and eventually understanding why everything works.

Recently, I decided to create a local database lab using Docker Desktop on my Mac. My goal was straightforward: build an environment where I could experiment with MySQL, PostgreSQL, Percona XtraDB Cluster (PXC), and Percona Monitoring and Management (PMM), without worrying about disrupting a production environment.

What started as a few containers quickly became a much more interesting learning experience.

Along the way, I ran into Docker networking problems, container persistence challenges, cluster bootstrap issues, and PostgreSQL monitoring configuration. This article shows the setup I recommend after working through those problems, so you can start with the right foundation instead of discovering it at the end.

1. Building the Lab

The foundation of my environment is Docker Desktop.

The lab includes:

• Percona Server for MySQL 8.4
• A three-node Percona XtraDB Cluster
• PostgreSQL 17
• PMM Server and PMM Client
• Persistent Docker volumes
• A shared Docker network

The architecture looks like this:

Docker Desktop
│
├── pmm-network
│ ├── pmm-server
│ ├── pmm-client
│ ├── percona-server
│ ├── postgres-server
│ ├── pxc1
│ ├── pxc2
│ └── pxc3
│
└── Persistent Docker Volumes
 ├── pmm-data
 ├── pmm-client-data
 ├── mysql-data
 ├── postgres-data
 ├── pxc1_data
 ├── pxc2_data
 └── pxc3_data

There are two ways to build this.

Option A (recommended): Docker Compose

Docker Compose lets you define every container, volume, and network attachment in one file. The whole lab becomes reproducible: one command brings it up, one command shuts it down, and nothing depends on remembering which docker run flags you used weeks ago.

I started out with individual docker run commands and later moved everything to Compose. If I were starting again, I would begin here.

First, keep passwords out of the Compose file by putting them in a .env file next to it (and out of version control):

# .env
MYSQL_ROOT_PASSWORD=change-me
PXC_ROOT_PASSWORD=change-me
POSTGRES_PASSWORD=change-me
PMM_ADMIN_PASSWORD=change-me
PG_MONITOR_PASSWORD=change-me
MYSQL_MONITOR_PASSWORD=change-me

Then define the lab in docker-compose.yml:

services:
 pmm-server:
 image: percona/pmm-server:3.9.1
 platform: linux/amd64
 container_name: pmm-server
 ports:
 - "443:8443"
 volumes:
 - pmm-data:/srv
 networks: [pmm-network]

 pmm-client:
 image: percona/pmm-client:3.9.1
 container_name: pmm-client
 depends_on: [pmm-server]
 environment:
 PMM_AGENT_SERVER_ADDRESS: pmm-server:8443
 PMM_AGENT_SERVER_USERNAME: admin
 PMM_AGENT_SERVER_PASSWORD: ${PMM_ADMIN_PASSWORD}
 PMM_AGENT_SERVER_INSECURE_TLS: 1
 PMM_AGENT_SETUP: 1
 PMM_AGENT_CONFIG_FILE: config/pmm-agent.yaml
 volumes:
 - pmm-client-data:/usr/local/percona/pmm/config
 networks: [pmm-network]

 percona-server:
 image: percona/percona-server:8.4.11
 container_name: percona-server
 environment:
 MYSQL_ROOT_PASSWORD: ${MYSQL_ROOT_PASSWORD}
 ports:
 - "3306:3306"
 volumes:
 - mysql-data:/var/lib/mysql
 networks: [pmm-network]

 postgres-server:
 image: postgres:17
 container_name: postgres-server
 environment:
 POSTGRES_PASSWORD: ${POSTGRES_PASSWORD}
 command: >
 postgres
 -c shared_preload_libraries=pg_stat_statements
 -c track_io_timing=on
 ports:
 - "5432:5432"
 volumes:
 - postgres-data:/var/lib/postgresql/data
 networks: [pmm-network]

 pxc1:
 image: percona/percona-xtradb-cluster:8.4.11
 container_name: pxc1
 environment:
 CLUSTER_NAME: pxc-lab
 MYSQL_ROOT_PASSWORD: ${PXC_ROOT_PASSWORD}
 volumes:
 - pxc1_data:/var/lib/mysql
 - ./certs:/etc/mysql/certs:ro
 - ./conf.d:/etc/percona-xtradb-cluster.conf.d:ro
 stop_grace_period: 120s
 healthcheck:
 test: ["CMD", "mysqladmin", "ping", "-uroot", "-p${PXC_ROOT_PASSWORD}"]
 interval: 10s
 retries: 30
 networks: [pmm-network]
 ports:
 - "3307:3306"

 pxc2:
 image: percona/percona-xtradb-cluster:8.4.11
 container_name: pxc2
 environment:
 CLUSTER_NAME: pxc-lab
 CLUSTER_JOIN: pxc1
 MYSQL_ROOT_PASSWORD: ${PXC_ROOT_PASSWORD}
 volumes:
 - pxc2_data:/var/lib/mysql
 - ./certs:/etc/mysql/certs:ro
 - ./conf.d:/etc/percona-xtradb-cluster.conf.d:ro
 stop_grace_period: 120s
 depends_on:
 pxc1:
 condition: service_healthy
 networks: [pmm-network]
 ports:
 - "3308:3306"

 pxc3:
 image: percona/percona-xtradb-cluster:8.4.11
 container_name: pxc3
 environment:
 CLUSTER_NAME: pxc-lab
 CLUSTER_JOIN: pxc1
 MYSQL_ROOT_PASSWORD: ${PXC_ROOT_PASSWORD}
 volumes:
 - pxc3_data:/var/lib/mysql
 - ./certs:/etc/mysql/certs:ro
 - ./conf.d:/etc/percona-xtradb-cluster.conf.d:ro
 stop_grace_period: 120s
 depends_on:
 - pxc2
 networks: [pmm-network]
 ports:
 - "3309:3306"

networks:
 pmm-network:
 name: pmm-network

volumes:
 pmm-data:
 pmm-client-data:
 mysql-data:
 postgres-data:
 pxc1_data:
 pxc2_data:
 pxc3_data:

A few design choices are worth calling out, because each one comes from a problem I hit:

• Volumes and the network are declared up front. Every database has persistent storage and is attached to pmm-network from the moment it is created, so there is nothing to wire up afterward.
• All PXC nodes share the same SSL certificates. Mismatched certificates were behind several of my early cluster join failures.
• stop_grace_period: 120s gives each PXC node time to shut down cleanly instead of being killed (exit code 137).
• depends_on defines the PXC order. Compose starts pxc1 first and waits for it to be healthy before the others join. On shutdown, Compose stops dependents first, so the nodes leave in the order pxc3, pxc2, pxc1.

Bring the lab up and down with:

docker compose up -d
docker compose stop # stops containers, keeps everything
docker compose down # removes containers, keeps named volumes

Option B (alternative): Individual docker run commands

If you prefer to build containers one at a time, or want to understand what Compose is doing for you, you can create the network yourself and attach containers to it:

docker network create pmm-network

docker run -d --name postgres-server \
 --network pmm-network \
 -e POSTGRES_PASSWORD="$POSTGRES_PASSWORD" \
 -v postgres-data:/var/lib/postgresql/data \
 -p 5432:5432 \
 postgres:17

An existing container can be attached later:

docker network connect pmm-network postgres-server

This works, but you are responsible for recreating each container with exactly the same flags, volumes, and network settings every time. That is the main reason I recommend Option A. Whichever option you choose, avoid mixing the two for the same lab, because it becomes hard to tell which containers are managed by Compose and which are not.

A networking lesson that applies either way

At one point, PMM couldn’t resolve my PostgreSQL container:

lookup postgres-server on 127.0.0.11:53: no such host

The problem wasn’t PostgreSQL or PMM. The PostgreSQL container was stopped, so it had no active network endpoint, and Docker’s internal DNS had nothing to return.

That was an important reminder: always verify the container and network state before troubleshooting the application.

2. Making Monitoring Persistent with PMM

Once the databases were running, I wanted visibility into their performance.

PMM provides dashboards and metrics for MySQL, PXC, and PostgreSQL. It consists of a server that stores and visualizes monitoring data and a client agent that collects database metrics.

The key thing to understand is that persistent storage and service registration are related, but they are not the same thing.

PMM Server maintains the monitoring inventory, while PMM Client has its own configuration and agent identity. If the client is recreated with a new identity, the association between the client and its registered services is lost. In my early setup, after recreating containers, the client listing showed only the basic agents:

pmm_agent Connected
node_exporter Running
vmagent Running

Two things prevent this, and both should be in place from the start:

1. A persistent volume for the client configuration. In the Compose file above, pmm-client-data keeps the agent identity across restarts and recreations.
2. An idempotent registration script. It adds each database only if it isn’t already registered, so it is safe to run after every docker compose up.

Before registering, create a monitoring user in each database. For PostgreSQL, for example:

CREATE USER pmm WITH PASSWORD 'change-me';
GRANT pg_monitor TO pmm;
CREATE EXTENSION IF NOT EXISTS pg_stat_statements;

Then the registration script, which reads credentials from the .env file rather than hardcoding them:

#!/usr/bin/env bash
# register-pmm.sh
set -euo pipefail
source .env

registered() {
 docker exec pmm-client pmm-admin list | grep -qw "$1"
}

if ! registered postgres-server; then
 docker exec pmm-client pmm-admin add postgresql \
 --username=pmm \
 --password="$PG_MONITOR_PASSWORD" \
 --host=postgres-server \
 --port=5432 \
 postgres-server
fi

for node in percona-server pxc1 pxc2 pxc3; do
 if ! registered "$node"; then
 docker exec pmm-client pmm-admin add mysql \
 --username=pmm \
 --password="$MYSQL_MONITOR_PASSWORD" \
 --host="$node" \
 --port=3306 \
 --query-source=perfschema \
 "$node"
 fi
done

docker exec pmm-client pmm-admin list

The takeaway: a monitoring system should be designed to survive a restart, not just work immediately after installation.

3. Learning PostgreSQL Through Real Workloads

With PostgreSQL running and connected to PMM, I wanted to do more than look at an idle dashboard.

I used sysbench to generate a workload against PostgreSQL 17.

My test environment included ten tables with approximately 750,000 rows each, giving me about 7.5 million rows to experiment with.

A sample workload command looked like this:

sysbench \
 --db-driver=pgsql \
 --pgsql-host=127.0.0.1 \
 --pgsql-port=5432 \
 --pgsql-user=postgres \
 --pgsql-db=sbtest \
 --tables=10 \
 --table-size=750000 \
 --threads=16 \
 --time=300 \
 --report-interval=5 \
 oltp_read_write run

With the workload running, PMM became much more interesting.

I could observe database activity, disk latency, query execution statistics, and changes in resource utilization.

This also helped me understand an important distinction: PostgreSQL tuple activity is not the same thing as SQL queries per second.

One query can read or return many tuples. Looking only at tuple counts without understanding the underlying queries can lead to incorrect conclusions about workload intensity.

4. Understanding PostgreSQL Query Performance

One of the most useful PostgreSQL features for troubleshooting is EXPLAIN.

For example:

EXPLAIN (ANALYZE, BUFFERS)
SELECT *
FROM sbtest1
WHERE id BETWEEN 100 AND 200;

This shows the execution plan, actual execution measurements, and buffer activity.

I started paying attention to several key pieces of information:

• Whether PostgreSQL used a sequential scan or an index
• Estimated versus actual rows
• Execution time
• Buffer hits and reads
• Number of execution loops

Because the Compose file loads pg_stat_statements at startup, query statistics are aggregated from the beginning.

A useful query is:

SELECT
 calls,
 total_exec_time,
 mean_exec_time,
 rows,
 LEFT(query, 100) AS query
FROM pg_stat_statements
ORDER BY total_exec_time DESC
LIMIT 10;

This provides a different perspective from looking at individual slow queries.

A query that takes only a fraction of a millisecond might still consume substantial cumulative execution time if it runs hundreds of thousands of times.

For a DBA, both individual latency and aggregate workload matter.

5. MVCC, Dead Tuples, and VACUUM

My next learning objective was understanding PostgreSQL’s Multi-Version Concurrency Control (MVCC).

Unlike an in-place overwrite model, PostgreSQL commonly creates a new row version when a row is updated.

Older versions may remain until PostgreSQL determines they are no longer needed.

That leads directly to one of PostgreSQL’s most important maintenance mechanisms: VACUUM.

I experimented with updating large numbers of rows:

UPDATE sbtest1
SET k = k + 1
WHERE id <= 300000;

Then checked the table statistics:

SELECT
 relname,
 n_live_tup,
 n_dead_tup,
 last_autovacuum,
 autovacuum_count
FROM pg_stat_user_tables
WHERE relname = 'sbtest1';

This gave me a practical way to observe dead tuple estimates and autovacuum activity.

I also explored what happens when a long-running transaction holds an old snapshot.

By starting a REPEATABLE READ transaction and leaving it open while another session updated rows, I could demonstrate how PostgreSQL may need to retain older row versions.

That helped connect several important DBA concepts:

Long-running transactions → old snapshots → delayed tuple cleanup → potential table bloat.

I also learned that normal VACUUM generally makes space reusable within PostgreSQL rather than immediately shrinking the table file.

VACUUM FULL can reclaim disk space by rewriting a table, but it requires an exclusive lock and is not something to run casually in production.

6. Operating the PXC Cluster Safely

My three-node Percona XtraDB Cluster taught me that database containers need database-aware operational procedures. That’s why startup and shutdown are built into the Compose file rather than left to chance.

Early on, I ran into certificate problems, donor timeouts, cluster membership issues, and shutdowns that ended with exit code 137 (the container being killed before MySQL finished). The shared certificate directory, the healthcheck on pxc1, and stop_grace_period address those directly.

The shutdown procedure is simply:

docker compose stop pxc3 pxc2 pxc1

If you use Option B, do the same thing explicitly, one node at a time, with a long timeout:

docker stop -t 120 pxc3
docker stop -t 120 pxc2
docker stop -t 120 pxc1

To start the cluster again, check grastate.dat on each node and understand the meaning of safe_to_bootstrap:

docker run --rm -v pxc1_data:/data alpine cat /data/grastate.dat

After a clean shutdown, the last node to leave the cluster (here, pxc1) will have safe_to_bootstrap: 1 and is the right node to start first.

After an unclean shutdown, selecting the correct bootstrap node requires recovery checks. Compare the recovered sequence numbers across nodes and bootstrap from the most advanced one. Simply changing safe_to_bootstrap to 1 without verifying that can lose data.

7. My Most Useful Commands

I kept a command cheat sheet from the start of the project, and it has become one of the most useful outcomes. It captures the commands I actually needed while learning and troubleshooting. I recommend starting one on day one.

For Docker:

# Running containers
docker ps

# All containers
docker ps -a

# Status of the Compose-managed lab
docker compose ps

# Follow logs
docker logs -f postgres-server

# Inspect network configuration
docker inspect postgres-server \
 --format ''

# Resource usage
docker stats

# List Docker volumes
docker volume ls

# Check PMM services
docker exec pmm-client pmm-admin list

These commands are my first line of investigation whenever something isn’t behaving as expected.

For PostgreSQL, my daily reference includes:

-- Current sessions
SELECT pid, usename, state, query
FROM pg_stat_activity;

-- Long-running transactions
SELECT
 pid,
 state,
 now() - xact_start AS transaction_age,
 query
FROM pg_stat_activity
WHERE xact_start IS NOT NULL;

-- Dead tuples
SELECT relname, n_live_tup, n_dead_tup
FROM pg_stat_user_tables
ORDER BY n_dead_tup DESC;

-- Blocking sessions
SELECT
 pid,
 pg_blocking_pids(pid) AS blocked_by
FROM pg_stat_activity
WHERE cardinality(pg_blocking_pids(pid)) > 0;

These are simple queries, but they form a useful starting point for troubleshooting.

8. Design Principles Behind the Lab

To summarize, these are the principles the setup above is built on:

1. Define the whole lab in Docker Compose, rather than mixing docker run commands with Compose-managed containers.
2. Declare persistent volumes and network attachments before creating any databases.
3. Treat startup and shutdown as part of the design, especially for PXC.
4. Make PMM registration reproducible with a persistent client volume and an idempotent script, and keep credentials out of scripts and command history.
5. Keep a command cheat sheet from the beginning.

Final Thoughts

What began as an effort to build a small local database environment turned into a practical DBA learning experience.

I now have a place to explore PostgreSQL performance, MVCC, autovacuum, MySQL replication concepts, PXC clustering, and database monitoring without risking a production system.

Perhaps the biggest lesson is that the most valuable learning moments often happen when something breaks.

A Docker DNS failure teaches networking. A missing PMM registration teaches persistence and identity. A PXC bootstrap problem teaches cluster recovery. A growing dead tuple count teaches PostgreSQL MVCC.

And once you understand why something broke, you’re much better prepared to troubleshoot it the next time.

My next goal is to continue expanding the lab with PostgreSQL replication, backup and recovery, and more deliberate performance troubleshooting exercises.

This article describes a local learning environment. The commands and configurations are intended for experimentation and should be reviewed and adapted before use in production. ∎