MySQL Index Optimization: How to Speed Up Your Database Queries

https://webyog.com/wp-content/uploads/2026/08/Index-optimization.png

If there is one change that can turn a 10-second MySQL query into a millisecond one, it is adding the right index. And if there is one change that can quietly slow down your write heavy database over time, it is adding too many. 

Indexes are the most powerful performance tool in MySQL. Understanding how they work — and how they fail — is foundational to managing database performance at any scale.

How Indexes Work

An index is a separate data structure MySQL maintains alongside your table. Think of it like a book’s index: instead of reading every page to find a topic, you go directly to the right page number. 

Without an index on a filtered column, MySQL reads every row in the table on every query. With the right index, it locates matching rows in a fraction of the reads.

Reading EXPLAIN

EXPLAIN is your primary tool for understanding whether MySQL is using an index and how.

EXPLAIN SELECT * FROM customers WHERE email = ‘[email protected]’; 

The most important column in the output is type.

Other columns to watch: 

•  key — which index MySQL chose. NULL means no index used. 

•  rows — estimated rows MySQL will examine. Lower is better. 

•  Extra: Using filesort — sorting without an index. Can be expensive on large result sets.

•  Extra: Using index — the query was satisfied entirely from the index. Very efficient.

Index Types

Creating Effective Indexes

Single-Column Indexes 

The simplest case: a column you filter on frequently.

ALTER TABLE customers ADD UNIQUE INDEX idx_email (email);

Before and after:

Composite Indexes and the Left-Prefix Rule

When queries filter on multiple columns, a composite index is usually more effective than separate single-column indexes.

ALTER TABLE orders ADD INDEX idx_status_created (status, created_at);

MySQL can use this index for queries that start with the leftmost column:

Order composite index columns by how they appear in your most common queries, with the most selective column first.

Covering Indexes

A covering index includes all the columns a query needs. MySQL satisfies the entire query from the index without reading the table at all — shown as Using index in EXPLAIN’s Extra column.

— Query only needs these three columns 
SELECT id, email, status FROM customers WHERE status = ‘active’; 

— Covering index includes all three 
ALTER TABLE customers ADD INDEX idx_covering (status, id, email);

On large tables, the performance difference between a regular index lookup and a covering index can be significant.

Common Mistakes

Over-Indexing

Every index you add slows down writes. MySQL updates all indexes on every INSERT, UPDATE, and DELETE. 

Find and remove unused indexes regularly:

SELECT * FROM sys.schema_unused_indexes;
SELECT * FROM sys.schema_redundant_indexes;

Functions on Indexed Columns

Using a function on an indexed column prevents MySQL from using the index:

Low-Cardinality Columns

Columns with very few distinct values (such as a boolean or a status with three possible values) are rarely worth indexing on their own. MySQL may choose a full table scan if it estimates the index would return a large fraction of rows anyway.

Maintaining Indexes

Want to analyze query performance visually? SQLyog’s Query Profiler shows detailed index usage and execution metrics for every query you run. Try SQLyog free.

Frequently Asked Questions

How do I know if a column needs an index?

Check your slow query log for queries with type: ALL in EXPLAIN. Any column that appears frequently in WHERE, JOIN, or ORDER BY clauses is a candidate. The test is always: run EXPLAIN before and after adding the index to confirm it is being used.

How many indexes should a table have?

There is no universal number, but most well-optimized tables have 3–8 indexes. Start with the primary key, add indexes for your most common query patterns, and stop before you over-index. Audit unused indexes quarterly.

Will adding an index lock my table in production?

In MySQL 5.6+ with InnoDB, most index additions are online and do not block reads or writes during the build. There is a brief metadata lock at the start and end. For very large tables, run the operation during low-traffic windows as a precaution.

My query shows an index in EXPLAIN but is still slow — why?

A few possibilities: the index has low cardinality and MySQL still reads many rows through it; you need a covering index (MySQL uses the index for filtering but reads full table rows for each match); or there is a Using filesort in EXPLAIN’s Extra column indicating additional sorting work after the index lookup.

What is the difference between a composite index and two separate indexes?

A composite index on (col1, col2) allows MySQL to filter on both columns in a single index lookup. Two separate indexes on col1 and col2 require MySQL to use one or perform a merge operation. For queries filtering on multiple columns together, a composite index almost always performs better.

How do I find duplicate indexes?

Query sys.schema_redundant_indexes . It shows indexes made unnecessary by another index on the same table — for example, if you have indexes on (a) and (a, b) , the first is redundant because all queries it can satisfy can also be satisfied by (a, b) .

Does MySQL automatically create indexes?

MySQL automatically indexes the primary key and any column declared with the UNIQUE constraint. It does not create indexes on foreign key columns or frequently queried columns automatically — those you must add manually based on your query patterns.

What does Using index in EXPLAIN’s Extra column mean?

It means MySQL satisfied the entire query from the index without reading the actual table rows. This is called a covering index and is one of the most efficient query execution patterns. When you see Using index , the query is well-optimized for index usage.

Planet for the MySQL Community

Still simple, still powerful, BBEdit is a Mac app legend

https://media.appleinsider.com/gallery/68460-144241-000-lead-BBEdit-xl.jpg

You may like Pages and you may tolerate Microsoft Word, but text editor BBEdit has been rightly earning devotees on the Mac for 34 years. Here’s how.

There are very few Mac apps that go back this far, and even fewer that remain both actively developed and actually beloved by its users. And there is only one with the trademark "It doesn’t suck."

BBEdit is just a text editor, but rather than this meaning it is limited, it means that BBEdit is focused. It is and always has been a plain text editor, it does not also try to be a mind-mapping tool or an image editor or a chatbot.

This is going to read like an unqualified hymn of praise, although there are actually a few qualifications. But this is what BBEdit does to you. While there are other apps I write in for many different purposes, BBEdit does what it does extraordinarily well.

I don’t remember when I first came across it, but it would’ve been before 2000 and possibly before Steve Jobs came back to Apple. Its original version was written in 1989 because of a problem with editing text on the Mac.

After it was released as BBEdit in 1992, Bare Bones Software never stopped developing it.

The problem back then was just that Apple offered an API for handling text, but it couldn’t handle large files. The creators of BBEdit were working on a version of the Pascal programming language for Mac that effectively demanded longer files.

While Apple’s Pascal faded away, their text editor grew into this favorite for programmers and writers.

If you’ve never used it, though, it’s hard to convey just why BBEdit is so popular among its users. There are people who will never leave the Mac because there isn’t a PC version of BBEdit, for instance.

And as you use it, you have such a clear sense that so do its developers. This is not a day job or a side project, BBEdit keeps getting better because its developers add what they want it to do themselves.

"I use BBEdit for everything I produce that involves text: code (C, C++, Objective-C), web site content (HTML, Markdown, Textile), and more," said Rich Siegel, developer of the app, in a 2011 interview. "In a very real sense, BBEdit is written using itself — my daily-use build is the same one that’s being beta tested or used in the field by our customers."

Text editor on macOS showing a hex data view, with a Multi-File Search dialog open in front and a menu of text processing options such as Remove Blank Lines expanded

If BBEdit’s search feature can’t find something, it isn’t there.

So an app that has been available for more than three decades, two hardware architectures ago, is still being actively developed. An app that was meant to be just a "bare bones" text editor is still simple to use, yet it has so many features a keystroke away that it’s hard not to gush about this app.

To be fair, there are actually reasons not to use it, there are writers for whom BBEdit is just the wrong choice. But if it suits your writing or it suits how you like to work, BBEdit can make you into a passionate fan.

"BBEdit is legendary because we keep improving it to serve the needs of our customers," Siegel said on the release of version 9. It’s now on version 16, which is promoted as having "millions* of new features and refinements (*estimated)."

With marketing like that, BBEdit remains a taste of those early times with the Mac some 34 years after it was first launched. It’s serious software that doesn’t take itself too seriously.

What BBEdit does and how

The short version is that BBEdit is a text editor. That’s not the same thing as a word processor which offers formatting options like styles or margin control, as a text editor is just for getting the words down.

BBEdit Application Settings window on macOS showing sidebar categories and main panel options for opening documents, changing workspaces, reopening documents, automatic updates, sandbox access, and a Restore Defaults button

There aren’t countless settings you can choose in BBEdit, but there are more than you’d like to count.

As such a text editor, BBEdit is fast and responsive, and it generally doesn’t get in your way. You don’t have to set any settings, you can just open it and start typing.

When you have written a lot of text, maybe even over several years, you will start to appreciate a feature called grep. This is a form of regular expression, the feature takes you beyond normal search and replace.

For instance, in any word processor you could search for a date like 01/12/2026. But in BBEdit, you can search for all dates in the document, matching every one of them simultaneously.

It’s a form of regular expression. As well as being powerful, though, the BBEdit version includes an on-screen guide to help you through it as simply as possible.

"Our base started as Mac software developers, scientists, system administrators, and other technical users," Rich Siegel told Apple in 2019. "The first big change was popularity among HTML authors and web back-end developers."

Web design software window showing a Background settings dialog with selector, file path, color picker grid, and positioning options, overlaid on HTML/CSS code and a list of webpage files

Web designs can write CSS files in BBEdit – image credit: Bare Bones Software

"As word spread, we were able to help folks understand that the internet was built with text that you could treat as data or you could treat it as a document," he continued. "So the next wave brought in internet architects, cryptanalysts, and scientists from unexpected disciplines."

Thos unexpected disciplines, he said, included "writers and other content creators-folks who see their text not as data but as words, and who want as little as possible between them and their words."

Overall design

Beyond the simplicity of writing in BBEdit, part of this get-on-with-it design is in how it stores your documents. Unlike most apps other than Ulysses, BBEdit can in theory be a single repository for all of your writing.

So rather than opening and saving documents all the time, you can just open BBEdit, press Command-N and be writing in a new document. Everything you’ve written before is still there, listed in a navigation bar on the left, until you choose to close it.

You don’t even have to save that previous work before you start a new one.

But then if you want to, you can. Anything you write can remain in BBEdit’s list on the left of the screen, but you can also save any of it out to separate, individual .txt files.

Mac desktop showing a text editor window titled

You can keep all of your recent documents open in BBEdit. Or don’t even ever save them, just write in this text editor and export the final text to wherever you need it to go.

Then it might not require you to set anything up, and it might not get in your way, but BBEdit is built to help you. So for example, if you type an opening quote mark, BBedit immediately adds a closing one right after it and leaves your cursor in the middle.

It’s the same with brackets. Type an opening bracket, and you get that plus a closing one for free. That’s chiefly convenient in that it means you can’t ever forget to include a closing speech mark or bracket, you’d have to choose to delete them.

If you’re a programmer, that can also make the difference between your code working or not. It’s easy to forget to close brackets, unless you’re writing in BBEdit.

But it is also a little pain when you’re writing dialog. So often you’re going to follow a speech with a phrase like "she said," and to do that you have to cursor over the automatically-inserted end quote mark.

Split computer screen showing Markdown text editor on the left and rendered Markdown documentation with browser developer tools panel open on the right, demonstrating live preview and HTML structure

You can set BBEdit to wrap words and it also supports Markdown, plus a split-screen view.

You may well have just shrugged at that, and if you did, it’s because you are the true target audience for BBEdit. While any type of writer can use it, and every type of writer does, it is best for coders and programmers.

These are people whose writing tends to be in short lines rather than paragraphs. So they may never even need to notice something about BBEdit lines that will strike a novelist or a journalist immediately.

That’s how, by default, BBEdit does not wrap text at the end of the screen or at some margin stop. You could write a novel on the same long single line if you wanted to.

But then you could alternatively go to Settings, Editor Defaults, and turn on Soft Wrap. Set it to wrap your text at a specific character count, at the width of your default page size for printing, or the width of the window.

There are hundreds upon hundreds of settings you can change, but none that you have to in order to start writing. You can readily believe the gag that it has "millions" of new features, but this is still not an app where you have to study the manual.

Although, actually, there is a manual. In another throwback to the early days of the Mac, there exists to this day a 456-page BBEdit manual, and it is constantly updated.

How it got here

BBEdit began as this way to address a problem on the Mac. Today you know TextEdit as the name of Apple’s free text editing app, but back in the day, it was what the company called its text handling API.

If you wanted to include text in your app, from a word processor to just labels on a drawing, you could use this API and it did the job. But it wouldn’t handle anything over 32K, it wouldn’t handle anything long.

BBEdit was made to show that longer files could be handled on the Mac. It did the job, it was popular, and in 1992 when it first came out, BBEdit was also free.

Code editor showing CSS file with syntax highlighting and an autocomplete dropdown listing border-related properties as a developer types a style rule for a website layout

As well as automatically adding closing brackets for any user, if you’re a programmer or web designer then BBEdit will offer auto-completion suggestions – image credit: Bare Bones Software

That didn’t last. Around a year later, Bare Bones Software launched version 2.5 and began selling it for $99 (around $230 in today’s money). Users of the previous free ones, or owners of various programming languages such as THINK Pascal, could buy it for $49.

At the same time, Bare Bones Software released a cut-down edition called BBEdit Lite. That was free, and users of it could upgrade to the full BBEdit for $49.

They could get it on a floppy disk by mailing — not emailing, regular postal mailing — the developer on an Massachusetts address. Please enclose a check for $5.

BBEdit Lite no longer exists, and nor does its brief replacement, TextWrangler. That was released in 2003 for $49, offering what Bare Bones Software thought would be a compelling mix of BBEdit features for programmers and database administrators.

Compared to some apps, TextWrangler did well, sticking around 14 years, even if it quickly became free. It may have seemed like it spent much of that time nagging users to upgrade to BBEdit, but come 2017, there was no longer any choice.

TextWrangler was ostensibly folded back into BBEdit and users were encouraged to move to the full app.

"We promise that you will feel right at home, because BBEdit and TextWrangler are identical in every way that you’re used to," said the company at the time. "They’re built on the same foundation, by the same developers, with the same care; and they work the same way."

Behind the scenes, that must’ve been a reason to drop TextWrangler. Maintaining the same code base across two apps was surely taking up time that could be spent developing new features.

It certainly wasn’t the lure of cash from upgraders that made Bare Bones Software do this. As welcome as that would doubtlessly be, the company also sweetened the deal.

If you bought or upgraded to BBEdit at that time, you could use it for free for 30 days. It’s not unusual to have such a trial period, but what happens afterward is much more rare.

Bare Bones Software made it so that you could continue using the app for free, with just a limited feature set. What it left in was generous, and really it created what was effectively a new, free, BBEdit Lite in all but name.

BBEdit and Apple

If you’re a long-time Mac user, you may distantly remember technologies such as OpenDoc, but you’re unlikely to have used any of it. Bare Bones Software has, it even added OpenDoc support to BBEdit.

BBEdit has been a good Mac citizen from the start but sometimes you don’t know what you’ve got until it’s gone. That happened with Apple and BBEdit over the Mac App Store.

While it’s never really seen a fraction of the business that the iPhone App Store has, the Mac version has been around longer than you think. It debuted in 2011 and BBEDit was right there as one of the very first products on the Mac App Store.

By 2014, though, it was gone.

The reasons given at the time were not very specific, but they centered on how the Mac App Store sandboxes apps. Any app on the store is deliberately limited to prevent it being able to control the user’s Mac or other apps.

In BBEdit’s case, the app was reportedly not allowed full access to the user’s files and folders. It needed to be able to open and save wherever the user wanted, and at the time, Apple put limits on this.

There was also, though, an issue of money. While BBEdit was a one-time purchase, Bare Bones Software has always issued paid upgrades at intervals.

The Mac App Store did not allow for paid upgrades once a user had bought the app. So the company could have gained a true believer but never know it, never profit from it, and that user would never know what new features were available.

By 2019, enough developers had complained about sandboxing that Apple made some changes. They were sufficient to mean that BBEdit came back on the Mac App Store, but perhaps chiefly because there was now a solution to the upgrade problem.

Three colorful article cards: a serious man portrait titled Where Respect Is Due, a stylized crowd cutting paper titled BFGs: A Writer's Secret Weapon, and abstract coders titled Code. Write. Munge.

Apple welcomed BBEdit back to the Mac App Store with a trio of promotions – image credit: Apple

While Apple still does not allow paid upgrades, it both allows and encourages subscriptions. So today you can buy BBEdit 16 from the developer for a one-off $60, but you can subscribe via the Mac App Store for $5/month, or $50/year.

And if you were looking when BBEdit returned to the Mac App Store in 2019, you will have seen Apple rushing to make up for arguably ignoring it. That return saw an App Store front page profile of its main developer, Siegel, plus an article about BBEdit’s search features, and another about its overall feature set.

At that time, Apple’s senior vice president of worldwide marketing was Phil Schiller, who oversaw the App Store. He tweeted a welcome back to BBEdit, although that tweet is no longer available since Schiller quit Twitter/X back in 2022.

There’s something missing

BBEdit sounds perfect. Unless you’re a novelist or a screenwriter, or perhaps even a non-fiction author, maybe it is. And if you are any of those things, you can make BBEdit work for you if its comprehensive yet no-nonsense feature set appeals.

What you can’t do is make BBEdit work on anything other than a Mac. There’s no PC version, for example, although if you cared about that you wouldn’t be reading this far into an AppleInsider article.

The real missing piece is the iPad.

It’s been some years since Bare Bones Software has talked about why it’s stayed Mac-only. So perhaps things will change, but 37 years after starting it, and 34 since it went on sale, BBEdit is a legend among macOS apps.

Enough so that it’s one of the very, very few apps I would enthuse about this much. Bare Bones Software and Naomi Pearce helped out with finding images like that ancient photo of the boxed version, but they haven’t read this prior to publication, and they certainly didn’t pay for it.

Try it out for yourself. Once again, it may not be the right writing tool for you, but if it is, you’ll become a fan.

BBEdit 16 for Mac is available for $60 direct from the developer as a one-time purchase. You can also get it from the Mac App Store on subscription for $5/month, or $50/year.

AppleInsider News

Why are databases so hard?

https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgCLW1tV9SsA-5gIBDDhoN-blfIzAECOag2YxyZZGigmK7V-6u6wi3EYODcxfDMvTILiGpw3m1AuN_uX2D-PwX3XZeCpWwJhlUM9zv7nFxbkljJBml754KlanoZOyA9z8xNMO0smkuFuyvnkjVU1WNkXvCKRu4kSdE5nSQmwbCr28kjHs8hjTpbTbQfBaw/s320/bobross.webp

You’ve probably all experienced it; another outage and the database is the root cause.  Why are databases such a frequent cause of problems in most tech stacks? Why can’t we seem to solve these problems industry-wide?  Are database engineers and database admins just bad at their jobs?

Over my career as a database reliability engineer, I’ve come to a conclusion which I don’t see repeated often:

All practical implementations have to balance the opposing concerns of correctness vs. performance & availability (this is kind of similar to CAP theorem , but not exactly the same).   Perfect correctness with no data loss across geographic distances would result in a database which is too slow or too costly to be useful for most applications. And these constraints cannot be overcome because it’s the physical bounds of reality which imposes these limits.

Why geographic distances? I’ll explain this step-by-step below.

Step 1: A Single Isolated Database Instance

You start with installing a copy of PostgreSQL or MySQL on a single instance, or using a managed database product from a cloud provider such as AWS’s RDS.  At this scale the database performs everything you need.  Full ACID compliance, fast reads and writes, transactions all work beautifully.  We’re done, right?

A happy little database on its own.  You’ll never see one this happy again.

Well only if you ignore that hardware and VM instances are not flawless.  Despite the database software being more than adequate, sometimes that underlying hardware or VM infrastructure will fail.  The naive response is to just wait until the original instance can be restored and you continue on your merry way.  However this could take minutes, or hours, or days.  If you have customers paying to use your service, they’re not going to be so patient.  So now you need High Availability! 

Step 2: High Availability

High Availability means that you want your system to recover to a usable state as quickly as possible.  How quick? It depends on the specific implementation.  For me, <10 seconds is common.  <1 second is the goal.  For systems like RDS their default gives you ~2 minutes average recovery time, with options that will get that down to ~30 seconds.  Not really HA-enough for my taste, but for most people it’s a vast improvement over hours or days of outage!

BUT THERE IS A TRADEOFF — A COST!

Do you see it yet? It’s that improving reliability means having another copy of your database ready to take over when the primary fails. Maintaining that copy takes time. Time is the tradeoff.

Every time you write data to your primary/live instance, that data must be recorded somewhere else where it will be available when another database instance takes over to maintain availability. Well, you could certainly build a system where this isn’t true, but imagine your customer’s surprise when you flip from one database instance to another and data they thought they had persisted suddenly disappears.  Even worse is the problems you would invite when you flip back to the original copy of the database and that data suddenly appears again.  This is our "correctness" problem.

 The correctness problem is this: when we have to maintain multiple copies of our "source of truth" data, how can we make sure they all stay in-sync?

I have some good news and bad news on that front: the good news is that we absolutely can keep all our copies perfectly in-sync and ensure perfect correctness.  The bad news is our database system will now be so slow it’s probably unusable on a practical level.

 It works like this:

 When a request comes to our primary/active database to update/insert/delete data, we can pause the transaction at the time of commit and go transfer that transaction data to our other copies.  Only once we have confirmed the data has been durably persisted to our other copies do we finish the commit and return a success to the original transaction’s client.  This adds time to the client’s request.  They have to wait for data to be transferred over the network between our databases.  For two database instances in the same datacenter, this could be microseconds — not terrible, maybe not even noticeable.

 However, that’s not the end of the problems we’ve added.  Now what happens if our backup database fails and can no longer accept updates, even when it’s not being actively used? To maintain correctness we would have to stop accepting writes to the primary database as well!  It’s the only way to ensure they always remain perfectly in sync is to treat a failure of one node as a whole-system failure.

 Wait, we were supposed to be increasing availability.  Did we just actually decrease it instead? Also when the primary database fails and we flip to the secondary we now no longer have a backup copy and we lose HA properties until the other instance is restored.

 We could just run more backup copies, but now we have more data transfers to keep everything in-sync.  We could just say we only need 2 out of N nodes to be in-sync at all times and mark the others as unusable temporarily until they can re-sync.  Or is that 3 out of N, so that we have a backup-for-the backup.  And our cost to serve a single copy of our data set has gone up to what? 3x? 5x?  It’s starting to get more expensive now too.

Now I hope you’re starting to see the complexity of the problem here.  We could go into permutations of redundant architectures until the cows come home, but I’ll spare you.  Suffice it to say that every single architecture we could examine or invent is going to run into the same fundamental limit — it takes time to keep copies of our data up-to-date perfectly. And the only way to mitigate the time constraint is to relax the correctness constraint. There is no way around this.

 And we’re not even done yet, because our highly available system still only operates in a single physical datacenter. A single disaster which takes out the whole datacenter still means we’re hosed.  Many companies just call this good enough and accept the risk (after all us-east-1 never goes down, right?)  But for others, their customers won’t be happy with an extended outage even if you can claim it’s not your fault.  For true fault-tolerance you need yet another copy of your data in some other physical location, usually far enough so that the same hurricane, or earthquake or power grid outage doesn’t affect both locations.  This is how we arrive at geographic distribution.

Step 3: Disaster Recovery & Geographic Databases

 The astute will note that this is just an extension of the same problem we have with transit times in our HA setup, now with larger distances involved.  This should be easy! How much more time could we possibly have to manage?

Let’s take New York to Los Angeles as an example.  If you were able to send data at the speed of light, it would take 16 milliseconds! And that’s a one-way trip.  To let our primary database receive a confirmation that the data was received we need a minimum of 32 ms.  And keep in mind this is the theoretical maximum the laws of physics allow for a straight-line path.  In practice even if our network was fiber from end-to-end, we still have stops at various routers along the way for processing.  A real network request therefore takes more like 66ms per trip, and a 130ms round-trip time.

I have yet to experience any commercial enterprise willing to accept database write latency of 130ms.  Using cloud services you might end up paying thousands, or even tens of thousands of dollars per year for a system that can process <100 write transactions per second.

 

Sad databases, so far apart.

 So what do you do?  How do you surmount the laws of physics? You don’t. You MUST compromise something.

And that’s the entire point of this article — you cannot escape the fundamental laws of physics.  You only choose what properties are desirable and know that you will be giving up other things.  If you absolutely cannot tolerate data loss, then your system will be incredibly slow (or costly).  If you want great performance and efficiency, there will be ways you can lose data.

 I talk about the laws of physics because someone might see the 130ms round-trip-time and think that we just need to do some fancy computing to optimize that.  Or change how we build networks. But even if we did that we’d get at most ~4x improvement. There is not even a single order of magnitude left between our current performance and the maximum allowed by the laws of physics.  We cannot optimize time much more than we already have!  No matter how advanced technology of the future becomes, this same problem will still exist until the end of the universe. Even at the speed of light, it takes a significant amount of time to move data.

 Most companies use a strategy of creating an HA cluster with strong consistency guarantees within a single datacenter only, and then using an "eventual consistency" approach to shipping data to another geographic location. If the need arises to run their application from a different geographic location, they call it "disaster recovery" and let clients know that recovery could take hours and some data loss is expected. This applies equally if you’re using async replication to a warm-standby database or if you’re taking backups or snapshots and filling in the gaps between full backups with transaction logs.  If your primary datacenter fails while processing user requests, there will always be some window of time where data written to your database at that location won’t make it to your backup. It could be seconds, or minutes. No matter what you cannot guarantee consistency with an async update model.

 This is why databases are hard — there’s never a perfect solution which will work all the time for all use-cases.  And this also ignores the entire other class of database problems which relate to availability which is what happens when someone writes a bad query that DOSes your database.  Just scale up your database, or just partition it, right? But as we see in this article, scaling a database is hard because everything takes time.  Partitioning is hard because you create new consistency problems.  Availability is hard because keeping things in-sync is hard.

A Postscript 

Communicating this is honestly the most challenging part of my job.  Initial development of most software projects starts with the single isolated database either from a cloud provider or just on someone’s laptop.  They develop against a non-distributed database system and a tiny database size and everything works! It’s blazing fast, it’s perfectly consistent. No durability issues, etc.  When engineers take this to production they are soon frustrated by the production database which seems slower and less reliable.  They underestimate that the production database has so much more demands and constraints against it.

Yet I will talk with an engineering team one week that stresses how important consistency guarantees are for them.  Sure, I can do that.  Then the next week I’ll talk to another team that demands the fastest performance possible.  Now we have a challenge.  Then after an incident I’ll be yelled at by a manager who says we need to make availability our highest priority because our largest customer is threatening to churn.  Then as we approach the end of our fiscal year I’ll have other people breathing down my neck saying we need to cut costs. Then this whole cycle repeats.

 You can try to fix this by running multiple database systems.  The slow-yet-scalable system; the ultra-fast-but-lossy system; the perfectly-consistent-but-tiny metadata store.  This is why so many companies run several different types of databases.  Redis for ephemeral data, MySQL/PostgreSQL for transactional guarantees, key-value stores for easy scaling of simple data.  But now the job is to make sure engineers are choosing the right location for their data.  Inevitably not all data will find the right home on the first try and migrating data from one system to another is always a big task which isn’t fun.  It all feels a bit like Sisyphus some days, but at least it’s job security!

Planet MySQL

The diagnostic data MongoDB Atlas doesn’t hand you

The diagnostic data MongoDB Atlas doesn’t hand you

Every MongoDB server keeps a flight recorder. It’s called FTDC, Full Time Diagnostic Data Capture, and it writes about 5,700 metrics every second into a folder called diagnostic.data, right next to your log. It’s delta-encoded and compressed so aggressively that days of history fit in a few hundred megabytes.

You’ve probably never looked at it. But if you’ve ever opened a performance ticket with MongoDB, it’s the first thing they asked you for, and there’s a good reason: when your cluster goes strange for twenty minutes on a Tuesday, this is usually the only artifact that can tell you what actually happened. Not a five-minute average. The WiredTiger ticket pool, second by second.

If you run your own servers, Percona Server for MongoDB, community MongoDB, whatever you manage yourself, that file is just sitting on disk. You copy it and you look at it.

On Atlas, the same file is written by the same code on a machine you’re paying for, and you can’t get to it. It isn’t in the UI. The log download gives you mongodb.gz and your audit logs and nothing else. Somebody asked how to do this on GitHub back in February 2021 and nobody ever answered.

There is a way. It just isn’t where you’d look. Everything below I ran against an Atlas M10 on MongoDB 8.0.29.


Ask the Admin API to build you a bundle

There’s an endpoint that packages FTDC on demand. Three calls and you have it:

BASE="https://cloud.mongodb.com/api/atlas/v1.0/groups/$GROUP_ID"
AUTH=(-u "$PUB:$PRIV" --digest -sS)

# 1. create the job
curl "${AUTH[@]}" -X POST "$BASE/logCollectionJobs" -H 'Content-Type: application/json' \
  -d '{"resourceType":"REPLICASET","resourceName":"<rs-name>","redacted":true,
      "sizeRequestedPerFileBytes":100000000,"logTypes":["FTDC"]}'

# 2. poll until it says SUCCESS
curl "${AUTH[@]}" "$BASE/logCollectionJobs/<job_id>"

# 3. download
curl "${AUTH[@]}" "$BASE/logCollectionJobs/<job_id>/download" -o ftdc.tar.gz

 

A few things that will trip you up. <rs-name> is the internal replica set name, not the display name you gave your cluster, run GET $BASE/processes and you’ll see it next to each host. You need a programmatic API key, not a database user. And if your organization requires an access list for API keys, add your IP first or the very first call comes back with ORG_REQUIRES_ACCESS_LIST.

What you get is the real thing: one diagnostic.data directory per replica set member, in exactly the layout every FTDC tool already understands.

Don’t ignore metrics.interim. That’s the chunk the server hasn’t flushed to a numbered file yet, and it holds your most recent samples. In my bundle the newest numbered file stopped at 22:33 while the interim carried data all the way to 22:38. If you’re chasing something that just happened, that’s the file you need.

 

Your data has a shorter shelf life than you think

Here’s the part that will hurt you if you don’t know it.

The bundle includes a metadata document with getCmdLineOpts in it, which tells you how Atlas starts mongod. There it is:

diagnosticDataCollectionDirectorySizeMB: 400

 

That’s twice the mongod default of 200 MB, and it’s a hard ceiling. When the directory fills up, the oldest file gets deleted. No warning, no archive.

How long 400 MB lasts depends entirely on how hard your cluster is working, because FTDC compresses by delta, a metric that sits still costs you almost nothing, a metric that moves every second costs real bytes. On an idle cluster I measured about 0.93 MB per 32 minutes per node, which works out to something close to ten days. On a busy production cluster, expect two to five.

So picture the usual sequence. Something goes wrong on a Thursday night. Nobody’s sure how bad it was. The postmortem gets scheduled for the following week, somebody finally asks what the cluster was actually doing at 3 AM, and the answer is gone. Not archived somewhere. Gone.

Collect during the incident, not during the retrospective. On your own servers this is a setting you control: raise diagnosticDataCollectionDirectorySizeMB, or copy the directory somewhere durable on a cron. On Atlas it’s a ceiling somebody else picked for you, and the only way around it is to pull the data yourself before it rolls off.


This shouldn’t have taken an afternoon

None of what I just showed you is documented anywhere.

Think about what FTDC actually is. It’s the first artifact MongoDB support asks for on a performance ticket. It contains no user data, it’s counters, and you can verify that yourself: I looked at 228 KB of diagnostic document and found 38 distinct strings, not one of them the name of a database or collection on the cluster. It’s the single most useful thing you can hand to somebody debugging your server, and it’s safe to share.

And on MongoDB’s own managed platform, the only way to get it is an endpoint that appears nowhere in the log download UI, isn’t mentioned in the Atlas docs, and sits on an API version that never made it to v2.

What you end up with is a two-tier arrangement. The engineers supporting your cluster work from the full second-by-second record. You work from a metrics page with a few dozen series and seven days of retention.

I don’t think anyone decided this. It reads like a capability nobody owned the job of surfacing, the endpoint exists and it works, after all. But intent doesn’t change what it costs you, and right now the gap is being filled by community projects with single-digit star counts. A “Download diagnostic data” button next to “Download logs” would close it in an afternoon.

This is the kind of thing that gets abstracted away when your database becomes somebody else’s service, and it’s almost never what anyone evaluates up front. You compare features and uptime. You don’t think to ask whether you’ll still be able to see what your own server was doing.


Now go read it

Once you have the folder, you need something to open it with, and the tooling here is thinner than the data deserves.

keyhole (https://github.com/simagix/keyhole) has been the reference for years and renders FTDC through Grafana. If you want dashboards and don’t mind standing up the stack, start there.

I built Big Hole (https://github.com/zelmario/Big-hole) for the other case, opening a capture the way you open a log file. It runs entirely in your browser: no backend, no container, nothing uploaded. You drop the folder in and it decodes on your machine. That turns out to matter a lot with this particular file, because the captures worth analysing usually belong to somebody else’s production cluster, and “nothing leaves your machine” is often the difference between being allowed to look at it and not. It opens the Atlas tarball as-is, puts every replica set member on one time axis, shows you who was primary when, overlays your mongod.log on the same timeline, and runs automated checks for the usual suspects, ticket pool exhaustion, cache pressure, flow control. MIT licensed, tested against MongoDB 4.4 through 8.0. You can see a live demo here: https://zelmario.github.io/Big-hole/

Pick whichever you like. Just don’t wait until you need it, by then the data you wanted is already gone.

 

 

The post The diagnostic data MongoDB Atlas doesn’t hand you appeared first on Percona.

Blog – Percona

MySQL Best Practice : not using date / time types, nor ENUM

Today, I was reminded of a MySQL Best Practice, probably generalizable to all databases : using simple types, not complex types.&nbsp; Such complex types to avoid include the date and time data types (including TIMESTAMP) and ENUM.&nbsp; Let’s see why.
A little history about this, Baron Schwartz, a MySQL Legend who is not involved in the community anymore, compared using the TIMESTAMP type toPlanet MySQL

How to Prepare a Database for a PCI-DSS Audit (2026)

https://cdn.prod.website-files.com/6717800cb1e973b8fc433b03/6a732bd845fb0c9df7fd7e71_How%20to%20Prepare%20a%20Database%20for%20a%20PCI-DSS%20Audit%20(2026).avif

A PCI-DSS audit lives or dies on what your database logs, encrypts, and restricts, not on what your firewall does. This guide walks through the exact sequence to get a MySQL, PostgreSQL, MongoDB, or MariaDB environment audit-ready before a QSA shows up in 2026.

TL;DR

  • How to prepare a database for a PCI-DSS audit in 2026 starts with scoping the CDE, not encryption.
  • Requirement 10 logging failures cause more audit findings than missing patches; centralize logs 90 days before the assessment.
  • PCI-DSS certified database audits catch scope creep and weak access control before the QSA does.
  • Skip native DB password policies for production PCI systems in 2026; enforce RBAC with unique service accounts instead.

Why this matters

Database findings account for a disproportionate share of failed PCI-DSS assessments, and the pattern repeats every cycle: teams harden the network perimeter and forget that Requirement 3 (protect stored cardholder data), Requirement 8 (unique IDs and access control), and Requirement 10 (log and monitor access) all point straight at the database layer. A QSA doesn’t ask if you have a firewall. They ask for a list of every database user with SELECT access to the cardholder data environment (CDE), and they want 90 days of audit log history to prove it.

If your database team has never sat through a PCI-DSS assessment, the gaps show up in access control and logging, not encryption. Most teams over-invest in TDE and under-invest in Requirement 10. Mydbops runs PCI-DSS and ISO certified PCI-DSS database services for fintech for exactly this reason: audit prep is a database engineering problem before it’s a compliance paperwork problem.

What you’ll need

  • A complete CDE inventory – every table, schema, replica, backup, and log stream that stores, processes, or transmits PAN, track data, or CVV. Missing a read replica is the single most common scoping mistake in 2026 audits.
  • Config file access – my.cnf, postgresql.conf, mongod.conf, or equivalent, plus current audit/log plugin settings.
  • An audit date – from your QSA engagement letter or internal compliance calendar. Work backward from it; most fixes below need 4 to 6 weeks of lead time.
  • Encryption documentation – key rotation schedule, TDE or column-level encryption config, and who holds the keys.
  • A DBA who can change production config without downtime – access control and audit logging changes touch live systems; this is where a database consulting services engagement usually gets pulled in.

The steps

1. Scope the cardholder data environment (CDE) first

Scoping tells you which databases the rest of PCI-DSS even applies to. Get it wrong and you either over-engineer systems that don’t touch cardholder data or, worse, miss one that does.

List every primary, replica, and backup instance that stores PAN, track data, or CVV, plus any staging or analytics database that receives a copy through ETL. In 2026, the most common scoping failure on compliance-heavy audits isn’t a production database, it’s a forgotten reporting replica or a QA environment seeded with production data.

Expected outcome: a signed-off CDE diagram with every database instance, replica, and backup location labeled in-scope or out-of-scope, reviewed by whoever owns the QSA relationship.

Common mistake: treating "out of scope" as a one-time label instead of re-verifying it after every schema change or new replica.

2. Encrypt cardholder data at rest and in transit

Requirement 3.4 and 4.2 require PAN to be unreadable wherever it’s stored and encrypted in transit between the application and the database.

Turn on TDE at the storage engine level for MySQL InnoDB, or use PostgreSQL’s pgcrypto or pg_tde, and enforce TLS 1.2 or higher on every connection string, including replication traffic between primary and replica. Truncate or tokenize PAN wherever full card numbers aren’t operationally necessary; most databases only need the last four digits for display.

Expected outcome: zero plaintext PAN in table dumps, query logs, or replication streams, with TLS enforced on 100% of client and inter-node connections.

Common mistake: encrypting the primary database and forgetting the same setting on read replicas, backups, or a disaster-recovery instance in another region.

3. Enforce least-privilege access control (Requirement 8)

Requirement 8.2.3 and 7.2 require unique IDs per user and access restricted to what a role actually needs. Shared "app_user" accounts with blanket SELECT and UPDATE across every table fail this every time.

Audit every database account, remove shared credentials, and map each remaining account to a named individual or service with the minimum grants that job requires. For fintech platforms in particular, database security audit services engagements usually start here, because payment schemas accumulate broad grants over years of ad-hoc fixes.

Expected outcome: a role matrix where every account is traceable to a person or a service, with no account holding write access to tables it doesn’t touch in production.

Common mistake: fixing application-layer roles but leaving DBA and monitoring tooling accounts with god-mode grants.

4. Turn on and centralize Requirement 10 audit logging

Requirement 10 needs a record of every access to cardholder data, available for at least 12 months, with the most recent 3 months immediately accessible.

Enable the MySQL Enterprise Audit plugin, Percona’s audit_log, pgAudit for PostgreSQL, or MongoDB’s native auditing, and ship logs to a central, tamper-evident store outside the database host itself. Set retention to 12 months minimum, and test that you can pull a 90-day log slice for a specific table in under an hour.

Expected outcome: centralized, queryable audit logs covering every SELECT, INSERT, UPDATE, and DELETE against CDE tables, retained for the full 12-month window.

Common mistake: enabling audit logging two weeks before the audit. You need 90 days of continuous history by the time the QSA asks for it, which makes this the step with the longest lead time on the list.

5. Patch and harden the database engine (Requirement 6)

Requirement 6.3.3 in PCI-DSS 4.0 requires critical and high-severity vulnerabilities patched within a defined window, and an unpatched CVE on a database engine is one of the fastest ways to fail a technical review.

Check the current version of MySQL, PostgreSQL, MongoDB, or MariaDB against the vendor’s security advisory list, patch anything flagged critical, and disable default accounts, sample databases, and unused stored procedures. Restrict network exposure so the database only accepts connections from the application and ProxySQL or PgBouncer tiers, never directly from the internet.

Expected outcome: no database instance running an end-of-life version or carrying an unpatched critical CVE.

Common mistake: patching production but skipping a staging or DR replica running an older engine version; QSAs check both.

6. Validate backup encryption and retention (Requirement 3.4, 9.5)

Encryption and retention rules extend to backups, not just live databases. An unencrypted backup file sitting on a shared drive is a full-scope finding on its own.

Confirm every backup job encrypts the output, verify the encryption key isn’t stored next to the backup, and check retention against your data retention policy rather than just your recovery-point objective. Restore-test at least one backup before the audit; a backup that can’t restore doesn’t count as a control.

Expected outcome: every backup, including cross-region and cold-storage copies, encrypted with keys managed separately from the storage location.

Common mistake: encrypting nightly backups but leaving an old manual export from a migration project sitting unencrypted on a file share.

7. Run a pre-audit vulnerability and configuration scan (Requirement 11.3)

Requirement 11.3 requires quarterly internal vulnerability scans and an annual penetration test. Running one against the database tier before the QSA does is the cheapest way to fix findings on your own terms.

Scan for default ports, weak TLS ciphers, missing patches, and misconfigured grants, then re-run the scan after remediation to confirm a clean pass. Budget one to two weeks between the scan and the audit date to fix what it finds.

Expected outcome: a documented scan report with zero open critical or high findings on any in-scope database.

Common mistake: scanning the production database but skipping the ProxySQL, HAProxy, or connection-pooling layer in front of it; that layer gets audited too.

8. Document everything the QSA will ask for

A QSA works from evidence, not intent. A control that isn’t documented with a timestamp and an owner might as well not exist during the interview.

Assemble the CDE diagram, access control matrix, audit log samples, patch records, encryption key management policy, and backup test results into one package before the assessment starts. Assign one owner who can answer follow-up questions on each document without pulling in three other people.

Expected outcome: a single evidence folder that maps one-to-one to the PCI-DSS requirements being tested.

Common mistake: having the right controls in place with no paper trail proving when they were implemented; QSAs test for "since when," not just "does it exist."

Troubleshooting

  • Audit logs are eating disk space fast. Ship logs off the database host to a log aggregator or object storage with lifecycle rules. Don’t disable logging to save space; that’s a bigger finding than the storage cost.
  • A legacy MySQL 5.7 instance is past end-of-life and can’t be patched. Isolate it behind stricter network controls immediately and put a migration date in the audit evidence package. QSAs accept a documented remediation plan more readily than silence.
  • A shared database mixes CDE and non-CDE data. Segment by schema or move non-PCI workloads to a separate instance. Mixing scope inflates the audit and increases the chance of a finding bleeding across systems.
  • Native database password policies don’t meet Requirement 8.3.6 (12+ characters, complexity). Enforce policy at the identity provider or connection layer instead of relying on database-native password rules, which are inconsistent across MySQL, PostgreSQL, and MongoDB.
  • A connection pooler sits between the app and database and isn’t covered by any audit log. Add logging at the proxy layer too; QSAs increasingly ask for the full connection path, not just the database engine.
  • Backup encryption keys are stored in the same bucket as the backups. Move keys to a separate KMS or vault; co-located keys and backups fail Requirement 3.4 on inspection.

Tools and resources

  • MySQL Enterprise Audit plugin or Percona’s audit_log plugin for MySQL and MariaDB installations
  • pgAudit for PostgreSQL, tuned to log DML on CDE tables specifically to avoid log bloat
  • ProxySQL or PgBouncer configured with connection-level logging in front of the database tier
  • pt-query-digest and Percona Toolkit for auditing query patterns against CDE tables before the scan
  • E-commerce platforms processing card payments directly on the database layer carry particular scoping risk; see the MySQL RDS audit-log retention guide for a practical retention and archival pattern that supports PCI-DSS logging without unnecessary storage cost

What to do next

Once the database-specific controls above are in place, audit prep expands to application-layer tokenization, network segmentation, and vendor risk management, all dependent on the CDE inventory built in step one. Teams operating high-volume MySQL RDS workloads can review the Swiggy AWS DMS RDS case study; the same audit-prep principle applies: nail the CDE map before anything else.

FAQ

What is PCI-DSS Requirement 10 for databases?

Requirement 10 requires you to log and monitor every access to cardholder data, including who accessed it and when. For databases this means audit logging on every SELECT, INSERT, UPDATE, and DELETE against CDE tables, retained for at least 12 months with 3 months immediately accessible.

How long must database audit logs be retained for PCI-DSS?

PCI-DSS Requirement 10 requires 12 months of audit log history, with the most recent 3 months available for immediate analysis. Logs older than 3 months can move to colder storage as long as they’re retrievable within a reasonable window.

Does encryption at rest satisfy PCI-DSS Requirement 3?

Encryption at rest covers part of Requirement 3, but you also need key management separate from the encrypted data, truncation or tokenization where full PAN isn’t needed, and encryption extended to backups and replicas, not just the primary database.

Can a shared database pass a PCI-DSS audit?

A database mixing cardholder data with unrelated workloads can pass, but only if access controls and logging are scoped to the CDE tables specifically. Most teams find it faster to segment the CDE into its own schema or instance than to prove isolation inside a shared one.

How much does a PCI-DSS database audit cost in 2026?

Cost depends on the number of in-scope database instances, the QSA’s scope, and how much remediation is needed before the assessment starts. Check current pricing directly with your QSA or compliance partner rather than assuming a flat rate.

What database engines are PCI-DSS compliant?

PCI-DSS doesn’t certify specific database engines. MySQL, PostgreSQL, MongoDB, MariaDB, and MSSQL can all pass an audit when configured with encryption, access control, and Requirement 10 logging in place. Compliance is a configuration and process outcome, not a feature of the engine itself.

How long does PCI-DSS database audit prep take?

Most database audit prep takes 4 to 6 weeks minimum, driven mainly by the 90-day audit log history Requirement 10 expects to see by assessment time. Scoping and access control cleanup can move faster, but logging history can’t be backfilled.

Is MongoDB PCI-DSS compliant?

MongoDB can support a PCI-DSS compliant deployment when native auditing, field-level encryption, and role-based access control are configured correctly. The database itself isn’t compliant out of the box; the configuration and monitoring around it determines the audit outcome.

One last thing

The finding that kills most database audits isn’t encryption, it’s Requirement 10 logging turned on too late. Teams spend weeks hardening TDE and TLS, then discover during the audit that centralized audit logs only cover the last 3 weeks instead of the 90 days a QSA expects. Turn on logging the day audit prep starts in 2026, not the week before the assessment.

Need a PCI-DSS ready database review?

PCI-DSS and ISO certified audits across MySQL, PostgreSQL, and MongoDB.

Planet for the MySQL Community

MySQL Backup Best Practices: A Complete Guide for DBAs in 2026

https://webyog.com/wp-content/uploads/2026/08/Backup-best-practices.png

Most teams believe they have a solid backup strategy. In practice, many have a backup process they set up once, have not tested recently, and would discover is broken only when they actually needed it most. 

This guide covers how to build a MySQL backup strategy that works reliably — not just in theory, but when it counts.

The One Rule That Matters Most

An untested backup is not a backup. It is a file you hope works. 

Before anything else, commit to testing your restore process at least once a month. Everything in this guide supports that goal.

Your Three Backup Options

Logical backup (mysqldump) exports your database as SQL statements. It is portable, straightforward, and easy to restore individual tables or databases from. Use –single transaction with InnoDB tables to avoid locking:

mysqldump –single-transaction mydb > mydb_backup.sql

Physical backup copies the raw database files. Faster for large databases, but must be restored to the same MySQL version and platform. Minimal locking, supports incremental backups. 

Binary log backup records every change to the database continuously. Combined with a full backup, binary logs let you restore to any point in time — not just the last full backup snapshot.

Designing Your Strategy

Three questions define your backup strategy: 

RPO and RTO Explained

Know both numbers before an incident, not during one. If restoring a full backup takes 6 hours and your RTO is 2 hours, you have a problem to solve before it becomes an emergency.

Automate Your Backups

Manual backups fail when people are sick, busy, or on vacation. Automation removes the dependency on memory.

Linux Automation 

A basic automated backup script using cron and mysqldump:

#!/bin/bash 
DATE=$(date +%F) 
mysqldump –single-transaction –all-databases \ 
| gzip > /backups/mysql/backup_$DATE.sql.gz 
if [ $? -ne 0 ]; then 
echo “Backup failed on $(hostname)” \ 
| mail -s “ALERT: MySQL Backup Failed” [email protected] 
fi

Scheduled via cron to run nightly at 2 AM:

0 2 * * * /usr/local/bin/mysql_backup.sh

Windows Automation with SQLyog

On Windows, SQLyog provides built-in backup scheduling with no scripting required. Configure the databases, output location, and schedule through the GUI. SQLyog handles execution and can send email notifications on completion or failure.

Compress and Protect Backups

Compress every backup before storing it. SQL dumps compress extremely well — typically 70–90% size reduction. Pipe mysqldump output through gzip:

mysqldump mydb | gzip > mydb_backup.sql.gz

For backups stored offsite or in cloud storage, encrypt them. A backup file containing sensitive data is a liability if storage access is ever misconfigured.

The Backup Checklist

Monitor Backup Health

Automated backups still need oversight. Common failure modes to watch for:

MONyog can alert you when disk space on monitored servers approaches a critical threshold — catching the “disk full” failure mode before your next backup runs.

Want to automate MySQL backups without writing scripts? Try SQLyog free and set up your first scheduled backup in minutes.

Frequently Asked Questions

How often should I back up my MySQL database?

Depends on your RPO — how much data loss is acceptable in a worst case. Nightly full backups with binary log shipping in between gives you both a daily recovery point and the ability to restore to any specific moment. For lower-stakes databases, weekly full backups may suffice.

What is the difference between RPO and RTO?

RPO (Recovery Point Objective) defines how much data you can afford to lose — it drives backup frequency. RTO (Recovery Time Objective) defines how quickly you must restore — it drives how you set up your restore process and whether you need standby infrastructure.

Is it safe to run mysqldump on a live production database?

Yes, if you use –single-transaction with InnoDB tables. This creates a consistent snapshot without locking tables during the backup. Avoid lock-based options in production — they block writes for the duration of the dump.

How do I know if a backup is valid?

The only way to know for certain is to restore it. File size monitoring catches obvious failures, but a file that looks correct can still be corrupt. Schedule regular restore tests to a staging instance.

How long should I keep backups?

At minimum, 7 days of daily backups. For regulated industries, compliance may require 30, 90, or 365 days. Compressed MySQL backups are inexpensive to store in cloud object storage, so err on the side of longer retention.

Where should I store MySQL backups?

Never only on the same server as the database — disk failure, ransomware, or an accidental deletion can destroy both simultaneously. Use a separate server or cloud storage (AWS S3, Azure Blob Storage, Google Cloud Storage). Follow the 3-2-1 rule.

Can SQLyog handle backups for multiple databases?

Yes. SQLyog’s backup scheduler lets you select individual databases or all databases on the server, schedule the job at your preferred time, and send notifications on completion or failure — all from a single interface.

Does replication replace the need for backups?

No. Replication copies changes from your source to replicas in near real time — including accidental DELETE statements or schema changes that break your application. Replication protects against hardware failure. Backups protect against human error and logical data corruption. You need both.

Planet for the MySQL Community

MySQL development as it happens

https://blogs.oracle.com/mysql/wp-content/uploads/sites/102/2026/08/MySQLDev_HI.png

Henrik Ingo
MySQL Community Architect

In May we hosted the first MySQL Contributor Summit 2026. We arrange such Summits every quarter, and it’s a forum where contributors come together and make proposals, and then discuss them, on what they wish to work on, or see someone else work on, in future MySQL versions. 

All of the presentations are now published as well, so you can follow the discussion even if you weren’t at the summit yourself. 

Proposed features were categorized into 4 work streams, and you can now find each of them on the MySQL Community Roadmap project board on GitHub. 

Why is this important? 

Nurturing an active contributor community is an important factor in the success of the MySQL open-source project. There are many ways and reasons you can contribute to MySQL:  

  1. To avoid having to maintain a patch, which fixes some problem you had with MySQL 
  1. To increase your career market value and professional development
  1. To get feedback and users for your ideas and projects, as MySQL is the most wide-spread open source database 

Regardless of your reasons, MySQL thrives off of your contributions. With your contributions:  

  • Bugs get fixed faster and more efficiently,  
  • Users can gain custom features that were requested by users, but that would never have been developed if the only option was to wait for the MySQL engineering team at Oracle to do it, 
  • MySQL can be a platform that helps you and your ideas reach and collaborate directly with a much larger user community 
  • Can grow a larger base of developers, who over time grow into senior MySQL or database developers 

Where can you participate? 

To follow what’s happening , or better yet, if you want to participate in the planning and development of the next version of MySQL , the best place to start is MySQL Community GitHub Discussions . These GitHub discussion threads are the central point for discussions related to actually writing code that goes into MySQL Server. But the discussion is not just about code, rather also including technical topics, proposals, roadmap feedback, and contribution process related questions.  

The MySQL Community Slack is a popular space for general community discussions and engagement. In other words, this is a good place for general discussions related to using MySQL as your database. 

Depending on your intended level of involvement, you should also sign up for a centralized Oracle user account, which gives you access to the discussion forums, the MySQL bug database, and other MySQL services. We keep maintaining the good old bugs database, and it will remain relevant but we will use GitHub issues for new development of features.   

And last but not least, more static information is found on the MySQL Community project’s Wiki. Here you can find the new MySQL Governance Model, the Developer Guide, The developer guide takes you through the whole process of submitting a Feature Request or Proposal, and all the way to a Pull Request. 

More resources about contributing to the MySQL community are also available in the Developer Zone on mysql.com

Early Access releases and Monthly Code Drops 

As reported on June 25, we now have a documented governance process, that is based on the assumption that there will be contributors, committers and project leads from various employers, and hopefully also some participating as their individual self. 

We have made two Early Access releases, 9.7.0 Early Access and 26.7.0 Early Access. Work is already underway to match those releases with a monthly code drop. This is of course essential for anyone who wants to work on some code contribution, to have a relatively recent snapshot of the MySQL source. A lot of work is also going into setting up all the workflow automation into the mysql-server github repository. 

Only when all of that is done, and operating somewhat smoothly, is it time for the open development of the MySQL development work to happen working live in the public mysql-server repo. This is scheduled to happen later this year in Phase 3 of the Community Engagement Plan. But for now we are already in good shape to receive pull requests from the MySQL community as part of Phase 2! 

From talks to action! 

So what is actually happening on the development frontier, now that we know where to look? Let me give you a quick summary of where each of those work streams are, at this moment in time. 

But first…MySQL 26.7. is out, and we are excited to report that the first improvement arising out of the May Contributor Summit, (Bug #119222 ) is already merged and will be part of the 26.7. release. The patch is a performance improvement for how MySQL returns data back to mysqlsh after a mysqlsh util.dumpInstance() call. In case of composite primary keys, it now uses all the primary key columns to break the result set into a smoother, more evenly distributed sequence of chunks. 

The authors of this first patch to make it into a release, are Sveta Smirnova and Kamil Holubicki from Percona! 

Percona – as most readers will know – have over the years already contributed many bug fixes and features to MySQL Server. So it is perhaps no surprise that they were quick to do one also in this new era of community contributions! 

Thank you Percona! And we hope to see many more contributions, small and large, as we continue to develop new versions of MySQL, together and in the open. 

AI and Cloud 

Most of the focus here is on bringing full featured vector indexes, distance functions and search, to MySQL. There were several alternatives on the table in the past months, that have been discussed back and forth. So far there seems to be consensus on the direction that this should be implemented natively, as a builtin index type that is a first class citizen both as far as extending the SQL syntax and parser, and the query optimizer are concerned. 

The design phase is still ongoing, and the team is looking into different vector index structures and algorithms in the field. In addition the the widely used HNSW (Hierarchical Navigable Small World) algorithm, there’s considerable interest in an algorithm developed at Google and known as ScaNN (Scalable Nearest Neighbor). Following the syntax design discussion, it also is looking increasingly likely that the team is thinking of supporting multiple different index  types. 

Notably there also exists a project by Alkin Teyzusal called MyVector, that is implemented as a MySQL Plugin and User Defined Function. We’ll take a closer look at both of these vector indexing algorithms and implementations in the next blog post. 

With vector support clearly leading the way as a high priority work item, there’s also a second projectin this category: Official MCP Server for  MySQL

This will allow various LLM based assistants and coding agents to access and explore MySQL in a standard way. The first version will be “read-only” in the sense that it is focused on exposing metrics for observability and on the other hand allowing the AI agents to discover the schema and query the data. 

Performance and Observability 

Performance improvements remain a constant topic of interest in the MySQL user community. As we have met with many of you both individually and at events like the contributor summit, often the leading topic are various performance improvements. 

Currently in the GitHub backlog there are a range of sub-components of the MySQL Server, ranging from InnoDB B+Tree performance to Replication, where contributors proposed performance improvements. But hhere are a couple of interesting themes I thought are worth highlighting: 

As a former sales engineer it warms my heart to see proposal from AWS separate performance improvements into “real issues”, that actually happen and cause problems in production, and “perceived problems”, that aren’t really a big problem in production, but cause the database to perform poorly in benchmarks.  (In this case, HammerDB.) So with that introduction, you might be thinking the proposal wanted us to focus on the issues that make production slower. But no, the whole point was that users evaluate databases with benchmarks, or at the very least they read about others who did run a benchmark. And this makes the “perceived” problem a very real problem, because it may cause the user to choose some other database, and then they won’t run MySQL in production at all! 

And what is it that promises not just 2x or even 4x improvement in HammerDB  results just like that: the RETURNING Clause. More specifically this is an extension to DELETE and UPDATE. If you add the RETURNING clause to a DML command, then you will be able select the column who’s values are returned to the caller. In practice this means you avoid doing a separate SELECT, since you can get the result within the single UPDATE or DELETE statement. Single roundtrip instead of many! 

In #24 InnoDB-based binary log Vinicius Grippa from ReadySet, presented a proposal to move the entire binary log inside of InnoDB. The main point of this move would be that for the typical MySQL transaction, both the tables and the binlog are now inside and managed by InnoDB, without the need for coordinating an XA-transaction between 2 separate subsystems. And in particular, removing the need for each of those subsystems doing their own fsync to disk. This ticket is one of the most active ones, with engineers from Oracle, Google and Alibaba discussing different approaches. 

The Hypergraph optimizer continues to receive a lot of attention. On the one hand there are improvements to the explain output and other observability improvements. Ultimately the goal is to make this optimizer strictly faster than the old MySQL optimizer for every conceivable use case. 

Issue #34 describes a service that should monitor also system events, such as disk or memory utilization. At first sight this appears to be an unnecessary feature, since there are already plenty of tools available that know how to monitor the Operating System. But it turns out that a) when you are diagnosing a database, in production, the server may not have the tools installed that you are used to working with, nor is it possible to easily install them either, and b) it’s just really convenient to have system level performance metrics available through the same channel where you get your database performance metrics from too. A single source for all your dashboards and performance comparisons. 

Developer and DBA Experience 

MySQL Workbench work falls under the Developer and DBA Experience bucket and issue #10 covers various updates needed to this popular GUI tool. 

Issue #12 captures a long standing topic that was also brought up at the Contributor Summit: In MySQL the keyword BOOL and BOOLEAN are really just aliases for TINYINT(1). It also proposes to add ARRAY and UUID as native data types. 

There’s a built-in tension in that discussion, since generally we encourage new datatypes to be developed as components! But for very primitive types like booleans and arrays, there’s still an argument they should be always available and therefore in the native set. 

A very interesting feature in this category is the proposal to add System versioned tables to MySQL. This means the capability to define snapshots in time of a table, and makes it possible to query past snapshots of the table. The proposal is to implement this as defined in the SQL:2011 standard. 

Extensibility and tooling 

A well designed modular architecture is important for any software project, but doubly so for popular open source projects. The MySQL Component framework is what allows contributors to create new data types, functions, and of course even entire storage engines, in a relatively independent fashion. To ensure that we can scale MySQL as an open source project, we will have to continue to evolve the MySQL Component Framework. 

MySQL was always modular 

Over its 30 year history, MySQL has introduced several frameworks that allow new functionality to be developed in a modular way. Most famously of course MySQL gave us the concept of pluggable storage engines, which has allowed developers of different database engines to benefit from MySQL providing the standard SQL user interface and functionality, and MySQL and MySQL users in turn benefited from having a rich collection of database engines packaged into a familiar user experience, yet allowing them to choose the best tool for each job.  

Another extension framework that’s been around for a long time is the API for adding User Defined Functions. UDFs allow users, or open source contributors, to extend the set of functions – both scalar and aggregate functions – from the default / built in set.  

MySQL 5.1 introduced the Plugin API, which made it possible to distribute and install features separately from the MySQL Server itself.  

MySQL Component Framework 

And finally from MySQL 8.0 onwards we have introduced the MySQL Component Framework, and since then every major version has added APIs to various aspects of MySQL Server functionality, that can be extended with components. For example various methods for password validation, or logging and auditing, have been implemented as components.  

The main improvement that components bring to the table: a well defined and enforced API. While for example the storage engine API is well defined in the sense that a storage engine needs to implement and provide certain things, it’s nevertheless the case that a storage engine plugin or a UDF plugin are loaded and live in the same address space as the rest of mysql-server. In principle there’s nothing stopping a plugin from calling any function and accessing any data it wants to. 

Components on the other hand do operate against an API, and these client components can only use the methods made available through that API. Of course, there are a couple hundred different such APIs – called services – so a component could call many of those APIs to do what it needs to do. 

Doubling down on the MySQL Component Framework 

Following the contributor summit, the focus has been on doubling down on the MySQL Component Framework as the mechanism to write extensions to the core MySQL server. There are still areas of the MySQL Server that aren’t yet available through MCF APIs. A big one is the query optimizer (#22).   

But what contributors have focused their efforts on so far is a feature that at the same time is much more basic and fundamental, and on the other hand is actually quite sweeping: User Defined Types (#674). The goal here is not just to allow end users to define new custom types for their use cases, but the goal is to make this API so complete and useful, that also built in (aka “native”) types could be implemented as components, without loss of functionality. 

For example the Vector data type has been mentioned as an example: The work to add Vector support is implementing it as a native or built in type, because it is such an important and widely requested feature. But this should not be a choice we need to make. In the future, there should not be any downsides to adding a type via the Component Framework. 

A vision for MySQL extensibility 

While current work is mostly focused on extending the “surface area” of MySQL Component Framework, we are also aware that this is nowhere near the full scope of what we need to build in the coming years. 

The ultimate power of the Component Framework is not just its use as a healthy way to practice good, modular, software architecture. The real goal is to unleash a mode of contributing to MySQL Server in a way where a majority of new code and features needn’t be contributed to the core MySQL server at all. In the future, a typical MySQL “contribution”: 

  • Should be possible to create and develop independently, without needing to coordinate or get “permission” with anyone employed at Oracle 
  • (In fact…) Should be possible and quite okay for two contributors to both implement the same feature, in slightly different ways, and let the users decide which one they want to use 
  • Should be possible for such extensions to clearly declare which component APIs they need to use, which they implement, and which versions they are compatible with 
  • And probably we need to have a way for one component to express the fact that it needs another component to be present. (dependencies) 
  • Finally, it should be possible for the independent 3rd party developer to publish their extension to MySQL functionality, and for end users to discover and install the extension, pretty much just as easily as they had installed core MySQL itself. 

A great example of a project that is showing us the way is MyVector, by Alkin Tezuysal. Alkin presented a talk on his experience implementing HNSW based vector indexing in MySQL, using the MySQL Component Framework

Utilities are extensions too 

But the above is not yet all: MySQL Server can also be extended by software that is completely separate from the MySQL Server. A great example are the different *SQL Proxy products that have provided us with so many powerful features from thread pooling to encryption.  

Tools used for observability, schema design, querying, reporting, dashboards… Various shells and clients… All of these are arguably “components” that extend and add value in the MySQL Ecossystem. 

And Storage Engines! 

While most of the development focus in the extensibility track is on extending the MySQL Component Framework, we shouldn’t forget that MySQL already has the tried and trusted framework that gave us so much cool database innovation over many decades already, The Storage Engine API. 

Our efforts to open up MySQL development to community contributions would not be complete without someone proposing to add a new storage engine. And we couldn’t be more excited about the fact that in the past weeks we’ve added not just one, but two, rather powerful and very state of the art engines to our roadmap dashboard: DuckDB (proposed by Alibaba), the database that has been taking the analytics / OLAP market by storm, and TidesDB, a new and very state of the art LSM engine. LSM architecture is typically used for write-optimized, MVCC type OLTP workloads. 

Both of these are scheduled to present at the next Contributor Summit.  

Upcoming Contributor Summits 

If you’ve read this far, and if you are anywhere near as excited as yours truly… You might be asking the question: When and  where are these contributor summits happening and where can you buy tickets? 

The answer is that these are by invitation request; if you would like to attend, reach out to the Community Team. These events are by and for developers who already are working on new MySQL features, or have made a proposal that they will be presenting and getting feedback on. 

The next summit is just around the corner: 

  •  August 5-6 in Colorado and virtually. 
  • November will be a 100% virtual summit 
  • February will be in Europe and adjacent to Fosdem and MySQL Belgium Days 
  • And after that likely in May at a location still to be determined

If you are working on or intend to propose some contribution to MySQL, and you would like to present in the contributor summit, or attend a summit, you can email HENRIK.INGO@ORACLE.COM or LENKA.KASPAROVA@ORACLE.COM.  

Get Involved 

And if you just want to roll your sleeves and start submitting issues and PRs… Sure, you can do that, there’s no hard requirement to present slides first. You can create a proposal in the mysql/mysql-community repo, and when you are ready to submit your first Pull Request, you’d typically do that against trunk in the mysql/mysql-server repo

And as a reminder, to try to keep some focus while we open up the development, we’ve chosen four strategic roadmap categories. As you submit your proposals, try to think of which of these your contribution most closely fits into: 

In the following blog posts I want to dive deeper into each of these four work streams, and give a bit more context around the rather technical discussions you will see if you start following the discussions in each ticket. Stay tuned for the next episode in MySQL development as it happens. 

Henrik Ingo
MySQL Community Architect

Henrik started his career in open source databases working at MySQL AB, shortly after graduating from college. Now, two decades and seven other database startups later, he is back to where he started, on the Oracle MySQL team. Prior to Oracle, he worked at DataStax where he led the project of porting DataStax Enterprise features back into the open source Apache Cassandra project, and at MongoDB, where he introduced the use of change point detection to automate discovery of performance regressions. Henrik is the author of the book “Open Life: The Philosophy of Open Source”, and its namesake blog.

Planet for the MySQL Community