My Journey from Traditional Monolithic Architecture to Distributed SQL

Editor’s note: Bill Kendall wrote this post during his time at PingCAP. It was originally published in January 2022 and updated in August 2026 to reflect TiDB’s agent state stack.

Key Takeaways

• Monolithic databases hit a hard ceiling: one machine, one failure domain, and a cost curve that steepens as you scale up.
• Distributed database architecture scales out instead of up, spreading data across nodes that tolerate failure and rebalance automatically.
• Distributed SQL keeps transactions, joins, and relational modeling on top of that architecture, so you don’t trade correctness for scale.
• AI agents face the same gap now. Durable memory, state, and files need the same distributed foundation, not a fourth system bolted on.

Like many data management engineers, I started my career on Oracle. It was a pragmatic choice. Oracle was mature, reliable, and feature-rich, and I could build fast, very large, vertically scalable systems with it. When a database needed more resources, I scaled up: a bigger machine, all the way to Oracle’s flagship engineered system, Exadata.

Then the cloud arrived and the assumption underneath all of that work stopped holding. Distributed database architecture stopped being a topic I read about and became the job. I spent years moving large on-premises applications and their databases onto cloud infrastructure, and the database was always the hard part. That work convinced me the shift is not a stylistic preference. For any workload that has to scale out, span regions, and stay online through failure, distribution is the design that fits.

Four years after I first wrote this, the same shift is happening again in a different domain. AI agents are hitting the wall my applications hit in 2014: they have nowhere durable to keep what they know. This is the story of how I got from Oracle to distributed SQL, and where that road leads next.

Why Monolithic Database Architecture Worked for Its Era

Monolithic database architecture places compute and storage on a single server and scales vertically. One machine holds the data, runs the optimizer, executes transactions, and enforces consistency. When it runs out of headroom, you buy a larger machine. For decades that was the correct answer, and it is worth saying why before explaining why it stopped being enough.

Traditional relational databases were built for workloads that were largely predictable. A general ledger, an ERP install, or an order-entry system had a known user population, a known peak, and a growth curve a DBA could budget a year ahead. Centralizing compute and storage on one node made those workloads simpler to reason about. Transactions had exactly one place to commit. Consistency was a property of the machine rather than a property of a protocol. Tuning, backups, and capacity planning all pointed at a single target.

Oracle-style enterprise environments took that model close to its limit. Exadata paired the database with purpose-built storage and networking so one system could carry workloads that would have needed a room full of hardware a decade earlier. I built plenty of those, and they worked. When the workload fits inside one machine and the machine is engineered well, scale-up remains the simplest and often the cheapest architecture available.

Monolithic databases were not badly built. The shape of the workload changed underneath them.

Where Monolithic Systems Start to Break Down

Monolithic systems break down when the workload stops fitting on one machine, and no amount of tuning fixes that. Vertical scaling has a hard ceiling set by the largest server you can buy, and the cost curve toward that ceiling is not linear. The last increment of capacity is always the most expensive one.

Moving the applications to the cloud was relatively trivial. I could distribute them geographically, and most of my clients had already moved to microservices designed to be cloud-native and horizontally distributed. The Oracle database was far harder. It was designed from the beginning to scale vertically. I could scale up, but not out to where the cloud required me to be. Traditional relational systems were becoming the bottleneck for mission-critical applications, so I pivoted in a big way.

The failure modes are consistent across the migrations I have worked on:

• Hardware ceilings. Once you are on the largest available instance, the only lever left is rewriting the application.
• The cost of vertical scale. Each doubling of capacity costs more than the last, and you pay for peak around the clock.
• Availability risk. A single primary is a single point of failure. Failover means a window where writes stop.
• Operational complexity. Teams that outgrow one node shard by hand, pushing routing, rebalancing, and cross-shard joins into application code.
• Global application demands. Users on three continents reading from one region pay for that geography in every request.
• Mixed workloads. Analytics against a transactional primary degrades both, so most teams add a warehouse and accept a lag measured in tens of minutes.

None of these stay technical for long. Manual sharding slows releases, peak-provisioned hardware inflates cost, and failover windows become customer-facing incidents. The architecture sets the ceiling on what the product can do.

What Distributed Database Architecture Changes

Distributed database architecture spreads data and query processing across many nodes that behave as one logical database. Instead of replacing a server to grow, you add nodes and the system rebalances data across them. That single change, scale out rather than scale up, cascades into how availability, sharding, and consistency all work.

In early 2014 I was sent a newly released Google Research paper, Spanner: Google’s Globally-Distributed Database. It described a clustered distributed database that could span one data center or several. Data was sharded automatically. Copies were distributed and balanced across the cluster. Applications could connect to any node and reach any data. Nodes could be added or removed to scale in either direction. It was the first design I had read that treated the database the way my applications were already being treated.

The mechanics matter more than the label:

• Horizontal scaling and data distribution. The data set is divided into ranges spread across nodes, and the cluster moves ranges to keep load even as nodes join or leave.
• Automatic sharding. Splitting and placement happen inside the database, not in application code.
• Replication and fault tolerance. Each range keeps multiple replicas. A consensus protocol, Raft in TiDB’s case, elects a leader per range so the cluster keeps serving when a node dies.
• Coordination of compute and storage. Separating the SQL layer from the storage layer lets each scale on its own axis, which makes elastic capacity practical.
• Consistency models. Eventual consistency is cheaper to build and harder to reason about. Strong consistency costs coordination and returns correctness.
• Operational resilience. Rolling upgrades, node replacement, and region failure become routine operations rather than change-control events.

Monolithic vs. Distributed Architecture

Set side by side, the two architectures differ on nearly every axis that matters in production:

Dimension	Monolithic architecture	Distributed architecture
Scaling model	Scale up. Replace the server with a larger one.	Scale out. Add nodes and let the cluster rebalance.
Data placement	One copy on one machine, plus backups and standbys.	Data sharded and replicated across many nodes.
Failure domain	The server is the failure domain. Losing it means failover or downtime.	A node is the failure domain. The cluster keeps serving on surviving replicas.
Capacity ceiling	The largest machine you can buy and afford.	Practically bounded by cluster size, not by any single machine.
Mixed workloads	Analytics competes with transactions for the same resources, or moves to a separate warehouse.	Row and column stores can serve transactions and analytics from the same cluster.
Operational model	Tune one instance. Shard manually at the application layer if you outgrow it.	Manage a cluster. Sharding, rebalancing, and failover happen inside the database.

The Rise of Distributed SQL

Distributed SQL is the branch of distributed database architecture that keeps SQL, relational modeling, and transactional consistency while distributing data across nodes. It sits one level below the broader category: every distributed SQL database is a distributed database, but not every distributed database gives you transactions and joins.

That distinction exists because the first wave of distributed systems dropped the relational model to get scale. NoSQL stores solved real problems, and for append-heavy or schema-loose workloads they are still the right tool. What many teams discovered is that they had traded away things they could not do without. Multi-row transactions moved into application code, where they became a source of subtle bugs. Ad hoc queries required a second system. Referential integrity became a convention rather than a guarantee. For a payments ledger or an inventory system, “the write probably landed” is not an acceptable answer.

SQL survived because the relational model is a good fit for how businesses describe their data, and because thirty years of tooling, drivers, ORMs, and hiring pools sit on top of it. Distributed SQL is the attempt to keep that surface while changing what runs underneath: familiar relational semantics and ACID transactions on an architecture that scales horizontally and survives node loss.

At PingCAP we took the multi-node auto-sharding architecture at the core of distributed SQL and extended it for flexibility beyond what Spanner and other distributed SQL databases offered. That database is TiDB: an open source, MySQL-compatible distributed SQL database. The MySQL wire protocol compatibility is deliberate. It means an existing application, driver, or DBA workflow arrives already knowing how to talk to the database.

TiDB Design Tenets for Distributed Database Architecture

TiDB operationalizes the properties described above rather than simply claiming the distributed label. In 2015 our founders built TiDB from scratch alongside a large and active open source community on GitHub, working from a set of design tenets meant to produce a cloud-native distributed SQL architecture. These are the core tenets:

• Scale without complexity. The database scales horizontally without operator intervention. Nodes join the cluster and TiDB rebalances data onto them automatically. Scaling in works the same way.
• Resiliency always on. TiDB survives node failures while applications continue processing transactions on the surviving nodes, which drives time to recovery toward zero.
• Consistent ACID transactions. For transactional systems you must always read the most recent correct copy of the data. That matters most in financial services and inventory. TiDB guarantees high isolation levels at the transaction level to preserve consistency.
• Support for both OLTP and OLAP workloads. Many distributed SQL databases handle only transactional traffic. TiDB serves transactional, analytical, and mixed workloads by adding a columnar store, TiFlash, alongside the row store to handle large aggregations and analytical functions. This is hybrid transactional/analytical processing, or HTAP, and the cost-based optimizer decides which store answers each query.
• A common API interface. TiDB was built by developers for developers, so access happens through a widely understood SQL interface. We chose wire compatibility with MySQL, which lets application developers and DBAs use one of the most established SQL protocols in production today.
• Install and run anywhere. The database deploys on premises, in a single cloud, or across multiple clouds, on VMs or containerized with Kubernetes. Deployment architecture can change as the organization’s requirements change.

Operational Features That Make the Tenets Usable

Design tenets produce a capable database. Operational features make it something a team can actually run. TiDB exposes rich metrics and monitoring through Prometheus and Grafana plus a built-in TiDB dashboard. Role-based access control and TLS encryption protect data in flight and at rest. Built-in asynchronous change data capture moves data into and out of the cluster, which matters for migrations and for feeding downstream systems. TiDB Cloud takes the same architecture and runs it as a managed service, which is how most teams start today.

Why Distributed Database Architecture Matters for AI Agents

AI agents are stateless by default, the same architectural gap monolithic databases left behind. A model call takes input and returns output. Nothing about it remembers the last call. The context window looks like memory, but it is a buffer: bounded, scoped to one session, and gone when the process ends. An agent that ran for six hours yesterday starts today knowing nothing about it.

Production agents need more than a transcript. The state that must survive falls into categories:

• Facts and preferences learned in earlier sessions, which must persist across machines and users.
• Session and task history, so the agent can answer what it did, when, and why.
• Tool outputs and intermediate results, expensive to regenerate and needed several steps later.
• Files and artifacts: repositories, generated code, logs, and documents produced during a run.
• Retrieval metadata tying embeddings to their source, version, and access rules.

Two access patterns run against that state, and both have to work. Agents need exact lookups with filters, joins, and transactions: which user owns this record, has this step already run. They also need semantic retrieval when nobody knows the exact key. A single agent turn frequently needs both.

The common workaround is to stitch systems together: Postgres for records, a vector database for embeddings, object storage for files, and Redis for session state. Every boundary adds a consistency problem. When memory is revised, superseded, or deleted, the change has to land everywhere or the agent acts on stale facts. This is why we treat agent memory and state as a database problem rather than a retrieval add-on. A distributed SQL database with native vector search can hold structured records, embeddings, and searchable text in one place, run hybrid retrieval in a single query, and give mutable memory transactional guarantees.

The TiDB Agent State Stack: Zero, mem9, and drive9

The TiDB agent state stack is three layers that together give an agent durable state, all powered by TiDB Cloud. TiDB Cloud Zero provides instant SQL state, mem9 provides persistent memory, and drive9 provides a persistent workspace for files. Because they share one backend, memory, operational records, retrieval metadata, files, and execution history stay connected as a deployment grows.

TiDB Cloud Zero for Instant SQL State

TiDB Cloud Zero is a zero sign-up, zero configuration way to provision a real MySQL-compatible TiDB Cloud database in seconds. There is no registration step and no cluster to size. The instance is TLS-secured and supports joins, transactions, and standard SQL, along with native vector search and full-text search, so an agent can store embeddings next to relational data instead of splitting them across systems.

Zero instances are disposable by design, with a 30-day lifespan. That fits agent sessions, CI runs, demos, and prototypes. When a prototype turns into something you want to keep, a three-click claim converts the instance into a persistent TiDB Cloud Starter database and carries the data and schema over. Zero also works well as the database behind an MCP server or custom agent tooling; it is not itself an MCP server, but it gives MCP tools a SQL backend they can provision on demand. TiDB Cloud Zero is currently in public preview.

mem9 for Persistent Agent Memory

mem9 is a persistent memory layer for AI agents. It gives coding agents, custom tools, and multi-agent systems a shared memory space that survives across sessions, machines, and users, so what one agent learns is available to the next one. It supports hybrid retrieval, combining vector similarity and keyword search in the same query, and generates embeddings server-side so there is no separate embedding pipeline to run.

Two design decisions matter more than the API surface. First, mem9 treats memory as a system rather than a store: ingestion, deduplication, ranking, retrieval, and evaluation are all part of it, because cognitive memory gets revised, superseded, expired, and purged. Mutable memory needs strong consistency, which is why it runs on TiDB Cloud. Second, memory is inspectable. The mem9 interface lets you visualize, manage, import, and export what an agent remembers, which is what makes it possible to trust and correct. mem9 integrates with agent harnesses including OpenClaw, OpenCode, Claude Code, Codex, and Dify apps, and any HTTP client can read and write to the same memory space.

drive9 for Persistent Agent Files

drive9 is a durable filesystem for agents that do real work on disk. Coding agents clone repositories, install dependencies, edit files, run tests, produce logs, and try several approaches in parallel. The sandbox they run in may last minutes; the work usually needs to last longer. drive9 keeps that workspace state outside the sandbox, so it survives resets and can be mounted from a different environment entirely.

You mount a drive9 workspace as a local filesystem and use normal tools against it. Workspaces can be forked for parallel attempts, checkpointed, rolled back, and committed with server-side conflict detection. Files are embedded and indexed automatically on write, so an agent can search its own workspace by meaning instead of guessing at paths. Local disks run the process and Git stores the final result; drive9 holds the working state in between.

Framed against the rest of this article, none of the three is a departure. They are the same architectural shift applied to a new workload: state moves out of a fragile, session-bound, single-machine home and into a distributed system that expects to outlive any one process.

From Monolithic Databases to Agent-Ready Architecture

After more than twenty years around Oracle, I can say plainly that distributed SQL changed what I am able to build. I have watched teams solve problems that were not solvable under a scale-up architecture, and I have watched that unlock a level of creativity in the community that keeps surprising me.

The lesson that has held up is that architectural decisions compound. Choosing a scale-up database in 2010 was defensible, and it still set the ceiling on what those systems could do in 2020. The same compounding is underway right now in AI infrastructure. Teams building agents on ephemeral state are making a decision whose cost arrives later, when the agent has to answer what it knew last Tuesday, or when memory has to move across machines, or when a sandbox reset takes an afternoon of work with it. The architecture that let databases scale for the cloud era is what makes durable agent state practical now, and for the same reasons: distribution, replication, consistency, and retrieval in one place instead of four.

If you want to see what that feels like in practice, spin up a TiDB Cloud Zero instance. It takes seconds, requires no sign-up, and gives you a MySQL-compatible database with vector search that you can point an agent at immediately.

FAQ

What Is Distributed Database Architecture?

Distributed database architecture spreads data and processing across multiple nodes instead of concentrating them on a single centralized server. This architecture improves scalability, availability, fault tolerance, and performance for applications that need to serve users or workloads at scale.

How Is Distributed Database Architecture Different from Monolithic Database Architecture?

Monolithic database architecture typically relies on a centralized system that scales vertically, while distributed database architecture scales horizontally across multiple nodes. Distributed systems handle larger workloads, tolerate failures, and support more flexible deployment models.

Why Did Companies Move from Monolithic Databases to Distributed Databases?

Companies moved toward distributed databases because modern applications require higher availability, global scale, real-time data access, and resilience than traditional scale-up architectures can easily provide. Distributed databases reduce single points of failure and make it easier to scale as workloads grow.

Where Does Distributed SQL Fit into Distributed Database Architecture?

Distributed SQL is a type of distributed database architecture that preserves SQL, relational data modeling, and transactional consistency while distributing data across multiple nodes. It serves teams that need horizontal scale without giving up familiar relational database capabilities.

Why Do AI Agents Need a Distributed Database for Memory?

AI agents are stateless between calls, and a context window is a temporary buffer rather than durable memory. Agents need persistent state that outlives a session: facts learned earlier, task history, tool outputs, files, and retrieval metadata. A distributed database can hold structured records and embeddings together, serve exact lookups and semantic search from one query, and apply transactional guarantees to memory that gets updated, superseded, or deleted.