Databricks CTO Matei Zaharia "said that Databricks had to keep track of scheduling a million things," remembers adjunct MIT professor Michael Stonebraker. " He said that this can’t be done with traditional operating system scheduling, and so this was done out of a Postgres database. And then he started to whine that Postgres was too slow, and I told him we can do better than that…." This resulted in DBOS — short for "database operating system" — which they teamed up to build with teams Stanford and MIT, according to The Next Platform:
They founded a company to commercialize the idea in April 2023 and secured $8.5 million initial seed funding to start building the real DBOS. Engine Ventures and Construct Capital led the funding, along with Sinewave and GutBrain Ventures… "The state that the operating system has to keep track of — memory, files, messages, and so on — is approximately linear to the resources you have got," says Stonebraker. "So without me saying another word, keeping track of operating system state is a database problem not addressed by current operating system schedulers. Moreover, OLTP [Online Transaction Processing] database performance has gone up dramatically, and that is why we thought instead of running the database system in user space on top of the operating system, why don’t we invert our thinking 180 degrees and run the operating system on top of the database, with all of the operating services are coded in SQL…?" For now, DBOS can give the same kind of performance as that full blown Linux operating system, and thanks to the distributed database underpinnings of its kernel, it can do things that a Linux kernel just cannot do… One is provide reliable execution, which means that if a program running atop DBOS is ever interrupted, it starts where it left off and does not have to redo its work from some arbitrary earlier point and does not crash and have to start from the beginning. And because every little bit of the state of the operating system — and therefore the applications that run atop it — is preserved, you can go backwards in time in the system and restart the operating system if it experiences some sort of anomaly, such as a bad piece of application software running or a hack attack. You can use this "time travel" feature, as Stonebraker calls it, to reproduce what are called heisenbugs — ones that are very hard to reproduce precisely because there is no shared state in the distributed Linux and Kubernetes environment and that are increasingly prevalent in a world of microservices. The other benefit of the DBOS is that it presents a smaller attack surface for hackers, which boosts security, and that you analyze the metrics of the operating system in place since they are already in a NoSQL database that can be queried rather than aggregating a bunch of log files from up and down the software stack to try to figure out what is going on… There is also a custom tier for DBOS, which we presume costs money, that can use other databases and datastores for user application data, stores more than three days of log data, can have multiple users per account, that adds email and Slack support with DBOS techies, and that is available on other clouds as well as AWS. The operating system kernel/scheduler "is itself largely a database," with services written in TypeScript, according to the article. The first iteration used the FoundationDB distributed key-value store for its scheduling core (open sourced by Apple in 2018), according to the article — "a blazingly fast NoSQL database… Stonebraker says there is no reason to believe that DBOS can’t scale across 1 million cores or more and support Java, Python, and other application languages as they are needed by customers…" And the article speculates they could take things even further. "There is no reason why DBOS cannot complete the circle and not only have a database as an operating system kernel, but also have a relational database as the file system for applications."
Read more of this story at Slashdot.
Slashdot