A Year of BonsaiDb: A retrospective and looking to the future

Written by Jonathan Johnson. Published 2022-03-19.

What is BonsaiDb?

BonsaiDb is a new database aiming to be the most developer-friendly Rust database. BonsaiDb has a unique feature set geared at solving many common data problems. We have a page dedicated to answering the question: What is BonsaiDb?. All source code is dual-licensed under the MIT and Apache License 2.0 licenses.

Today marks the one year anniversary of the initial commit to BonsaiDb. This project is largely a written by me (@Ecton), although I'm very thankful to have been accompanied along the way by @daxpedda. He is responsible for the QUIC networking layer, our currently removed OPAQUE-KE support, and above all, countless long discussions and debates that not only led to starting this project, but also to the easy-to-use API designs we have today.

In the last few months, we've also had two additional contributors: @ModProg and @vbmade2000. I'm grateful for their additions to the project, and I hope this year we see even more contributors as BonsaiDb gains momentum.

Beginning a real bonsai cluster

The name BonsaiDb was suggested by @daxpedda after countless times where we stumbled trying to pronounce the project's former name: PliantDb. When I announced the new name, I researched trying to grow a bonsai tree in the Las Vegas climate. My initial findings were that many of the iconic types are difficult to grow here, and I ended up shelving the idea.

As March was approaching, I knew I wanted to commemorate the anniversary of the first commit. I researched again and found two species that were great candidates for being able to be grown mostly indoors. After some deliberation, I decided to start with a Golden Gate (Tiger Bark) Ficus:

One Year Anniversary Golden Gate Ficus

After giving the tree another week or two of adjustment to its new home, I'll be pruning it for the first time and attempting to grow a second tree from its cuttings. Within the limits of being able to provide good living conditions for each tree, I hope to expand the "cluster" each year and will share progress pictures along the way!

The first year: Inspiration

The inspiration for BonsaiDb came from weeks and weeks of discussions between myself and @daxpedda. We were wanting to build an cluster architecture for an MMO that:

At the start of our discussions, I was dead-set on using PostgreSQL and Redis for our database/caching needs. However both PostgreSQL and Redis aren't easy to deploy and maintain in a highly available cluster, and if I chose managed solutions, they definitely weren't affordable.

When starting my last company, I chose CouchDB for various reasons that aren't worth diving into. Through the ten years of growing that company, I became intimately familiar with the simple yet powerful design of CouchDB. Several times I pondered rewriting CouchDB atop PostgreSQL using JSONB columns to see how it would perform. I never ended up taking any of those prototypes very far, but the thought process helped me appreciate and understand CouchDB's inner workings better.

One day while working on Cosmic Verge, it dawned on me: I had built a small wrapper around Sled that made it feel like a higher-level database. What if I went further and built a CouchDB-like API atop Sled?

The first year: Initial Goals

Initially, I was hesitant to tell anyone I was building a database. Despite relying on Sled for the low-level transactional storage, it still felt like one of those projects that is usually ill-advised. I finally felt confident enough in my reasoning of why I was building BonsaiDb that I wrote a detailed devlog describing the motivation for tackling this new project.

Early on, I never had high ambitions for BonsaiDb. After all, this was the first database I had ever written. The likelihood of me building a high-performance database seemed low. Instead, I wanted to focus primarily on developer experience, reliability, and ease of deployment. I tried to make intelligent choices based on my expectations of what would be efficient, but I intentionally avoided comparing BonsaiDb against other databases until January of this year.

To me the selling points of BonsaiDb were (and still are):

BonsaiDb isn't just a toy

In October a series of events made me look at porting the append-only B-Tree implementation from CouchDB to Rust as a potential replacement for Sled in our architecture. In the end, the bug that I was experiencing on Mac was unrelated to Sled, but its symptoms manifested in such a way that resembled issues others have had with Sled in the past.

After getting some basic unit tests working, I did something really stupid: I benchmarked it against Sled and SQLite. Why was it stupid? It kicked off a viscious few weeks, nearly reaffirming the common wisdom: why was I trying to write my own database? I would stare at microbenchmarks and profiling results, tweak something, and try again. Over and over. For some reason it just wasn't occurring to me: the fact Nebari was even in the same realm as these highly optimized databases was exciting enough. Eventually, I convinced myself it was better for the future of BonsaiDb to adopt Nebari.

In January I was preparing to release our first alpha. I knew if I didn't have benchmarks available, it would be one of the first questions asked. I wrote a benchmark suite aimed at comparing BonsaiDb against PostgreSQL in a simple eCommerce setup. I wasn't prepared for the results: BonsaiDb is significantly faster than PostgreSQL in this benchmark suite.

Last month I set my focus testing large datasets and was able to achieve performance again that exceeded my expectations. I no longer am viewing BonsaiDb as an easy-to-use database that will be "good enough" for many people. I now view BonsaiDb as having the potential to be a serious database option for most people.

For being just one year old, I'm extremely happy with where BonsaiDb is today.

Year Two: My Goals

One thing that surprised me was how much interest Nebari garnered. Despite not heavily advertising it, it's accumulated a fair number of stars on GitHub. It's popular enough that I've been rethinking how I was planning on implementing certain features. Outside of bug fixes and minor features, I haven't spent much time working on Nebari as it has been meeting BonsaiDb's needs. I would like to spend more time in this coming year making Nebari a better crate for developers to consume directly. Ultimately, the more users Nebari has, the more confidence everyone can have in BonsaiDb's ACID compliance.

For BonsaiDb, there are several goals I have:

Help grow BonsaiDb

Building BonsaiDb has reawakened many of the passions I felt in my first full-time computer-related job: working on REALbasic (now Xojo). As I started getting schema design questions and feature requests from potential users, I reflected on the joy those interactions provided me. It is clear to me that one of the reasons I look back so fondly at my first programming job is the wide array of problems I helped users solve.

For the longest time, I held off on allowing any means of sponsoring development of BonsaiDb. Until recently, I was still hoping to get back to writing an MMO once BonsaiDb was mature enough. My dreams have changed in the past months: I want to focus on bringing safe, high-level data-oriented application development to Rust. Initially, that means getting BonsaiDb to a maturity level where it meets my original vision.

Many readers may not be aware that the early days of BonsaiDb development were overlapped with my development of Gooey. To this day, it is the only framework that meets my personal desires of application development. However, it's lacking many critical features/widgets. I paused development because I want to focus on BonsaiDb. Yet, I also want to begin writing GUI administration tools for BonsaiDb soon.

I would love to continue dedicating my focus to these areas of the Rust ecosystem:

If you would like to ensure my ability to continue working on these projects full-time, I am now accepting sponsorship through GitHub Sponsors. I have been working on open-source Rust projects since November 2019 all funded by the sale of my previous startup. I would love the opportunity to continue working on open-source full time without needing to focus on building another startup. As I am not in dire need of finances, please only sponsor me if you have truly disposable income.

Another important way to help grow BonsaiDb is to contribute. I've been working on adding "Good First Issue" tasks. If you're looking to contribute to an open-source Rust project, I would be honored to have you as part of our team.

Trying out BonsaiDb

Our homepage has basic setup instructions and a list of examples. We have started writing a user's guide, and we have tried to write good documentation.

I would love to hear from you if you have questions or feedback. We have community Discourse forums and a Discord server, but also welcome anyone to open an issue with any questions or feedback.

Lastly, if you build something with one of our libraries, we would love to hear about it. Nothing makes us happier than hearing people are building things with our crates!

Thank you to all of the wonderful people in the Khonsu Labs community and the Rust ecosystem.