Written by Jonathan Johnson. Published 2023-03-19.
What is BonsaiDb?
BonsaiDb is a new database aiming to be the most developer-friendly Rust database. BonsaiDb has a unique feature set geared at solving many common data problems. We have a page dedicated to answering the question: What is BonsaiDb?. All source code is dual-licensed under the MIT and Apache License 2.0 licenses.
Today marks two years since the initial commit to BonsaiDb. It has also been nearly a full year since the last release. I want to take some time to highlight the major events for BonsaiDb over the past year, as well as discuss what is coming for BonsaiDb in the next year.
Last May, I discovered mistakes in my benchmarks and set off on a journey to improve BonsaiDb's transactional performance. BonsaiDb and its underlying architecture are largely maintained by me. I focused on trying to solve the speed issue by creating a new storage layer, and in the meantime, I had been continuing to work on BonsaiDb off and on.
Last October I hit some burnout, and I thought it was time to try to release an
update to BonsaiDb without the new storage layer. While trying to update the
currently-hidden benchmarks page, I was running into a panic that I only could
reproduce on a virtual machine after letting it run for nearly an hour. My
attempts to simplify were often met with the panic vanishing -- a classic
symptom of a "rare" race condition. Simply running the benchmark with debug = true in the Cargo.toml prevented the bug from occurring.
When coupled with the burnout I had at the time, I needed a break. At the beginning of January, I released the first version of OkayWAL, and in early February, I finished my rewrite of Sediment (more on that later). Still, I knew there was a lot of remaining work.
So I resolved to diagnose and fix the issue. It took a few days, but I
identified a bug that I had already fixed in Nebari's main branch. I
released Nebari v0.5.5 with the fix backported. This finally unblocked
my ability to release an update: no known issues!
Since then, I've been working on a couple of projects dogfooding BonsaiDb. One of those projects, Dossier, served the page you're reading! Not only has it been fun using my own database, it has helped me find a few rough edges and fix them in the process. I'm also feeling more and more confident in the huge list of unreleased changes.
I'm currently hoping to have a new version of BonsiaDb released in the next few weeks. Some notable changes include:
Key trait and composite keys, including the
ability to derive the trait. These changes make using custom primary keys or
view keys significantly easier and more efficient.BLAKE3-powered token authentication.I will be publishing a blog post going into more details when the new release is out.
In last year's anniversary post, I discussed some goals. The storage layer rewrite definitely impacted my goals. Despite this, I wanted to review my previous goals before talking about what I hope to accomplish this year.
The first goal was to implement a lot of features. Of the list, I only completed two:
One of the biggest features I really wanted to complete when writing last year's post was replication. To me, it's somewhat irresponsible for a production application to run without at least the ability to fail-over to a "warm-standby". Replication was the path I hoped to enable a form of high-availability, which was the underlying inspiration of having replication on the goals list.
I had begun looking at this feature, and realized that to make it efficient, I would need to change how documents are stored. This work has been completed in the branch that is working towards improving transactional write performance. So, while replication hasn't been completed, some prerequisite work has been completed.
The second goal from last year was stability.
KhonsuLabs.com is a site that is powered by
BonsaiDb. It's a simple site that shows
off the GitHub activity across all of our projects. I realized the other day I
hadn't even checked to see if it was still running. Sure enough, it's been
working great without any maintenance being done. systemd reports that the
service last was launched on Mon 2023-03-06 10:02:50 UTC -- over a year of
uptime!
The remaining two goals were tooling and community. The tooling for BonsaiDb hasn't changed much, but there are features and changes to APIs in v0.5 that are aimed at making more kinds of tooling possible. Our community has been growing steadily. I must admit I shied away from growing the community. I hadn't felt comfortable promoting BonsaiDb due to the looming storage rewrite.
Despite my lack of effort, our Discord server has continued to grow,
and I find myself hanging out in the #coworking channel with people from the
community several times a week. Additionally, BonsaiDb and its related projects
have gained a few more contributors this year. I'm always so grateful when
others take their time to help improve any of our projects.
The major goals I have for the next year are:
Ship the storage rewrite. At this point, I feel like I have all the puzzle pieces lined up. I just need to finish putting the puzzle together.
Sediment is a new storage format whose goal is to provide ACID-compliant storage of "blobs" of data. Sediment's purpose is to replace Nebari's current append-only file format. Nebari is the library that BonsaiDb uses to store data on-disk.
Sediment is still in development, but it is mostly complete (nearly 95% unit test code coverage!), and most importantly, its performance is meeting my goals. I am hopeful that after this is all done that BonsaiDb will be competitive with other ACID-compliant databases.
Release regular BonsaiDb updates and continue building the community. This past year, I initially lost track of something important: BonsaiDb is already pretty nice to use. While I know I am biased, I've really enjoyed some of the recent dogfooding I've been doing. It may sound alarming that transactional insert speed is an order of magnitude slower, but we're talking about measurements that are fractions of a second long. For many applications, the difference in performance will not be impactful.
One of the benefits of releasing regular updates is that they provide opportunities for new people to hear of BonsaiDb for the first time. At this early stage of development, I am hoping to find other Rust developers who share a similar vision for easily building and deploying data-driven applications.
Not only would it be great to potentially find additional contributors, I see increasing the bus factor beyond one as a critical step in ensuring BonsaiDb's long-term viability.
Eat my own dogfood more. In the past few months, I've been creating projects using BonsaiDb both to dogfood and have to fun. This year I would like to put more projects into production that are built using BonsaiDb. I even have some loose plans to start building a pared-down MMO-like game with a friend. Regardless of what projects I end up working on, I want to find reasons to use BonsaiDb to test it more and find ways to make it better.
Continue to work towards high-availability. I would love to finish shipping replication in the next year, but at a minimum, I want to have made some progress towards having a warm-standby style high-availibity offering. Minimizing downtime when hardware inevitably fails is an important part of taking BonsaiDb from a hobby database to an offering worth considering seriously.
Work towards better tooling. Being able to browse and edit your database without writing custom tooling is important. Because BonsaiDb doesn't have a query language, there isn't a simple way to interact with BonsaiDb databases outside of writing Rust code. This needs to change to make BonsaiDb easier to use, understand, and maintain.
Our homepage has basic setup instructions and a list of examples. We have started writing a user's guide, and we have tried to maintain good documentation.
I would love to hear from you if you have questions or feedback. We have community Discourse forums and a Discord server, but also welcome anyone to open an issue with any questions or feedback.
Lastly, if you build something with one of our libraries, we would love to hear about it. Nothing makes us happier than hearing people are building things with our crates!
If you're interested in contributing to an open-source Rust project, I have been keeping a list of "Good First Issue" tasks. Working on BonsaiDb can feel surprisingly high-level. Regardless of your experience level, if you are interested in contributing, please reach out!
If you want to hear more frequent updates, I post shorter updates occasionally
on our community Discord. I also stream in the #coworking channel
several times a week. I've also been trying to be more active on
Mastodon.
Regardless of whether you're looking to contribute or are just interested in something we've written, I am always happy to answer any questions and greatly appreciate hearing any constructive feedback.