Architecting an open-source observability tool

The problem

The project started as a side experiment: a Rust-based metrics pipeline that could ingest OpenTelemetry data without the JVM warmup tax of the dominant tool in the space. The maintainer had built a working prototype but couldn’t get the architecture stable enough for outside contributors, and the project was stalling at 200 stars.

The challenge wasn’t writing more code. It was designing a system that other people could extend without breaking each other, and that could scale to the workloads the early users were asking about — a million metrics per second across a hundred nodes.

The work

I joined as the architect, not the implementer. The first month was spent rewriting the module boundaries so that ingestion, storage, and query were three independently deployable crates with stable, documented APIs. I wrote the contributor guide from scratch — not the docs explaining what the code does, but the docs explaining why the code is shaped the way it is.

The storage layer was the architectural bet. We went with columnar Apache Arrow buffers, compressed and sharded by tenant, with a write-ahead log per shard for durability. The query engine was a streaming vectorized interpreter — no JIT, no LLVM dependency, just tight Rust loops over Arrow arrays.

The outcome

The project shipped v1.0 nine months after the architecture reset. The first release reached 8.2k stars, and forty-seven contributors had merged PRs by the v1.0 cutoff. Two major cloud providers adopted the design as the basis for their reference observability deployments — not by licensing it, but by hiring contributors and writing their own implementations against the same interface.

The maintainer later told me the most useful thing I’d done wasn’t the architecture — it was writing the contributor guide. That, and forcing him to delete the prototype module before v1.0 shipped.