I have a new blog post out on honeycomb.io, diving into the insights and lessons of completing a major, multi-quarter Kafka migration project at Honeycomb.

https://www.honeycomb.io/blog/transforming-how-we-run-kafka-honeycomb

There are some things that I talk about in the post that I’d like to speak to more of here. I want to give some shout-outs, and I’d like to do a little bit of personal reflection around the importance of the sociotechnical outcomes of this project.

In the blog, I talked about how building on institutional knowledge and wisdom set this project up for success. Specifically, there was a lot of material around Kafka’s history at Honeycomb that my predecessor, Terra Field, left for me when I started at Honeycomb. As far as I’m concerned, she played a critical role in this project’s success, because it was her documentation, RFCs, proposals, moonshots, and more, that informed me about Kafka’s contextual history.

This goes directly back to something I’ve talked about before that Dr. Cat Hicks has written about: the duty of the traveler from Citation as Pilgrimage https://www.drcathicks.com/post/citation-as-pilgrimage

I continue to feel very strongly that it is my duty, personally and professionally, to share my knowledge with others. To connect the past to the present, and for me to build roads and tend to the gardens that connect the present to the future. Thank you Terra for entrusting the person following you with that knowledge. Oh, when we ceremoniously terminated the last remaining prod Kafka brokers from the old cluster, I did terminate one in your name as well, as you requested.

My team, Core Services, did an incredible job with this project. I hope that pride comes through in the blog post. They should feel extremely proud for what they did. Same goes for the many adjacent teams, managers, directors, and contributors who were also essential to this project’s success.

Thank you also to Liz Fong-Jones who helped me with the shape of this blog post. Through our brainstorming, she helped me bring key sociotechnical narratives and stories of this project into clear view. Thank you very much for your continued guidance and influence on me. Also, Liz’s Kafka domain expertise as a guide was incredibly helpful and inspiring to me.

As I hope you’ll see in the blog post, this project was a huge journey which I suspect will carry echoes for a long time. There are significant accomplishments to be proud of, and there are hard lessons for us to learn from. The technical achievements were substantial, but my most durable takeaways are around how this project reinforced and valued learning culture.

My colleagues can operate Kafka with confidence, are empowered to answer Kafka questions, and can meaningfully contribute to the cultivation of our library of Kafka knowledge. We can and do meaningfully work on difficult things. Strong, sustainable engineering practices of collaboration and problem solving are still very much alive today.

My hope is that you remember that you can do this too.