Some time in early 2017, I got a meeting invite from Debezium’s founder, Randall Hauch. He was about to begin a new chapter in his professional career and was looking for someone to take over as the project lead for Debezium. So we hopped on a call to talk things through, and I was immediately sold on the concept of change data capture, its large number of potential use cases and applications, and the idea of making this available to the community as open-source. After some short consideration I decided to take up this opportunity, and without a doubt this has been one of the best decisions I’ve ever made in my job.
Today, five years and two major releases (1.0, 2.0) later, I am feeling really proud of what the Debezium community has accomplished, having established itself as the leading open-source platform for change data capture. The number of officially supported databases has grown from three to eight. Further Debezium-based CDC connectors are developed externally by database vendors like ScyllaDB and Yugabyte, making Debezium’s change event format kind of a de-facto standard for CDC. The project is used in production by companies such as Reddit, Shopify, Ubisoft, and Zalando. Debezium became part of Red Hat’s commercially supported product offerings (on-prem, as well as fully managed in the cloud), with multiple other vendors providing Debezium-backed offers as well. During the keynote of this year’s Current conference, Debezium was recognized as one of the most impactful open-source projects in the Apache Kafka space.
The most important part to me though is the tremendous growth of the Debezium community itself. To this day, more than 450 individuals have contributed to the code base. A big thank you to all the people and organizations who’ve worked tirelessly to make the vision of open-source change data capture a reality and continue to improve it every day: Red Hat — as the project’s main sponsor — Stripe, Instaclustr, SugarCRM, Redis, and many other companies and individual contributors!
After ten amazing years at Red Hat, I felt that it was about time for a change for me and start some new adventure, and I am going to join a start-up in the data streaming space next month. As part of this transition, I am also stepping down from the role as the project lead for Debezium. While I’ll be less active in the project on a daily basis, I definitely plan to stay involved and hopefully still send the one or other pull request.
My partner in crime Jiri Pechanec will take over as the acting engineering lead. Or, I should say, has taken over, since in fact he has had that role since earlier this year already. Jiri has been a member of the project for many years, working on several key features such as incremental snapshots and MongoDB change streams support. He’s an outstanding software engineer, with a unique insight into the problem space of CDC and decades of experience working in open source, and he will be an amazing lead for the Debezium project and community.
With the Debezium 2.0 release just through the door, addressing several consistency issues and getting rid of a fair chunk of technical debt, the project is in an excellent position for its future evolution. There are plans for another community-led connector which should be announced very soon, there’ll be support for exactly-once semantics as recently introduced in Kafka Connect (KIP -618), a Kubernetes operator for Debezium Server, a JDBC sink connector, and much more.
The best is yet to come, and I can’t wait to see what this amazing community will build next!
Gunnar is a software engineer at Decodable and an open-source enthusiast by heart. He has been the project lead of Debezium over many years. Gunnar has created open-source projects like kcctl, JfrUnit, and MapStruct, and is the spec lead for Bean Validation 2.0 (JSR 380). He’s based in Hamburg, Germany.
Debezium is an open source distributed platform that turns your existing databases into event streams, so applications can see and respond almost instantly to each committed row-level change in the databases. Debezium is built on top of Kafka and provides Kafka Connect compatible connectors that monitor specific database management systems. Debezium records the history of data changes in Kafka logs, so your application can be stopped and restarted at any time and can easily consume all of the events it missed while it was not running, ensuring that all events are processed correctly and completely. Debezium is open source under the Apache License, Version 2.0.
We hope you find Debezium interesting and useful, and want to give it a try. Follow us on Twitter @debezium, chat with us on Zulip, or join our mailing list to talk with the community. All of the code is open source on GitHub, so build the code locally and help us improve ours existing connectors and add even more connectors. If you find problems or have ideas how we can improve Debezium, please let us know or log an issue.