Featured Posts

Tags
apache kafka apache kafka apicurio avro aws batch caassandra camel cassandra cdc channels community community stories connectors containers cqrs custom datalake db2 ddd debezium debezium server debezium ui deduplication discussion docker elasticsearch event sourcing exactly once semantics example examples features fedora flink hiring ibmi iceberg informix integration introduction jaeger jdbc json kafka kafka streams kafka streams kogito ksql kubernetes lakehouse machine learning mariadb microservices mongo mongodb mysql news newsletter notifications online learning operator oracle outbox performance postgres presentation production quarkus questdb rds releases schema scylla secrets sentry serialization signaling smt snapshots spanner spark sql sqlserver tensorflow testcontainers tests time series timescaledb topics tracing transactions vagrant vitess website

It’s with great please that I’m announcing the release of Debezium 1.3.0.Final!

As per Debezium’s quarterly release cadence, this wraps up the work of the last three months. Overall, the community has fixed 138 issues during that time, including the following key features and changes:

  • A new incubating LogMiner-based implementation for ingesting change events from Oracle

  • Support for Azure Event Hubs in Debezium Server

  • Upgrade to Apache Kafka 2.6

  • Revised filter option names

  • A new SQL Server connector snapshot mode, initial_only

  • Support for database-filtered columns for SQL Server

  • Additional connection options for the MongoDB connector

  • Improvements to ByteBufferConverter for implementing the outbox pattern with Avro as the payload format

Please refer to the announcements of the preview releases (Alpha1, Beta1, Beta2, CR1) for more details. Since last week’s CR1 release, we’ve been focusing on ironing out some remaining bugs and improvements to the documentation. To learn more about procedures for upgrading from earlier Debezium versions, please take a look the release notes.

Thank you to everyone testing the preview releases, this is of invaluable help for spotting and fixing short-comings in new features as well as regressions. And of course I’d also like to thank all the community members contributing to this release: Aaron Brady, Abhishek Gupta, Andrey Ignatenko, Arik Cohen, Bingqin Zhou, Björn Häuser, Cory Harper, Denis Liseichykau, Eric Weaver, Grant Cooksey, Grzegorz Kołakowski, Guillaume Smet, Guy Pascarella, James Gormley, Jeremy Finzel, Jonas Lins, Jos Huiting, Justin Hiza, Lukasz Korzeniowski, Luis Garcés-Erice, Matt Beary, Mickaël Isaert, Moira Tagle, Nathan Mills, Peter Junos, Ruslan Gibaiev, Thomas Prelle, and Victor Xiang!

Overall, more than 220 individuals have contributed to the Debezium project at this point.

But not only that, also the number of Debezium users is constantly growing, as e.g. documented on our reference list of Debezium users (let us know if you want to be added). There’s also several new entries in our compilation of public talks and blog posts touching on Debezium, e.g. a highly recommendable talk by Marta Paes about change data capture with Flink SQL and Debezium, a blog post by Cemal Turkoglu about [making sense of change data capture pipelines for Postgres with the Debezium Kafka Connector, and a nice piece on implementing the outbox pattern with Debezium in Quarkus by Iain Porter. Abdellatif Bouchama did an amazing job by creating a cheat sheet for running Debezium on OpenShift.

Outlook

With the 1.3 Final release out, planning for the 1.4 version (due by the end of the year) is happening right now. The roadmap still is in flux, so make sure to chime in and let us know about your requirements and feature requests. Some of the things we’re likely going to work on include:

  • The community-led connector for Vitess; the initial contribution has already been merged and we plan to ship the first release of this as part of Debezium 1.4 Alpha1 later this month

  • Moving the MySQL connector to the CDC connector framework shared by most other Debezium connectors; this will drastically reduce maintenance burden of this connector in the future

  • Exploring more powerful snapshotting options (e.g. for parallelization and re-doing snapshots of selected tables)

  • Improving the new LogMiner-based implementation for Oracle

And lastly, there’s one other area of activity which I’m particularly excited to share here today for the first time: a proof-of-concept of how a potential future Debezium user interface might look like. In that PoC we’re exploring how a graphical UI could help with the set-up and operation of Debezium connectors. We’ve got quite a few ideas in that field and will share more details in a blog post very soon. If you feel adventureous in the meantime, you could grab the current PoC code and take it for spin!

Until then, happy change data streaming, onwards and upwards!

Gunnar Morling

Gunnar is a software engineer at Decodable and an open-source enthusiast by heart. He has been the project lead of Debezium over many years. Gunnar has created open-source projects like kcctl, JfrUnit, and MapStruct, and is the spec lead for Bean Validation 2.0 (JSR 380). He’s based in Hamburg, Germany.

   

About Debezium

Debezium is an open source distributed platform that turns your existing databases into event streams, so applications can see and respond almost instantly to each committed row-level change in the databases. Debezium is built on top of Kafka and provides Kafka Connect compatible connectors that monitor specific database management systems. Debezium records the history of data changes in Kafka logs, so your application can be stopped and restarted at any time and can easily consume all of the events it missed while it was not running, ensuring that all events are processed correctly and completely. Debezium is open source under the Apache License, Version 2.0.

Get involved

We hope you find Debezium interesting and useful, and want to give it a try. Follow us on Twitter @debezium, chat with us on Zulip, or join our mailing list to talk with the community. All of the code is open source on GitHub, so build the code locally and help us improve ours existing connectors and add even more connectors. If you find problems or have ideas how we can improve Debezium, please let us know or log an issue.