Featured Posts

Tags
apache kafka apache kafka apicurio avro aws batch caassandra camel cassandra cdc channels community community stories connectors containers cqrs custom datalake db2 ddd debezium debezium server debezium ui deduplication discussion docker elasticsearch event sourcing exactly once semantics example examples features fedora flink hiring ibmi iceberg informix integration introduction jaeger jdbc json kafka kafka streams kafka streams kogito ksql kubernetes lakehouse machine learning mariadb microservices mongo mongodb mysql news newsletter notifications online learning operator oracle outbox performance postgres presentation production quarkus questdb rds releases schema scylla secrets sentry serialization signaling smt snapshots spanner spark sql sqlserver tensorflow testcontainers tests time series timescaledb topics tracing transactions vagrant vitess website

I’m pleased to announce the release of Debezium 1.6.0.Final!

This release is packed full with tons of new features, including support for incremental snapshotting that can be toggled using the new the Signal API. Based on the excellent paper DBLog: A Watermark Based Change-Data-Capture Framework by Netflix engineers Andreas Andreakis and Ioannis Papapanagiotou, the notion of incremental snapshotting addresses several requirements around snapshotting that came up repeatedly in the Debezium community:

  • Ability to resume an on-going snapshot after a connector restart

  • Re-snapshot selected tables during streaming, e.g. to re-bootstrap Kafka topics with change events for specific tables

  • Snapshot tables newly added to the list of captured tables after changing the filter configuration

  • Begin to stream changes while an initial snapshot is running

Incremental snapshotting is an incubating feature as of Debezium 1.6, and we’re looking forward to your feedback on this feature. To learn more about this functionality, please refer to the individual connector docs, e.g. for the Debezium MySQL connector. There are already some follow-up improvements in this area in the workings, for instance the usage of MySQL GTIDs for setting the high/low watermarks required for this snapshotting approach, which will avoid the need for write access to the database by the connector. You can expect these improvements to be rolled out in one of the upcoming 1.7 preview releases.

Besides incremental snapshotting, other new features in Debezium 1.6 include two brand new Debezium Server sinks, one for Apache Kafka and another for Pravega, as well as several notable enhancements to the Debezium connector for Oracle which include reacting to DDL schema changes and an opt-in, incubating feature to emit BLOB and CLOB column data types. There’s also improvements to the community-led connectors for Vitess and Apache Cassandra, as well as wide range of bug fixes and other smaller improvements. We’ve also upgraded the Debezium Quarkus extension for implementing the outbox pattern to Quarkus 2.0.

Across all the 1.6 preview and the final releases, a grand total of 188 issues has been addressed.

For more details, please see the earlier announcements for the 1.6.0 Alpha1, Beta1, Beta2, and CR1 releases.

Since the CR1 release, we’ve primarily focused on documentation improvements and some bug fixes. But there was one last-minute feature addition, too, which allows you to specify archive log locations (DBZ-3661) for the Oracle connector.

Please refer to the release notes of Debezium 1.6.0.Final for the complete list of resolved issues as well as procedures for upgrading from earlier versions.

As always, a big thank you to all the members from the community who helped with this release, be it via code contributions, bug report, testing, providing insight and expertise, etc. Kudos to the following individuals from the community which contributed to Debezium 1.6, bringing the overall number of contributors to the Debezium core repository to 277:

Anisha Mohanty, Bingqin Zhou, Bob Roldan, Camile Sing, Cao Manh Dat, Chris Collingwood, Derek Moore, Eric Slep, Gilles Vaquez, Hossein Torabi, Hyunjin Oh, Jakub Cechacek, JapuDCret, Jiri Novotny, Kevin Pullin, Mike, Olivier Jacquemart, Patrick Chu, Robert Roldan, Sara Fonseca, Sergei Morozov, SiuFay, Stefan Miklosovic, Teo Stocco, Thomas Aregger, and Vadzim Ramanenka!

Outlook

As always, Debezium will be following its normal quarterly release cadence with Debezium 1.7 planned by the end of September. There are a couple of key issues we intend to work on over the course of this version including support for incremental snapshots with the MongoDB and Oracle connectors. Additionally, we plan to explore some new buffering options for the Oracle connector’s LogMiner-based implementation, work on a tool for compacting large schema history topics, expand the feature set of Debezium UI, and much more.

In the coming week(s), keep an eye out on our roadmap as we’ll be refining this in preparations for Debezium 1.7. If you have any specific feature requests or other input for the roadmap and future releases, please let us know via the mailing list!

Chris Cranford

Chris is a software engineer at Red Hat. He previously was a member of the Hibernate ORM team and now works on Debezium. He lives in North Carolina just a few hours from Red Hat towers.

   

About Debezium

Debezium is an open source distributed platform that turns your existing databases into event streams, so applications can see and respond almost instantly to each committed row-level change in the databases. Debezium is built on top of Kafka and provides Kafka Connect compatible connectors that monitor specific database management systems. Debezium records the history of data changes in Kafka logs, so your application can be stopped and restarted at any time and can easily consume all of the events it missed while it was not running, ensuring that all events are processed correctly and completely. Debezium is open source under the Apache License, Version 2.0.

Get involved

We hope you find Debezium interesting and useful, and want to give it a try. Follow us on Twitter @debezium, chat with us on Zulip, or join our mailing list to talk with the community. All of the code is open source on GitHub, so build the code locally and help us improve ours existing connectors and add even more connectors. If you find problems or have ideas how we can improve Debezium, please let us know or log an issue.