It’s my pleasure to announce the release of Debezium 1.6.0.Beta2!
This release adds support for Pravega to Debezium Server, expands the snapshotting options of the Debezium Oracle connector, and provides a range of bug fixes and other improvements across different Debezium connectors.
Pravega Support for Debezium Server
With Debezium Server, we’re on a mission to bring open-source change data capture to all the users, no matter which data streaming platform or commit log they are using. So we are very happy to receive a contribution which adds support for Pravega to Debezium Server. A Cloud Native Computing Foundation (CNCF) sandbox and Apache 2.0 licensed open-source project, Pravega describes itself as a "storage abstraction for continuously generated and unbounded data".
The Debezium Server Pravega sink adapter offers two modes: non-transactional and transactional. The non-transactional mode individually writes each event in a Debezium batch to Pravega. The transactional mode writes the Debezium batch to a Pravega transaction that commits when the batch is completed.
Oracle Snapshotting Improvements
The Debezium connector for Oracle received two improvements related to snapshotting:
Support for the
snapshot.include.collection.listoption (DBZ-3062); this allows to create an initial snapshot only for a subset of all those tables captured by the connector
snapshot.locking.modewhich provides control over the locking behavior when the connector captures the schema of the tables (DBZ-3557); in particular, this allows to disable locking completely, which is very useful if you can guarantee that no DDL changes are happening while the connector is taking the (schema) snapshot
Further Improvements and Bugfixes
The Debezium connector for SQL Server saw two performance-related improvments (3486, DBZ-3515). The schemas of change events from the Postgres connector contain default values now, based on the source column definition (DBZ-2790). This comes in handy for instance when deriving downstream table schemas from a change event stream.
Other fixes include correct identification of primary members in MongoDB replica sets (DBZ-3522), support for the
JSON function in the MySQL connector’s DDL parser (DBZ-3559), and the upgrade of the Debezium Quarkus extension for implementing the outbox pattern to Quarkus 2.0 (DBZ-3550).
Overall, 48 issues have been addressed in Debezium 1.6.0.Beta2. We’re deeply grateful to all the community members contributing to this release:
Anisha Mohanty, Bingqin Zhou, Bob Roldan, Chris Collingwood, Chris Cranford, Derek Moore, Eric Slep, Gilles Vaquez, Gunnar Morling, Jackey Zhang, Jakub Cechacek, Jiri Novotny, Jiri Pechanec, Kevin Pullin, Patrick Chu, Robert Roldan, Sara Fonseca, Sergei Morozov, Stefan Miklosovic, and Vadzim Ramanenka!
With Beta2 through the door, we’re entering the stabilization phase for the 1.6 release cycle. You can expect one or two CRs (candidate releases), before the final release, which is planned for the end of the month, barring any unforeseen complications of cause. Besides some more bug fixes and documentation improvements we’re also intending to upgrade to Apache Kafka 2.8, which will allow you to take a sneak peak at using Debezium with ZooKeeper-less Kafka!
In parallel, we’re going to work on the roadmap for Debezium 1.7 (due by the end of September). Please get in touch via the mailing list if you have specific feature requests for this release!
Gunnar is a software engineer at Decodable and an open-source enthusiast by heart. He has been the project lead of Debezium over many years. Gunnar has created open-source projects like kcctl, JfrUnit, and MapStruct, and is the spec lead for Bean Validation 2.0 (JSR 380). He’s based in Hamburg, Germany.
Debezium is an open source distributed platform that turns your existing databases into event streams, so applications can see and respond almost instantly to each committed row-level change in the databases. Debezium is built on top of Kafka and provides Kafka Connect compatible connectors that monitor specific database management systems. Debezium records the history of data changes in Kafka logs, so your application can be stopped and restarted at any time and can easily consume all of the events it missed while it was not running, ensuring that all events are processed correctly and completely. Debezium is open source under the Apache License, Version 2.0.
We hope you find Debezium interesting and useful, and want to give it a try. Follow us on Twitter @debezium, chat with us on Zulip, or join our mailing list to talk with the community. All of the code is open source on GitHub, so build the code locally and help us improve ours existing connectors and add even more connectors. If you find problems or have ideas how we can improve Debezium, please let us know or log an issue.