What’s better than getting Java 9? Getting Java 9 and a new version of Debezium at the same time! So it’s with great happiness that I’m announcing the release of Debezium 0.6 today.
What’s in it?
Debezium is now built against and tested with Apache Kafka 0.11.0. Also the Debezium Docker images have been updated do that version (DBZ-305). You should make sure to read the Kafka update guide when upgrading from an earlier version.
To improve integration with existing Kafka sink connectors such as the JDBC sink connector or the Elasticsearch connector, Debezium provides a new single message transformation (DBZ-226). This SMT converts Debezium’s CDC event structure into a more conventional structure commonly used in other sink and non-CDC source connectors where the message represents the state of the inserted or updated row, or null in the case of a deleted row. This lets your for instance capture the changes from a table in MySQL and update a corresponding table in a Postgres database accordingly. We’ll provide a complete example showing the usage of that new SMT in the next few days.
New connector features
Now let’s take a look at some of the changes around the specific Debezium connectors. The MySQL connector has seen multiple improvements, e.g.:
For the MongoDB connector, the way of serializing ids into the key payload of CDC events has changed (DBZ-306). The new format allows to read back ids into the correct type. We also took the opportunity and made the id field name consistent with the other connectors, i.e. it’s "id" now. Note: that change may break existing consumers, so some work on your end may be required, depending on the implementation of your consumer. The details are discussed in the release notes and the format of message keys is described in depth in the connector documentation. Kudos to Hans-Peter Grahsl who contributed on this feature!
Another nice improvement for this connector is support for SSL connections (DBZ-343).
Finally, the Postgres connector learned some new tricks, too:
Please refer to the changelog for an overview of all the 20 issues fixed in Debezium 0.6.0.
High on our agenda is exploring support for Oracle (DBZ-20). We are also looking into using another logical decoding plug-in (wal2json) for the Postgres connector, which would enable to use Debezium with Postgres instances running on Amazon RDS. Another feature being worked on by community member Moira Tagle is support for updates to the
table.whitelist for existing connector instances. Finally, we’ve planned to test and adapt the existing MySQL connector for providing CDC functionality to MariaDB.
Debezium 0.7 with one or more out of those features as well as hopefully some others will be released later this year. We’ll likely also do further 0.6.x releases with bug fixes as required.
You’d like to contribute? That’s great - let us know and we’ll get you started. Check out the details below on how to get in touch.
Gunnar is a software engineer at Decodable and an open-source enthusiast by heart. He has been the project lead of Debezium over many years. Gunnar has created open-source projects like kcctl, JfrUnit, and MapStruct, and is the spec lead for Bean Validation 2.0 (JSR 380). He’s based in Hamburg, Germany.
Debezium is an open source distributed platform that turns your existing databases into event streams, so applications can see and respond almost instantly to each committed row-level change in the databases. Debezium is built on top of Kafka and provides Kafka Connect compatible connectors that monitor specific database management systems. Debezium records the history of data changes in Kafka logs, so your application can be stopped and restarted at any time and can easily consume all of the events it missed while it was not running, ensuring that all events are processed correctly and completely. Debezium is open source under the Apache License, Version 2.0.
We hope you find Debezium interesting and useful, and want to give it a try. Follow us on Twitter @debezium, chat with us on Zulip, or join our mailing list to talk with the community. All of the code is open source on GitHub, so build the code locally and help us improve ours existing connectors and add even more connectors. If you find problems or have ideas how we can improve Debezium, please let us know or log an issue.