The Debezium community is on the homestretch towards the 0.10 release and we’re happy to announce the availability of Debezium 0.10.0.CR1!
Besides a number of bugfixes to the different connectors, this release also brings a substantial improvement to the way initial snapshots can be done with Postgres. Unless any major regressions show up, the final 0.10 release should follow very soon.
Exported Snapshots for Postgres
One capability of Postgres' logical decoding facility that has not been leveraged by Debezium so far is the notion of exported snapshots: when a replication slot is created, a transaction with SNAPSHOT
isolation mode can be started, which allows to export tables from the database without taking any locks, exactly at the moment when the slot is created.
Based on earlier work which allows for much more flexibility in regards to how snapshot are handled by the Debezium Postgres connector, the exported snapshot functionality can now be used via the new snapshot mode exported
(DBZ-1035, DBZ-1440). We encourage you to give this a test ride and report back any issues.
Eventually, this mode should become the default snapshotting behavior, as it doesn’t require the connector to obtain any locks.
Bugfixes
Due to a value overflow multiple users had reported issues when capturing the values of temporal columns with values far out in the future, e.g. years after 3000 (DBZ-949, DBZ-1205, DBZ-1255).
While such values are perhaps not too common in typical enterprise applications (after all, who wants to have their purchase order delivery scheduled in a thousand years from now), there’s still several use cases working with such values, e.g. think of potentially large half-life times when modelling nuclear decay processes. So we did a larger refactoring of the code dealing with temporal values, and we are happy to report that these issues have been fixed now.
Other bugfixes were done in regards to dealing with "no-op" events in MongoDB (DBZ-1464) and the recently added propagation of source
fields to outgoing message when using the new record state extraction SMT (DBZ-1448).
Cassandra Connector
After the initial release of the Debezium connector for Cassandra, work has begun to further align it with the other Debezium connectors (unlike the relational connectors and the one for MongoDB, the Cassandra connector currently is not based on Apache Kafka Connect, but runs as a stand-alone process). The first outcome of this is that its configured via a properties file now (similar to using Kafka Connect in standalone mode) and not via a YAML file. (DBZ-1406).
The next step will be to make the aspect of message serialization configurable: while currently only Avro is supported by the connector, it will eventually support the notion of pluggable converters, allowing you to use the JSON, Avro and any other converters you may already know from Kafka Connect.
Reworked Website and Documentation
When reading this blog post, it’s hard to miss: the Debezium website has received a facelift.
Information for the current stable and development releases (0.9 and 0.10 at this time) is much easier to find now. Also the documentation has been re-organized and is published in version-specific approach now, i.e. you can now obtain the specific documentation applying for a particular release.
The blog section of the website has been reworked, too: the main page shows an introductory snippet for the most recent posts, whereas a number of "featured" blog posts is listed on the left. These are typically earlier blog posts which explore advanced topics such as the outbox pattern in depth and which we wanted to make easier to discover and consume. We hope you like the new website and documentation structure and it helps you to find all the information you’re looking for better than before. If you run into any issues (formatting glitches, broken links etc.), please let us know.
Refer to the 0.10.0.CR1 release notes for the complete list of addressed issues and the upgrading procedure.
As always, many thanks to all the awesome people from the Debezium community who contributed to this release: Andrew Garrett, Bingqin Zhou Cyril Scetbon, Guillaume Rosauro, Ivan Luzyanin, Lev Zemlyanov and Renato Mefi!
Gunnar Morling
Gunnar is a software engineer at Decodable and an open-source enthusiast by heart. He has been the project lead of Debezium over many years. Gunnar has created open-source projects like kcctl, JfrUnit, and MapStruct, and is the spec lead for Bean Validation 2.0 (JSR 380). He’s based in Hamburg, Germany.
About Debezium
Debezium is an open source distributed platform that turns your existing databases into event streams, so applications can see and respond almost instantly to each committed row-level change in the databases. Debezium is built on top of Kafka and provides Kafka Connect compatible connectors that monitor specific database management systems. Debezium records the history of data changes in Kafka logs, so your application can be stopped and restarted at any time and can easily consume all of the events it missed while it was not running, ensuring that all events are processed correctly and completely. Debezium is open source under the Apache License, Version 2.0.
Get involved
We hope you find Debezium interesting and useful, and want to give it a try. Follow us on Twitter @debezium, chat with us on Zulip, or join our mailing list to talk with the community. All of the code is open source on GitHub, so build the code locally and help us improve ours existing connectors and add even more connectors. If you find problems or have ideas how we can improve Debezium, please let us know or log an issue.