Reaching the home stretch towards Debezium 0.9, it’s with great pleasure that I’m announcing the first release of Debezium in 2019, 0.9.0.CR1!
For this release we’ve mainly focused on sorting out remaining issues in the Debezium connector for SQL Server; the connector comes with greatly improved performance and has received a fair number of bug fixes.
Other changes include a new interface for event handlers of Debezium’s embedded engine, which allows for bulk handling of change events, an option to export the scale of numeric columns as schema parameter, as well as a wide range of bug fixes for the Debezium connectors for MySQL, Postgres and Oracle.
SQL Server Connector Improvements
The SQL Server connector supports blacklisting of specific columns now (DBZ-1067). That’s useful in cases where you’d like to exclude specific columns from emitted change data messages, e.g. to data protection considerations.
The "snapshot locking mode" option has been reworked (DBZ-947) and is named "snapshot isolation mode" now, better reflecting its semantics. A new mode "repeatable_read" has been added, and "none" has been renamed to "read_uncommitted". Please see the connector documentation and the migration notes for more details.
The connector allows for a much higher through-put now, thanks to caching of timestamps for the same LSN (DBZ-1078). Please refer to the change log for details on bugs fixed in this connector. A massive "Thank You" is in order to Grzegorz Kołakowski, for his tireless work on and testing of this connector!
New Embedded Engine Handler Interface
Debezium’s embedded engine now comes with a new interface ChangeConsumer
, which event handlers can implement if they’d like to process change events in bulks (DBZ-1080). That can result in substantial performance improvements when pushing change events to APIs that apply batch semantics themselves, such as the Kinesis Producer Library. You can learn more in the embedded engine docs.
Misc. Changes and Bug Fixes
All the relational connectors allow now to propagate the scale of numeric columns as a schema parameter (DBZ-1073). This is controlled via the column.propagate.source.type
option and builds on the exposure of type name and width added in Debezium 0.8. All these schema parameters can be used when creating the schema of corresponding tables in sink databases.
Debezium’s container image for Apache Kafka allows to create and watch topics now (DBZ-1057). You also can specify a clean-up policy when creating a topic (DBZ-1038).
The Debezium MySQL connector handles unsigned SMALLINT
columns as expected now. (DBZ-1063). For nullable columns with a default value, NULL
values are correctly exported (DBZ-1064; previously, the default value would have been exported in that case).
The Postgres connector handles tables without a primary key correctly now (DBZ-1029). We’ve also applied a fix to make sure that the connector works with Postgres on Amazon RDS, which recently was broken due to an update of wal2json in RDS (DBZ-1083). Going forward, we’re planning to set-up CI jobs to test against Postgres on RDS in all the versions supported by the Debezium connector. This will help us to spot similar issues early on and react quickly.
Please see the change log for the complete list of all addressed issues.
This release wouldn’t have been possible without all the contributions by the following members of the Debezium community: Addison Higham, Amit Sela, Gagan Agrawal, Grzegorz Kołakowski, Ilia Bogdanov, Ivan Kovbas, Moira Tagle, Renato Mefi and Tony Rizko.
Thanks a lot!
Next Steps
The CR1 release took us a bit longer than anticipated. The release of Debezium 0.9.0.Final will therefore be moved to early February. Rather quickly thereafter we’re planning to release Debezium 0.9.1, which will provide improvements and potential bugfixes to the features added in 0.9.
For further plans beyond that, check out our road map. If you got any feedback or suggestions for future additions, please get in touch via the mailing list or in the comments below.
Gunnar Morling
Gunnar is a software engineer at Decodable and an open-source enthusiast by heart. He has been the project lead of Debezium over many years. Gunnar has created open-source projects like kcctl, JfrUnit, and MapStruct, and is the spec lead for Bean Validation 2.0 (JSR 380). He’s based in Hamburg, Germany.
About Debezium
Debezium is an open source distributed platform that turns your existing databases into event streams, so applications can see and respond almost instantly to each committed row-level change in the databases. Debezium is built on top of Kafka and provides Kafka Connect compatible connectors that monitor specific database management systems. Debezium records the history of data changes in Kafka logs, so your application can be stopped and restarted at any time and can easily consume all of the events it missed while it was not running, ensuring that all events are processed correctly and completely. Debezium is open source under the Apache License, Version 2.0.
Get involved
We hope you find Debezium interesting and useful, and want to give it a try. Follow us on Twitter @debezium, chat with us on Zulip, or join our mailing list to talk with the community. All of the code is open source on GitHub, so build the code locally and help us improve ours existing connectors and add even more connectors. If you find problems or have ideas how we can improve Debezium, please let us know or log an issue.