I’m very happy to announce the release of Debezium 1.3.0.Beta2!
In this release we’ve improved support for column filtering for the MySQL and SQL Server connectors, and there’s a brand-new implementation for ingesting change events from Oracle, using the LogMiner package. As we’re on the home stretch towards Debezium 1.3 Final, there’s also a wide range of smaller improvements, bug fixes and documentation clarifications; overall, not less than 44 issues have been resolved for this release.
Column Filtering Improvements
Addressing a long standing feature request (DBZ-1068), the Debezium connector for SQL Server supports now server-side column filtering: capture instances in the database itself can be configured so to only contain a sub-set of the captured table’s columns. That way, specific columns can be excluded by the CDC process right away, instead of only removing them in the Debezium connector, which is much more efficient for large BLOB for instance.
The MySQL connector adds support for the
column.include.list option already known from the Debezium Postgres connector (DBZ-2508).
Related to the matter of filtering, following up on the work begun in the 1.3 Beta1 release around replacing the terms "master/slave", "blacklist" and "whitelist" with more inclusive alternatives, also all the incubating connectors (Oracle, Db2, Cassandra) use the new terms like "database.include.list", "primary/replica", etc. now (DBZ-2462).
LogMiner-based Ingestion Engine for Oracle
The Debezium Oracle connector can now use the LogMiner package to ingest change events. As this package comes with the Oracle database itself, it’s a very attractive alternative to the existing XStream-based implementation. Discussions and work towards LogMiner support have been happening in the Debezium community for a long time (as you already might have guessed from the very low issue number DBZ-137), so we’re particularly excited about this work being merged eventually and being part of this release.
Note that there’s several follow-up tasks to be resolved related to the LogMiner-based ingestion implementation; while it is not recommended for production usage at this point, we’d love to get your feedback from testing and evaluating!
A massive thank you to everyone involved with this: Andrey Ignatenko and his team for the main work, Andrey Pustovetov for his ideas around transaction buffering, Chris Cranford for picking up the PR and preparing it to get merged, Milo vd Zee for his extensive review, as well as everyone else commenting and providing feedback on the PR and Jira issue.
Misc. Features and Bug Fixes
In addition to these key features, the community has completed the work on some other features and fixes, too:
The MySQL connector supports the
LOCK TABLES FOR BACKUPlock mode when being used with Percona Server for MySQL (DBZ-2466), which reduces contention during snapshots
The Postgres connector snapshot SPI got more flexible, allowing for custom implementations now that e.g. can re-snapshot selected tables (DBZ-2094)
The prefix of additional headers and fields produced by the event flattening SMTs is customizeable now (DBZ-2504)
Support for JSON functions in MySQL DDL statements (DBZ-2453)
Improved exception logging for the Cassandra connector (DBZ-2498)
As always, please refer to the release notes for the complete list of resolved issues as well as procedures for upgrading from earlier Debezium versions.
With the first cut of LogMiner support being merged and released, we’re now planning to focus on stabilization and bug fixing, with the Debezium 1.3 Final release to be expected around the end of the month.
In parallel, work is happening on a new connector contributed by the community for the Vitess (which will - depending on progress of review - be released as an incubating connector either in Debezium 1.3 or 1.4), and we’re going to share some exciting efforts around a proof-of-concept for a potential future Debezium UI with you very soon!
Gunnar is a software engineer at Decodable and an open-source enthusiast by heart. He has been the project lead of Debezium over many years. Gunnar has created open-source projects like kcctl, JfrUnit, and MapStruct, and is the spec lead for Bean Validation 2.0 (JSR 380). He’s based in Hamburg, Germany.
Debezium is an open source distributed platform that turns your existing databases into event streams, so applications can see and respond almost instantly to each committed row-level change in the databases. Debezium is built on top of Kafka and provides Kafka Connect compatible connectors that monitor specific database management systems. Debezium records the history of data changes in Kafka logs, so your application can be stopped and restarted at any time and can easily consume all of the events it missed while it was not running, ensuring that all events are processed correctly and completely. Debezium is open source under the Apache License, Version 2.0.
We hope you find Debezium interesting and useful, and want to give it a try. Follow us on Twitter @debezium, chat with us on Zulip, or join our mailing list to talk with the community. All of the code is open source on GitHub, so build the code locally and help us improve ours existing connectors and add even more connectors. If you find problems or have ideas how we can improve Debezium, please let us know or log an issue.