With only a few days left for the year, it’s about time for another Debezium release; so it’s with great pleasure that I’m announcing Debezium 0.9.0.Beta2!
This release comes with support for MySQL 8 and Oracle 11g; it includes a first cut of metrics for monitoring the SQL Server and Oracle connectors, several improvements to the MongoDB event flattening SMT as well as a wide range of bug fixes. Overall, not less than 42 issues were addressed; very clearly, there has to be some deeper sense in that ;)
A big shout out goes to the following members Debezium’s amazing community, who contributed to this release: Eero Koplimets, Grzegorz Kołakowski, Hanlin Liu, Lao Mei, Renato Mefi, Tautvydas Januskevicius, Wout Scheepers and Zheng Wang!
In the following, let’s take a closer look at some of the changes coming with the 0.9 Beta2 release.
Monitoring and Metrics for the SQL Server and Oracle Connectors
Following the example of the MySQL connector, the connectors for SQL Server and Oracle now expose a range of metrics for monitoring purposes via JMX (DBZ-978). This includes values like the time since the last CDC event, offset of the last event, the total number of events, remaining and already scanned tables while doing a snapshot and much more. Please see the monitoring documentation for details on how to enable JMX. The following image shows an example of displaying the values in OpenJDK’s Mission Control tool:
We’re planning to expand the set of exposed metrics in future versions and also make them available for Postgres and MongoDB. Please let us know about the metrics you’d like to see by commenting on JIRA issue DBZ-1040.
As a bonus, we’ve also created a Grafana dashboard for visualizing all the relevant metrics:
We’ll blog about monitoring and the dashboard in more detail soon; but if you are interested, you already can take a look at this demo in our examples repository.
Misc. Features
The "snapshot.delay.ms" option already known from the Debezium MySQL connector is now available for all other Debezium connectors, too (DBZ-966). This comes in handy when deploying multiple connectors to a Kafka Connect cluster, which may cause rebalancing the connectors in the cluster, interrupting and restarting running snapshots of already deployed connector instances. This can be avoided by specifying a delay which allows to wait with the snapshotting until the rebalancing phase is completed.
The MongoDB CDC Event Flattening transformation received a number of improvements:
-
Support for MongoDB’s
$unsetoperator (DBZ-612) -
Support for full document updates (DBZ-987)
-
New option for dropping delete and tombstone messages (DBZ-563)
-
Option to convey the original type of operation as a header parameter (DBZ-971); that option is also available for the Flattening SMT for the relational connectors and can be useful in case sink connectors need to differentiate between inserts and updates
Bug fixes
As always, we’ve also fixed a good number of bugs reported by Debezium users. The set of fixed issues includes:
-
Several bugs related to streaming changes from MySQL in GTID mode (DBZ-923, DBZ-1005, DBZ-1008)
-
Handling of tables with reserved names in the SQL Server connector (DBZ-1031)
-
Potential event loss after MySQL connector restart (DBZ-1033)
-
Unchanged values of TOASTed columns caused the Postgres connector to fail (DBZ-842)
Please see the change log for the complete list of addressed issues.
Next Steps
We’re planning to do a candidate release of Debezium 0.9 in early January. Provided no critical issues show up, Debezium 0.9.0.Final should be out by the end of January. For the CR we’ve mostly scheduled a number of further bug fixes, improvements to the SQL Server connector and the addition of further metrics.
In parallel, we’ll focus our attention on the Oracle connector again, finally getting back to the long-awaited LogMiner-based capture implementation (DBZ-137). This will be a primary feature of Debezium 0.10.
In addition, we’ll spend some cycles on the blogging and demo side of things; namely we’re thinking about writing on and demoing the new monitoring and metrics support, HA architectures including failover with MySQL, HAProxy and Debezium, as well as enriching CDC events with contextual information such as the current user or use case identifiers. Stay tuned!
Also going beyond 0.10, we got some great plans for Debezium in the coming year. If you’d like to bring in your ideas, too, please let us know on the mailing list or in the comments below, we’re looking forward to hearing from you.
And with that, all that remains to be said, is "Happy Festivus for the rest of us!"
Happy change data streaming and see you in 2019!
Gunnar Morling
Gunnar is a software engineer at Decodable and an open-source enthusiast by heart. He has been the project lead of Debezium over many years. Gunnar has created open-source projects like kcctl, JfrUnit, and MapStruct, and is the spec lead for Bean Validation 2.0 (JSR 380). He’s based in Hamburg, Germany.
About Debezium
Debezium is an open source distributed platform that turns your existing databases into event streams, so applications can see and respond almost instantly to each committed row-level change in the databases. Debezium is built on top of Kafka and provides Kafka Connect compatible connectors that monitor specific database management systems. Debezium records the history of data changes in Kafka logs, so your application can be stopped and restarted at any time and can easily consume all of the events it missed while it was not running, ensuring that all events are processed correctly and completely. Debezium is open source under the Apache License, Version 2.0.
Get involved
We hope you find Debezium interesting and useful, and want to give it a try. Follow us on Twitter @debezium, chat with us on Zulip, or join our mailing list to talk with the community. All of the code is open source on GitHub, so build the code locally and help us improve ours existing connectors and add even more connectors. If you find problems or have ideas how we can improve Debezium, please let us know or log an issue.
