It’s my pleasure to announce the release of Debezium 1.6.0.CR1!
This release adds skipped operations optimizations for SQL Server, introduces Heartbeat support to the Oracle connector, Oracle BLOB/CLOB support is now opt-in only, and provides a range of bug fixes and other improvements across different Debezium connectors.
Skipped operations optimizations - SQL Server
Debezium introduced the notion of
skipped.operations some time ago. This feature allows connectors to control what change events are emitted during the streaming phase, optionally omitting certain change types such as updates or deletes.
skipped.operations feature was originally implemented to act more like an SMT where it was a filter that was applied toward the end of the CDC pipeline just before an event was emitted. While this worked well, there was an optimization that could be achieved here because SQL Server records the change type in the capture table along side the change data. Using this knowledge, the query to fetch changes from the capture table was adjusted so that the changes returned from SQL Server are pre-filtered based on these "skipped operations".
Overall, this optimization should improve both the time spent during iterations of capturing changes when the connector is configured to use
skipped.operations as well as reduce the overall garbage collection that the connector may do over its lifetime due to generating fewer objects that are later discarded.
Many thanks to Hossein Torabi for identifying this optimization and contributing it!
Heartbeat Support for the Oracle Connector
Debezium supports heartbeats across its other connectors, this feature is now available to Oracle!
A heartbeat is a critical part of insuring that a connector’s offsets remain synchronized with Kafka. Normally, the offsets are automatically maintained by Kafka Connect when we emit an event to a topic. But there are circumstances where event emission can be infrequent enough that stale offsets become a concern.
Regardless which adapter the Debezium Oracle connector is deployed to use, there are situations where events may be seen by the connector and in-memory offset state is updated but because we don’t emit an event to Kafka, the offsets aren’t synchronized to Kafka Connect. A great example is when the connector could be capturing changes for a table that changes very infrequently compared to other tables in the data source.
In order to guarantee that offsets remain synchronized with Kafka Connect, specifically during periods of low capture activity, the
heartbeat.interval.ms configuration option can be set to periodically keep offsets in sync.
Oracle BLOB/CLOB Support now Opt-In
While we added support for
CLOB data types to the Debezium connector for Oracle in an earlier 1.6 pre-release, we believe that the work towards fully supporting these types is still an ongoing one.
There is a certain amount of overhead the connector has when handling BLOB and CLOB fields. Transactions which contain these data types require a reconciliation step at commit-time to merge certain events into a single logical emitted event. Additionally, LOB data types require a certain amount of memory footprint for their values to be kept in memory while the event to be emitted is constructed. Furthermore, users may not want to have LOB data emitted at all due to the size of the data.
So with that, the Oracle connector’s support for LOB data types is now an opt-in only feature. This means that OOTB the connector won’t attempt to capture LOB column data. If LOB columns need to be captured by the connector, the connector option,
lob.enabled, must be set to
true so that the connector will pickup and process those column types.
Lastly, we also felt that since there is still some ongoing effort for BLOB and CLOB support, it made sense to denote LOB support as incubating for the short-term until we believe the solution has matured. We strongly encourage users who want to capture LOB-based columns to enable LOB support and give it test drive and provide any and all feedback, both good or bad, so that we can continue to improve support for LOB columns.
Further Improvements and Bugfixes
With the recent release of Quarkus 2.0.0.Final, the Quarkus Outbox extension is now based on Quarkus 2.0 (DBZ-3602).
Overall, 27 issues have been addressed in Debezium 1.6.0.CR1. We’re deeply grateful to all the community members contributing to this release:
With CR1 out, we’re nearing the end of the stablization phase for the 1.6 release cycle. You can expect possibly one more CR (candidate release), before the final release, which is planned for the end of the month, barring any unforeseen complications of cause. Besides some more bug fixes and documentation improvements we’re also intending to upgrade to Apache Kafka 2.8, which will allow you to take a sneak peak at using Debezium with ZooKeeper-less Kafka!
In parallel, we’re going to work on the roadmap for Debezium 1.7 (due by the end of September). Please get in touch via the mailing list if you have specific feature requests for this release!
Debezium is an open source distributed platform that turns your existing databases into event streams, so applications can see and respond almost instantly to each committed row-level change in the databases. Debezium is built on top of Kafka and provides Kafka Connect compatible connectors that monitor specific database management systems. Debezium records the history of data changes in Kafka logs, so your application can be stopped and restarted at any time and can easily consume all of the events it missed while it was not running, ensuring that all events are processed correctly and completely. Debezium is open source under the Apache License, Version 2.0.
We hope you find Debezium interesting and useful, and want to give it a try. Follow us on Twitter @debezium, chat with us on Zulip, or join our mailing list to talk with the community. All of the code is open source on GitHub, so build the code locally and help us improve ours existing connectors and add even more connectors. If you find problems or have ideas how we can improve Debezium, please let us know or log an issue.