I’m pleased to announce the immediate availability of Debezium 2.6.2.Final. This release is the second maintenance release that focuses on addressing several critical stability issues with the 2.6.1.Final release, support for Oracle database query filtering with more than one thousand tables, fixed race condition with PostgreSQL offset flushing, fixed Avro compatibility, and more.
Let’s take a few moments and dive into these and more…
New features and improvements
Debezium 2.6.2.Final introduces one new feature, lets take a look at it.
Oracle database query filtering with 1000+ tables
The Debezium Oracle connector allows configuring the table include and exclude lists to be applied as part of the database query for efficiency reasons using the log.mining.query.filter.mode. When setting this configuration property to a value of in, the query is constructed using a SQL in-clause. In Oracle, an in-clause is restricted to one thousand elements, and if your connector configuration exceeded this limit, the query would fail to execute.
Debezium 2.6 addresses this by creating sub-buckets of exactly one thousand elements and constructs the query using multiple in-clauses separated by a disjunction if the table include and exclude lists are too large. This improves the user experience, allowing a single connector to capture more than one thousand tables with a single deployment, using the query filter mode, and without resorting to regular expressions. (DBZ-7847).
Fixes and stability improvements
Debezium 2.6.2.Final introduces several new bug fixes and stability improvements, lets take a look at a few of them.
PostgreSQL offset flush race condition
The Debezium PostgreSQL connector requires that not only do we publish offset metadata to Kafka to handle connector restarts, but the connector must also flush these details to the database’s underlying replication slot to prevent unbounded WAL growth.
Using an example, let’s propose we have two changes for LSN 1 and 2. Both of these changes are written to two different partitions, A and B, respectively. As this happens asynchronously, it’s possible that the write to partition B succeeds while the write to partition A fails.
If Debezium receives acknowledgement for partition B before A, the connector will synchronize the WAL confirmed LSN as 2. Should the connector fail before acknowledging partition A or get restarted due to a rebalance, upon connector restart the change for LSN 1 is lost.
This problem has been fixed in Debezium 2.6 onward, and the replication slot’s confirmed LSN will correctly match the state in the topics (DBZ-7816).
Avro compatibility
In Debezium 2.5, we introduced several new timestamp fields, ts_us, and ts_ns, which represent the millisecond-based time values in microseconds and nanoseconds. Unfortunately, these fields were not introduced in a compatible way for users who use Avro (DBZ-7880).
We have adjusted the change event schema so that the new fields introduced are optional, making the change event schema compatible with old versions so that users can upgrade to Debezium 2.6 or later without issues using Avro.
Other fixes
-
After the initial deployment of Debezium, if a new table is added to MSSQL, its schema is was captured DBZ-7697
-
NO_DATA snapshot mode validation throw DebeziumException on restarts if snapshot is not completed DBZ-7780
-
DDL statement couldn’t be parsed DBZ-7788
-
old class reference in ibmi-connector services DBZ-7795
-
Ad-hoc blocking snapshot not working through file channeling without inserting a row in the database. DBZ-7806
-
Postgres: Potential data loss on connector restart DBZ-7816
-
Add c3p0 timeout configuration example to JDBC sink DBZ-7822
-
Sql Server incorrectly applying quoted snapshot statement overrides DBZ-7828
-
Debezium JDBC Sink not handle order correctly DBZ-7830
-
in the Cassandra documentation, there is a typo which should have been disable not Dusable. DBZ-7851
-
Debezium MySQL Snapshot Connector Fails DBZ-7858
-
Support Oracle DDL Alter Audit Policy DBZ-7864
-
Support Oracle DDL Create Audit Policy DBZ-7865
-
Default value of error retries not interpreted correctly DBZ-7870
-
Too much logs after Debezium update DBZ-7871
-
Avro schema compatibility issues when upgrading from Oracle Debezium 2.5.3.Final to 2.6.1.Final DBZ-7880
-
ParsingException (MySQL 8): create trigger if exists DBZ-7881
-
Debezium can’t handle columns with # in its name DBZ-7893
-
Oracle interval default values are not properly parsed DBZ-7898
In total there were 20 issues resolved in Debezium 2.6.2.Final. The list of changes can also be found in our release notes.
A big thank you to all the contributors from the community who worked diligently on this release: Anil Dasari, Chris Cranford, Duc Le Tu, Ilyas Ahsan, Jiri Pechanec, Jochen Schalanda, Mario Fiore Vitale, Michal Augustýn, Pradeep Nain, Robert Roldan, Robin Moffatt, Roman Kudryashov, and Vadzim Ramanenka!
About Debezium
Debezium is an open source distributed platform that turns your existing databases into event streams, so applications can see and respond almost instantly to each committed row-level change in the databases. Debezium is built on top of Kafka and provides Kafka Connect compatible connectors that monitor specific database management systems. Debezium records the history of data changes in Kafka logs, so your application can be stopped and restarted at any time and can easily consume all of the events it missed while it was not running, ensuring that all events are processed correctly and completely. Debezium is open source under the Apache License, Version 2.0.
Get involved
We hope you find Debezium interesting and useful, and want to give it a try. Follow us on Twitter @debezium, chat with us on Zulip, or join our mailing list to talk with the community. All of the code is open source on GitHub, so build the code locally and help us improve ours existing connectors and add even more connectors. If you find problems or have ideas how we can improve Debezium, please let us know or log an issue.
