I am pleased to announce the first maintenance release of Debezium 2.7, 2.7.1.Final. This maintenance release focuses on addressing a number of stability issues, including improvements to ad-hoc snapshots, closing of transformations in the embedded engine, improvements to the Oracle LogMiner implementation, Vitess epoch calculations, and more…​

Let’s dive into these changes…​

Fixes and stability improvements

Debezium 2.7.1.Final introduces several new bug fixes and stability improvements, lets take a look at a few of them.

MariaDB is now included in Debezium container images

In Debezium 2.7.0.Final, we mistakenly missed including the new Debezium for MariaDB connector in the container images, which necessitated users to manually install the MariaDB connector. This has been fixed and Debezium 2.7.1.Final’s container images now include all connectors like they did previously (DBZ-8008).

Support for MariaDB RENAME TABLE

In Debezium 2.7 with the separation of MariaDB and MySQL into their unique connectors, each also use a unique grammar to parse SQL statements. A regression was identified that the new MariaDB grammar did not support the SQL syntax for RENAME TABLE (DBZ-8066). This has been fixed and is fully supported in both MySQL and MariaDB.

JDBC sink partition rebalance fixed

In prior versions of the Debezium JDBC sink, users may have noticed several strange messages in their connector logs that may resemble the following output:

Ignoring invalid task provided offset topicXYZ-1 OffsetAndMetadata{offset=4966, leaderEpoch=null, metadata=''} — partition not assigned, assignment=[topicXYZ-1] (org.apache.kafka.connect.runtime.WorkerSinkTask:434)

Thanks to a user report (DBZ-7946), it was identified that the JDBC sink connector was not properly handling the rebalance of partitions across multiple tasks. This regression has been fixed in 2.7.1.Final, and when a partition rebalance occurs, the partition will be closed by the task that no longer will manage the partition; ergo, the above messages will no longer be written to the connector logs.

Ad-hoc snapshot improvements

In Debezium 2.7, users may have noticed that if you picked the insert_delete watermark strategy for handling signal table entries, that the connector would eventually fail with a StackOverflowError (DBZ-8011). With Debezium 2.7.1.Final or later, this regression has been fixed and users can once again use the insert_delete watermark strategy.

In addition, there was also a report where sending an ad-hoc snapshot signal that contained a regular expression that matches no database tables would cause the ad-hoc snapshot process to fail with an "Unexpected result ." error (DBZ-8050). This has also been fixed so the ad-hoc snapshot will gracefully skip the signal as no matching tables were found.

Finally, it was also identified that ad-hoc blocking snapshots could lead to a CommunicationsException due to how the connection was being managed by the initial and blocking snapshot code paths (DBZ-8058). This has been fixed in Debezium 2.7.1.Final by having the blocking snapshot validate and re-open the connection if required.

Embedded engine now closes transformations

The new asynchronous embedded engine was recently released by the Debezium team and continues to improve across releases. As a part of our continuous improvements, it was identified that single message transforms did not have their close() method being called. This could easily lead to resource leaks depending on what the transformation does.

This release introduces a fix to automatically close transformations to avoid this problem (DBZ-8106).

Oracle connector could produce a NullPointerException

A change was added to the Debezium Oracle connector (DBZ-8055) that to address several race concerns with Oracle LogMiner and capturing changes. Unfortunately, that change introduced a corner case where the connector would recalculate the upper mining range, and in such cases would lead to a NullPointerException.

A new fix (DBZ-8119) was included in Debezium 2.7.1.Final to correct the problem. Oracle connector users are recommended to upgrade to 2.7.1.Final and to skip 2.7.0.Final to avoid this regression.

Vitess epoch calculations fixed

It was identified that there were some corner cases where the transaction epoch value was being reset when a transaction ID was unavailable (DBZ-8087). This is unexpected behavior as the transaction epoch should monotonically increase, and the resetting to zero broke that expectation. A fix has been added to avoid the reset and to restore the expected transaction epoch behavior.

Other fixes

  • Document new MariaDB connector DBZ-7786

  • Clarify that Oracle connector does not read from physical standby DBZ-7895

  • Issue with Hybrid mode and DDL change DBZ-7991

  • Incorrect offset/history property mapping generated DBZ-8007

  • Bump Infinispan to 14.0.29.Final DBZ-8010

  • JDBC primary.key.fields cannot be empty when i set insert.mode to upsert and primary.key.mode record_value DBZ-8018

  • Unable to acquire buffer lock, buffer queue is likely full DBZ-8022

  • ORA-65090: operation only allowed in a container database when connecting to a non-CDB database DBZ-8023

  • Add disclaimer that PostProcessors and CustomConverters are Debezium source connectors only DBZ-8031

  • Added type to Prometheus JMX exporter DBZ-8036

  • Add kafka.producer metrics to debezium-server jmx exporter config DBZ-8037

  • Oracle connector inconsistency in redo log switches DBZ-8055

  • Oracle histogram metrics are no longer printed in logs correctly DBZ-8068

  • In hybrid log.mining.strategy reconstruction logs should be set to DEBUG DBZ-8070

  • Support capturing BLOB column types during snapshot for MySQL/MariaDB DBZ-8076

  • Conditionalize reference to the MySQL default value in description of schema.history.internal.store.only.captured.databases.ddl DBZ-8081

  • After changing the column datatype from int to float the Debezium fails to round it and i get a null value for this field in the stream DBZ-8089

  • MySQL and MariaDB keyword YES cannot be parsed as a column name DBZ-8092

  • Update third-party LICENSE with LGPL forMariaDB Connector/J DBZ-8099

  • NotificationIT tests seemingly seem to fail due to stepping on one another DBZ-8100

  • ORA-26928 - Unable to communicate with XStream apply coordinator process should be retriable DBZ-8102

  • Converters documentation uses incorrect examples DBZ-8104

  • Remove reference to`additional condition` signal parameter from ad hoc snapshots doc DBZ-8107

  • Don’t close connection after loading timescale metadata in TimescaleDb SMT DBZ-8109

  • Primary Key Update/ Snapshot Race Condition DBZ-8113

  • Docs: connect-log4j.properties instead log4j.properties DBZ-8117

  • TimescaleDbDatabaseTest.shouldTransformCompressedChunks is failing DBZ-8123

  • Update Oracle connector doc to describe options for restricting access permissions for the Debezium LogMiner user DBZ-8124

In total there were 37 issues resolved in Debezium 2.7.1.Final. The list of changes can also be found in our release notes.

Chris Cranford

Chris is a software engineer at Red Hat. He previously was a member of the Hibernate ORM team and now works on Debezium. He lives in North Carolina just a few hours from Red Hat towers.

   


About Debezium

Debezium is an open source distributed platform that turns your existing databases into event streams, so applications can see and respond almost instantly to each committed row-level change in the databases. Debezium is built on top of Kafka and provides Kafka Connect compatible connectors that monitor specific database management systems. Debezium records the history of data changes in Kafka logs, so your application can be stopped and restarted at any time and can easily consume all of the events it missed while it was not running, ensuring that all events are processed correctly and completely. Debezium is open source under the Apache License, Version 2.0.

Get involved

We hope you find Debezium interesting and useful, and want to give it a try. Follow us on Twitter @debezium, chat with us on Zulip, or join our mailing list to talk with the community. All of the code is open source on GitHub, so build the code locally and help us improve ours existing connectors and add even more connectors. If you find problems or have ideas how we can improve Debezium, please let us know or log an issue.