Debezium 2.5.3.Final Released

Breaking changes

The team aims to avoid any potential breaking changes between minor releases; however, such changes are sometimes inevitable.

SQL Server: The SQL Server connector was not capturing all schemas when the connector was first deployed, and instead, was only capturing the schemas based on the tables defined in the configuration’s include list. This was a bug that could prevent users from easily adding new tables to the connector when expecting that the new table’s schema would already exist in the schema history topic. The connector now correctly honors the store.only.captured.tables.ddl configuration option (DBZ-7593).

For existing connector deployments, if you do not specifically set the store.only.captured.tables.ddl property for the schema history topic, the connector will begin capturing schema changes for all relevant tables in your database. If you want to prevent this and retain the prior behavior, you will need to adjust your connector configuration by adding schema.history.internal.store.only.captured.tables.ddl with a value of true.

Improvements and changes

TRACE level logging for Debezium Server

Debezium Server is a ready-made runtime for Debezium source connectors that uses the Quarkus framework to manage the source and sink deployments. As most Debezium Server users are aware who have reached out with questions or bugs, we often ask for TRACE-level logs and this has often proven difficult as it requires a full rebuild of Debezium Server due to how minimum logging level is a build-time configuration in Quarkus.

With Debezium 2.5.3.Final and the upcoming Debezium 2.6.0.Final releases, this will no longer be required. The build time configuration has been adjusted by default to include TRACE logging levels, so moving forward users can simply set the log level to TRACE and restart Debezium Server to obtain the logs (DBZ-7369).

Informix appends LSN to Transaction Identifier

Informix databases only increases the transaction identifier when there are concurrent transactions, otherwise the value remains identical for sequential transactions. This can prove difficult for users who may want to utilize the transaction metadata to order change events in a post processing step.

In Debezium 2.5.3.Final and the upcoming 2.6, Informix will now append the log sequence number (LSN) to the transaction identifier so that users can easily sort change events based on the transaction metadata. The transaction identifier field will now use the format <id>:<lsn> (DBZ-7454). This change affects transaction metadata events and the source information block for change events, as shown below:

Transaction Begin Event

{
  "status": "BEGIN",
  "id": "571:53195829",
  ...
}

Transaction End Event

{
  "status": "END",
  "id": "571:53195832",
  ...
}

Change Events

{
  ...
  "source": {
    "id": "571:53195832"
    ...
  }
}

PostgreSQL improvements

Updated JDBC driver: The PostgreSQL driver was upgraded to version 42.6.1.
Improved logging: During the start-up of the PostgreSQL connector, some users reported the connector stuck or acting as though it could be caught in an infinite loop. (DBZ-7581). The issue appeared that the pg_replication_slot_advance was taking longer than the read database timeout, and the exception thrown wasn’t satisfactory to identify the problem. In this case, the logging has been significantly improved to identify this root-cause so that users can make necessary adjustments if they encounter this problem.

Other changes

Altogether, 25 issues were fixed in this release:

NullPointerException in MongoDB connector DBZ-6434
Numeric default value decimal scale mismatch DBZ-7562
Documentation conflict DBZ-7565
Oracle connector always brings OLR dependencies DBZ-7579
Correct JDBC connector dependencies DBZ-7580
Unavailable Toasted HSTORE Json Storage Mode column causes serialization failure DBZ-7582
Incorrect value of TIME(n) replicate from MySQL if the original value is negative DBZ-7594
Re-select Post Processor not working for complex types DBZ-7596
Serialization of XML columns with NULL values fails using Infinispan Buffer DBZ-7598
Null instead of toast placeholder written for binary types when "hex" mode configured DBZ-7599
Poor snapshot performance during schema snapshot DDL processing DBZ-7608
Re-select post processor performance DBZ-7611
Uncaught exception during config validation in Engine DBZ-7614
Incremental snapshot query doesn’t honor message.key.columns order DBZ-7617
Metric ScnFreezeCount never increases DBZ-7619
Upgrade Debezium Server to Quarkus 3.2.10 DBZ-7624
Cassandra can have misaligned Jackson dependencies DBZ-7629
Numeric value without mantissa cannot be parsed DBZ-7643
Fix MySQL image fetch for tests DBZ-7651

A big thank you to all the contributors from the community who worked on this release: Amirmohammad Sadat Shokouhi, Andreas Martens, Andrey Pustovetov, Anisha Mohanty, Chris Cranford, Enzo Cappa, Indra Shukla, Jiri Pechanec, Mario Fiore Vitale, Nick Golubev, Razvan Laurus, Robert Roldan, and Sean Wu!

Outlook & What’s next?

We are only just a few weeks away from the final build of Debezium 2.6. I would anticipate 2.6.0.CR1 build later in the week, with 2.6.0.Final sometime next week, barring no unforeseen issues.

In a few short weeks, the team will begin to shift focus to what lies ahead with Debezium 2.7. This will be the last minor release of the Debezium 2.x release stream, which will include read-only incremental snapshots for other connectors, a new dedicated MariaDB source connector, and a MongoDB sink connector, just to name a few highlights. We will also be carrying over any work from 2.6 that we’re still working on but didn’t make it into that release.

And in closing, the last half of the year will be focusing on Debezium 3, where we’ll be moving to Java 17, work on new time-series source connectors, and more. Please take a moment and review our road map. If you have any suggestions or ideas, please feel free to get in touch with us on our mailing list or in our Zulip chat.

Until next time…

Chris Cranford

Chris is a software engineer at Red Hat. He previously was a member of the Hibernate ORM team and now works on Debezium. He lives in North Carolina just a few hours from Red Hat towers.

About Debezium

Debezium is an open source distributed platform that turns your existing databases into event streams, so applications can see and respond almost instantly to each committed row-level change in the databases. Debezium is built on top of Kafka and provides Kafka Connect compatible connectors that monitor specific database management systems. Debezium records the history of data changes in Kafka logs, so your application can be stopped and restarted at any time and can easily consume all of the events it missed while it was not running, ensuring that all events are processed correctly and completely. Debezium is open source under the Apache License, Version 2.0.

Get involved

We hope you find Debezium interesting and useful, and want to give it a try. Follow us on Twitter @debezium, chat with us on Zulip, or join our mailing list to talk with the community. All of the code is open source on GitHub, so build the code locally and help us improve ours existing connectors and add even more connectors. If you find problems or have ideas how we can improve Debezium, please let us know or log an issue.