Release Notes for Debezium 2.6

All notable changes for Debezium releases are documented in this file. Release numbers follow Semantic Versioning.

Release 2.6.0.Alpha2 (February 13rd 2024)

Kafka compatibility

This release has been built against Kafka Connect 3.6.1 and has been tested with version 3.6.1 of the Kafka brokers. See the Kafka documentation for compatibility with other versions of Kafka brokers.

Upgrading

Before upgrading any connector, be sure to check the backward-incompatible changes that have been made since the release you were using.

When you decide to upgrade one of these connectors to 2.6.0.Alpha2 from any earlier versions, first check the migration notes for the version you’re using. Gracefully stop the running connector, remove the old plugin files, install the 2.6.0.Alpha2 plugin files, and restart the connector using the same configuration. Upon restart, the 2.6.0.Alpha2 connectors will continue where the previous connector left off. As one might expect, all change events previously written to Kafka by the old connector will not be modified.

If you are using our container images, then please do not forget to pull them fresh from Quay.io.

Breaking changes

In 2.6, the task config format for Vitess connector was changed because the previous approach could de-stabilize Kafka Connect cluster (DBZ-7250). In some cases, when upgrading this change will cause NullPointerException and the error Couldn’t instantiate task <task-id> because it has an invalid task configuration. This task will not execute until reconfigured.. To fix this, delete & recreate each connector, using the same name & config. The connector(s) will start up and reuse the offsets last stored since they are using the same connector name (but will not try to reuse old task configs, which causes the error).

New features

  • Add Number of records captured and processed as metrics for Debezium MongoDB Connector DBZ-6432

  • Add timezone conversion to metadata in Timezone Converter SMT DBZ-7022

  • Create new implementation of DebeziumEngine DBZ-7024

  • Error when fail converting value with internal schema DBZ-7143

  • Provide alternative direct query for faster execution DBZ-7273

  • MongoDb connector doesn’t use post-images DBZ-7299

  • Support DECFLOAT in Db2 connector DBZ-7362

  • Create PubSub example for DS deployed via operator DBZ-7370

  • Support connector scoped trustore/keystore for MongoDB DBZ-7379

  • Put transaction id in offsets only when it’s present DBZ-7380

  • Replace additional rolebinding definition in kubernetes.yml with @RBACRule DBZ-7381

  • Reduce size of docker image for Debezium 2.6 and up DBZ-7385

  • Allow the C3P0ConnectionProvider to be customized via configuration DBZ-7431

  • Need to be able to set an ordering key value DBZ-7435

  • Evaluate container image size for Debezium UI served by nginx DBZ-7447

  • Support UUID as document key for incremental snapshotting DBZ-7451

  • Consolidate version management DBZ-7455

Fixes

  • Connector is getting stopped while processing bulk update(50k) records in debezium server 2.0.1.Final DBZ-6955

  • Debezium fails after table split operation DBZ-7360

  • Informix-Connector breaks on table with numerical default value DBZ-7372

  • MSSQL wrong default values in db schema for varchar, nvarchar, char columns DBZ-7374

  • Fix mysql version in mysql-replication container images DBZ-7384

  • Duplicate Debezium SMT transform DBZ-7416

  • Kinesis Sink Exception on PutRecord DBZ-7417

  • ParsingException (MariaDB Only): alterSpec drop foreign key with 'tablename.' prefix DBZ-7420

  • Poor performance with incremental snapshot with long list of tables DBZ-7421

  • Oracle Snapshot mistakenly uses LogMiner Offset Loader by default DBZ-7425

  • Reselect columns should source key values from after Struct when not using event-key sources DBZ-7429

  • Stopwatch throw NPE when toString is called without having statistics DBZ-7436

  • ReselectColumnsPostProcessor filter not use exclude predicate DBZ-7437

  • Adhoc snapshots are not triggered via File channel signal when submitted before the start of the application DBZ-7441

  • LogMiner batch size does not increase automatically DBZ-7445

  • Reduce string creation during SQL_REDO column read DBZ-7446

  • Oracle connector does not ignore reselection for excluded clob/blob columns DBZ-7456

  • The expected value pattern for table.include.list does not align with the documentation DBZ-7460

  • SQL Server queries with special characters fail after applying DBZ-7273 DBZ-7463

  • Signals actions are not loaded for SQLServer DBZ-7467

  • MySQL connector cannot parse table with WITH SYSTEM VERSIONING PARTITION BY SYSTEM_TIME DBZ-7468

  • Postgres images require clang-11 DBZ-7475

  • Make readiness and liveness proble timouts configurable DBZ-7476

  • Snapshotter SPI wrongly loaded on Debezium Server DBZ-7481

Other changes

  • Remove obsolete MySQL version from TF DBZ-7173

  • Correctly handle METADATA records DBZ-7176

  • Move Snapshotter interface to core module as SPI DBZ-7300

  • Implement Snapshotter SPI MySQL/MariaDB DBZ-7301

  • Update the Debezium UI repo with local development infra and readme file. DBZ-7353

  • Update QOSDK to the latest version DBZ-7361

  • Upstream artefact server image preparation job failing DBZ-7371

  • Tests in RHEL system testsuite fail to initialize Kafka containers DBZ-7373

  • Fix logging for schema only recovery mode in mysql connector DBZ-7376

  • Records from snapshot delivered out of order DBZ-7382

  • Upgrade json-path to 2.9.0 DBZ-7383

  • Remove the use of Lombok in Debezium testsuite DBZ-7386

  • Use Java 17 as compile-time dependency DBZ-7387

  • Upgrade Outbox Extension to Quarkus 3.7.0 DBZ-7388

  • Add dependancy update bot to the UI Repo DBZ-7392

  • Fix the unit test cases DBZ-7423

  • Adopt Oracle 23 to Testing Farm DBZ-7439

  • Upgrade protobuf to 3.25.2 DBZ-7442

  • Correct debezium.sink.pubsub.flowcontrol.* variable names in Debezium Server docs site DBZ-7443

  • Upgrade Quarkus for Debezium Server to 3.2.9.Final DBZ-7449

  • Fix TimescaleDbDatabaseTest to run into test container DBZ-7452

  • Upgrade example-mongo image version to 6.0 DBZ-7457

  • Test Db2ReselectColumnsProcessorIT randomly fails DBZ-7471

Release 2.6.0.Alpha1 (January 18th 2024)

Kafka compatibility

This release has been built against Kafka Connect 3.6.1 and has been tested with version 3.6.1 of the Kafka brokers. See the Kafka documentation for compatibility with other versions of Kafka brokers.

Upgrading

Before upgrading any connector, be sure to check the backward-incompatible changes that have been made since the release you were using.

When you decide to upgrade one of these connectors to 2.6.0.Alpha1 from any earlier versions, first check the migration notes for the version you’re using. Gracefully stop the running connector, remove the old plugin files, install the 2.6.0.Alpha1 plugin files, and restart the connector using the same configuration. Upon restart, the 2.6.0.Alpha1 connectors will continue where the previous connector left off. As one might expect, all change events previously written to Kafka by the old connector will not be modified.

If you are using our container images, then please do not forget to pull them fresh from Quay.io.

Breaking changes

MongoDB no longer supports replica_set connection mode (DBZ-7260).

Re-select columns post-processor used for the key defined by message.key.columns for query building. This is not correct for most tables with primary key. The default behaviour has changed and the table primary key is used by default. A new configuration option was introduced to allow user to choose is primary key or generated key should be used (DBZ-7358).

New features

  • Provide a public API from the connector implementations to retrieve the list of matching collections or tables based on the different include-/exclude lists DBZ-7167

  • Notifications are Missing the ID field in log channel DBZ-7249

  • Provide config option to customize CloudEvents.data schema name DBZ-7284

  • Clarify comment on serialization of document ids DBZ-7287

  • Unittest for hasCommitAlreadyBeenHandled in CommitScn Class DBZ-7288

  • Oracle Infinispan abandoned trasactions minor enhancements DBZ-7313

  • Add support for NEW_ROW_AND_OLD_VALUES value capture type. DBZ-7348

Fixes

  • Empty object sent to GCP Pub/Sub after DELETE event DBZ-7098

  • Debezium-ddl-parser crashes on parsing MySQL DDL statement (subquery with UNION) DBZ-7259

  • Oracle DDL parsing error in PARTITION REFERENCE DBZ-7266

  • Enhance Oracle’s CREATE TABLE for Multiple Table Specifications DBZ-7286

  • PostgreSQL ad-hoc blocking snapshots fail when snapshot mode is "never" DBZ-7311

  • Ad-hoc blocking snapshot dies with "invalid snapshot identifier" immediately after connector creation DBZ-7312

  • Specifying a table include list with spaces between elements cause LogMiner queries to miss matches DBZ-7315

  • Debezium heartbeat.action.query does not start before writing to WAL: part 2 DBZ-7316

  • errors.max.retries is not used to stop retrying DBZ-7342

  • Oracle connector is ocasionally unable to find SCN DBZ-7345

  • Initial snapshot notifications should use full identifier. DBZ-7347

  • MySqlJdbcSinkDataTypeConverterIT#testBooleanDataTypeMapping fails DBZ-7355

Other changes

  • Add service loader manifests for all Connect plugins DBZ-7298

  • Update Groovy version to 4.x DBZ-7340

  • Upgrade Antora to 3.1.7 DBZ-7344

  • Upgrade Outbox Extension to Quarkus 3.6.5 DBZ-7352