Release Notes for Debezium 3.0

All notable changes for Debezium releases are documented in this file. Release numbers follow Semantic Versioning.

Release 3.0.0.Final (October 2nd 2024)

Kafka compatibility

This release has been built against Kafka Connect 3.8.0 and has been tested with version 3.8.0 of the Kafka brokers. See the Kafka documentation for compatibility with other versions of Kafka brokers.

Upgrading

Before upgrading any connector, be sure to check the backward-incompatible changes that have been made since the release you were using.

When you decide to upgrade one of these connectors to 3.0.0.Final from any earlier versions, first check the migration notes for the version you’re using. Gracefully stop the running connector, remove the old plugin files, install the 3.0.0.Final plugin files, and restart the connector using the same configuration. Upon restart, the 3.0.0.Final connectors will continue where the previous connector left off. As one might expect, all change events previously written to Kafka by the old connector will not be modified.

If you are using our container images, then please do not forget to pull them fresh from Quay.io.

Breaking changes

The deprecated additional-condition field of execute-snapshot signal was removed. The field is fully replaced with previously introduced additional-conditions field (DBZ-8278).

New features

  • Add documentation for custom converters in PG DBZ-7820

  • Create REST bridge for DBZ signal channels DBZ-8101

  • Support int/bigint arrays in reselect colums postprocessors DBZ-8212

  • Log the record key when debezium fails to send the record to Kafka DBZ-8282

Fixes

  • Custom convert (all to strings) and SQLServer default '0' type issue DBZ-7045

  • UnsupportedClassVersionError while running debezium-connector docker Image DBZ-7751

  • Error writing data to target database. (Caused by: java.lang.RuntimeException: org.postgresql.util.PSQLException: The column index is out of range: 140, number of columns: 139.) DBZ-8221

  • Debezium Server messages not being sent to Pub/Sub after restart DBZ-8236

  • An aborted ad-hoc blocking snapshot leaves the connector in a broken state DBZ-8244

  • JDBC Sink truncate event also add event to updateBufferByTable DBZ-8247

  • mysql-binlog-connector-java doesn’t compile with java 21 DBZ-8253

  • DDL statement couldn’t be parsed. 'mismatched input 'NOCACHE' expecting {'AS', 'USAGE', ';'} DBZ-8262

  • journal processing loops after journal offset reset DBZ-8265

Other changes

  • Add async engine config options to server documentation DBZ-8133

  • Bump apicurio schema registry to 2.6.2.Final DBZ-8145

  • Correct description of the all_tables option for the PG publication.autocreate.mode property DBZ-8268

  • Test docs for productization and fix broken links and rendering errors DBZ-8284

Release 3.0.0.CR2 (September 25th 2024)

Kafka compatibility

This release has been built against Kafka Connect 3.8.0 and has been tested with version 3.8.0 of the Kafka brokers. See the Kafka documentation for compatibility with other versions of Kafka brokers.

Upgrading

Before upgrading any connector, be sure to check the backward-incompatible changes that have been made since the release you were using.

When you decide to upgrade one of these connectors to 3.0.0.CR2 from any earlier versions, first check the migration notes for the version you’re using. Gracefully stop the running connector, remove the old plugin files, install the 3.0.0.CR2 plugin files, and restart the connector using the same configuration. Upon restart, the 3.0.0.CR2 connectors will continue where the previous connector left off. As one might expect, all change events previously written to Kafka by the old connector will not be modified.

If you are using our container images, then please do not forget to pull them fresh from Quay.io.

Breaking changes

There are no breaking changes in this release.

New features

  • Snapshot isolation level options for postgres DBZ-1252

  • Retry flush records if LockAcquisitionException occured in mysql DBZ-7291

  • Add support for MAX_STRING_SIZE set to EXTENDED DBZ-8039

  • Add invalid value logger for dates to Debezium Vitess Connector DBZ-8235

  • Support BLOB with EMPTY_BLOB() as default DBZ-8248

Fixes

  • Debezium does not restart automatically after throwing an ORA-00600 krvrdccs30 error DBZ-8223

  • JDBC sink doesn’t include fields as per documentation DBZ-8224

  • Unbounded number of processing threads in async engine DBZ-8237

  • Streaming metrics are stuck after an ad-hoc blocking snapshot DBZ-8238

  • DDL statement couldn’t be parsed with IF EXISTS DBZ-8240

  • Random engine factory used by default DBZ-8241

  • JDBC sink test suite should use the debezium/connect:nightly image for e2e tests DBZ-8245

  • Performance Regression in Debezium Server Kafka after DBZ-7575 fix DBZ-8251

  • Error Prone library included in MySQL connector DBZ-8258

  • Debezium.text.ParsingException: DDL statement couldn’t be parsed DBZ-8259

Other changes

  • Test and check compatibility with ojdbc11 DBZ-3658

  • Broken link to Streams doc about configuring logging DBZ-8231

  • Document passthrough hibernate.* properties for the JDBC connector DBZ-8232

  • Bump Infinispan to 15.0.8.Final DBZ-8246

  • AbstractConnectorTest consumeRecordsUntil may prematurely exit loop DBZ-8250

  • Add a note to the docs about JDBC batch retry configs DBZ-8252

  • Fix conditionalization in shared MariaDB/MySQL file DBZ-8254

  • Add Oracle FUTC license DBZ-8260

  • Remove Oracle libs from product assembly package DBZ-8261

  • debezium-connector-binlog does not need MariaDB dependency DBZ-8263

  • Provide subset package for Debezium Server DBZ-8264

  • Bump container images to Fedora 40 DBZ-8266

Release 3.0.0.CR1 (September 13rd 2024)

Kafka compatibility

This release has been built against Kafka Connect 3.8.0 and has been tested with version 3.8.0 of the Kafka brokers. See the Kafka documentation for compatibility with other versions of Kafka brokers.

Upgrading

Before upgrading any connector, be sure to check the backward-incompatible changes that have been made since the release you were using.

When you decide to upgrade one of these connectors to 3.0.0.CR1 from any earlier versions, first check the migration notes for the version you’re using. Gracefully stop the running connector, remove the old plugin files, install the 3.0.0.CR1 plugin files, and restart the connector using the same configuration. Upon restart, the 3.0.0.CR1 connectors will continue where the previous connector left off. As one might expect, all change events previously written to Kafka by the old connector will not be modified.

If you are using our container images, then please do not forget to pull them fresh from Quay.io.

Breaking changes

JMX signalling and notification did not work correctly for deployments with SQL Server configured for multiple tasks. To fix the issue it was necessary to change the naming of signalling and notification MBeans to make them unique per each task (DBZ-8137).

Deprecated Oracle connector configuration options were removed from the project (DBZ-8181).

Vector datatype names introduced for PostgreSQL were too related to PostgreSQL datatype naming. The type names were changed to more generic ones and are shared between PostgreSQL and MySQL (DBZ-8183).

New features

  • Add support for MySQL 9 DBZ-8030

  • Add support for MySQL vector datatype DBZ-8157

  • Refactor engine signal support DBZ-8160

  • Add feature to inherit shard epoch DBZ-8163

  • Avoid 3 second delay in Oracle when one of the RAC nodes is offline DBZ-8177

  • Truncate byte buffer should return a new array DBZ-8189

  • Support for older MongoDb versions DBZ-8202

  • Add VECTOR functions to MySQL grammar DBZ-8210

  • Support MariaDB 11.4.3 DBZ-8226

  • Add information about helm chart installation to operator readme DBZ-8233

Fixes

  • Make ORA-00600 - krvrdccs10 automatically retriable DBZ-5009

  • Incremental snapshot fails with NPE if surrogate key doesn’t exist DBZ-7797

  • MySQL 8.4 incompatibility due to removed SQL commands DBZ-7838

  • Postgres connector - null value processing for "money" type column. DBZ-8027

  • Using snapshot.include.collection.list with Oracle raises NullPointerException DBZ-8032

  • Performance degradation when reconstructing (log.mining.stragtegy hybrid mode) DBZ-8071

  • The source data type exceeds the debezium data type and cannot deserialize the object DBZ-8142

  • Incorrect use of generic types in tests DBZ-8166

  • Postgres JSONB Fields are not supported with Reselect Post Processor DBZ-8168

  • NullPointerException (schemaUpdateCache is null) when restarting Oracle engine DBZ-8187

  • XStream may fail to attach on retry if previous attempt failed DBZ-8188

  • Exclude Oracle 23 VECSYS tablespace from capture DBZ-8198

  • AbstractProcessorTest uses an incorrect database name when run against Oracle 23 Free edition DBZ-8199

  • DDL statement couldn’t be parsed: REVOKE IF EXISTS DBZ-8209

  • System testsuite fails with route name being too long DBZ-8213

  • Oracle TableSchemaBuilder provides wrong column name in error message DBZ-8217

  • Using ehcache in Kafka connect throws an XMLConfiguration parse exception DBZ-8219

  • OcpJdbcSinkConnectorIT fails DBZ-8228

  • Container image does not install correct apicurio deps DBZ-8230

Other changes

  • Documentation for signals provides incorrect data-collection format for some connectors DBZ-8090

  • Latest Informix JDBC Driver DBZ-8167

  • upgrade Adobe s3mock to version 3.10.0 DBZ-8169

  • Include Jackson libraries to JDBC connector Docker image distribution DBZ-8175

  • Ehcache fails to start, throwing "Invaild XML Configuration" DBZ-8178

  • Enable snapshot.database.errors.max.retriesEnable during Oracle tests DBZ-8184

  • Change event for a logical decoding message doesn’t contain transaction field DBZ-8185

  • Add MariaDB connector server distribution DBZ-8186

  • Update Vitess example to Debezium 2.7/Vitess 19 DBZ-8196

  • OracleConnectorIT test shouldGracefullySkipObjectBasedTables can timeout prematurely DBZ-8197

  • Reduce log verbosity of OpenLogReplicator SCN confirmation DBZ-8201

  • Implement separate source and sink connector sections in documentation navigation DBZ-8220

Release 3.0.0.Beta1 (August 22nd 2024)

Kafka compatibility

This release has been built against Kafka Connect 3.8.0 and has been tested with version 3.8.0 of the Kafka brokers. See the Kafka documentation for compatibility with other versions of Kafka brokers.

Upgrading

Before upgrading any connector, be sure to check the backward-incompatible changes that have been made since the release you were using.

When you decide to upgrade one of these connectors to 3.0.0.Beta1 from any earlier versions, first check the migration notes for the version you’re using. Gracefully stop the running connector, remove the old plugin files, install the 3.0.0.Beta1 plugin files, and restart the connector using the same configuration. Upon restart, the 3.0.0.Beta1 connectors will continue where the previous connector left off. As one might expect, all change events previously written to Kafka by the old connector will not be modified.

If you are using our container images, then please do not forget to pull them fresh from Quay.io.

Breaking changes

Debezim Kafka sink could wait indefinitely in case of Kafka broker unavailablity. A support for configurable timeout was added and the default behaviour is timeout after 30 seconds (DBZ-7575).

RabbitMQ native stream sink was sending changes all messages into a static single stream. With the new default behaviour the changes are sent to a distinct stream for each table (DBZ-8118).

New features

  • Implement Ehcache event buffer DBZ-7758

  • Expose a metric for number of create, update, delete events per table DBZ-8035

  • Log additional details about abandoned transactions DBZ-8044

  • Introduce timeout for replication slot creation DBZ-8073

  • ConverterBuilder doesn’t pass Headers to be manipulated DBZ-8082

  • Add SMT to decode binary content of a logical decoding message DBZ-8103

  • Support DECIMAL(p) Floating Point DBZ-8114

  • Support for PgVector datatypes DBZ-8121

  • Implement in process signal channel DBZ-8135

  • Validate log position method missing gtid info from SourceInfo DBZ-8140

  • Vitess Connector Epoch should support parallelism & shard changes DBZ-8154

  • Add an option for publication.autocreate.mode to create a publication with no tables DBZ-8156

Fixes

  • Incremental snapshots don’t work with CloudEvent converter DBZ-7601

  • Snapshot retrying logic falls into infinite retry loop DBZ-7860

  • Primary Key Update/ Snapshot Race Condition DBZ-8113

  • Docs: connect-log4j.properties instead log4j.properties DBZ-8117

  • Recalculating mining range upper bounds causes getScnFromTimestamp to fail DBZ-8119

  • ORA-00600: internal error code, arguments: [krvrdGetUID:2], [18446744073709551614], [], [], [], [], [], [], [], [], [], [] DBZ-8125

  • ConvertingFailureIT#shouldFailConversionTimeTypeWithConnectModeWhenFailMode fails randomly DBZ-8128

  • ibmi Connector does not take custom properties into account anymore DBZ-8129

  • Unpredicatable ordering of table rows during insertion causing foreign key error DBZ-8130

  • schema_only crashes ibmi Connector DBZ-8131

  • Support larger database.server.id values DBZ-8134

  • Open redo thread consistency check can lead to ORA-01291 - missing logfile DBZ-8144

  • SchemaOnlyRecoverySnapshotter not registered as an SPI service implementation DBZ-8147

  • When stopping the Oracle rac node the Debezium server throws an expections - ORA-12514: Cannot connect to database and retries DBZ-8149

  • Issue with Debezium Snapshot: DateTimeParseException with plugin pgoutput DBZ-8150

  • JDBC connector validation fails when using record_value with no primary.key.fields DBZ-8151

  • Taking RAC node offline and back online can lead to thread inconsistency DBZ-8162

Other changes

  • MySQL has deprecated mysql_native_password usage DBZ-7049

  • Upgrade to Apicurio 2.5.8 or higher DBZ-7357

  • Write and publish Debezium Orchestra blog post DBZ-7972

  • Move Debezium Conductor repository under Debezium Organisation DBZ-7973

  • Decide on name, jira components, etc…​ for Debezium Orchestra platform DBZ-7975

  • Migrate Postgres testsuite to async engine DBZ-8077

  • Conditionalize reference to the MySQL default value in description of schema.history.internal.store.only.captured.databases.ddl DBZ-8081

  • Bump Debezium Server to Quarkus 3.8.5 DBZ-8095

  • Converters documentation uses incorrect examples DBZ-8104

  • Remove reference to`additional condition` signal parameter from ad hoc snapshots doc DBZ-8107

  • TimescaleDbDatabaseTest.shouldTransformCompressedChunks is failing DBZ-8123

  • Update Oracle connector doc to describe options for restricting access permissions for the Debezium LogMiner user DBZ-8124

  • Use SQLSTATE to handle exceptions for replication slot creation command timeout DBZ-8127

  • Re-add check to test for if assembly profile is active DBZ-8138

  • Add LogMiner start mining session retry attempt counter to logs DBZ-8143

  • Reduce logging verbosity of XStream DML event data DBZ-8148

  • Upgrade Outbox Extension to Quarkus 3.14.0 DBZ-8164

Release 3.0.0.Alpha2 (August 2nd 2024)

Kafka compatibility

This release has been built against Kafka Connect 3.8.0 and has been tested with version 3.8.0 of the Kafka brokers. See the Kafka documentation for compatibility with other versions of Kafka brokers.

Upgrading

Before upgrading any connector, be sure to check the backward-incompatible changes that have been made since the release you were using.

When you decide to upgrade one of these connectors to 3.0.0.Alpha2 from any earlier versions, first check the migration notes for the version you’re using. Gracefully stop the running connector, remove the old plugin files, install the 3.0.0.Alpha2 plugin files, and restart the connector using the same configuration. Upon restart, the 3.0.0.Alpha2 connectors will continue where the previous connector left off. As one might expect, all change events previously written to Kafka by the old connector will not be modified.

If you are using our container images, then please do not forget to pull them fresh from Quay.io.

Breaking changes

Debezium is now build with Kafka 3.8.0. There were few changes in Kafka internal APIs Debezium is using. The codebase was modified to run with both pre-3.8.0 and 3.8.0 versions (DBZ-8105).

New features

  • Add Status ObservedGeneration to Operator DBZ-8025

  • Support Custom Converters in Debezium Server DBZ-8040

  • Support FLOAT32 type in debezium-connector-spanner DBZ-8043

  • Debezium should auto exclude empty shards (no tablets) and not crash on keyspaces with empty shards DBZ-8053

  • Refactor LogMining implementation to allow alternative cache implementations DBZ-8054

  • Standard Webhooks signatures for HTTP sink DBZ-8063

  • Vitess-connector should provide a topic naming strategy that supports separate connectors per-table DBZ-8069

  • Update third-party LICENSE with LGPL forMariaDB Connector/J DBZ-8099

  • Rabbitmq native stream Failed DBZ-8108

Fixes

  • Embedded Infinispan tests fail to start with Java 23 DBZ-7840

  • Clarify that Oracle connector does not read from physical standby DBZ-7895

  • StackOverflow exception on incremental snapshot DBZ-8011

  • JDBC primary.key.fields cannot be empty when i set insert.mode to upsert and primary.key.mode record_value DBZ-8018

  • Unable to acquire buffer lock, buffer queue is likely full DBZ-8022

  • Release process sets incorrect images for k8s for the next development version DBZ-8041

  • Use recrate as (default) rollout strategy for deployments DBZ-8047

  • "Unexpected input: ." when snapshot incremental empty Database DBZ-8050

  • Debezium Operator Using RollingUpdate Strategy DBZ-8051

  • Debezium Operator Using RollingUpdate Strategy DBZ-8052

  • Oracle connector inconsistency in redo log switches DBZ-8055

  • Blocking snapshot can fail due to CommunicationsException DBZ-8058

  • FakeDNS not working with JDK version > 18 DBZ-8059

  • Debezium Operator with a provided Service Account doesn’t spin up deployment DBZ-8061

  • ParsingException (MySQL/MariaDB): rename table syntax DBZ-8066

  • Oracle histogram metrics are no longer printed in logs correctly DBZ-8068

  • In hybrid log.mining.strategy reconstruction logs should be set to DEBUG DBZ-8070

  • Support capturing BLOB column types during snapshot for MySQL/MariaDB DBZ-8076

  • Standard Webhooks auth secret config value is not marked as PASSWORD_PATTERN DBZ-8078

  • Vitess transaction Epoch should not reset to zero when tx ID is missing DBZ-8087

  • After changing the column datatype from int to float the Debezium fails to round it and i get a null value for this field in the stream DBZ-8089

  • MySQL and MariaDB keyword YES cannot be parsed as a column name DBZ-8092

  • NotificationIT tests seemingly seem to fail due to stepping on one another DBZ-8100

  • ORA-26928 - Unable to communicate with XStream apply coordinator process should be retriable DBZ-8102

  • Transformations are not closed in emebdded engine DBZ-8106

  • Don’t close connection after loading timescale metadata in TimescaleDb SMT DBZ-8109

Other changes

  • Bump Infinispan to 14.0.29.Final DBZ-8010

  • Write a blog post about async engine DBZ-8013

  • Test offset/history store configurations DBZ-8015

  • Upgrade postgres server version to 15 DBZ-8062

  • Disable DebeziumResourceNoTopicCreationIT - no longer compatible with Java 21 DBZ-8067

  • Speed-up PostgresShutdownIT DBZ-8075

  • Add MariaDB to debezium/connect image DBZ-8088

Release 3.0.0.Alpha1 (July 11st 2024)

Kafka compatibility

This release has been built against Kafka Connect 3.7.0 and has been tested with version 3.7.0 of the Kafka brokers. See the Kafka documentation for compatibility with other versions of Kafka brokers.

Upgrading

Before upgrading any connector, be sure to check the backward-incompatible changes that have been made since the release you were using.

When you decide to upgrade one of these connectors to 3.0.0.Alpha1 from any earlier versions, first check the migration notes for the version you’re using. Gracefully stop the running connector, remove the old plugin files, install the 3.0.0.Alpha1 plugin files, and restart the connector using the same configuration. Upon restart, the 3.0.0.Alpha1 connectors will continue where the previous connector left off. As one might expect, all change events previously written to Kafka by the old connector will not be modified.

If you are using our container images, then please do not forget to pull them fresh from Quay.io.

Breaking changes

Debezium connectors now require Java 17 for runtime and Java 21 for building. Debezium Server, Debezium Operator, and Debezium Outbox extension require Java 21 both for build and runtime (DBZ-6795).

New features

  • Provide MongoDB sink connector DBZ-7223

  • Extends process of finding Bundle path DBZ-7992

  • Support FLOAT32 type in debezium-connector-spanner DBZ-8043

Fixes

  • Debezium postgres jdbc sink not handling infinity values DBZ-7920

  • JdbcSinkTask doesn’t clear offsets on stop DBZ-7946

  • ibmi as400 connector config isn’t prefixed with "database." DBZ-7955

  • Duplicate downstream annotation comments incorrectly refer to Db2 connector DBZ-7968

  • Issue with Hybrid mode and DDL change DBZ-7991

  • Incorrect offset/history property mapping generatated DBZ-8007

  • Debezium Server Operator on minikube with java.lang.NullPointerException': java.lang.NullPointerException DBZ-8019

  • ORA-65090: operation only allowed in a container database when connecting to a non-CDB database DBZ-8023

  • Added type to Prometheus JMX exporter DBZ-8036

  • Add kafka.producer metrics to debezium-server jmx exporter config DBZ-8037

Other changes

  • Use Java 17 as baseline DBZ-7224

  • Document new MariaDB connector DBZ-7786

  • Move to Maven 3.9.8 as build requirement DBZ-7965

  • Add disclaimer that PostProcessors and CustomConverters are Debezium source connectors only DBZ-8031

  • Typos in Bug report template DBZ-8038

  • Find an alternative way to manually deploy the connector with local changes that is compatible with Debezium 3 DBZ-8046