Release Notes for Debezium 2.0

All notable changes for Debezium releases are documented in this file. Release numbers follow Semantic Versioning.

Release 2.0.0.Alpha3 (July 1st 2022)

Kafka compatibility

This release has been built against Kafka Connect 3.2.0 and has been tested with version 3.2.0 of the Kafka brokers. See the Kafka documentation for compatibility with other versions of Kafka brokers.

Upgrading

Before upgrading any connector, be sure to check the backward-incompatible changes that have been made since the release you were using.

When you decide to upgrade one of these connectors to 2.0.0.Alpha3 from any earlier versions, first check the migration notes for the version you’re using. Gracefully stop the running connector, remove the old plugin files, install the 2.0.0.Alpha3 plugin files, and restart the connector using the same configuration. Upon restart, the 2.0.0.Alpha3 connectors will continue where the previous connector left off. As one might expect, all change events previously written to Kafka by the old connector will not be modified.

If you are using our container images, then please do not forget to pull them fresh from DockerHub.

Breaking changes

There are no breaking changes in this release.

New features

  • Mysql Commit Timestamp DBZ-5170

  • Include event scn in Oracle records DBZ-5225

  • Redis Store does not work with GCP Managed Redis DBZ-5268

Fixes

  • Incorrect loading of LSN from offsets DBZ-3942

  • Database history recovery will retain old tables after they’ve been renamed DBZ-4451

  • Adding new table with incremental snapshots not working DBZ-4834

  • BigDecimal has mismatching scale value for given Decimal schema DBZ-4890

  • Debezium has never found starting LSN DBZ-5031

  • Data duplication problem using postgresql source on debezium server DBZ-5070

  • Cursor fetch is used for all results during connection DBZ-5084

  • Debezuim connector fails at parsing select statement overrides when table name has space DBZ-5198

  • DDL statement couldn’t be parsed 2 - Oracle connector 1.9.3.Final DBZ-5230

  • Debezium server duplicates scripting jar files DBZ-5232

  • Cannot convert field type tinyint(1) unsigned to boolean DBZ-5236

  • Oracle unparsable ddl create table DBZ-5237

  • Postgres Incremental Snapshot on parent partitioned table not working DBZ-5240

  • Character set influencers are not properly parsed on default values DBZ-5241

  • Dupicate SCNs on Oracle RAC installations incorrectly processed DBZ-5245

  • NPE when using Debezium Embedded in Quarkus DBZ-5251

  • Oracle LogMiner may fail with an in-progress transaction in an archive log that has been deleted DBZ-5256

  • Order of source block table names in a rename schema change event is not deterministic DBZ-5257

  • Debezium fails to connect to replicaset if a node is down DBZ-5260

  • No changes to commit_scn when oracle-connector got new lob data DBZ-5266

  • Invalid date 'SEPTEMBER 31' DBZ-5267

  • database.history.store.only.captured.tables.ddl not suppressing logs DBZ-5270

  • io.debezium.text.ParsingException: DDL statement couldn’t be parsed DBZ-5271

  • Deadlock during snapshot with Mongo connector DBZ-5272

  • Mysql parser is not able to handle variables in KILL command DBZ-5273

  • Debezium server fail when connect to Azure Event Hubs DBZ-5279

  • ORA-01086 savepoint never established raised when database history topic cannot be created or does not exist DBZ-5281

  • Enabling database.history.store.only.captured.tables.ddl does not restrict history topic records DBZ-5285

Other changes

  • Add script SMT test case to OCP test suite DBZ-2581

  • Confusing example for schema change topic DBZ-4713

  • Update cache-invalidation example DBZ-4754

  • Switch from static yaml descriptors to dynamic objects DBZ-4830

  • Verify that snapshot deployments build and deploy javadocs DBZ-4875

  • DelayStrategy should accept Duration rather than long ms DBZ-4902

  • Use maven 3.8.4 version with enforcer plugin DBZ-5069

  • Add option for '*' wildcard usage testsuite preparation jenkins jobs DBZ-5190

  • Use the Maven wrapper in the Github and Jenkins workflows DBZ-5207

  • Improve performance of OracleConnectorIT shouldIgnoreAllTablesInExcludedSchemas test DBZ-5226

  • Document use of JAR artifact to build Debezium scripting SMT into Kafka Connect DBZ-5227

  • Create shared adoc fragments for specifying MBean name format in connector metrics sections DBZ-5233

  • Build Oracle connector by default without Maven profiles DBZ-5234

  • Remove reference to removed case insensitive option in Oracle README.md DBZ-5250

  • Several Oracle tests do not get database name from TestHelper DBZ-5258

  • Upgrade to Quarkus 2.10.0.Final DBZ-5259

  • Upgrade PostgreSQL driver to 42.4.0 DBZ-5261

  • Refactor ChangeEventQueue to better support n:1 threads DBZ-5277

  • Upgrade MongoDB driver to 4.6.1 DBZ-5287

Release 2.0.0.Alpha2 (June 9th 2022)

Kafka compatibility

This release has been built against Kafka Connect 3.2.0 and has been tested with version 3.2.0 of the Kafka brokers. See the Kafka documentation for compatibility with other versions of Kafka brokers.

Upgrading

Before upgrading any connector, be sure to check the backward-incompatible changes that have been made since the release you were using.

When you decide to upgrade one of these connectors to 2.0.0.Alpha2 from any earlier versions, first check the migration notes for the version you’re using. Gracefully stop the running connector, remove the old plugin files, install the 2.0.0.Alpha2 plugin files, and restart the connector using the same configuration. Upon restart, the 2.0.0.Alpha2 connectors will continue where the previous connector left off. As one might expect, all change events previously written to Kafka by the old connector will not be modified.

If you are using our container images, then please do not forget to pull them fresh from DockerHub.

Breaking changes

Debezium MySQL connector represented FLOAT datatype as FLOAT64. Now the correct type FLOAT32 is used (DBZ-3865).

MongoDB connector no longer supports streaming from oplog. The connector fully switched to change streams streaming (DBZ-4951).

All connectors now use multi-partitioned codebase. This has impact only on project developers (DBZ-5042).

All deprecated configuration options were removed (DBZ-5045).

New features

  • Provide a signal to stop the running incremental snapshot DBZ-4251

  • SQL Server - Fail connector when a user doesn’t have the right permission (CDCReader) DBZ-4346

  • Allow mongodb-connector to decode Binary payloads DBZ-4600

  • Add UI backend tests for SQL Server connector DBZ-4867

  • direct usage of debezium engine ignores ChangeConsumer.supportsTombstoneEvents DBZ-5052

  • Config the cache size property for ByLogicalTableRouter caches DBZ-5072

  • Introduce a new extension api for query debezium version DBZ-5092

  • Introduce a new field "ts_ms" to identify the process time for schema change event DBZ-5098

  • MongoDB Connector should use RawBsonDocument instead of Document DBZ-5113

Fixes

  • Postgres existing publication is not updated with the new table DBZ-3921

  • Error and connector stops when DDL contains lateral DBZ-4780

  • Schema changes should flush SCN to offsets if there are no other active transactions DBZ-4782

  • Connector stops streaming after a re-balance DBZ-4792

  • MySQL connector increment snapshot failed parse datetime column lenth when connector set "snapshot.fetch.size": 20000 DBZ-4939

  • [MySQL Debezium] DDL Parsing error - CREATE OR REPLACE TABLE DBZ-4958

  • InstanceAlreadyExistsException during MongoDb connector metrics registration DBZ-5011

  • DateTimeParseException: Text 'infinity' could not be parsed in Postgres connector DBZ-5014

  • PostgreSQL ENUM default values are missing from generated schema DBZ-5038

  • Debezium official documentation typo DBZ-5040

  • Fix inconsistent transaction id when handling transactional messages in Vitess connector DBZ-5063

  • 4 Connections per connector (postgres) DBZ-5074

  • Oracle documentation refers to archive_log_target rather than archive_lag_target DBZ-5076

  • 'ALTER TABLE mytable DROP FOREIGN KEY IF EXISTS mytable_fk' no viable alternative at input 'ALTER TABLE mytable DROP FOREIGN KEY IF' DBZ-5077

  • Oracle Logminer: records missed during switch from snapshot to streaming mode DBZ-5085

  • Interrupting a snapshot process can hang for some JDBC drivers DBZ-5087

  • Debezium fails to undo change event due to transaction id ending in ffffffff with LogMiner DBZ-5090

  • Table changes are not filled in schema changes from snapshot DBZ-5096

  • Postgresql connector does not retry one some errors when postgres is taken offline DBZ-5097

  • Parsing zero day fails DBZ-5099

  • Cannot Set debezium.sink.kafka.producer.ssl.endpoint.identification.algorithm to empty value DBZ-5105

  • Debezium connector failed with create table statement DBZ-5108

  • Current version of surefire/failsafe skips tests on failure in BeforeAll DBZ-5112

Other changes

  • Restructure documentation for custom converters DBZ-4588

  • Document xmin.fetch.interval.ms property for Postgres connector DBZ-4734

  • Update to Quarkus 2.9.2.Final DBZ-4806

  • Upgrade Oracle driver to 21.5.0.0 DBZ-4877

  • Execute Debezium UI build when core library is changed DBZ-4947

  • Remove unused Oracle connector code DBZ-4973

  • Links to cassandra 3 and 4 artifacts no longer work for Debezium 1.9+ DBZ-5055

  • Align Postgresql driver with Quarkus DBZ-5060

  • Outdated links in Javadoc documentation DBZ-5075

  • Rename "Mysql" to "MySql" in related MysqlFieldReader interface DBZ-5078

  • Create CI job for maven repository verification DBZ-5082

  • Remove database.server.id default value handler, no longer auto-generated. DBZ-5100

  • Upgrade Jackson Databind to 2.13.2.2 DBZ-5107

  • Switch to released version of Fixture5 extension in System testsuite DBZ-5114

Release 2.0.0.Alpha1 (April 28th 2022)

Kafka compatibility

This release has been built against Kafka Connect 3.1.0 and has been tested with version 3.1.0 of the Kafka brokers. See the Kafka documentation for compatibility with other versions of Kafka brokers.

Upgrading

Before upgrading any connector, be sure to check the backward-incompatible changes that have been made since the release you were using.

When you decide to upgrade one of these connectors to 2.0.0.Alpha1 from any earlier versions, first check the migration notes for the version you’re using. Gracefully stop the running connector, remove the old plugin files, install the 2.0.0.Alpha1 plugin files, and restart the connector using the same configuration. Upon restart, the 2.0.0.Alpha1 connectors will continue where the previous connector left off. As one might expect, all change events previously written to Kafka by the old connector will not be modified.

If you are using our container images, then please do not forget to pull them fresh from DockerHub.

Breaking changes

Java 11 is required to run Debezium in any form: Kafka Connect plug-in, Debezium engine and Debezium Server (DBZ-4949).

Maven 3.8.4 is required to build Debezium (DBZ-5064).

PostgreSQL connector no longer supports old versions of protobuf decoding plug-in that do not have transaction and default value support (DBZ-703).

PostgreSQL connector no longer supports wal2json decoding plug-in (dehttps://issues.redhat.com/browse/DBZ-4156[DBZ-4156]). pgoutput decoding plug-in is recommended as the replacement.

Legacy implementation of MySQL connector was removed (DBZ-4950).

Confluent Avro converters are no longer present in ebezium container images (DBZ-4952).

JDBC legacy date/time properties support was removed from MySQL connector (DBZ-4965).

New features

  • Implement Pub/Sub Lite change consumer DBZ-4450

  • Include Instant Client in Docker build for Oracle in Tutorial examples DBZ-1013

  • Add Google Pub/Sub emulator support DBZ-4491

  • Making Postgres PSQLException: This connection has been closed. retriable DBZ-4948

  • ORA-04030: out of process memory when trying to allocate 65568 bytes (Logminer LCR c,krvxrib:buffer) DBZ-4963

  • Should store event header timestamp in HistoryRecord DBZ-4998

  • DBZ-UI: In the Edit/Duplicate connector flow make the access/secret key/password/Client Secret filed as editable. DBZ-5001

  • adjust LogMiner batch size based on comparison with currently used batch size DBZ-5005

Fixes

  • Connector throws java.lang.ArrayIndexOutOfBoundsException DBZ-3848

  • Document no relevant tables should be in the SYS or SYSTEM tablespaces. DBZ-4762

  • Getting java.sql.SQLException: ORA-01291: missing logfile while running with archive log only DBZ-4879

  • Debezium uses wrong LCR format for Oracle 12.1 DBZ-4932

  • Oracle duplicates on connector restart DBZ-4936

  • Oracle truncate causes exception DBZ-4953

  • NPE caused by io.debezium.connector.oracle.antlr.listener.ColumnDefinitionParserListener.resolveColumnDataType DBZ-4976

  • Oracle connector may throw NullPointerException when stopped after an unsuccessful startup DBZ-4978

  • NPE for non-table related DDLs DBZ-4979

  • CTE statements aren’t parsed by MySQL connector DBZ-4980

  • Missing SSL configuration option in the debezium mongodb connector UI DBZ-4981

  • Unsupported MySQL Charsets during Snapshotting for fields with custom converter DBZ-4983

  • Outbox Transform does not allow expanded payload with additional fields in the envelope DBZ-4989

  • Redis Sink - clientSetname is taking place before auth DBZ-4993

  • CLOB with single quotes causes parser exception DBZ-4994

  • Oracle DDL parser fails on references_clause with no column list DBZ-4996

  • Can’t use 'local' database through mongos DBZ-5003

  • Triggering Incremental Snapshot on MongoDB connector throws json parsing error DBZ-5015

  • Jenkins jobs fail to download debezium-bom DBZ-5017

  • Redis Sink - Check if client is not null before closing it DBZ-5019

  • Cassandra 3 handler does not process partition deletions correctly DBZ-5022

  • Keyspaces should be initialised in all schema change listeners on sessions startup. DBZ-5023

  • SQL Server in multi-partition mode fails if a new database is added to an existing configuration DBZ-5033

  • Mysql tests start before MySQL DB constainer is running DBZ-5054

  • Debezium server configuration properties not rendered correctly DBZ-5058

Other changes

  • Add integration test for Oracle database.url configurations DBZ-3318

  • Build Cassandra 3.x connector with Java 11 DBZ-4910

  • Add ignoreSnapshots build option to release pipeline DBZ-4957

  • Update Pulsar client version used by Debezium Server DBZ-4961

  • Intermittent failure of RedisStreamIT.testRedisConnectionRetry DBZ-4966

  • Add triggers for 2.x paths in Github CI DBZ-4971

  • Debezium raised an exception and the task was still running DBZ-4987

  • Nexus Staging Maven plugin is incompatible with OpenJDK 17 DBZ-5025

  • Duplicate definition of Maven plugins DBZ-5026

  • OracleOffsetContextTest should be scoped to LogMiner only DBZ-5028

  • Scope several new Oracle tests to LogMiner only DBZ-5029

  • Failure in jdk outreach jobs DBZ-5041

  • Update artifact server job listing script DBZ-5051

  • Add FAQ about ORA-01882 and Oracle 11 to documentation DBZ-5057

  • Upgrade to Quarkus 2.8.2.Final DBZ-5062