Release Notes for Debezium 1.8

All notable changes for Debezium releases are documented in this file. Release numbers follow Semantic Versioning.

Release 1.8.0.Beta1 (November 30th, 2021)

Kafka compatibility

This release has been built against Kafka Connect 3.0.0 and has been tested with version 3.0.0 of the Kafka brokers. See the Kafka documentation for compatibility with other versions of Kafka brokers.

Upgrading

Before upgrading any connector, be sure to check the backward-incompatible changes that have been made since the release you were using.

When you decide to upgrade one of these connectors to 1.8.0.Beta1 from any earlier versions, first check the migration notes for the version you’re using. Gracefully stop the running connector, remove the old plugin files, install the 1.8.0.Beta1 plugin files, and restart the connector using the same configuration. Upon restart, the 1.8.0.Beta1 connectors will continue where the previous connector left off. As one might expect, all change events previously written to Kafka by the old connector will not be modified.

If you are using our container images, then please do not forget to pull them fresh from DockerHub.

Breaking changes

Vitess connector no longer depends on vtcld (DBZ-4324).

Oracle connector inserts a placeholder value for unavaliable CLOB/BLOB columns (DBZ-4276).

New Features

  • Support pg_logical_emit_message DBZ-2363

  • Outbox Event Router for MongoDB DBZ-3528

  • Improve interval type support in Oracle DBZ-1539

  • money data type should be controlled by decimal.handling.mode DBZ-1931

  • Support for Multiple Databases per SQL Server Connector DBZ-2975

  • Debezium server stops with wrong exit code (0) DBZ-3570

  • Change Debezium UI configurations property names DBZ-4066

  • Extend configuration support for Infinispan caches DBZ-4169

  • Support schema changes during incremental snapshot DBZ-4196

  • Handle login failure during instance upgrade as retriable DBZ-4285

  • Modify the type of aggregateid in MongoDB Outbox Event Router DBZ-4318

  • Explicit the MS SQL Materialized view limitation DBZ-4330

Fixes

  • PostgresConnector does not allow a numeric slot name DBZ-1042

  • False empty schema warning for snapshot mode never DBZ-1344

  • Tutorial shows incorrectly shows "op": "c" for initial change events DBZ-3786

  • SQL Server fails to read CDC events if there is a schema change ahead DBZ-3992

  • Once user click on "Review and finish" button that step in link in not enabled in wizard side menu. DBZ-4119

  • DDL statement couldn’t be parsed DBZ-4224

  • The lastOffset variable in MySqlStreamingChangeEventSource is always null DBZ-4225

  • Unknown entity: io.debezium.outbox.quarkus.internal.OutboxEvent DBZ-4232

  • Signal based incremental snapshot is failing when launched right after a schema change DBZ-4272

  • SQL Server connector doesn’t handle multiple capture instances for the same table with equal start LSN DBZ-4273

  • Debezium UI - some issues with browser support for replaceAll DBZ-4274

  • AbstractDatabaseHistory.java has typo DBZ-4275

  • OracleConnectorIT - two tests fail when using Xstream DBZ-4279

  • ParsingException: DDL statement couldn’t be parsed DBZ-4280

  • Topic Group UI step does not refresh correctly after setting properties DBZ-4293

  • Add MariaDB specific username for MySQL parser DBZ-4304

  • NullPointerException may be thrown when validating table and column lengths DBZ-4308

  • RelationalChangeRecordEmitter calls "LoggerFactory.getLogger(getClass())" for each instance of the emitter DBZ-4309

  • support for JSON function in MySQL index DBZ-4320

  • Avoid holding table metadata lock in read-only incremental snapshots DBZ-4331

  • Convert mysql time type default value error DBZ-4334

  • Wrong configuration option name for MongoDB Outbox SMT DBZ-4337

  • Incremental Snapshot does not pick up table DBZ-4343

  • Oracle connector - Cannot parse column default value 'NULL ' to type '2' DBZ-4360

Other changes

  • Add canonical URL links to older doc versions DBZ-3897

  • Set up testing job for MongoDB 5.0 DBZ-3938

  • Misc. documentation changes for the Debezium MySQL connector DBZ-3974

  • Promote Outbox SMT to GA DBZ-4012

  • Test failure: SchemaHistoryTopicIT::schemaChangeAfterSnapshot() DBZ-4082

  • Jenkins job for creating image snapshot used by new Jenkins nodes DBZ-4122

  • Use SMT/Transformation UI backend endpoint DBZ-4146

  • Create GH Action for tearing down abandoned website preview environments DBZ-4214

  • Unify Memory and Infinispan event processor implementations DBZ-4236

  • Update system-level testsuite CI job DBZ-4267

  • Upgrade MySQL JDBC driver to 8.0.27 DBZ-4286

  • Only build debezium-core and dependences in cross-repo builds DBZ-4289

  • Reduce log verbosity DBZ-4291

  • Vitess connector should expose vstream flags DBZ-4295

  • Vitess connector should allow client to config starting VGTID DBZ-4297

  • Layout glitch on docs landing page DBZ-4299

  • Provide outbox routing example for MongoDB DBZ-4302

  • Fix wrong option names in examples DBZ-4311

  • Update functional test CI to work with downstream source archive DBZ-4316

  • Provide example showing usage of remote Infinispan cache DBZ-4326

  • Provide CI for MongoDB 4.4 DBZ-4327

  • Test case for schema migration in Vitess connector DBZ-4353

  • Enable transaction metadata for vitess connector DBZ-4355

  • io.debezium.data.VerifyRecord.isValid(SourceRecord) is a no-op DBZ-4364

  • SignalsIT times out after 6h on CI DBZ-4370

  • Document incremental chunk size setting DBZ-4127

Release 1.8.0.Alpha2 (November 11th, 2021)

Kafka compatibility

This release has been built against Kafka Connect 2.8.0 and has been tested with version 2.8.0 of the Kafka brokers. See the Kafka documentation for compatibility with other versions of Kafka brokers.

Upgrading

Before upgrading any connector, be sure to check the backward-incompatible changes that have been made since the release you were using.

When you decide to upgrade one of these connectors to 1.8.0.Alpha2 from any earlier versions, first check the migration notes for the version you’re using. Gracefully stop the running connector, remove the old plugin files, install the 1.8.0.Alpha2 plugin files, and restart the connector using the same configuration. Upon restart, the 1.8.0.Alpha2 connectors will continue where the previous connector left off. As one might expect, all change events previously written to Kafka by the old connector will not be modified.

If you are using our container images, then please do not forget to pull them fresh from DockerHub.

Breaking changes

Previous versions stored buffered events in an Infinispan cache as a part of transaction data that could lead to memory issues upon load. This version separates transaction metadata and buffered events in different caches so the data is not loaded with transaction metadata processing (DBZ-4159).

New Features

  • TableChangesSerializer ignored defaultValue and enumValues DBZ-3966

  • Support for heartbeat action queries for MySQL DBZ-4029

  • Expose the transaction topicname as a config DBZ-4077

  • Improvement to the topic creation step DBZ-4172

  • Process transaction started/committed in MySQL read-only incremental snapshot DBZ-4197

  • Ability to use base image from authenticated registry with KC build mechanism DBZ-4227

  • Remove SqlServerConnector database.user Required Validator DBZ-4231

  • Specify database hot name as 0.0.0.0 for Oracle connector tests CI DBZ-4242

  • Suport all charsets in MySQL parser DBZ-4261

Fixes

  • "table" is null for table.include.list and column.include.list DBZ-3611

  • Debezium server crashes when deleting a record from a SQLServer table (redis sink) DBZ-3708

  • Invalid default value error on captured table DDL with default value DBZ-3710

  • Incremental snapshot doesn’t work without primary key DBZ-4107

  • Error: PostgresDefaultValueConverter - Cannot parse column default value 'NULL::numeric' to type 'numeric'. Expression evaluation is not supported. DBZ-4137

  • Container images for Apache Kafka and ZooKeeper fail to start up DBZ-4160

  • Debezium 1.7 image disables unsecure algorithms. Breaks unpatched databases DBZ-4167

  • DDL statement couldn’t be parsed - Modify Column DBZ-4174

  • DML statement couldn’t be parsed DBZ-4194

  • Debezium log miner processes get terminated with ORA-04030 error in idle database environment. DBZ-4204

  • DDL with Oracle SDO_GEOMETRY cannot be parsed DBZ-4206

  • DDL with Oracle sequence as default for primary key fails schema generation DBZ-4208

  • io.debezium.text.ParsingException: DDL statement couldn’t be parsed. Please open a Jira issue with the statement 'DROP TABLE IF EXISTS condition' DBZ-4210

  • Support MySQL Dual Passwords in DDL Parser DBZ-4215

  • Debezium Metrics not being set correctly DBZ-4222

  • CREATE PROCEDURE DDL throws ParsingException DBZ-4229

  • Exception ORA-00310 is not gracefully handled during streaming DBZ-4230

  • CHAR / NCHAR precision is not correctly derived from DDL statements DBZ-4233

  • Oracle connector parses NUMBER(*,0) as NUMBER(0,0) in DDL DBZ-4240

  • Signal based incremental snapshot is failing if database name contains dash DBZ-4244

  • SQL Server connector doesn’t handle retriable errors during database state transitions DBZ-4245

  • Does Debezium support database using charset GB18030? DBZ-4246

  • Broken anchors in Debezium Documentation DBZ-4254

  • Reduce verbosity of logging Oracle memory metrics DBZ-4255

  • When Debezium executes select * in the snapshot phase, it does not catch the sql exception, resulting in confusing exceptions and logs DBZ-4257

Other changes

  • Rename "master" branches to "main" for remaining repos DBZ-3626

  • Support Oracle Logminer docker image in system level test-suite DBZ-3929

  • Missing documentation for max.iteration.transactions option DBZ-4129

  • Use topic auto-creation UI backend endpoint DBZ-4148

  • Remove superfluous build triggers DBZ-4200

  • Tag debezium/tooling:1.2 version DBZ-4238

  • Rework MySqlTimestampColumnIT test DBZ-4241

  • Remove unused code DBZ-4252

  • Optimize tooling image DBZ-4258

  • Change DB2 image in testsuite to use private registry DBZ-4268

Release 1.8.0.Alpha1 (October 27th, 2021)

Kafka compatibility

This release has been built against Kafka Connect 2.8.0 and has been tested with version 2.8.0 of the Kafka brokers. See the Kafka documentation for compatibility with other versions of Kafka brokers.

Upgrading

Before upgrading any connector, be sure to check the backward-incompatible changes that have been made since the release you were using.

When you decide to upgrade one of these connectors to 1.8.0.Alpha1 from any earlier versions, first check the migration notes for the version you’re using. Gracefully stop the running connector, remove the old plugin files, install the 1.8.0.Alpha1 plugin files, and restart the connector using the same configuration. Upon restart, the 1.8.0.Alpha1 connectors will continue where the previous connector left off. As one might expect, all change events previously written to Kafka by the old connector will not be modified.

If you are using our docker images then do not forget to pull them fresh from Docker registry.

Breaking changes

There are no breaking changes in this release.

New Features

  • Provide MongoDB CDC implementation based on 4.0 change streams DBZ-435

  • No option fullDocument for the connection to MongoDB oplog.rs DBZ-1847

  • Make antora playbook_author.yml use current branch DBZ-2546

  • Support Kerberos for Debezium MS SQL plugin DBZ-3517

  • Make "snapshot.include.collection.list" case insensitive like "table.include.list" DBZ-3895

  • Exclude usernames at transaction level DBZ-3978

  • [oracle] Add the SCHEMA_ONLY_RECOVERY snapshot mode DBZ-3986

  • Support parse table and columns comment DBZ-4000

  • Upgrade postgres JDBC driver to version 42.2.24 DBZ-4046

  • Support JSON logging formatting DBZ-4114

  • Upgrade mysql-binlog-connector-java to v0.25.4 DBZ-4152

  • Wrong class name in SMT predicates documentation DBZ-4153

  • Log warning when table/column name exceeds maximum allowed by LogMiner DBZ-4161

  • Add Redis to debezium-server-architecture.png DBZ-4190

  • wrong variable naming in an unit test for Outbox Event Router SMT DBZ-4191

  • MongoDB connector support user defined topic delimiter DBZ-4192

  • Parse the "window" keyword for agg and nonagg function in mysql8 DBZ-4193

  • wrong field on change event message example in MongoDB Connector documentation DBZ-4201

  • Add a backend service for UI to fetch the SMT and topic auto-creation configuration properties DBZ-3874

Fixes

  • Debezium build is unstable for Oracle connector DBZ-3807

  • Row hashing in LogMiner Query not able to differentiate between rows of a statement. DBZ-3834

  • The chunk select statement is incorrect for combined primary key in incremental snapshot DBZ-3860

  • Crash processing MariaDB DATETIME fields returns empty blob instead of null (Snapshotting with useCursorFetch option) DBZ-4032

  • column.the mask.hash.hashAlgorithm.with…​. data corruption occurs when using this feature DBZ-4033

  • Compilation of MySQL grammar displays warnings DBZ-4034

  • Infinispan SPI throws NPE with more than one connector configured to the same Oracle database DBZ-4064

  • Extra double quotes on Kafka message produced by Quarkus Outbox Extension DBZ-4068

  • Debezium Server might contain driver versions pulled from Quarkus DBZ-4070

  • Connection failure while reading chunk during incremental snapshot DBZ-4078

  • Postgres 12/13 images are not buildable DBZ-4080

  • Postgres testsuite hangs on PostgresConnectorIT#exportedSnapshotShouldNotSkipRecordOfParallelTx DBZ-4081

  • CloudEventsConverter omits payload data of deleted documents DBZ-4083

  • Database history is constantly being reconfigured DBZ-4106

  • projectId not being set when injecting a custom PublisherBuilder DBZ-4111

  • Oracle flush table should not contain multiple rows DBZ-4118

  • Can’t parse DDL for View DBZ-4121

  • SQL Server Connector fails to wrap in flat brackets DBZ-4125

  • Oracle Connector DDL Parsing Exception DBZ-4126

  • Debezium deals with Oracle DDL appeared IndexOutOfBoundsException: Index: 0, Size: 0 DBZ-4135

  • Oracle connector throws NPE during streaming in archive only mode DBZ-4140

  • debezium-api and debezium-core jars missing in NIGHTLY Kafka Connect container image libs dir DBZ-4147

  • Trim numerical defaultValue before converting DBZ-4150

  • Possible OutOfMemoryError with tracking schema changes DBZ-4151

  • DDL ParsingException - not all table compression modes are supported DBZ-4158

  • Producer failure NullPointerException DBZ-4166

  • DDL Statement couldn’t be parsed DBZ-4170

  • In multiple connect clusters monitoring, no matter which cluster is selected from the dropdown list, the detailed information is always for the first cluster. DBZ-4181

  • Remove MINUSMINUS operator DBZ-4184

  • OracleSchemaMigrationIT#shouldNotEmitDdlEventsForNonTableObjects fails for Xstream DBZ-4186

  • Certain LogMiner-specific tests are not being skipped while using Xstreams DBZ-4188

  • Missing debezium/postgres:14-alpine in Docker Hub DBZ-4195

  • nulls for some MySQL properties in the connector-types backend response DBZ-3108

Other changes

  • Test with new deployment mechanism in AMQ Streams DBZ-1777

  • Incorrect documentation for message.key.columns DBZ-3437

  • Re-enable building PostgreSQL alpine images DBZ-3691

  • Upgrade to Quarkus 2.2.3.Final DBZ-3785

  • Document awareness of Oracle database tuning DBZ-3880

  • Publish website-builder and tooling images once per week DBZ-3907

  • Intermittent test failure on CI - RecordsStreamProducerIT#shouldReceiveHeartbeatAlsoWhenChangingNonWhitelistedTable() DBZ-3919

  • Please fix vulnerabilites DBZ-3926

  • Error processing binlog event DBZ-3989

  • Upgrade Java version for GH actions DBZ-3993

  • Replace hard-coded version of MySQL example image with getStableVersion() DBZ-4005

  • Handle SCN gap DBZ-4036

  • Upgrade to Apache Kafka 3.0 DBZ-4045

  • Recreate webhook for linking PRs to JIRA issues DBZ-4065

  • Recipient email address should be a variable in all Jenkins jobs DBZ-4071

  • Allow [ci] tag as commit message prefix DBZ-4073

  • Debezium Docker build job fails on rate limiter DBZ-4074

  • Add Postgresql 14 container image (Alpine) DBZ-4075

  • Add Postgresql 14 container image DBZ-4079

  • Fail Docker build scripts on error DBZ-4084

  • Display commit SHA in page footer DBZ-4110

  • Handle large comparisons results from GH API to address missing authors in release workflow DBZ-4112

  • Add debezium-connect-rest-extension module to GH workflows DBZ-4113

  • Display commit SHA in documentation footer DBZ-4123

  • Add Debezium Kafka Connect REST Extension to Debezium Kafka Connect NIGHTLY container image DBZ-4128

  • Migrate from Gitter to Zulip DBZ-4142

  • Postgres module build times out after 6h on CI DBZ-4145

  • Misc. MongoDB connector docs fixes DBZ-4149

  • Document Oracle buffering solutions DBZ-4157

  • Close open file handle DBZ-4164

  • Outreach jobs should test all connectors DBZ-4165

  • Broken link in MySQL docs DBZ-4199

  • Expose outbox event structure at level of Kafka Connect messages DBZ-1297