Release Notes for Debezium 2.0

All notable changes for Debezium releases are documented in this file. Release numbers follow Semantic Versioning.

Release 2.0.1.Final (December 7th 2022)

Kafka compatibility

This release has been built against Kafka Connect 3.3.1 and has been tested with version 3.3.1 of the Kafka brokers. See the Kafka documentation for compatibility with other versions of Kafka brokers.

Upgrading

Before upgrading any connector, be sure to check the backward-incompatible changes that have been made since the release you were using.

When you decide to upgrade one of these connectors to 2.0.1.Final from any earlier versions, first check the migration notes for the version you’re using. Gracefully stop the running connector, remove the old plugin files, install the 2.0.1.Final plugin files, and restart the connector using the same configuration. Upon restart, the 2.0.1.Final connectors will continue where the previous connector left off. As one might expect, all change events previously written to Kafka by the old connector will not be modified.

If you are using our container images, then please do not forget to pull them fresh from DockerHub.

Breaking changes

There are no breaking changes in this release.

New features

  • Improve LogMiner query performance by reducing REGEXP_LIKE disjunctions DBZ-5648

  • Replace obsolete DebeziumDownload attribute DBZ-5835

  • Reduce container image sizes by consolidating operations per layer DBZ-5864

Fixes

  • Using snapshot boundary mode "all" causes DebeziumException on Oracle RAC DBZ-5302

  • ORA-01003: no statement parsed DBZ-5352

  • Missing snapshot pending transactions DBZ-5482

  • Db2 documentation refers to invalid SMALLMONEY and MONEY data types DBZ-5504

  • UNIQUE INDEX with NULL value throws exception when lob.enabled is true DBZ-5682

  • Oracle SQL parsing error when collation used DBZ-5726

  • Unparseable DDL statement DBZ-5734

  • NullPointerException thrown during snapshot of tables in Oracle source connector DBZ-5738

  • Remove note from snapshot metrics docs file that flags incremental snapshots as TP feature DBZ-5748

  • Exclude Oracle Compression Advisor tables from capture to avoid infinite loop DBZ-5756

  • More Oracle logging DBZ-5759

  • Oracle should only log row contents at TRACE level DBZ-5760

  • Outbox Router documentation outdated regarding value converter DBZ-5770

  • Using DBMS_LOB.ERASE by itself can lead to an unexpected UPDATE with null BLOB value DBZ-5773

  • PostgreSQL missing metadata info DBZ-5789

  • Oracle connector does not attempt restart when ORA-01089 exception is nested DBZ-5791

  • Message with LSN 'LSN{XYZ}' not present among LSNs seen in the location phase DBZ-5792

  • Mysql connector alter table with database name parse failed DBZ-5802

  • Conflicting documentation for snapshot.mode property in MongoDB connector v2.0 DBZ-5812

  • 'topic.prefix' default value in MongoDB connector v2.0 DBZ-5817

  • Quarkus outbox extention never finishes the open tracing span DBZ-5821

  • fix names of range fields in schema to comply with Avro standard DBZ-5826

  • CREATE/ALTER user does not support COMMENT token DBZ-5836

  • Invalid Java object for schema with type FLOAT64: class java.lang.Float DBZ-5843

  • IllegalStateException is thrown if task is recovering while other tasks are running DBZ-5855

  • CREATE/ALTER user does not support ATTRIBUTE token DBZ-5876

Other changes

  • SQL Server connector docs should mention multi-task support DBZ-5714

  • Debezium connectors ship with an old version of google-protobuf vulnerable to CVE-2022-3171 DBZ-5747

  • Upgrade to Quarkus 2.14.CR1 DBZ-5774

  • Upgrade postgres driver to version 42.5.0 DBZ-5780

  • Upgrade to Quarkus 2.14.0.Final DBZ-5786

  • Doc Typo in cloudevents DBZ-5788

  • Add ORA-01555 to Oracle documentation DBZ-5816

  • GitHub Actions: Deprecating save-state and set-output commands DBZ-5824

  • Upgrade wildfly-elytron to 1.15.5 / 1.16.1 due to CVE-2021-3642 DBZ-5854

Release 2.0.0.Final (October 14th 2022)

Kafka compatibility

This release has been built against Kafka Connect 3.3.1 and has been tested with version 3.3.1 of the Kafka brokers. See the Kafka documentation for compatibility with other versions of Kafka brokers.

Upgrading

Before upgrading any connector, be sure to check the backward-incompatible changes that have been made since the release you were using.

When you decide to upgrade one of these connectors to 2.0.0.Final from any earlier versions, first check the migration notes for the version you’re using. Gracefully stop the running connector, remove the old plugin files, install the 2.0.0.Final plugin files, and restart the connector using the new configuration parameter names. Upon restart, the 2.0.0.Final connectors will continue where the previous connector left off. As one might expect, all change events previously written to Kafka by the old connector will not be modified.

If you are using our container images, then please do not forget to pull them fresh from DockerHub.

Breaking changes

PostgreSQL transaction id is 32-bit integer and gets rolled over. To simplify deduplication of transactions the LSN was added as a part of the identifier (DBZ-5329).

New features

There are no new features in this release.

Fixes

  • ORA-01289: cannot add duplicate logfile DBZ-5276

  • Function DATE_ADD can be used as an identifier DBZ-5679

  • MySqlConnector parse create view statement failed DBZ-5708

  • The DDL_FILTER of SchemaHistory doesn’t work for including break lines ddl statement DBZ-5709

  • Debezium Server 1.9.6 is using MSSQL JDBC 7.2.2 instead of 9.4.1 DBZ-5711

  • Invalid prop names in MongoDB outbox router docs DBZ-5715

  • tests are running forever DBZ-5718

  • cassandra connector first startup ever may fail DBZ-5719

  • Vitess: Handle Vstream error: unexpected server EOF DBZ-5722

  • ParsingException: DDL statement couldn’t be parsed (index hints) DBZ-5724

Other changes

  • Remove whilelisted/blacklisted from log messages DBZ-5710

  • MySqlSchemaMigrationIT runs failed DBZ-5728

Release 2.0.0.CR1 (October 7th 2022)

Kafka compatibility

This release has been built against Kafka Connect 3.3.1 and has been tested with version 3.3.1 of the Kafka brokers. See the Kafka documentation for compatibility with other versions of Kafka brokers.

Upgrading

Before upgrading any connector, be sure to check the backward-incompatible changes that have been made since the release you were using.

When you decide to upgrade one of these connectors to 2.0.0.CR1 from any earlier versions, first check the migration notes for the version you’re using. Gracefully stop the running connector, remove the old plugin files, install the 2.0.0.CR1 plugin files, and restart the connector using the new configuration parameter names. Upon restart, the 2.0.0.CR1 connectors will continue where the previous connector left off. As one might expect, all change events previously written to Kafka by the old connector will not be modified.

If you are using our container images, then please do not forget to pull them fresh from DockerHub.

Breaking changes

Default behaviour for schema.name.adjustment.mode is now none. While the avro was safer option in case the user was using Avro converter it was confusing in the more frequent cases when default JSON converter was used. (DBZ-5541).

New features

  • Implement retries for Debezium embedded engine DBZ-4629

  • MySqlErrorHandler should handle SocketException DBZ-5486

  • Traditional snapshot process setting source.ts_ms DBZ-5591

  • Clean up "logical name" config DBZ-5594

  • Upgrade Kafka client to 3.3.1 DBZ-5600

  • When writing docs, use website stylesheet for IDE preview in IntelliJ DBZ-5616

  • Support READ ONLY/ENCRYPTION options for alter database statment DBZ-5622

  • Clarify semantics of include/exclude options DBZ-5625

  • Added support for Mongo pre-image in change stream DBZ-5628

  • Support for seting stats_sample_pages=default in alter table statements DBZ-5631

  • support for using any expression in kill statements DBZ-5636

  • Logging enhancement for non-incremental snapshot in postgres connector DBZ-5639

  • Support set statement in mariadb DBZ-5650

  • Add Mongo-initiator 6.0 container image DBZ-5666

  • Remove logic name parameter from sub connector config DBZ-5671

Fixes

  • ConvertingEngineBuilder looses the accents DBZ-4213

  • Debezium Db2 Connector fails to handle default values in schema when is making the snapshot DBZ-4990

  • Debezium 2.0.0.Beta1 Azure SQL breaking change DBZ-5496

  • Oracle connector parsing SELECT_LOB_LOCATOR event missing constant unavailable.value.placeholder DBZ-5581

  • Starting Embedded Engine swallows ClassNotFoundException so user cannot see why engine does not work DBZ-5583

  • Message with LSN foo larger than expected LSN bar DBZ-5597

  • Fix broken anchors in docs DBZ-5618

  • DDL Parsing Error DBZ-5623

  • MySQL connector cannot parse default value of decimal colum enclosed in double quotes DBZ-5630

  • Support grant LOAD FROM S3, SELECT INTO S3, INVOKE LAMBDA with aws mysql DBZ-5633

  • Continuously WARNs about undo transactions when LOB is enabled DBZ-5635

  • Literal "${project.version}" in the source record instead of the actual version DBZ-5640

  • TABLE_TYPE keyword can be used as identifier DBZ-5643

  • Large numbers of ROLLBACK transactions can lead to memory leak when LOB is not enabled. DBZ-5645

  • Race in DebeziumContainer during startup DBZ-5651

  • Outbox pattern nested payload leads to connector crash DBZ-5654

  • Allow the word STATEMENT to be a table / column name DBZ-5662

  • ValidatePostgresConnectionIT.testInvalidPostgresConnection fails DBZ-5664

  • Hardcoded driver task properties are not being passed to underlying connections DBZ-5670

  • Keyword virtual can be used as an identifier DBZ-5674

  • MongoDB Connector with DocumentDB errors with "{$natural: -1} is not supported" DBZ-5677

Other changes

  • Align connector properties to have an empty default cell if property has no default DBZ-3327

  • Improve Filter SMT documentation / examples DBZ-4417

  • Test failure on CI: SqlServerConnectorIT#updatePrimaryKeyTwiceWithRestartInMiddleOfTx DBZ-4475

  • Intermittent test failure: SqlServerConnectorIT#updatePrimaryKeyWithRestartInMiddle() DBZ-4490

  • Edit content newly added to the MongoDB connector doc DBZ-5542

  • Upgrade apicurio to 2.2.5.Final DBZ-5549

  • Modify the Instantiator to not require classloader DBZ-5585

  • Use quay.io in test containers DBZ-5603

  • Remove records from being logged at all levels DBZ-5612

  • Upgrade binary log client to 0.27.2 DBZ-5620

  • Allow to change docker maven properties from command line DBZ-5657

  • Update docker maven plugin DBZ-5658

  • Run UI tests on all connector changes DBZ-5660

  • Cleanup UI e2e tests after removing default value for topic.prefix DBZ-5667

Release 2.0.0.Beta2 (September 16th 2022)

Kafka compatibility

This release has been built against Kafka Connect 3.2.1 and has been tested with version 3.2.1 of the Kafka brokers. See the Kafka documentation for compatibility with other versions of Kafka brokers.

Upgrading

Before upgrading any connector, be sure to check the backward-incompatible changes that have been made since the release you were using.

When you decide to upgrade one of these connectors to 2.0.0.Beta2 from any earlier versions, first check the migration notes for the version you’re using. Gracefully stop the running connector, remove the old plugin files, install the 2.0.0.Beta2 plugin files, and restart the connector using the new configuration parameter names. Upon restart, the 2.0.0.Beta2 connectors will continue where the previous connector left off. As one might expect, all change events previously written to Kafka by the old connector will not be modified.

If you are using our container images, then please do not forget to pull them fresh from DockerHub.

Breaking changes

Object sizes for memory queue limits are no longer calculated using reflection but estimated based on message schema. This is not supported for Cassandra connector (DBZ-2766).

All schemas used by Debezium are now defined in a central point, properly named and versioned (DBZ-4365, DBZ-5044). This can lead to schema compatibility issues if a schema registry is used.

The connector parameter naming was overhauled and separated into distinct trees. The connector configuration must be updated (DBZ-5043).

Most Debezium connectors now restart by default when an exception related to communication (SqlException, IOException) is thrown (DBZ-5244).

The skipped.operations configuration option now defaults to truncate t (DBZ-5497).

Java 17 is no longer supported for writing tests. As some of the developer tools have issues when different Java level is used for main and test code, Debezium now uses Java 11 for the whole codebase (DBZ-5568).

New features

  • Support binlog compression for MySQL DBZ-2663

  • Limit log output for "Streaming requested from LSN" warnings DBZ-3007

  • Redis Sink - Change the format of the message sent to the stream DBZ-4441

  • Debezium UI frontend should use new URLs and new JSON schema descriptors DBZ-4619

  • Provide a signal to pause/resume a running incremental snapshot DBZ-4727

  • support mongodb connection string as configuration option DBZ-4733

  • Update Readme on github for Cassandra 4.x support DBZ-4839

  • Debezium Server verifies existence and format of the config file DBZ-5116

  • Include Oracle Debezium Connector in Debezium Server distribution DBZ-5122

  • Smart Backfills | Ability to backfill selective data DBZ-5327

  • Support multiple tasks in vitess connector DBZ-5382

  • Enhancing Cassandra 4 Connector to read incremental changes and not wait for Commit Log file to be marked complete DBZ-5410

  • Unsupported non-relational tables should be gracefully skipped by the connector during streaming DBZ-5441

  • Support incremental snapshot stop-snapshot signal sourced from Kafka topic DBZ-5453

  • Upgrade Kafka client to 3.2.1 DBZ-5463

  • Restart SQL Server task on "Socket closed" exception DBZ-5478

  • Augment a uniqueness key filed/value in regex topic naming strategy DBZ-5480

  • Support wait/nowait clause in mariadb DBZ-5485

  • Adapt create function syntax of mariadb DBZ-5487

  • add schema doc from column comments DBZ-5489

  • My connector parse the mariadb relevant sequence statement failed DBZ-5505

  • Expose default values and enum values in schema history messages DBZ-5511

  • Simplify passing of SINK config properties to OffsetBackingStore DBZ-5513

  • Support BASE64_URL_SAFE in BinaryHandlingMode DBZ-5544

  • Handle Vstream Connection reset DBZ-5551

  • Supply partition when comiting offsets with source database DBZ-5557

  • Vitess: Filter table.include.list during VStream subscription DBZ-5572

  • Improve documentation editing experience by setting attributes for the preview DBZ-5576

Fixes

  • Source info of incremental snapshot events exports wrong data DBZ-4329

  • "No maximum LSN recorded" log message can be spammed on low-activity databases DBZ-4631

  • Redis Sink config properties are not passed to DB history DBZ-5035

  • HTTP sink not retrying failing requests DBZ-5307

  • Translation from mongodb document to kafka connect schema fails when nested arrays contain no elements DBZ-5434

  • Duplicate SCNs on same thread Oracle RAC mode incorrectly processed DBZ-5439

  • Typo in postgresql document. DBZ-5450

  • Unit test fails on Windows DBZ-5452

  • Missing the regex properties validation before start connector of DefaultRegexTopicNamingStrategy DBZ-5471

  • Create Index DDL fails to parse when using TABLESPACE clause with quoted identifier DBZ-5472

  • Outbox doesn’t check array consistecy properly when it detemines its schema DBZ-5475

  • Misleading statistics written to the log DBZ-5476

  • Debezium connector task didn’t retry when failover in mongodb 5 DBZ-5479

  • ReadOnlyIncrementalSnapshotIT testStopSnapshotKafkaSignal randomly fails DBZ-5483

  • Better error reporting for signal table failures DBZ-5484

  • Oracle DATADUMP DDL cannot be parsed DBZ-5488

  • Mysql connector parser the ddl statement failed when including keyword "buckets" DBZ-5499

  • duplicate call to config.validateAndRecord() in RedisDatabaseHistory DBZ-5506

  • DDL statement couldn’t be parsed : mismatched input 'ENGINE' DBZ-5508

  • Use “database.dbnames” in SQL Server docs DBZ-5516

  • LogMiner DML parser incorrectly interprets concatenation operator inside quoted column value DBZ-5521

  • Mysql Connector DDL Parser does not parse all privileges DBZ-5522

  • SQL Server random test failures - EventProcessingFailureHandlingIT DBZ-5525

  • CREATE TABLE with JSON-based CHECK constraint clause causes MultipleParsingExceptions DBZ-5526

  • SQL Server test failure - verifyOffsets DBZ-5527

  • Unit test fails on Windows DBZ-5533

  • EmbeddedEngine should initialize Connector using SourceConnectorContext DBZ-5534

  • Unclear validation error when required field is missing DBZ-5538

  • Testsuite is missing server.id in MySQL connector’s configuration DBZ-5539

  • Support EMPTY column identifier DBZ-5550

  • Testsuite doesn’t reflect changes to SQLServer connector DBZ-5554

  • Use TCCL as the default classloader to load interface implementations DBZ-5561

  • max.queue.size.in.bytes is invalid DBZ-5569

  • Language type for listings in automatic topic creation DBZ-5573

  • Vitess: Handle VStream close unepectedly DBZ-5579

  • Unreliable RedisDatabaseHistoryIT DBZ-5582

  • Error when parsing alter sql DBZ-5587

  • Field validation errors are misleading for positive, non-zero expectations DBZ-5588

  • Mysql connector can’t handle the case sensitive of rename/change column statement DBZ-5589

  • LIST_VALUE_CLAUSE not allowing TIMESTAMP LITERAL DBZ-5592

  • Orcale DDL does not support comments on materialized views DBZ-5595

  • Oracle DDL does not support DEFAULT ON NULL DBZ-5605

  • Datatype mdsys.sdo_geometry not supported DBZ-5609

Other changes

  • Add signal table automatically to include list DBZ-3293

  • No documentation for snapshot.include.collection.list property for Db2 connector DBZ-4345

  • Deprecate internal key/value converter options DBZ-4617

  • Run system testsuite inside OpenShift DBZ-5165

  • Upgrade SQL Server driver to 10.2.1.jre8 DBZ-5290

  • Rewrite oracle tests pipeline job to matrix job DBZ-5412

  • Debezium on ROSA sanity testing DBZ-5416

  • Update link format in shared tutorial file DBZ-5422

  • Deprecate legacy topic selector for all connectors DBZ-5457

  • Remove community conditionalization in signaling doc for Oracle incremental and ad hoc snapshots content DBZ-5458

  • Remove the dependency of JdbcConnection on DatabaseSchema DBZ-5470

  • Remove SQL Server SourceTimestampMode DBZ-5477

  • Maintanence branch builds on connector repos should build against proper branch DBZ-5492

  • Upgrade PostgreSQL driver to 42.4.1 DBZ-5493

  • Force updating snapshots when building the UI in the workflow DBZ-5501

  • Restrict connector workflows based on individual grammar changes in DDL module DBZ-5528

  • Disable preferring DDL before logical schema in history recovery DBZ-5535

  • Disable Eager loading for federated module bundles. DBZ-5545

  • Missing format value option in debezium-server doc DBZ-5546

  • Debezium inputs with number types have the wrong name of the input DBZ-5553

  • MySQL read.only property incorrectly appears in downstream documentation DBZ-5555

  • Add the Fed module running script and update readme DBZ-5560

  • Logging improvements in TestSuite DBZ-5563

  • Formatting characters in properties tables rendered in published content DBZ-5565

  • Upgrade mysql-binlog-connector-java library version DBZ-5574

  • MySQL database.server.id indicates default value is random but that no longer applies DBZ-5577

  • Switch test containers to Debezium nightly DBZ-5601

  • GitHub CI fails for DB2 connector DBZ-5606

  • ValidateSqlServerFiltersIT fails in CI DBZ-5613

Release 2.0.0.Beta1 (July 26th 2022)

Kafka compatibility

This release has been built against Kafka Connect 3.2.0 and has been tested with version 3.2.0 of the Kafka brokers. See the Kafka documentation for compatibility with other versions of Kafka brokers.

Upgrading

Before upgrading any connector, be sure to check the backward-incompatible changes that have been made since the release you were using.

When you decide to upgrade one of these connectors to 2.0.0.Beta1 from any earlier versions, first check the migration notes for the version you’re using. Gracefully stop the running connector, remove the old plugin files, install the 2.0.0.Beta1 plugin files, and restart the connector using the same configuration. Upon restart, the 2.0.0.Beta1 connectors will continue where the previous connector left off. As one might expect, all change events previously written to Kafka by the old connector will not be modified.

If you are using our container images, then please do not forget to pull them fresh from DockerHub.

Breaking changes

Debezium previously exposed connector metrics as a single tuple of snapshot, streaming, and history-based beans. With this release, connector metrics have migrated to a multi-partition scheme, which means that the naming and how metrics are exposed has changed as a part of DBZ-4726. Please be sure to review your metrics gathering processes if you’re using tools like Grafana, Prometheus, or other JMX metrics gathering frameworks.

Debezium previously provided support for reading and storing offsets, history, and other bits as a part of the debezium-core module. With this release, we’ve introduced a new module called debezium-storage with implementations for local file and Kafka based storage approaches (DBZ-5229). This approach provides a wonderful extension point going forward to introduce other storage implementations as the need arises. When upgrading, you may need to adjust your application’s dependencies depending on what storage module implementations your code depends upon.

New features

  • Pluggable topic selector DBZ-4180

  • Read Debezium Metrics From Debezium Server Consumer DBZ-5235

  • Treat SQLServerException with "Broken pipe (Write failed)" exception message as a retriable exception DBZ-5292

  • Include user that committed change in metadata (oracle) DBZ-5358

  • UI Add debezium-ui i18n zh translation DBZ-5379

  • Support storing extended attributes in relational model and JSON schema history topic DBZ-5396

  • Validate topic naming strategy relative topic name properties DBZ-5414

  • Verify the unique index whether including function or arbitrary expression DBZ-5424

  • Remove the duplicated SimpleDdlParserListener from mysql connector DBZ-5425

Fixes

  • MongoConnector’s field exclusion configuration does not work with fields with the same name but from different collections DBZ-4846

  • User input are not consistent on Filter step for the DBZ connectors DBZ-5246

  • KafkaDatabaseHistory without check database history topic create result caused UnknowTopicOrPartitionException DBZ-5249

  • Lob type data is inconsistent between source and sink, after modifying the primary key DBZ-5295

  • Caused by: java.io.EOFException: Failed to read next byte from position 2005308603 DBZ-5333

  • Incremental Snapshot: Oracle table name parsing does not support periods in DB name DBZ-5336

  • Support PostgreSQL default value function calls with schema prefixes DBZ-5340

  • Unsigned tinyint conversion fails for MySQL 8.x DBZ-5343

  • Log a warning when an unsupported LogMiner operation is detected for a captured table DBZ-5351

  • NullPointerException thrown when unique index based on both system and non-system generated columns DBZ-5356

  • MySQL Connector column hash v2 does not work DBZ-5366

  • Outbox JSON expansion fails when nested arrays contain no elements DBZ-5367

  • docker-maven-plugin needs to be upgraded for Mac Apple M1 DBZ-5369

  • AWS DocumentDB (with MongoDB Compatibility) Connect Fail DBZ-5371

  • Oracle Xstream does not propagate commit timestamp to transaction metadata DBZ-5373

  • UI View connector config in non-first cluster return 404 DBZ-5378

  • CommitScn not logged in expected format DBZ-5381

  • org.postgresql.util.PSQLException: Bad value for type timestamp/date/time: CURRENT_TIMESTAMP DBZ-5384

  • Missing "previousId" property with parsing the rename statement in kafka history topic DBZ-5386

  • Check constraint introduces a column based on constraint in the schema change event. DBZ-5390

  • The column is referenced as PRIMARY KEY, but a matching column is not defined in table DBZ-5398

  • Clarify which database name to use for signal.data.collection when using Oracle with pluggable database support DBZ-5399

  • Timestamp with time zone column’s default values not in GMT DBZ-5403

  • Upgrade to Kafka 3.1 broke build compatibility with Kafka 2.x and Kafka 3.0 DBZ-5404

  • PostgresConnectorIT#shouldRecoverFromRetriableException fails randomly DBZ-5408

Other changes

  • Clean-up unused documentation variables DBZ-2595

  • Intermittent test failures on CI: EventProcessingFailureHandlingIT DBZ-4004

  • Clarify whether SQL Server on Azure is a supported configuration or not DBZ-4312

  • Remove redundant setting of last events DBZ-5047

  • Rename docker-images repository and JIRA component to container-images DBZ-5048

  • Update instructions for deploying Debezium on RHEL (downstream-only change) DBZ-5293

  • Add ts_ms field to examples of transaction boundary events and examples and update property description in documentation DBZ-5334

  • Oracle GitHub actions workflow no longer run tests on pushes DBZ-5349

  • Unify job names in jenkins system-tests DBZ-5392

  • Build stable branches for connector-specific repos DBZ-5409

  • Oracle non-cdb builds do not use the correct environment settings DBZ-5411

  • Update the topic naming strategy doc to all connectors DBZ-5413

  • Address User guide review comments for Oracle connector DBZ-5418

  • OracleSchemaMigrationIT fails on non-pluggable (non-CDB) databases DBZ-5419

Release 2.0.0.Alpha3 (July 1st 2022)

Kafka compatibility

This release has been built against Kafka Connect 3.2.0 and has been tested with version 3.2.0 of the Kafka brokers. See the Kafka documentation for compatibility with other versions of Kafka brokers.

Upgrading

Before upgrading any connector, be sure to check the backward-incompatible changes that have been made since the release you were using.

When you decide to upgrade one of these connectors to 2.0.0.Alpha3 from any earlier versions, first check the migration notes for the version you’re using. Gracefully stop the running connector, remove the old plugin files, install the 2.0.0.Alpha3 plugin files, and restart the connector using the same configuration. Upon restart, the 2.0.0.Alpha3 connectors will continue where the previous connector left off. As one might expect, all change events previously written to Kafka by the old connector will not be modified.

If you are using our container images, then please do not forget to pull them fresh from DockerHub.

Breaking changes

There are no breaking changes in this release.

New features

  • Mysql Commit Timestamp DBZ-5170

  • Include event scn in Oracle records DBZ-5225

  • Redis Store does not work with GCP Managed Redis DBZ-5268

Fixes

  • Incorrect loading of LSN from offsets DBZ-3942

  • Database history recovery will retain old tables after they’ve been renamed DBZ-4451

  • Adding new table with incremental snapshots not working DBZ-4834

  • BigDecimal has mismatching scale value for given Decimal schema DBZ-4890

  • Debezium has never found starting LSN DBZ-5031

  • Data duplication problem using postgresql source on debezium server DBZ-5070

  • Cursor fetch is used for all results during connection DBZ-5084

  • Debezuim connector fails at parsing select statement overrides when table name has space DBZ-5198

  • DDL statement couldn’t be parsed 2 - Oracle connector 1.9.3.Final DBZ-5230

  • Debezium server duplicates scripting jar files DBZ-5232

  • Cannot convert field type tinyint(1) unsigned to boolean DBZ-5236

  • Oracle unparsable ddl create table DBZ-5237

  • Postgres Incremental Snapshot on parent partitioned table not working DBZ-5240

  • Character set influencers are not properly parsed on default values DBZ-5241

  • Dupicate SCNs on Oracle RAC installations incorrectly processed DBZ-5245

  • NPE when using Debezium Embedded in Quarkus DBZ-5251

  • Oracle LogMiner may fail with an in-progress transaction in an archive log that has been deleted DBZ-5256

  • Order of source block table names in a rename schema change event is not deterministic DBZ-5257

  • Debezium fails to connect to replicaset if a node is down DBZ-5260

  • No changes to commit_scn when oracle-connector got new lob data DBZ-5266

  • Invalid date 'SEPTEMBER 31' DBZ-5267

  • database.history.store.only.captured.tables.ddl not suppressing logs DBZ-5270

  • io.debezium.text.ParsingException: DDL statement couldn’t be parsed DBZ-5271

  • Deadlock during snapshot with Mongo connector DBZ-5272

  • Mysql parser is not able to handle variables in KILL command DBZ-5273

  • Debezium server fail when connect to Azure Event Hubs DBZ-5279

  • ORA-01086 savepoint never established raised when database history topic cannot be created or does not exist DBZ-5281

  • Enabling database.history.store.only.captured.tables.ddl does not restrict history topic records DBZ-5285

Other changes

  • Add script SMT test case to OCP test suite DBZ-2581

  • Confusing example for schema change topic DBZ-4713

  • Update cache-invalidation example DBZ-4754

  • Switch from static yaml descriptors to dynamic objects DBZ-4830

  • Verify that snapshot deployments build and deploy javadocs DBZ-4875

  • DelayStrategy should accept Duration rather than long ms DBZ-4902

  • Use maven 3.8.4 version with enforcer plugin DBZ-5069

  • Add option for '*' wildcard usage testsuite preparation jenkins jobs DBZ-5190

  • Use the Maven wrapper in the Github and Jenkins workflows DBZ-5207

  • Improve performance of OracleConnectorIT shouldIgnoreAllTablesInExcludedSchemas test DBZ-5226

  • Document use of JAR artifact to build Debezium scripting SMT into Kafka Connect DBZ-5227

  • Create shared adoc fragments for specifying MBean name format in connector metrics sections DBZ-5233

  • Build Oracle connector by default without Maven profiles DBZ-5234

  • Remove reference to removed case insensitive option in Oracle README.md DBZ-5250

  • Several Oracle tests do not get database name from TestHelper DBZ-5258

  • Upgrade to Quarkus 2.10.0.Final DBZ-5259

  • Upgrade PostgreSQL driver to 42.4.0 DBZ-5261

  • Refactor ChangeEventQueue to better support n:1 threads DBZ-5277

  • Upgrade MongoDB driver to 4.6.1 DBZ-5287

Release 2.0.0.Alpha2 (June 9th 2022)

Kafka compatibility

This release has been built against Kafka Connect 3.2.0 and has been tested with version 3.2.0 of the Kafka brokers. See the Kafka documentation for compatibility with other versions of Kafka brokers.

Upgrading

Before upgrading any connector, be sure to check the backward-incompatible changes that have been made since the release you were using.

When you decide to upgrade one of these connectors to 2.0.0.Alpha2 from any earlier versions, first check the migration notes for the version you’re using. Gracefully stop the running connector, remove the old plugin files, install the 2.0.0.Alpha2 plugin files, and restart the connector using the same configuration. Upon restart, the 2.0.0.Alpha2 connectors will continue where the previous connector left off. As one might expect, all change events previously written to Kafka by the old connector will not be modified.

If you are using our container images, then please do not forget to pull them fresh from DockerHub.

Breaking changes

Debezium MySQL connector represented FLOAT datatype as FLOAT64. Now the correct type FLOAT32 is used (DBZ-3865).

MongoDB connector no longer supports streaming from oplog. The connector fully switched to change streams streaming (DBZ-4951).

All connectors now use multi-partitioned codebase. This has impact only on project developers (DBZ-5042).

All deprecated configuration options were removed (DBZ-5045).

New features

  • Provide a signal to stop the running incremental snapshot DBZ-4251

  • SQL Server - Fail connector when a user doesn’t have the right permission (CDCReader) DBZ-4346

  • Allow mongodb-connector to decode Binary payloads DBZ-4600

  • Add UI backend tests for SQL Server connector DBZ-4867

  • direct usage of debezium engine ignores ChangeConsumer.supportsTombstoneEvents DBZ-5052

  • Config the cache size property for ByLogicalTableRouter caches DBZ-5072

  • Introduce a new extension api for query debezium version DBZ-5092

  • Introduce a new field "ts_ms" to identify the process time for schema change event DBZ-5098

  • MongoDB Connector should use RawBsonDocument instead of Document DBZ-5113

Fixes

  • Postgres existing publication is not updated with the new table DBZ-3921

  • Error and connector stops when DDL contains lateral DBZ-4780

  • Schema changes should flush SCN to offsets if there are no other active transactions DBZ-4782

  • Connector stops streaming after a re-balance DBZ-4792

  • MySQL connector increment snapshot failed parse datetime column lenth when connector set "snapshot.fetch.size": 20000 DBZ-4939

  • [MySQL Debezium] DDL Parsing error - CREATE OR REPLACE TABLE DBZ-4958

  • InstanceAlreadyExistsException during MongoDb connector metrics registration DBZ-5011

  • DateTimeParseException: Text 'infinity' could not be parsed in Postgres connector DBZ-5014

  • PostgreSQL ENUM default values are missing from generated schema DBZ-5038

  • Debezium official documentation typo DBZ-5040

  • Fix inconsistent transaction id when handling transactional messages in Vitess connector DBZ-5063

  • 4 Connections per connector (postgres) DBZ-5074

  • Oracle documentation refers to archive_log_target rather than archive_lag_target DBZ-5076

  • 'ALTER TABLE mytable DROP FOREIGN KEY IF EXISTS mytable_fk' no viable alternative at input 'ALTER TABLE mytable DROP FOREIGN KEY IF' DBZ-5077

  • Oracle Logminer: records missed during switch from snapshot to streaming mode DBZ-5085

  • Interrupting a snapshot process can hang for some JDBC drivers DBZ-5087

  • Debezium fails to undo change event due to transaction id ending in ffffffff with LogMiner DBZ-5090

  • Table changes are not filled in schema changes from snapshot DBZ-5096

  • Postgresql connector does not retry one some errors when postgres is taken offline DBZ-5097

  • Parsing zero day fails DBZ-5099

  • Cannot Set debezium.sink.kafka.producer.ssl.endpoint.identification.algorithm to empty value DBZ-5105

  • Debezium connector failed with create table statement DBZ-5108

  • Current version of surefire/failsafe skips tests on failure in BeforeAll DBZ-5112

Other changes

  • Restructure documentation for custom converters DBZ-4588

  • Document xmin.fetch.interval.ms property for Postgres connector DBZ-4734

  • Update to Quarkus 2.9.2.Final DBZ-4806

  • Upgrade Oracle driver to 21.5.0.0 DBZ-4877

  • Execute Debezium UI build when core library is changed DBZ-4947

  • Remove unused Oracle connector code DBZ-4973

  • Links to cassandra 3 and 4 artifacts no longer work for Debezium 1.9+ DBZ-5055

  • Align Postgresql driver with Quarkus DBZ-5060

  • Outdated links in Javadoc documentation DBZ-5075

  • Rename "Mysql" to "MySql" in related MysqlFieldReader interface DBZ-5078

  • Create CI job for maven repository verification DBZ-5082

  • Remove database.server.id default value handler, no longer auto-generated. DBZ-5100

  • Upgrade Jackson Databind to 2.13.2.2 DBZ-5107

  • Switch to released version of Fixture5 extension in System testsuite DBZ-5114

Release 2.0.0.Alpha1 (April 28th 2022)

Kafka compatibility

This release has been built against Kafka Connect 3.1.0 and has been tested with version 3.1.0 of the Kafka brokers. See the Kafka documentation for compatibility with other versions of Kafka brokers.

Upgrading

Before upgrading any connector, be sure to check the backward-incompatible changes that have been made since the release you were using.

When you decide to upgrade one of these connectors to 2.0.0.Alpha1 from any earlier versions, first check the migration notes for the version you’re using. Gracefully stop the running connector, remove the old plugin files, install the 2.0.0.Alpha1 plugin files, and restart the connector using the same configuration. Upon restart, the 2.0.0.Alpha1 connectors will continue where the previous connector left off. As one might expect, all change events previously written to Kafka by the old connector will not be modified.

If you are using our container images, then please do not forget to pull them fresh from DockerHub.

Breaking changes

Java 11 is required to run Debezium in any form: Kafka Connect plug-in, Debezium engine and Debezium Server (DBZ-4949).

Maven 3.8.4 is required to build Debezium (DBZ-5064).

PostgreSQL connector no longer supports old versions of protobuf decoding plug-in that do not have transaction and default value support (DBZ-703).

PostgreSQL connector no longer supports wal2json decoding plug-in (dehttps://issues.redhat.com/browse/DBZ-4156[DBZ-4156]). pgoutput decoding plug-in is recommended as the replacement.

Legacy implementation of MySQL connector was removed (DBZ-4950).

Confluent Avro converters are no longer present in ebezium container images (DBZ-4952).

JDBC legacy date/time properties support was removed from MySQL connector (DBZ-4965).

New features

  • Implement Pub/Sub Lite change consumer DBZ-4450

  • Include Instant Client in Docker build for Oracle in Tutorial examples DBZ-1013

  • Add Google Pub/Sub emulator support DBZ-4491

  • Making Postgres PSQLException: This connection has been closed. retriable DBZ-4948

  • ORA-04030: out of process memory when trying to allocate 65568 bytes (Logminer LCR c,krvxrib:buffer) DBZ-4963

  • Should store event header timestamp in HistoryRecord DBZ-4998

  • DBZ-UI: In the Edit/Duplicate connector flow make the access/secret key/password/Client Secret filed as editable. DBZ-5001

  • adjust LogMiner batch size based on comparison with currently used batch size DBZ-5005

Fixes

  • Connector throws java.lang.ArrayIndexOutOfBoundsException DBZ-3848

  • Document no relevant tables should be in the SYS or SYSTEM tablespaces. DBZ-4762

  • Getting java.sql.SQLException: ORA-01291: missing logfile while running with archive log only DBZ-4879

  • Debezium uses wrong LCR format for Oracle 12.1 DBZ-4932

  • Oracle duplicates on connector restart DBZ-4936

  • Oracle truncate causes exception DBZ-4953

  • NPE caused by io.debezium.connector.oracle.antlr.listener.ColumnDefinitionParserListener.resolveColumnDataType DBZ-4976

  • Oracle connector may throw NullPointerException when stopped after an unsuccessful startup DBZ-4978

  • NPE for non-table related DDLs DBZ-4979

  • CTE statements aren’t parsed by MySQL connector DBZ-4980

  • Missing SSL configuration option in the debezium mongodb connector UI DBZ-4981

  • Unsupported MySQL Charsets during Snapshotting for fields with custom converter DBZ-4983

  • Outbox Transform does not allow expanded payload with additional fields in the envelope DBZ-4989

  • Redis Sink - clientSetname is taking place before auth DBZ-4993

  • CLOB with single quotes causes parser exception DBZ-4994

  • Oracle DDL parser fails on references_clause with no column list DBZ-4996

  • Can’t use 'local' database through mongos DBZ-5003

  • Triggering Incremental Snapshot on MongoDB connector throws json parsing error DBZ-5015

  • Jenkins jobs fail to download debezium-bom DBZ-5017

  • Redis Sink - Check if client is not null before closing it DBZ-5019

  • Cassandra 3 handler does not process partition deletions correctly DBZ-5022

  • Keyspaces should be initialised in all schema change listeners on sessions startup. DBZ-5023

  • SQL Server in multi-partition mode fails if a new database is added to an existing configuration DBZ-5033

  • Mysql tests start before MySQL DB constainer is running DBZ-5054

  • Debezium server configuration properties not rendered correctly DBZ-5058

Other changes

  • Add integration test for Oracle database.url configurations DBZ-3318

  • Build Cassandra 3.x connector with Java 11 DBZ-4910

  • Add ignoreSnapshots build option to release pipeline DBZ-4957

  • Update Pulsar client version used by Debezium Server DBZ-4961

  • Intermittent failure of RedisStreamIT.testRedisConnectionRetry DBZ-4966

  • Add triggers for 2.x paths in Github CI DBZ-4971

  • Debezium raised an exception and the task was still running DBZ-4987

  • Nexus Staging Maven plugin is incompatible with OpenJDK 17 DBZ-5025

  • Duplicate definition of Maven plugins DBZ-5026

  • OracleOffsetContextTest should be scoped to LogMiner only DBZ-5028

  • Scope several new Oracle tests to LogMiner only DBZ-5029

  • Failure in jdk outreach jobs DBZ-5041

  • Update artifact server job listing script DBZ-5051

  • Add FAQ about ORA-01882 and Oracle 11 to documentation DBZ-5057

  • Upgrade to Quarkus 2.8.2.Final DBZ-5062