Release Notes for Debezium 1.7

All notable changes for Debezium releases are documented in this file. Release numbers follow Semantic Versioning.

Release 1.7.1.Final (November 5th, 2021)

Kafka compatibility

This release has been built against Kafka Connect 2.8.1 and has been tested with version 2.8.1 of the Kafka brokers. See the Kafka documentation for compatibility with other versions of Kafka brokers.

Upgrading

Before upgrading any connector, be sure to check the backward-incompatible changes that have been made since the release you were using.

When you decide to upgrade one of these connectors to 1.7.1.Final from any earlier versions, first check the migration notes for the version you’re using. Gracefully stop the running connector, remove the old plugin files, install the 1.7.1.Final plugin files, and restart the connector using the same configuration. Upon restart, the 1.7.1.Final connectors will continue where the previous connector left off. As one might expect, all change events previously written to Kafka by the old connector will not be modified.

If you are using our docker images then do not forget to pull them fresh from Docker registry.

Breaking changes

There are no breaking changes in this release.

New Features

  • Store buffered events in separate Infinispan cache DBZ-4159

  • Log warning when table/column name exceeds maximum allowed by LogMiner DBZ-4161

  • Process transaction started/committed in MySQL read-only incremental snapshot DBZ-4197

Fixes

  • Row hashing in LogMiner Query not able to differentiate between rows of a statement. DBZ-3834

  • The chunk select statement is incorrect for combined primary key in incremental snapshot DBZ-3860

  • Cassandra UUID handling DBZ-3885

  • Infinispan SPI throws NPE with more than one connector configured to the same Oracle database DBZ-4064

  • Debezium Server might contain driver versions pulled from Quarkus DBZ-4070

  • Database history is constantly being reconfigured DBZ-4106

  • Oracle flush table should not contain multiple rows DBZ-4118

  • SQL Server Connector fails to wrap in flat brackets DBZ-4125

  • Oracle Connector DDL Parsing Exception DBZ-4126

  • Debezium deals with Oracle DDL appeared IndexOutOfBoundsException: Index: 0, Size: 0 DBZ-4135

  • Error: PostgresDefaultValueConverter - Cannot parse column default value 'NULL::numeric' to type 'numeric'. Expression evaluation is not supported. DBZ-4137

  • Possible OutOfMemoryError with tracking schema changes DBZ-4151

  • DDL ParsingException - not all table compression modes are supported DBZ-4158

  • Producer failure NullPointerException DBZ-4166

  • DDL statement couldn’t be parsed - Modify Column DBZ-4174

  • OracleSchemaMigrationIT#shouldNotEmitDdlEventsForNonTableObjects fails for Xstream DBZ-4186

  • Certain LogMiner-specific tests are not being skipped while using Xstreams DBZ-4188

  • DML statement couldn’t be parsed DBZ-4194

  • io.debezium.text.ParsingException: DDL statement couldn’t be parsed. Please open a Jira issue with the statement 'DROP TABLE IF EXISTS condition' DBZ-4210

  • Support MySQL Dual Passwords in DDL Parser DBZ-4215

  • CREATE PROCEDURE DDL throws ParsingException DBZ-4229

  • CHAR / NCHAR precision is not correctly derived from DDL statements DBZ-4233

Other changes

  • Incorrect documentation for message.key.columns DBZ-3437

  • Document awareness of Oracle database tuning DBZ-3880

  • Please fix vulnerabilites DBZ-3926

  • Handle SCN gap DBZ-4036

  • Upgrade to Apache Kafka 2.8.1 DBZ-4108

  • Missing documentation for max.iteration.transactions option DBZ-4129

  • Misc. MongoDB connector docs fixes DBZ-4149

  • Document Oracle buffering solutions DBZ-4157

Release 1.7.0.Final (September 30th, 2021)

Kafka compatibility

This release has been built against Kafka Connect 2.8.0 and has been tested with version 2.8.0 of the Kafka brokers. See the Kafka documentation for compatibility with other versions of Kafka brokers.

Upgrading

Before upgrading any connector, be sure to check the backward-incompatible changes that have been made since the release you were using.

When you decide to upgrade one of these connectors to 1.7.0.Final from any earlier versions, first check the migration notes for the version you’re using. Gracefully stop the running connector, remove the old plugin files, install the 1.7.0.Final plugin files, and restart the connector using the same configuration. Upon restart, the 1.7.0.Final connectors will continue where the previous connector left off. As one might expect, all change events previously written to Kafka by the old connector will not be modified.

If you are using our docker images then do not forget to pull them fresh from Docker registry.

Breaking changes

For the incubating Debezium Cassandra connector, the field format emitted for UUID columns has changed; instead of using a base64 representation like "AA4AAFcZEqMAAAAAKDJ9Sg==", the more common UUID string representation is used now, such as "000e0000-5719-12a3-0000-000028327d4a" (DBZ-3885.

New Features

  • DBZ-UI - Provide list of configurations DBZ-3960

  • add ProtobufConverter for Cassandra CDC DBZ-3906

Fixes

  • java.lang.RuntimeException: com.microsoft.sqlserver.jdbc.SQLServerException: The connection is closed DBZ-3346

  • Oracle connector unable to start in archive only mode DBZ-3712

  • DDL statement couldn’t be parsed DBZ-4026

  • Question about handling Raw column types DBZ-4037

  • Fixing wrong log dir location in Kafka container image DBZ-4048

  • Incremental snapshotting of a table can be prematurely terminated after restart DBZ-4057

  • Documentation - Setting up Db2 - Step 10 (Start the ASN agent) is not accurate DBZ-4044

  • Debezium Server uses MySQL driver version as defined in Quarkus not in Debezium DBZ-4049

  • Events are missed with Oracle connector due to LGWR buffer not being flushed to redo logs DBZ-4067

  • Postgres JDBC Driver version causes connection issues on some cloud Postgres instances DBZ-4060

  • nulls for some MySQL properties in the connector-types backend response DBZ-3108

Other changes

  • Oracle IncrementalSnapshotIT invalid table test fails DBZ-4040

  • Document how to enable schema for JSON messages DBZ-4041

  • Trigger contributor check action only when PR is opened DBZ-4058

  • Provide JMH benchmark for ChangeEventQueue DBZ-4050

  • Commit message action fails for multi-line commit messages DBZ-4047

Release 1.7.0.CR2 (September 23rd, 2021)

Kafka compatibility

This release has been built against Kafka Connect 2.8.0 and has been tested with version 2.8.0 of the Kafka brokers. See the Kafka documentation for compatibility with other versions of Kafka brokers.

Upgrading

Before upgrading any connector, be sure to check the backward-incompatible changes that have been made since the release you were using.

When you decide to upgrade one of these connectors to 1.7.0.CR2 from any earlier versions, first check the migration notes for the version you’re using. Gracefully stop the running connector, remove the old plugin files, install the 1.7.0.CR2 plugin files, and restart the connector using the same configuration. Upon restart, the 1.7.0.CR2 connectors will continue where the previous connector left off. As one might expect, all change events previously written to Kafka by the old connector will not be modified.

If you are using our docker images then do not forget to pull them fresh from Docker registry.

Breaking changes

There are no breaking changes in this release.

New Features

  • Support read-only MySQL connection in incremental snapshot DBZ-3577

Fixes

  • Connection failure after snapshot wasn’t executed for a while DBZ-3951

  • Oracle-Connector fails parsing a DDL statement DBZ-3977

  • Oracle connector fails after error ORA-01327 DBZ-4010

  • Incorrect incremental snapshot DDL triggers snapshot that generates unending* inserts against signalling table DBZ-4013

  • Oracle-Connector fails parsing a DDL statement (truncate partition) DBZ-4017

Other changes

  • Jenkins build node is based on RHEL 8.0 and requires upgrade DBZ-3690

  • Remove GRANT ALTER ANY TABLE from Oracle documentation DBZ-4007

  • Update deploy action configuration for v3 DBZ-4009

  • Website preview via surge.sh DBZ-4011

  • Automate contributor check in COPYRIGHT.txt DBZ-4023

  • Provide an example of usage of snapshot.select.statement.overrides DBZ-3603

  • Throughput Bottleneck and Inefficient Batching in ChangeEventQueue DBZ-3887

  • Performance Bottleneck in TableIdParser String Replacement DBZ-4015

Release 1.7.0.CR1 (September 16th, 2021)

Kafka compatibility

This release has been built against Kafka Connect 2.8.0 and has been tested with version 2.8.0 of the Kafka brokers. See the Kafka documentation for compatibility with other versions of Kafka brokers.

Upgrading

Before upgrading any connector, be sure to check the backward-incompatible changes that have been made since the release you were using.

When you decide to upgrade one of these connectors to 1.7.0.CR1 from any earlier versions, first check the migration notes for the version you’re using. Gracefully stop the running connector, remove the old plugin files, install the 1.7.0.CR1 plugin files, and restart the connector using the same configuration. Upon restart, the 1.7.0.CR1 connectors will continue where the previous connector left off. As one might expect, all change events previously written to Kafka by the old connector will not be modified.

If you are using our container images then do not forget to pull them fresh from Docker registry.

Breaking changes

The container images for Apache ZooKeeper, Kafka and Connect are now based on Fedora base image (DBZ-3939). This change was introduced to stay synchronized with the latest Java releases.

A side effect of the Java upgrade is that old unsecure TLS algorithms, namely TLSv1 and TLSv1.1, are disabled by default on the 1.7 container images. If your database cannot accept modern TLS connections, Debezium will throw a SSLHandshakeException and fail to connect.

If a connection with older algorithms is necessary, then remove the entries "TLSv1" and/or "TLSv1.1" from the jdk.tls.disabledAlgorithms key in the following files of the Debezium container image for Kafka Connect:

  • /etc/crypto-policies/back-ends/java.config

  • /lib/jvm/jre/conf/security/java.security

New Features

There are no new features in this release.

Fixes

  • RedisStreamChangeConsumer - handleBatch - client.xadd should be wrapped with a try catch block DBZ-3713

  • Incorrect information in documentation about supplemental logging DBZ-3776

  • DML statement couldn’t be parsed DBZ-3892

  • DEBEZIUM producer stops unexpectedly trying to change column in table which does not exist DBZ-3898

  • "binary.handling.mode": "hex" setting works incorrectly for values with trailing zeros DBZ-3912

  • System test-suite is unable to work with unreleased Apicurio versions DBZ-3924

  • CI support for running Apicurio registry tests DBZ-3932

  • Incorrect validation of truncate handling mode DBZ-3935

  • protobuf decoder has sends unsigned long as signed for Postgres 13 DBZ-3937

  • Field#description() should return a proper java.lang.String when documentation/description is not set DBZ-3943

  • MySQL example image not working after upgrade to 8.0 DBZ-3944

  • Fix empty high watermark check DBZ-3947

  • Oracle Connector replicating data from all PDBs. Missing PDB filter during replication. DBZ-3954

  • Oracle connector Parsing Exception: DDL statement couldn’t be parsed DBZ-3962

  • FormSwitchComponent not working correctly in case of duplicate STM form DBZ-3968

  • Strings with binary collation shouldn’t be parsed as Types.BINARY by MySqlAntlrDdlParser. DBZ-3969

  • Openshift pods list image preview not found DBZ-3970

  • MySqlValueConvertes.java has typo DBZ-3976

  • Mysql-Connector fails parsing invalid decimal format DDL statement DBZ-3984

  • Connection Factory is not used when validating SQL Server Connector DBZ-4001

Other changes

  • Promote Outbox SMT to GA DBZ-3584

  • Clarify lifecycle of snapshot metrics DBZ-3613

  • Explore on building non-core repos with corresponding PR branch of core repo and vice-versa DBZ-3748

  • Upgrade to binlog-client 0.25.3 DBZ-3787

  • RelationalSnapshotChangeEventSource should accept a RelationalDatabaseSchema DBZ-3818

  • Create GH Action that flags "octocat" commits DBZ-3822

  • Publish Maven repo with downstream artifacts DBZ-3861

  • CI preparation for Apicurio Registry downstream DBZ-3908

  • Specify branch name on push/pull_request step in all GH action workflows DBZ-3913

  • Consistently order releases from new to old on the website DBZ-3917

  • Update RELEASING.md DBZ-3918

  • Update antora.yml file with new values for SMT attributes DBZ-3922

  • Documentation update should not trigger staging workflow build DBZ-3923

  • Upgrade to Jackson Databind version 2.10.5.1 DBZ-3927

  • Add top-level Transformation menu node for downstream docs DBZ-3931

  • Docker image serving plugin artifacts over HTTP for new Strimzi deployment mechanism DBZ-3934

  • Upgrade MySQL example image to 8.0 DBZ-3936

  • Gracefully handle DB history file stored in a sym-linked directory DBZ-3958

  • Update docs to specify that connectors track metadata only for transactions that occur after deployment DBZ-3961

  • Update and automate Jenkis Node setup DBZ-3965

  • Hyper-link references between options in the Outbox SMT options table DBZ-3920

  • Generify exclusion of columns from snapshotting DBZ-2525

  • PoC for adding transformations / SMT steps to the Debezium UI DBZ-3698

  • Use No match found of pf Empty state component in filter page. DBZ-3888

  • Update the "Skip to review" implementation as per PF new documented standard design pattern DBZ-3916

  • Set up MongoDB 5.0 image DBZ-3973

Release 1.7.0.Beta1 (August 25th, 2021)

Kafka compatibility

This release has been built against Kafka Connect 2.8.0 and has been tested with version 2.8.0 of the Kafka brokers. See the Kafka documentation for compatibility with other versions of Kafka brokers.

Upgrading

Before upgrading any connector, be sure to check the backward-incompatible changes that have been made since the release you were using.

When you decide to upgrade one of these connectors to 1.7.0.Beta1 from any earlier versions, first check the migration notes for the version you’re using. Gracefully stop the running connector, remove the old plugin files, install the 1.7.0.Beta1 plugin files, and restart the connector using the same configuration. Upon restart, the 1.7.0.Beta1 connectors will continue where the previous connector left off. As one might expect, all change events previously written to Kafka by the old connector will not be modified.

If you are using our docker images then do not forget to pull them fresh from Docker registry.

Breaking changes

The MySQL driver was updated to the latest version 8.0.26 (DBZ-3833). This update comes with a new timezone handling and configuration options.

Detailed information can be found in the MySQL docs.

New Features

  • Sink adapter for Nats Streaming DBZ-3815

  • Debezium Server’s run.sh startup script fails on msys or cygwin bash DBZ-3840

  • Upgrade Debezium Server Pravega sink to 0.9.1 DBZ-3841

Fixes

  • Create example for using self-managed Debezium with MK DBZ-2947

  • Exception when validating field.exclude.list for Mongo DB connectors DBZ-3028

  • In case of /api/connectors/1 takes longer time(more than pooling) to fail spinner keeps on loading. DBZ-3313

  • SQL Server CDC event timestamps do not get converted to UTC DBZ-3479

  • Debezium snapshot.select.statement.overrides overrides not used DBZ-3760

  • Server name pattern is unnecessarily restrictive. DBZ-3765

  • Crash when processing MySQL 5.7.28 TIME fields returns empty blob instead of null DBZ-3773

  • Debezium UI and CDC DBZ-3781

  • Disable "Next" if any field value is changed after the validation. DBZ-3783

  • Add DEFAULT to partition option engine DBZ-3784

  • Initiating MongoDB connector causes oplog table scan DBZ-3788

  • SRCFG00014: The config property debezium.sink.pravega.scope is required but it could not be found in any config source DBZ-3792

  • LSN component of Postgres sequence numbers is not updated DBZ-3801

  • Debezium 1.6.1 expecting database.port even when database.url is provided in config. DBZ-3813

  • Postgres numeric default value throwing exception DBZ-3816

  • SQL Server connector doesn’t handle retriable errors during task start DBZ-3823

  • Debezium OpenShift integration test-suite failure DBZ-3824

  • Debezium Server Kinesis Sink Cannot Handle Null Events DBZ-3827

  • Timeout when reading from MongoDB oplog cannot be controlled DBZ-3836

  • Snapshot locking mode "minimal_percona" incorrectly resets transaction & isolation state DBZ-3838

  • Properly skip tests when minor/patch are not specified DBZ-3839

  • Truncate validation should verify key schema is null and not value schema DBZ-3842

  • System test-suite fails if CRD already exist within the cluster DBZ-3846

  • Incorrect test-tags for OcpAvroDB2ConnectorIT DBZ-3851

  • System test-suite CI job does not have RHEL image parameter DBZ-3852

  • Typo with prodname asciidoc attribute usage DBZ-3856

  • SQL Server Connector finds tables for streaming but not snapshot DBZ-3857

  • Signaling table id column too small in example DBZ-3867

  • Oracle unparsable DDL issue DBZ-3877

  • Support AS clause in GRANT statement DBZ-3878

  • Error Parsing Oracle DDL dropping PK DBZ-3886

  • Q3 docs referencing Service Registry 2.0 docs DBZ-3891

  • EMPTY_CLOB() and EMPTY_BLOB() should be treated as empty LOB values DBZ-3893

  • Oracle DDL parsing issue DBZ-3896

Other changes

  • Debezium UI participating in upstream releases — follow-up DBZ-3169

  • Discuss SMT predicates in docs DBZ-3227

  • Test failure for SqlServerConnectorIT#excludeColumnWhenCaptureInstanceExcludesColumns DBZ-3228

  • Adjust to changed Strimzi CRDs DBZ-3385

  • Create a smoke test for Debezium with Kafka on RHEL DBZ-3387

  • Promote Debezium support on RHEL to GA DBZ-3406

  • Oracle Docs for TP DBZ-3407

  • Upgrade to Kafka 2.8 DBZ-3444

  • Update Debezium on RHEL documentation for GA DBZ-3462

  • Options in outbox router docs not linked DBZ-3649

  • Create Kafka related images based on UBI-8 for RHEL certification DBZ-3650

  • Error in description of the property column.mask.hash.hashAlgorithm.with.salt.salt DBZ-3802

  • Debezium does not provide up-to-date container images DBZ-3809

  • Change DBZ kafka image , so its start script can be used on QA Rhel kafka DBZ-3810

  • Test with Apicurio Registry 2.0 in system level test-suite DBZ-3812

  • Upgrade commons-compress from 1.20 to 1.21 DBZ-3819

  • Update jenkins job configuration to incorporate recent system-testsuite changes DBZ-3825

  • Test Failure - RecordsStreamProducerIT#testEmptyChangesProducesHeartbeat DBZ-3828

  • Upgrade UI proxy connectors to 1.6.1.Final DBZ-3837

  • Improperly constructed links generating downstream build errors DBZ-3858

  • CI Failure in VitessConnectorIT.shouldOutputRecordsInCloudEventsFormat DBZ-3863

  • CI Failure for StreamingSourceIT.shouldFailOnSchemaInconsistency DBZ-3869

  • Extract new top-level menu node for SMTs DBZ-3873

  • Introduce documentation variables for AMQ DBZ-3879

  • Don’t log error when dropping non-existent replication slot in tests DBZ-3889

  • Intermittent test failures on CI: VitessConnectorIT::shouldUseUniqueKeyAsRecordKey DBZ-3900

  • Intermittent test failures on CI: IncrementalSnapshotIT#updatesWithRestart DBZ-3901

  • Test shouldNotEmitDdlEventsForNonTableObjects randomly fails DBZ-3902

  • VOLUME instruction causes issue with recent Docker versions DBZ-3903

  • Provide ability to denote UI order in field metadata DBZ-3904

  • Make relocation.dir and offset.dir configs required. DBZ-2251

  • Create Debezium API Spec Generator and static API definitions for connectors DBZ-3364

  • Improve incremental snapshot metrics DBZ-3688

  • Import Pattern-fly CSS from @patternfly/patternfly DBZ-3779

  • Allow system testsuite to produce Strimzi image for arbitrary released version of Debezium DBZ-3826

  • PostgreSQL - Minor Performance bottleneck in PostgresChangeRecordEmitter DBZ-3870

  • Oracle - Provide a more user-friendly way to update SCN DBZ-3876

  • Test failure on CI - SqlServerConnectorIT#readOnlyApplicationIntent DBZ-2398

  • Test failure for SqlServerConnectorIT#EventProcessingFailureHandlingIT DBZ-3229

  • Remove underscore from Debezium Server NATS sink Java package name DBZ-3910

  • LogMinerDatabaseStateWriter causes a SQLException DBZ-3911

  • Maven release fails due to debezium-testing version handling DBZ-3909

  • Zookeeper image should not use archive.apache.org DBZ-3914

Release 1.7.0.Alpha1 (July 30th, 2021)

Kafka compatibility

This release has been built against Kafka Connect 2.7.0 and has been tested with version 2.7.0 of the Kafka brokers. See the Kafka documentation for compatibility with other versions of Kafka brokers.

Upgrading

Before upgrading any connector, be sure to check the backward-incompatible changes that have been made since the release you were using.

When you decide to upgrade one of these connectors to 1.7.0.Alpha1 from any earlier versions, first check the migration notes for the version you’re using. Gracefully stop the running connector, remove the old plugin files, install the 1.7.0.Alpha1 plugin files, and restart the connector using the same configuration. Upon restart, the 1.7.0.Alpha1 connectors will continue where the previous connector left off. As one might expect, all change events previously written to Kafka by the old connector will not be modified.

If you are using our docker images then do not forget to pull them fresh from Docker registry.

Breaking changes

There are no breaking changes in this release.

New Features

  • Implement incremental snapshotting for Oracle DBZ-3692

  • Implement a LogMiner event buffer SPI DBZ-3752

  • Remove artifacts.url property from UI config.js DBZ-3209

  • Do not mark offset for commit log files with error DBZ-3366

  • Support read-only MySQL connection in incremental snapshot DBZ-3577

  • CloudEventsConverter does not support Oracle, Db2, or Vitess DBZ-3668

  • Allow usernames to be excluded in logminer query DBZ-3671

  • Track Oracle session PGA memory consumption DBZ-3756

  • Performance issue due to inefficient ObjectMapper initialization DBZ-3770

  • Add more smoke tests DBZ-3789

Fixes

  • UI frontend build fails for exported checkout which has no .git dir DBZ-3265

  • Broken links in Avro and Outbox Event Router documentation DBZ-3430

  • Cassandra connector generates invalid schema name for its CDC records DBZ-3590

  • Support invisible columns with MySql 8.0.23+ DBZ-3623

  • Db2Connector is unable to establish validation connection DBZ-3632

  • Status stays in RUNNING for Postgres Connector after Postgres is stopped DBZ-3655

  • Change connection validation log level for better visibility DBZ-3677

  • OracleSchemaMigrationIT can throw false positive test failures if test artifacts remain DBZ-3684

  • MySQL Connector error after execute a "create role" statement DBZ-3686

  • ERROR in Entry module not found: Error: Can’t resolve './src' DBZ-3716

  • Error parsing query, even with database.history.skip.unparseable.ddl DBZ-3717

  • Support for TABLE_TYPE missing form MySQL grammar DBZ-3718

  • Oracle LogMiner DdlParser Error DBZ-3723

  • Debezium mysql connector plugin throws SQL syntax error during incremental snapshot DBZ-3725

  • DDL statement couldn’t be parsed DBZ-3755

  • Debezium Oracle connector stops with DDL parsing error DBZ-3759

  • Exception thrown from getTableColumnsFromDatabase DBZ-3769

  • Incorrect regex parsing in start script of kafka image DBZ-3791

  • Dropdown items list visibility blocked by wizard footer DBZ-3794

  • Permission issues with DB2 example image DBZ-3795

Other changes

  • Make consumer of outbox example more resilient DBZ-1709

  • Set up CI for debezium-examples repo DBZ-1749

  • Refactor LogMinerHelper and SqlUtils DBZ-2552

  • Implement tests for UI components DBZ-3050

  • Add documentation about new capturing implementation for the MySQL connector to downstream product DBZ-3140

  • Remove JSimpleParser DBZ-3155

  • Ability to build KC image with Apicurio converters DBZ-3433

  • Remove log.mining.history.xxx deprecated options DBZ-3581

  • Un-document deprecated options and metrics DBZ-3681

  • Capture changes made by connector user & document that SYS/SYSTEM changes are not captured DBZ-3683

  • Use Debezium thread factory for PG keep-alive DBZ-3685

  • Time for another community newsletter DBZ-3695

  • Improve signalling documentation DBZ-3699

  • Example end-to-end fails due to an API incompatibility with Maven 3.6+ DBZ-3705

  • Example debezium-server-name-mapper fails due to an API incompatibility with Maven 3.6+ DBZ-3706

  • Doc clarification on connector rewrite DBZ-3711

  • Support RHEL deployments in system-test tooling DBZ-3724

  • Misc. tutorial updates DBZ-3747

  • Update Oracle connector deployment instructions for consistency DBZ-3772