Release Notes for Debezium 1.5

All notable changes for Debezium releases are documented in this file. Release numbers follow Semantic Versioning.

Release 1.5.0.Beta1 (February 23rd, 2021)

Kafka compatibility

This release has been built against Kafka Connect 2.6.1 and has been tested with version 2.6.1 of the Kafka brokers. See the Kafka documentation for compatibility with other versions of Kafka brokers.

Upgrading

Before upgrading any connector, be sure to check the backward-incompatible changes that have been made since the release you were using.

When you decide to upgrade one of these connectors to 1.5.0.Beta1 from any earlier versions, first check the migration notes for the version you’re using. Gracefully stop the running connector, remove the old plugin files, install the 1.5.0.Beta1 plugin files, and restart the connector using the same configuration. Upon restart, the 1.5.0.Beta1 connectors will continue where the previous connector left off. As one might expect, all change events previously written to Kafka by the old connector will not be modified.

If you are using our docker images then do not forget to pull them fresh from Docker registry.

Breaking changes

A regression in the binlog client used by Debezium was identified where large JSON documents in a MySQL JSON column cause a severe performance degredation (DBZ-3106). This issue is under active discussion with the maintainer of the binlog client library.

In earlier versions of Debezium, the MySQL connector incorrectly emitted snapshot events using the c (create) operation type instead of the correct type r (read). If you have consumers which rely on that earlier behavior, you can use the io.debezium.connector.mysql.transforms.ReadToInsertEvent single message transform to emulate that earlier behavior (DBZ-2788). A connector option which accidentally was introduced in 1.4.0 for this same purpose, snapshot.events.as.inserts, got removed again, and the SMT should be used instead in this situation. This SMT is meant for migration purposes only and will be removed in a future Debezium version.

The (incubating) Debezium connector for Oracle emits transaction ids in lower-case now, differing from the previous behavior of returning them as upper-case (DBZ-3165).

The previously deprecated snapshot mode INITIAL_SCHEMA_ONLY of the Oracle connector has been removed. Please use SCHEMA_ONLY instead (DBZ-3034).

New Features

  • Make field descriptions consistent for time values (milliseconds, ms, sec, seconds, etc) DBZ-2858

  • DebeziumEngine RecordChangeEvents cannot be modified DBZ-2897

  • Add license headers and related checkstyle checks for Debezium UI files DBZ-2985

  • Display commit SHA of UI frontend/backend somewhere in the footer DBZ-3052

  • Implement UX suggestions for display of connector type DBZ-3054

  • SqlServerConnector does not implement validate DBZ-3056

  • Database History Producer does not close with a timeout DBZ-3075

  • Improve DML parser performance DBZ-3078

  • Connector list table UI improvement desktop/mobile DBZ-3079

  • Vitess Connector adds support for Vitess 9.0.0 GA DBZ-3100

  • Improve layout for Column Truncate - Mask Component DBZ-3101

  • Improve layout for Data options component and main wizard nav DBZ-3105

  • Add ability to skip tests based on available database options DBZ-3110

  • Support for Transaction Metadata in MySql connector DBZ-3114

  • Add support for JSON column type DBZ-3115

  • Add support for ENUM column type DBZ-3124

  • Enable easy downloading of Camel Kafka Connectors DBZ-3136

  • Capture LogMiner session parameters when session fails to start DBZ-3153

  • Process special values in temporal datatypes DBZ-2614

Fixes

  • Negative timestamps are converted to positive during snapshot DBZ-2616

  • Wrong reference to KafkaConnector in setting up Debezium DBZ-2745

  • Oracle Connector(Using Logminer) with Oracle RDS (v12) does not capture changes DBZ-2754

  • Oracle connector causes ORA-65090 when connecting to an Oracle instance running in non-CDB mode DBZ-2795

  • Warnings and notifications from PostgreSQL are ignored by the connector until the connection is closed DBZ-2865

  • Add support for MySQL to UI Backend DBZ-2950

  • ExtractNewRecord SMT incorrectly extracts ts_ms from source info DBZ-2984

  • Replication terminates with ORA-01291: missing log file DBZ-3001

  • Kafka Docker image the HEAP_OPTS variable is not used DBZ-3006

  • Support multiple schemas with Oracle LogMiner DBZ-3009

  • Function calls does not allow parentheses for functions with non-mandatory parentheses DBZ-3017

  • Complete support for properties that contain hyphens DBZ-3019

  • UI issues with connectors table row expansion state DBZ-3049

  • SQLException for Global temp tables from OracleDatabaseMetaData.getIndexInfo() makes Debezium snapshotting fail DBZ-3057

  • Cassandra Connector doesn’t support Cassandra version >=3.11.5 DBZ-3060

  • Make Cassandra Connector work with CommitLogTransfer better DBZ-3063

  • no viable alternative at input 'create or replace index' DBZ-3067

  • Connect image propagates env vars starting with CONNECT prefix DBZ-3070

  • PgOutputMessageDecoder doesn’t order primary keys DBZ-3074

  • Strange transaction metadata for Oracle logminer connector DBZ-3090

  • Getting RejectedExecutionException when checking topic settings from KafkaDatabaseHistory.checkTopicSettings DBZ-3096

  • Environment Variables with spaces are truncated when written to properties file DBZ-3103

  • Error: Supplemental logging not configured for table. Use command: ALTER TABLE DBZ-3109

  • Uncaught (in promise) TypeError: Cannot read property 'call' of undefined DBZ-3125

  • Final stage of snapshot analyzes tables not present in table.include.list thus stumbles upon unsupported XMLTYPE table DBZ-3151

  • Missing Prometheus port in kafka network policy DBZ-3170

  • XStream does not process NUMER(1) data DBZ-3172

Other changes

  • Setup CI job for DB2 DBZ-2235

  • Integration with Service Registry promoted to GA DBZ-2815

  • Remove DECIMAL string sanitisation once Vitess upstream bug is fixed DBZ-2908

  • Review format and configuration options for Db2 for GA DBZ-2977

  • Test with Postgres 13 DBZ-3022

  • Prepare Debezium UI to participate in upstream releases DBZ-3027

  • Upgrade testcontainers to 1.15.1 DBZ-3066

  • Use new deployment endpoint for releases to Maven Central DBZ-3069

  • Remove obsolete Awestruct container image DBZ-3072

  • "JDBC driver" doesn’t make sense for non-relational connectors DBZ-3076

  • Replace RecordMakers with MySqlChangeRecordEmitter DBZ-3077

  • Make CI builds resilient against disconnects on GH Actions infrastructure DBZ-3083

  • Separate SourceInfo and MySQL offset context DBZ-3086

  • Remove zero-width whitespace from option names DBZ-3087

  • Adapt UI for MySQL connector type DBZ-3091

  • Change MySQL database schema contract to support separate parsing and processing phase DBZ-3093

  • MySQL build stuck for 6h DBZ-3095

  • Rewrite legacy reader tests DBZ-3099

  • Intermittent test failure in Postgres PostgresConnectorIT#customSnapshotterSkipsTablesOnRestart DBZ-3107

  • Remove duplicate anchor links in Connector properties DBZ-3111

  • Upgrade to Quarkus 1.12.0.Final DBZ-3116

  • Config validation for Vitess DBZ-3117

  • Config validation for Oracle DBZ-3119

  • Avoid naming conflict between connection classes DBZ-3147

  • Set up commit message check for Vitess DBZ-3152

  • Put IIDR license requirement into NOTE box DBZ-3163

  • Consistent logging of connection validation failure DBZ-3164

  • Remove COLUMN_BLACK_LIST option in Oracle connector DBZ-3167

Release 1.5.0.Alpha1 (February 4th, 2021)

Kafka compatibility

This release has been built against Kafka Connect 2.6.1 and has been tested with version 2.6.1 of the Kafka brokers. See the Kafka documentation for compatibility with other versions of Kafka brokers.

Upgrading

Before upgrading any connector, be sure to check the backward-incompatible changes that have been made since the release you were using.

When you decide to upgrade one of these connectors to 1.5.0.Alpha1 from any earlier versions, first check the migration notes for the version you’re using. Gracefully stop the running connector, remove the old plugin files, install the 1.5.0.Alpha1 plugin files, and restart the connector using the same configuration. Upon restart, the 1.5.0.Alpha1 connectors will continue where the previous connector left off. As one might expect, all change events previously written to Kafka by the old connector will not be modified.

If you are using our docker images then do not forget to pull them fresh from Docker registry.

Breaking changes

A new capturing implementation for the Debezium MySQL connector has been created (DBZ-1865) based on the common connector framework used by all the other Kafka Connect connectors of Debezium. The connector behaviour is almost in parity with previous implementation, with the exception of the experimental parallel snapshotting feature (DBZ-175), which isn’t available with the new implementation yet and which is planned to be re-introduced later in a different form. If you encounter any issues with the new MySQL connector implementation, please log a Jira issue; in this case, you can use the legacy implementation by setting the internal.implementation=legacy connector configuration option.

New Features

  • Support emitting TRUNCATE events in PostgreSQL pgoutput plugin DBZ-2382

  • Migrate DebeziumContainer enhancements for DBZ-2950 and DBZ-2952 into main repository DBZ-3024

  • Implement meta tags DBZ-2620

  • Improve performance for very large postgres schemas DBZ-2575

Fixes

  • Extra connectors are not buildable unless main Debezium is built locally DBZ-2901

  • java.sql.SQLException: ORA-01333: failed to establish Logminer Dictionary DBZ-2939

  • Add support for connector/task lifecycle ops to UI backend DBZ-2951

  • Cassandra CDC failed to deserialize list<UserType> column correct DBZ-2974

  • Debezium Oracle Connector will appear stuck on large SCN jumps DBZ-2982

  • Invalid regex patterns should fail validation when validation database.include/exclude.list properties for MySQL connector DBZ-3008

  • Fix repository config for Jenkis snapshot deployment DBZ-3011

  • Unable to parse non-constant SIGNAL option value DBZ-3018

  • Cannot parse expression in DEFAULT column definition DBZ-3020

  • Key being used as value in pubsub batch handler DBZ-3037

  • Table creation DDL with CHARACTER SET = DEFAULT causes MySQL connector failure DBZ-3023

  • Missing some MariaDB existence predicates in ALTER TABLE DBZ-3039

Other changes

  • Improved resiliency of release process against OSS failures DBZ-2274

  • Pull up HOSTNAME, PORT, DATABASE_NAME, USER and PASSWORD to RelationalDatabaseConnectorConfig DBZ-2420

  • Db2 Connector doesn’t declare database related config options DBZ-2424

  • Fix build status badge in README files DBZ-2802

  • Merge and complete web components PR DBZ-2804

  • IBM Db2 Connector promoted to GA DBZ-2814

  • Document several Oracle frequently encountered problems DBZ-2970

  • No syntax highlighting on website listings DBZ-2978

  • Admonition icons missing DBZ-2986

  • Improve logging for Logminer adapter DBZ-2999

  • CI build not required for changes in README files DBZ-3012

  • Execute ZZZGtidSetIT as the last test DBZ-3047

  • Capture and report LogMiner state when mining session fails to start DBZ-3055