Release Notes for Debezium 1.6

All notable changes for Debezium releases are documented in this file. Release numbers follow Semantic Versioning.

Release 1.6.2.Final (September 2nd, 2021)

Kafka compatibility

This release has been built against Kafka Connect 2.7.1 and has been tested with version 2.7.1 of the Kafka brokers. See the Kafka documentation for compatibility with other versions of Kafka brokers.

Upgrading

Before upgrading any connector, be sure to check the backward-incompatible changes that have been made since the release you were using.

When you decide to upgrade one of these connectors to 1.6.2.Final from any earlier versions, first check the migration notes for the version you’re using. Gracefully stop the running connector, remove the old plugin files, install the 1.6.2.Final plugin files, and restart the connector using the same configuration. Upon restart, the 1.6.2.Final connectors will continue where the previous connector left off. As one might expect, all change events previously written to Kafka by the old connector will not be modified.

If you are using our docker images then do not forget to pull them fresh from Docker registry.

Breaking changes

There are no breaking changes in the release.

New Features

There are no new features in the release.

Fixes

  • Broken links in Avro and Outbox Event Router documentation DBZ-3430

  • Oracle LogMiner DdlParser Error DBZ-3723

  • DDL statement couldn’t be parsed DBZ-3755

  • Debezium Oracle connector stops with DDL parsing error DBZ-3759

  • Debezium snapshot.select.statement.overrides overrides not used DBZ-3760

  • Server name pattern is unnecessarily restrictive. DBZ-3765

  • Exception thrown from getTableColumnsFromDatabase DBZ-3769

  • Crash when processing MySQL 5.7.28 TIME fields returns empty blob instead of null DBZ-3773

  • Add DEFAULT to partition option engine DBZ-3784

  • Initiating MongoDB connector causes oplog table scan DBZ-3788

  • SRCFG00014: The config property debezium.sink.pravega.scope is required but it could not be found in any config source DBZ-3792

  • Debezium 1.6.1 expecting database.port even when database.url is provided in config. DBZ-3813

  • Postgres numeric default value throwing exception DBZ-3816

  • Snapshot locking mode "minimal_percona" incorrectly resets transaction & isolation state DBZ-3838

  • Typo with prodname asciidoc attribute usage DBZ-3856

  • Oracle unparsable DDL issue DBZ-3877

  • Support AS clause in GRANT statement DBZ-3878

  • Error Parsing Oracle DDL dropping PK DBZ-3886

  • EMPTY_CLOB() and EMPTY_BLOB() should be treated as empty LOB values DBZ-3893

  • Oracle DDL parsing issue DBZ-3896

  • DEBEZIUM producer stops unexpectedly trying to change column in table which does not exist DBZ-3898

  • "binary.handling.mode": "hex" setting works incorrectly for values with trailing zeros DBZ-3912

  • Incorrect validation of truncate handling mode DBZ-3935

Other changes

  • Discuss SMT predicates in docs DBZ-3227

  • Adjust to changed Strimzi CRDs DBZ-3385

  • Clarify lifecycle of snapshot metrics DBZ-3613

  • Error in description of the property column.mask.hash.hashAlgorithm.with.salt.salt DBZ-3802

  • Improperly constructed links generating downstream build errors DBZ-3858

  • Extract new top-level menu node for SMTs DBZ-3873

  • Upgrade to Jackson Databind version 2.10.5.1 DBZ-3927

  • Upgrade ZooKeeper in 1.6 container image DBZ-3950

  • Upgrade Kafka container image to 2.7.1 DBZ-3956

  • Performance issue due to inefficient ObjectMapper initialization DBZ-3770

Release 1.6.1.Final (July 23rd, 2021)

Kafka compatibility

This release has been built against Kafka Connect 2.7.0 and has been tested with version 2.7.0 of the Kafka brokers. See the Kafka documentation for compatibility with other versions of Kafka brokers.

Upgrading

Before upgrading any connector, be sure to check the backward-incompatible changes that have been made since the release you were using.

When you decide to upgrade one of these connectors to 1.6.1.Final from any earlier versions, first check the migration notes for the version you’re using. Gracefully stop the running connector, remove the old plugin files, install the 1.6.1.Final plugin files, and restart the connector using the same configuration. Upon restart, the 1.6.1.Final connectors will continue where the previous connector left off. As one might expect, all change events previously written to Kafka by the old connector will not be modified.

If you are using our docker images then do not forget to pull them fresh from Docker registry.

Breaking changes

There are no breaking changes in the release.

New Features

There are no new features in the release.

Fixes

  • Support invisible columns with MySql 8.0.23+ DBZ-3623

  • Db2Connector is unable to establish validation connection DBZ-3632

  • MySQL Connector error after execute a "create role" statement DBZ-3686

  • Error parsing query, even with database.history.skip.unparseable.ddl DBZ-3717

  • Support for TABLE_TYPE missing form MySQL grammar DBZ-3718

  • Debezium mysql connector plugin throws SQL syntax error during incremental snapshot DBZ-3725

Other changes

  • Add documentation about new capturing implementation for the MySQL connector to downstream product DBZ-3140

  • Doc clarification on connector rewrite DBZ-3711

  • Update Oracle connector deployment instructions for consistency DBZ-3772

Release 1.6.0.Final (June 30th, 2021)

Kafka compatibility

This release has been built against Kafka Connect 2.7.0 and has been tested with version 2.7.0 of the Kafka brokers. See the Kafka documentation for compatibility with other versions of Kafka brokers.

Upgrading

Before upgrading any connector, be sure to check the backward-incompatible changes that have been made since the release you were using.

When you decide to upgrade one of these connectors to 1.6.0.Final from any earlier versions, first check the migration notes for the version you’re using. Gracefully stop the running connector, remove the old plugin files, install the 1.6.0.Final plugin files, and restart the connector using the same configuration. Upon restart, the 1.6.0.Final connectors will continue where the previous connector left off. As one might expect, all change events previously written to Kafka by the old connector will not be modified.

If you are using our docker images then do not forget to pull them fresh from Docker registry.

Breaking changes

There are no breaking changes in the release.

New Features

  • Allow specifying of Oracle archive log location DBZ-3661

Fixes

  • Fix connect container build to be compatible with Oracle Instant Client DBZ-2547

  • Schema change events of excluded databases are discarded DBZ-3622

  • Provide a descriptive error when enabling log.mining.archive.log.only.mode with an offset SCN that isn’t yet in an archive log. DBZ-3665

  • When LOB support is disabled, use legacy SCN mining algorithm DBZ-3676

Other changes

  • Oracle connector error with tables using unique index keys: "key must not be null" DBZ-1211

  • Database history properties missing in connector docs DBZ-3459

  • Oracle connector doc fixes DBZ-3662

  • Change the reached max batch size log message to DEBUG level DBZ-3664

  • Remove unused code DBZ-3672

  • Update deprecated config for debezium smt DBZ-3673

  • Align Antlr versions used during testing DBZ-3675

Release 1.6.0.CR1 (June 24th, 2021)

Kafka compatibility

This release has been built against Kafka Connect 2.7.0 and has been tested with version 2.7.0 of the Kafka brokers. See the Kafka documentation for compatibility with other versions of Kafka brokers.

Upgrading

Before upgrading any connector, be sure to check the backward-incompatible changes that have been made since the release you were using.

When you decide to upgrade one of these connectors to 1.6.0.CR1 from any earlier versions, first check the migration notes for the version you’re using. Gracefully stop the running connector, remove the old plugin files, install the 1.6.0.CR1 plugin files, and restart the connector using the same configuration. Upon restart, the 1.6.0.CR1 connectors will continue where the previous connector left off. As one might expect, all change events previously written to Kafka by the old connector will not be modified.

If you are using our docker images then do not forget to pull them fresh from Docker registry.

Breaking changes

There are no breaking changes in the release.

New Features

  • Implement SKIPPED_OPERATIONS for SQLServer DBZ-2697

  • Handling database connection timeout during schema recovery DBZ-3615

  • Scope mined DDL events to include/exclude lists if provided DBZ-3634

  • Support heartbeats during periods of low change event activity DBZ-3639

Fixes

  • Fix exception on not found table DBZ-3523

  • Transaction commit event dispatch fails if no active transaction in progress. DBZ-3593

  • Additional unique index referencing columns not exposed by CDC causes exception DBZ-3597

  • GRANT/REVOKE for roles is not working DBZ-3610

  • ParsingException for ALTER TABLE against a table that is unknown to the connector. DBZ-3612

  • Oracle connector continually logging warnings about already processed transactions. DBZ-3616

  • StringIndexOutOfBoundsException thrown while handling UTF-8 characters DBZ-3618

  • DDL ParsingException - "SUPPLEMENTAL LOG DATA (UNIQUE INDEX) COLUMNS" DBZ-3619

  • Oracle transaction reconciliation fails to lookup primary key columns if UPDATE sets columns to only NULL DBZ-3631

  • Oracle DDL parser fails on CREATE TABLE: mismatched input 'maxtrans' expecting {'AS', ';'} DBZ-3641

  • Antlr version mismatch DBZ-3646

  • SQL Agent does not start in SqlServer image when deployed to openshift DBZ-3648

  • Java UBI image is lacking gzip utility DBZ-3659

Other changes

  • Upgrade to Apicurio Registry 2.0 DBZ-3171

  • Vitess: rename "master" branch to "main" DBZ-3275

  • Formatting updates to correct errors in documentation builds DBZ-3518

  • Prepare test-suite for Kafka on RHEL DBZ-3566

  • Upgrade to Quarkus 2.0.0.Final DBZ-3602

  • Some dependencies are broken in ocp testsuite after BOM introduction DBZ-3625

  • Handle updated json schema for connector passwords DBZ-3637

  • MySQL SourceInfo should be public DBZ-3638

  • Change CLOB/BLOB data type support to an opt-in feature DBZ-3645

  • Denote BLOB support as incubating DBZ-3651

Release 1.6.0.Beta2 (June 10th, 2021)

Kafka compatibility

This release has been built against Kafka Connect 2.7.0 and has been tested with version 2.7.0 of the Kafka brokers. See the Kafka documentation for compatibility with other versions of Kafka brokers.

Upgrading

Before upgrading any connector, be sure to check the backward-incompatible changes that have been made since the release you were using.

When you decide to upgrade one of these connectors to 1.6.0.Beta2 from any earlier versions, first check the migration notes for the version you’re using. Gracefully stop the running connector, remove the old plugin files, install the 1.6.0.Beta2 plugin files, and restart the connector using the same configuration. Upon restart, the 1.6.0.Beta2 connectors will continue where the previous connector left off. As one might expect, all change events previously written to Kafka by the old connector will not be modified.

If you are using our docker images then do not forget to pull them fresh from Docker registry.

Breaking changes

There are no breaking changes in the release.

New Features

  • Clarification on MySQL vs MariaDb Usage DBZ-1145

  • Pravega sink for Debezium Server DBZ-3546

  • Postgres - Column default values are not extracted DBZ-2790

  • Add support for snapshot.include.collection.list DBZ-3062

  • Apply filters with empty filter changes 'Exclude' selection to 'Include' DBZ-3102

  • Adjust OpenShift tests to support new version of Strimzi CRDs DBZ-3475

  • Remove SchemaProcessor From Cassandra Connector DBZ-3506

  • Provide a snapshot.locking.mode option for Oracle DBZ-3557

  • Implement support for JSON function in MySQL parser DBZ-3559

Fixes

  • AbstractConnectorTest should work in environment with longer latency DBZ-400

  • PostgreSQL connector task fails to resume streaming because replication slot is active DBZ-3068

  • SQL Server connector buffers all CDC events in memory if more than one table is captured DBZ-3486

  • SQLServer low throughput tables increase usage of TempDB DBZ-3515

  • Incorrectly identifies primary member of replica set DBZ-3522

  • Cannot enable binlog streaming when INITIAL_ONLY snapshot mode configured DBZ-3529

  • Connector CRD name and database.server.name cannot use the same value in OCP test-suite DBZ-3538

  • SelectLobParser checks for lowercase "is null" instead of uppercase "IS NULL" DBZ-3545

  • DDL ParsingException "mismatched input 'sharing'" for create table syntax. DBZ-3549

  • DDL ParsingException on alter table DBZ-3554

  • ORA-00310 when online redo log is archived and replaced by redo log with new sequence DBZ-3561

  • Server name pattern is unnecessarily restrictive DBZ-3562

  • ORA-01289 error encountered on Oracle RAC when multiple logs are mined with same sequence number DBZ-3563

  • MySQL metrics documentation refers to legacy implementation DBZ-3572

  • Update downstream MySQL doc to reference streaming metrics vs. binlog metrics DBZ-3582

  • No viable alternative at input "add COLUMN optional" DBZ-3586

  • NPE when OracleValueConverters get unsupported jdbc type DBZ-3587

  • SelectLobParser throws NullPointerException when parsing SQL for an unknown table DBZ-3591

  • Pulsar sink tries to convert null key to string DBZ-3595

  • Oracle RAC URL does not correctly substitute node IP addresses DBZ-3599

  • Oracle Connector - got InputMismatchException mismatched input 'CASCADE' expecting {'AS', 'PURGE', ';'} DBZ-3606

Other changes

  • Unsupported column types should be ignored as with other connectors DBZ-814

  • Make outbox extensions dependency on tracing extension optional DBZ-2834

  • Avoid copying in DML handling DBZ-3328

  • Document impact of using --hostname when starting Connect container DBZ-3466

  • Update external link to AMQ Streams documentation DBZ-3502

  • Update external links in downstream docs to AMQ Streams deployment information DBZ-3525

  • Debezium Server Core builds plugin artifact DBZ-3542

  • List contributors script fails when name contains a "/" character DBZ-3544

  • Upgrade to Quarkus 2.0.0.CR3 DBZ-3550

  • Reduce DB round-trips for LOB handling DBZ-3556

  • Oracle benchmark does not execute LogMiner parser performance tests DBZ-3560

  • Clarify purpose of database.history.retention.hours DBZ-3565

  • Improve documentation related to signalling table DDL DBZ-3568

  • cassandra-driver-core 3.5.0 managed in Debezium BOM too old for testcontainers 1.15.3 DBZ-3589

  • Remove some dead code in Postgres connector DBZ-3596

  • Debezium server sink oracle database to pulsar without default namespace "public/default" DBZ-3601

  • Document OffsetContext.incrementalSnapshotEvents() DBZ-3607

  • Database skipping logic isn’t correct DBZ-3608

Release 1.6.0.Beta1 (May 20th, 2021)

Kafka compatibility

This release has been built against Kafka Connect 2.7.0 and has been tested with version 2.7.0 of the Kafka brokers. See the Kafka documentation for compatibility with other versions of Kafka brokers.

Upgrading

Before upgrading any connector, be sure to check the backward-incompatible changes that have been made since the release you were using.

When you decide to upgrade one of these connectors to 1.6.0.Beta1 from any earlier versions, first check the migration notes for the version you’re using. Gracefully stop the running connector, remove the old plugin files, install the 1.6.0.Beta1 plugin files, and restart the connector using the same configuration. Upon restart, the 1.6.0.Beta1 connectors will continue where the previous connector left off. As one might expect, all change events previously written to Kafka by the old connector will not be modified.

If you are using our docker images then do not forget to pull them fresh from Docker registry.

Breaking changes

RENAME TABLE statement with multiple tables now emits multiple schema change events, one for each of the renamed tables (DBZ-3399).

New Features

  • Support ad hoc snapshots on MySQL connector DBZ-66

  • Support DDL operations DBZ-2916

  • Add support for RAW, LONG, LONG RAW, BLOB, and CLOB data types DBZ-2948

  • Update Doc For Cassandra Connector DBZ-3092

  • Document log.mining.strategy for Oracle connector DBZ-3393

  • Update DOC with the new NUM_OF_CHANGE_EVENT_QUEUES parameter DBZ-3480

  • Use date format model that does not depend on client NLS settings in integration tests DBZ-3482

  • Provide Japanese translation of README.md DBZ-3503

  • Better handling of invalid SQL Server connector configuration DBZ-3505

  • Allow table.include.list and table.exclude.list to be updated after a connector is created DBZ-1263

  • Allow retry when SQL Server is down temporarily DBZ-3339

Fixes

  • Database name should not be converted to lower case if tablenameCaseInsensitive=True in Oracle Connector DBZ-2203

  • Not able to configure Debezium Server via smallrye/microprofile environment variables DBZ-2622

  • Upgrading from debezium 1.2.2 to 1.4.0 stopped snapshotting new tables DBZ-2944

  • oracle logminer cannot add duplicate logfile DBZ-3266

  • Oracle connector does not correctly handle partially committed transactions DBZ-3322

  • Data loss when MongoDB snapshot take longer than the Oplog Window DBZ-3331

  • First online log query does not limit results to those that are available. DBZ-3332

  • Connector crashing after running for some time DBZ-3377

  • Broken links in downstream Monitoring chapter DBZ-3408

  • Broken links in User guide table of routing SMT configuration options DBZ-3410

  • Broken link to basic configuration example in downstream content-based routing topic DBZ-3412

  • Cassandra connector does not react on schema changes properly DBZ-3417

  • Debezium mapped diagnostic contexts doesn’t work DBZ-3438

  • source.timestamp.mode=commit imposes a significant performance penalty DBZ-3452

  • Timezone difference not considered in LagFromSourceInMilliseconds calculation DBZ-3456

  • "Found null value for non-optional schema" error when issuing TRUNCATE from Postgres on a table with a PK DBZ-3469

  • Connector crashes when table name contains '-' character DBZ-3485

  • Kafka Clients in Debezium Server is not aligned with Debezium Kafka version DBZ-3498

  • ReadToInsertEvent SMT needs to set ConfigDef DBZ-3508

  • Debezium configuration can be modified after instantiation DBZ-3514

  • Oracle redo log switch not detected when using multiple archiver process threads DBZ-3516

  • Cannot enable binlog streaming when INITIAL_ONLY snapshot mode configured DBZ-3529

  • Missing schema function in DDL Parser DBZ-3543

  • Retry logic for "No more data to read from socket" is too strict DBZ-3472

Other changes

  • Document new source block and fix formatting issues DBZ-1614

  • Re-connect after "too many connections" DBZ-2300

  • Modularize doc for MongoDB component DBZ-2334

  • Rebase Postgres snapshot modes on exported snapshots DBZ-2337

  • Enable continuous JFR recording DBZ-3082

  • Remove deprecated Oracle connector option "database.tablename.case.insensitive" DBZ-3240

  • Improve Oracle redo logs query to avoid de-duplication step DBZ-3256

  • Migrate Jenkins CI to OCP 4.0 in PSI cloud DBZ-3396

  • Remove Antlr-based DML Parser DBZ-3400

  • Update Oracle driver version DBZ-3460

  • Incremental snapshot follow-up tasks DBZ-3500

  • Unnecessary NPE due to autoboxing DBZ-3519

  • Upgrade actions/cache to v2 version for formatting check DBZ-3520

  • Improve documentation for Oracle supplemental logging requirements DBZ-3521

  • SignalsIT leave table artifacts that cause other tests to fail DBZ-3533

  • Mark xstream dependency as provided DBZ-3539

  • Add test for Oracle table without PK DBZ-832

Release 1.6.0.Alpha1 (May 6th, 2021)

Kafka compatibility

This release has been built against Kafka Connect 2.7.0 and has been tested with version 2.7.0 of the Kafka brokers. See the Kafka documentation for compatibility with other versions of Kafka brokers.

Upgrading

Before upgrading any connector, be sure to check the backward-incompatible changes that have been made since the release you were using.

When you decide to upgrade one of these connectors to 1.6.0.Alpha1 from any earlier versions, first check the migration notes for the version you’re using. Gracefully stop the running connector, remove the old plugin files, install the 1.6.0.Alpha1 plugin files, and restart the connector using the same configuration. Upon restart, the 1.6.0.Alpha1 connectors will continue where the previous connector left off. As one might expect, all change events previously written to Kafka by the old connector will not be modified.

If you are using our docker images then do not forget to pull them fresh from Docker registry.

Breaking changes

Debezium now requires Java 11 as a build and a runtime environment (DBZ-2875). The only exception is Debezium Cassandra connector that still uses Java 8.

New Features

  • Sink adapter for Apache Kafka DBZ-3382

  • Optimisation on MongoDB and MySQL connector for skipped.operations DBZ-3403

  • Incremental snapshotting DBZ-3473

Fixes

  • io.debezium.text.ParsingException: no viable alternative at input 'IDNUMBER(4)GENERATEDBY' DBZ-1721

  • SKIPPED_OPERATIONS is added to CommonConnectorConfig.CONFIG_DEFINITION although it’s not implemented in all connectors DBZ-2699

  • Snapshot fails when reading TIME, DATE, DATETIME fields in mysql from ResultSet DBZ-3238

  • Update to fabric8 kube client 5.x DBZ-3349

  • An exception in resolveOracleDatabaseVersion if system language is not English DBZ-3397

  • Change strimzi branch in jenkins openshift-test job to main DBZ-3404

  • Broken link in downstream Monitoring chapter 7.3 DBZ-3409

  • Broken link in content-based routing chapter to page for downloading the SMT scripting archive DBZ-3411

  • LogMinerDmlParser mishandles double single quotes in WHERE clauses DBZ-3413

  • Incorrectly formatted links in downstream automatic topic creation doc DBZ-3414

  • SMT acronym incorrectly expanded in Debezium User Guide DBZ-3415

  • MariaDB — support privilege DDL in parser DBZ-3422

  • Change oc apply in jenkins openshift-test job to oc create DBZ-3423

  • SQL Server property (snapshot.select.statement.overrides) only matches 1st entry if comma-separated list also contains spaces DBZ-3429

  • Permission issue when running docker-compose or docker build as user not having uid 1001 DBZ-3453

  • no viable alternative at input 'DROP TABLE IF EXISTS group' (Galera and MariaDB) DBZ-3467

  • Debezium MySQL connector does not process tables with partitions DBZ-3468

  • The building tools' version in README doc is outdated DBZ-3478

  • MySQL DATE default value parser rejects timestamp DBZ-3497

  • MySQL8 GRANT statement not parsable DBZ-3499

Other changes

  • Config validation for Db2 DBZ-3118

  • Add smoke test for UI DBZ-3133

  • Create new metric "CapturedTables" DBZ-3161

  • Handle deadlock issue for MySql build stuck for 6h DBZ-3233

  • Document using Connect REST API for log level changes DBZ-3270

  • User Guide corrections for SQL Server connector DBZ-3297

  • User Guide corrections for Db2 connector DBZ-3298

  • User Guide corrections for MySQL connector DBZ-3299

  • User Guide corrections for MongoDB connector DBZ-3300

  • Allow building the Oracle connector on CI DBZ-3365

  • Add tests for Protobuf Converter DBZ-3369

  • Use current SQL Server container image for testing and examples DBZ-3379

  • Reword prereq in downstream SQL Server connector doc DBZ-3392

  • Duplicate entry in MySQL connector properties table for mysql-property-skipped-operations DBZ-3402

  • Docs clarification around tombstone events DBZ-3416

  • Validate logical server name contains only alpha-numerical characters DBZ-3427

  • Provide a "quick" build profile DBZ-3449

  • Avoid warning about superfluous exclusion during packaging DBZ-3458

  • Upgrade binlog client DBZ-3463