Featured Posts

Tags
apache kafka apache kafka apicurio avro aws batch caassandra camel cassandra cdc channels community community stories connectors containers cqrs custom datalake db2 ddd debezium debezium server debezium ui deduplication discussion docker elasticsearch event sourcing exactly once semantics example examples features fedora flink hiring ibmi iceberg images informix integration introduction jaeger jdbc json kafka kafka streams kafka streams kogito ksql kubernetes lakehouse machine learning mariadb microservices mongo mongodb mysql news newsletter notifications online learning operator oracle outbox performance postgres presentation production quarkus questdb rds releases schema scylla secrets sentry serialization signaling smt snapshots spanner spark sql sqlserver tensorflow testcontainers tests time series timescaledb topics tracing transactions ui vagrant vitess website

The community is in the homestretch toward the next major milestone for Debezium, and we’re happy to announce the availability of Debezium 3.0.0.CR1!

Beyond a number of bugfixes to connectors, this release also brings several improvements for MySQL, Oracle, and the Vitess connectors. Unless any major regressions show up, we can expect Debezium 3 in the not too distant future.

Breaking changes

The team aims to avoid any potential breaking changes between minor releases; however, such changes are sometimes inevitable.

PostgreSQL

Vector data type names introduced for PostgreSQL were too database specific. In order to provide a more generic support for vector data types, the names were adjusted to support PostgreSQL and MySQL (DBZ-8183).

Oracle

Several deprecated configuration properties have been removed:

  • log.mining.transaction.retention.hours replaced by log.mining.transaction.retention.ms

  • log.mining.archive.destination.name replaced by archive.destination.name

  • log.mining.archive.log.hours replaced by archive.log.hours

Please be sure to update your Oracle connector configuration when using the deprecated configuration options to retain old behavior (DBZ-8181).

The default log.mining.strategy value has changed and is now online_catalog. As a vast majority of users typically use this strategy, and it generally performs better than redo_log_catalog, we felt this change made since in Debezium 3. If your deployments were previously relying on the default redo_log_catalog strategy, you will need to explicitly add log.mining.strategy to the connector configuration and specify the value redo_log_catalog when upgrading (DBZ-3656).

SQL Server

The JMX signaling and notifications for SQL Server did not work correctly when a connector was configured with multiple databases spawning multiple tasks. To resolve this issue, it was necessary to change the naming of signalling and notification MBean names to make sure they are unique per task (DBZ-8137).

New features and improvements

Debezium 3.0.0.CR1 also introduces several improvements and features, lets take a look at each individually.

MySQL 9

Oracle unveiled the first innovation release of MySQL 9.0 on July 1st, 2024. We are pleased to announce that we’ve tested and verified that MySQL 9.0 works and is supported starting with Debezium 3.0 (DBZ-8030). If you experience any issues or problems, please be sure to open an issue.

MySQL Vector data types

One of the newest features being added to relational databases is the introduction of vector data types. In addition to support for MySQL 9.0, Debezium 3 also introduces support for the new VECTOR(n) data type, which supports a list of floating-point values that can be expressed as a binary or list-formatted string. More information is available in the MySQL documentation about the vector data type (DBZ-8157).

In addition, the MySQL grammar has been updated to reflect support for the new MySQL 9.0 vector functions (DBZ-8210). More information about these functions are also in the MySQL documentation.

MariaDB 11.4.3

Debezium 3 also introduces support for the most recent non-rolling release of MariaDB, 11.4.3 (DBZ-8226). We are also closing monitoring the MariaDB 11.6 release cycle and plan to introduce vector data type support when MariaDB 11.6 becomes stable.

Oracle offline RAC node flush improvements

In recent improvements to the Oracle RAC node flush strategy, it was determined that a three-second delay was being forced when an Oracle RAC node was taken offline by the database administrator. Since an Oracle RAC node cannot perform any writes to the redo logs while offline, this three-second delay introduced an unnecessary amount of latency while the node remained offline.

In Debezium 3, the three-second delay is only imposed if a connection is active to an Oracle RAC node; however, the flush SQL operation was unsuccessful. This means that when database administrators take RAC nodes offline for maintenance, no latency overhead will be imposed by the connector (DBZ-8177).

Vitess inherit shard epoch

A new Vitess connector configuration property has been added to control whether epochs of a new shard, after a re-shard operation, inherits epochs from its parent shard. This new configuration property, vitess.inherit.epoch, defaults to false and isn’t enabled by default (DBZ-8163).

Other changes

Altogether, 45 issues were fixed in this release. Here are a list of some additional noteworthy changes:

  • Make ORA-00600 - krvrdccs10 automatically retriable DBZ-5009

  • Incremental snapshot fails with NPE if surrogate key doesn’t exist DBZ-7797

  • MySQL 8.4 incompatibility due to removed SQL commands DBZ-7838

  • Postgres connector - null value processing for "money" type column. DBZ-8027

  • Using snapshot.include.collection.list with Oracle raises NullPointerException DBZ-8032

  • Performance degradation when reconstructing (log.mining.stragtegy hybrid mode) DBZ-8071

  • Documentation for signals provides incorrect data-collection format for some connectors DBZ-8090

  • The source data type exceeds the debezium data type and cannot deserialize the object DBZ-8142

  • Refactor engine signal support DBZ-8160

  • Incorrect use of generic types in tests DBZ-8166

  • Latest Informix JDBC Driver DBZ-8167

  • Postgres JSONB Fields are not supported with Reselect Post Processor DBZ-8168

  • upgrade Adobe s3mock to version 3.10.0 DBZ-8169

  • Include Jackson libraries to JDBC connector Docker image distribution DBZ-8175

  • Ehcache fails to start, throwing "Invaild XML Configuration" DBZ-8178

  • Enable snapshot.database.errors.max.retriesEnable during Oracle tests DBZ-8184

  • Change event for a logical decoding message doesn’t contain transaction field DBZ-8185

  • Add MariaDB connector server distribution DBZ-8186

  • NullPointerException (schemaUpdateCache is null) when restarting Oracle engine DBZ-8187

  • XStream may fail to attach on retry if previous attempt failed DBZ-8188

  • Truncate byte buffer should return a new array DBZ-8189

  • Update Vitess example to Debezium 2.7/Vitess 19 DBZ-8196

  • OracleConnectorIT test shouldGracefullySkipObjectBasedTables can timeout prematurely DBZ-8197

  • Exclude Oracle 23 VECSYS tablespace from capture DBZ-8198

  • AbstractProcessorTest uses an incorrect database name when run against Oracle 23 Free edition DBZ-8199

  • Reduce log verbosity of OpenLogReplicator SCN confirmation DBZ-8201

  • Support for older MongoDb versions DBZ-8202

  • DDL statement couldn’t be parsed: REVOKE IF EXISTS DBZ-8209

  • System testsuite fails with route name being too long DBZ-8213

  • Oracle TableSchemaBuilder provides wrong column name in error message DBZ-8217

  • Using ehcache in Kafka connect throws an XMLConfiguration parse exception DBZ-8219

  • Implement separate source and sink connector sections in documentation navigation DBZ-8220

  • OcpJdbcSinkConnectorIT fails DBZ-8228

  • Container image does not install correct apicurio deps DBZ-8230

  • Add information about helm chart installation to operator readme DBZ-8233

A huge thank you to all contributors from the community who worked on this release: Ajit, Chris Cranford, Jakub Cechacek, Jiri Pechanec, Jochen Schalanda, Lars M. Johansson, Mario Fiore Vitale, Minh Son Nguyen, Mohamed El Shaer, Ondrej Babec, Oren Elias, René Kerner, Robert Roldan, Roman Kudryashov, Sean C. Sullivan, Sergei Kazakov, Sylvain Marty, Thomas Thornton, Tudor Plugaru, Vojtěch Juránek, and moyq5!

Chris Cranford

Chris is a software engineer at Red Hat. He previously was a member of the Hibernate ORM team and now works on Debezium. He lives in North Carolina just a few hours from Red Hat towers.

   

About Debezium

Debezium is an open source distributed platform that turns your existing databases into event streams, so applications can see and respond almost instantly to each committed row-level change in the databases. Debezium is built on top of Kafka and provides Kafka Connect compatible connectors that monitor specific database management systems. Debezium records the history of data changes in Kafka logs, so your application can be stopped and restarted at any time and can easily consume all of the events it missed while it was not running, ensuring that all events are processed correctly and completely. Debezium is open source under the Apache License, Version 2.0.

Get involved

We hope you find Debezium interesting and useful, and want to give it a try. Follow us on Twitter @debezium, chat with us on Zulip, or join our mailing list to talk with the community. All of the code is open source on GitHub, so build the code locally and help us improve ours existing connectors and add even more connectors. If you find problems or have ideas how we can improve Debezium, please let us know or log an issue.