This release shifts the Java requirements needed to both build and run Debezium. In addition, this release requires a later version of Maven to build Debezium from source.

All Debezium connectors require a runtime baseline of Java 17.

If you are using Debezium Server, Operator, or the Quarkus Outbox Extension, a runtime baseline of Java 21 is required.

If you intend to build Debezium from source, all Debezium projects require Java 21 and Maven 3.9.8 or later. When building Debezium from source using a Java version before Java 21, the build will fail, reporting that Java 21 or later is required.

Please see the following chart, showing an at-a-glance view of the Maven and Java requirements by component:

This release moves to Kafka 3.8 as our baseline for testing and building Debezium. Kafka 3.8 changed a number of internal APIs that needed to be adapted for Debezium’s use (DBZ-8105).

For most users, this change has no impact; however, if you are extending Debezium, it’s important to be aware of these changes.

In Debezium 2.4, the additional-condition field in the incremental snapshot signal payload was deprecated, replaced with the new additional-conditions property, allowing the specification for conditions by table. In this release, the old additional-condition field has been removed and is no longer supported (DBZ-8278). Please be sure to update your scripts, workflows, or documentation that may have referred to this old, deprecated field.

Debezium will now begin to track metrics based on the individual create, update, and delete operations performed per relational table. For some connectors such as PostgreSQL and Oracle, these new detailed metrics also track the truncate operations performed per relational table. This can be quite useful for situations where you need to detect specific mutation patterns or where you may want to integrate analytics or observability stacks where this detailed information could be valuable to identifying problems.

For users upgrading to Debezium 3, these new metrics are captured automatically. They are exposed using a map-based pattern of Map<String, Long> where the key is the table name and the value is the number of events observed. The new metrics names are NumberOfCreateEventsSeen, NumberOfDeleteEventsSeen, NumberOfUpdateEventsSeen, and NumberOfTruncateEventsSeen (DBZ-8035).

Debezium 3 shifts its baseline support for MariaDB for the most recent non-rolling release, 11.4.3 (DBZ-8226). We are also closely monitoring the MariaDB 11.6 release cycle and plan to introduce vector data type support when MariaDB 11.6 becomes stable.

Debezium introduced its first sink-based connector in Debezium 2.2, just over a year ago, and we’re pleased to announce the inclusion of another sink-based connector for MongoDB as a part of Debezium 3.

Unlike the JDBC sink relational connector that requires an additional plug-in to be installed to use it, the MongoDB sink connector is bundled alongside the MongoDB source connector in the same artifact. So if you have already installed or use the MongoDB source connector and are using Debezium 3 or later, you also have the MongoDB sink connector.

The configuration to get started with the MongoDB is quite straightforward, here’s an example:

The connection.string and sink.database configuration properties are mandatory. These define the details for connecting to the target MongoDB database and the name of the target database where the changes will be written.

Additionally, the topics configuration property is mandatory by Kafka Connect, and it describes a comma-separated list of regular expressions for the topics that the sink connector will observe.

Oracle unveiled the first innovation release of MySQL 9.0 on July 1st, 2024. We are pleased to announce that we’ve tested and verified that MySQL 9.0 works and is supported starting with Debezium 3.0 (DBZ-8030). If you experience any issues or problems, please be sure to open an issue.

One of the newest features being added to relational databases is the introduction of vector data types. In addition to support for MySQL 9.0, Debezium 3 also introduces support for the new VECTOR(n) data type, which supports a list of floating-point values that can be expressed as a binary or list-formatted string. More information is available in the MySQL documentation about the vector data type (DBZ-8157).

In addition, the MySQL grammar has been updated to reflect support for the new MySQL 9.0 vector functions (DBZ-8210). More information about these functions are also in the MySQL documentation.

Several deprecated configuration properties have been removed:

Please be sure to update your Oracle connector configuration when using the deprecated configuration options to retain old behavior (DBZ-8181).

The default log.mining.strategy value has changed and is now online_catalog. As a vast majority of users typically use this strategy, and it generally performs better than redo_log_catalog, we felt this change made since in Debezium 3. If your deployments were previously relying on the default redo_log_catalog strategy, you will need to explicitly add log.mining.strategy to the connector configuration and specify the value redo_log_catalog when upgrading (DBZ-3656).

Debezium 3 introduces as new Oracle connector transaction buffer implementation, based on Ehcache to provide off-heap storage of transaction processing and event data. This new implementation adds to the existing Java Heap, Infinispan Embedded, and Infinispan Remote buffer types.

To begin taking advantage of the Ehcache implementation, the log.mining.buffer.type must be set to ehcache. By default, the buffer type is memory to use the JVM’s heap for optimal performance.

In order to for the Ehcache library to start successfully, several additional configurations must be provided to explicitly configure the caches maintained by the cache manager. These new configuration options are:

Debezium creates the Ehcache configuration using XML, so each of these configurations provide XML snippets.

The global configuration is optional, and allows you to provide details about persistence and other Ehcache attributes, excluding specifying <cache> or <default-serializers> tags, which are handled separately. The other individual cache configurations are meant to supply the inner XML bits of a <cache> configuration tag, excluding its <key-type> and <value-type>, which are managed directly by Debezium.

In this example, Ehcache will maintain a combination of heap and off-heap storage for the caches, maintaining at most 256 entries in heap at all times and flushing to disk. The disk caches will be stored at the relative path ./data. This implies that you will need a persistent storage volume available when using disk-based caches.

This is a new feature and is experimental, so we would love your feedback on how we can improve this (DBZ-7758).

In recent improvements to the Oracle RAC node flush strategy, it was determined that a three-second delay was being forced when an Oracle RAC node was taken offline by the database administrator. Since an Oracle RAC node cannot perform any writes to the redo logs while offline, this three-second delay introduced an unnecessary amount of latency while the node remained offline.

In Debezium 3, the three-second delay is only imposed if a connection is active to an Oracle RAC node; however, the flush SQL operation was unsuccessful. This means that when database administrators take RAC nodes offline for maintenance, no latency overhead will be imposed by the connector (DBZ-8177).

Oracle extended strings is a feature that allows the traditional 4000 byte limit on character data to be raised to 32K. This is done by applying a database upgrade to set the database parameter max_string_size to EXTENDED. The extended string feature then allows using the same SQL syntax used for 4000 byte or smaller character data to be used for character data up to 32K without forcing you to use CLOB-based operations.

With Debezium 3, you can now use the Oracle connector with databases that use extended strings and capture the changes directly from the transaction logs (DBZ-8039). As extended strings are effectively CLOB operations on the database level, mining such column types require setting lob.enabled to true.

As this new feature is experimental, we’d love to hear any feedback from the community!

In some cases, Oracle users may define tables with a CLOB or BLOB as required, using the EMPTY_BLOB() or EMPTY_CLOB() function to define a default when the field isn’t supplied. In previous builds, these special functions were not evaluated by Debezium, and such columns would have been emitted as optional rather than not optional.

Starting with Debezium 3, when an EMPTY_BLOB() or EMPTY_CLOB() default value is specified, the field will be emitted as not optional. Additionally, the field contain the appropriate default value, an empty byte array or an empty string respectively (DBZ-8248).

When the PostgreSQL connector is first deployed, one of its first tasks is to create a replication slot in the database if it doesn’t already exist. The replication slot is pivotal to how the connector works and facilitates the capture and dispatch of changes to Debezium. Unfortunately, there are some database operations that will block the creation of replication slots, such as in-progress transactions, forcing the connector to block indefinitely while waiting for the transaction to conclude. For short-lived transactions, this isn’t generally a concern; however, for long-running transactions that’s an entirely different situation.

In order to improve this experience, a new internal option was added, internal.create.slot.command.timeout, which defaults to 90 seconds. If the creation of the replication slot does not complete within 90 seconds, it will retry up to slot.max.retries. Once the retries are exhausted, the connector will throw an unrecoverable error (DBZ-8073).

The pgvector extension introduces vector search functionality for PostgreSQL. There are three data types this extension introduces: vector, halfvec, and sparsevec.

In Debezium 3, all three data types will be streamed like any other data type. Each data type is emitted based on the following semantic mappings:

There is no additional configuration required after enabling the pgvector extension in your database. Please see the documentation for more details on the semantic mappings (DBZ-8121).

PostgreSQL is unique in that you can implement the Outbox pattern without creating an outbox table, by writing logical messages directly into the WAL using pg_logical_emit_message. The unfortunate part is that this data is then sent to Kafka as a series of bytes, which may not always be ideal for consumers who may be looking for structured messages.

Debezium 3 introduces a new PostgreSQL-specific transform called DecodeLogicalDecodingMessageContent. This transform is specifically meant to convert the pg_logical_emit_message event bytes to a structured event payload that consumer applications are capable of understanding.

Given the following configuration:

The event’s value of an event written using pg_logical_emit_message before the transform would be:

After applying the transformation, the event’s value now looks like:

So you can safely implement the Outbox pattern without the physical outbox table! (DBZ-8103).

A longstanding enhancement for snapshot isolation support for PostgreSQL is now here! A new connector configuration property, snapshot.isolation.mode, allows the connector to control the consistency used while executing the initial and ad-hoc blocking snapshot steps. There are four isolation levels: serializable (the default), repeatable_read, read_committed, and read_uncommitted. You can find details about these isolation levels and how they work with PostgreSQL in the documentation (DBZ-1252).

The ReselectPostProcessor is a useful tool to handle populating change events that contain TOAST columns (the oversized-attribute storage technique). By default, when a TOAST column is found and is not mutated by the SQL operation, Debezium populates these fields with placeholders, indicating that the value wasn’t provided, but also wasn’t changed. A host of data types use this storage mechanism, including int/bigint arrays. With Debezium 3, these int/bigint array data types can be reselected by the post processor so that these fields are always populated, even when they’re not changed in the SQL operation (DBZ-8212).

The JMX signaling and notifications for SQL Server did not work correctly when a connector was configured with multiple databases spawning multiple tasks. To resolve this issue, it was necessary to change the naming of signalling and notification MBean names to make sure they are unique per task (DBZ-8137).

The JDBC sink repository has been relocated from debezium-connector-jdbc to debezium main repository (DBZ-8008). With the introduction of the MongoDB sink connector in Debezium 3, this allows the team to easily share common contracts across our sink connectors.

Moving forward, to raise pull requests for the JDBC sink, please use the main Debezium repository, as the old repository has been archived and is only read-only.

The JDBC sink uses a set of buffers to improve the throughput writes to the target database. In some use cases, the flush operation of these buffers may face specific exceptions due to locks due to other applications that may have locked a specific row or table. To improve the user experience, two new configuration properties have been added:

The retry feature is enabled by default, attempting to retry up to a maximum 5 attempts, with a 1-second delay between retries. If you prefer retries disabled, setting flush.failure.max.retries to 0 would disable this feature (DBZ-7291).

In prior releases of Debezium Server, there were a finite number of converters that could be used for headers, keys, and values. These included Json, JsonByteArray, CloudEvents, Avro, Protobuf, Binary, and SimpleString. While these often satisfied a vast majority of use cases, it’s not uncommon that someone may have a unique requirement specific to their environment that is outside these options.

In this release, a new ClientProvided converter option has been added, which allows for extending the header, key, and value converters with a custom, user-supplied implementation (DBZ-8040).

The Kafka sink adapter will now log the record key when Debezium fails to send the record to the Kafka broker. This is useful to know what specific record was a problem without necessarily needing to increase the logging verbosity of the runtime (DBZ-8282).

The Google Spanner database introduced support for a 32-bit float data type. The Debezium Google Spanner connector has been adjusted to support this new data type (DBZ-8043).

In Vitess, it is possible for a keyspace to have shards that have no tablets. Debezium for Vitess has improved working with this use case, and now gracefully handles such a keyspace without fault (DBZ-8053).

A new Vitess connector configuration property has been added to control whether epochs of a new shard, after a re-shard operation, inherits epochs from its parent shard. This new configuration property, vitess.inherit.epoch, defaults to false and isn’t enabled by default (DBZ-8163).