Debezium 2.5 introduces a new configuration option, schema.history.internal.redis.max.attempts designed to limit the number of retry attempts while connecting to a Redis database when it becomes unavailable, previously it simply retried forever. This new option defaults to 10 but is user configurable. (DBZ-7120)

Debezium’s notification subsystem provides a simple way to integrate external processes and applications with certain Debezium tasks, such as taking a snapshot. In prior versions, the notifications for initial snapshots were quite basic and provided details such as when the snapshot started, when each table started and concluded, and finally when the snapshot ended.

A recent contribution that was part of DBZ-6878 extends on this base functionality to provide enhanced details about the snapshot. For example, the IN_PROGRESS notification will provide additional details about what tables are being captured and which table is currently in-progress, shown below:

Additionally, another initial snapshot notification called TABLE_SCAN_COMPLETED also provides similar details, as shown below:

For more details, please see the Notification documentation.

In some cases, because of the way that certain source databases function, when a Debezium connector emits a change event, the event might exclude values for specific column types. For example, values for TOAST columns in PostgreSQL, LOB columns in Oracle, or Extended String columns in Oracle Exadata, might all be excluded.

Debezium 2.5 introduces the ReselectColumnsPostProcessor providing a way to re-select one or more columns from a database table and fetch the current state. You can configure the post processor to re-select the following column types:

Configuring a PostProcessor is similar to configuring a CustomConverter or Transformation, except that it works on the mutable payload’s Struct rather than the SourceRecord.

The property incremental.snapshot.watermarking.strategy has been introduced to let users choose the watermarking strategy to use during an incremental snapshot.

The insert_insert (old behavior) approach lets Debezium creating two entries in the signaling data collection for each chunk during the snapshot to signal the opening of the snapshot window and another to mark its closure.

On the other hand, with the insert_delete option, a single entry is written in the signaling data collection for each chunk at the beginning of the window. After completion, this entry is removed, and no corresponding entry is added to signify the closure of the snapshot window. This method aids in more efficient management of the signaling data collection.

For more details, please see the Connector properties section of the connector of your interest.

Debezium has introduced several changes around large document processing in recent releases; however, those changes primarily focused on handling that use case with MongoDB 4 and 5. While these improvements certainly help for those older versions, the MongoDB community has introduced a way in MongoDB 6 to seamlessly deal with this at the database pipeline level.

Debezium 2.5’s MongoDB connector now uses the $changeStreamSplitLargeEvent aggregation feature, introduced as part of MongoDB 6.0.9. This avoids the BSONObjectTooLarge exception when working with documents that would exceed the 16MB document size limit of MongoDB. This new feature is controlled by the oversize.handling.mode option, which defaults to fail. Please adjust this configuration if you would like to take advantage of this new, opt-in feature. (DBZ-6726)

The MySQL community recently released a new innovation release, MySQL 8.2.0 at the end of October 2023. This new release has been tested with Debezium, and we’re happy to announce that we officially support this new innovation release. (DBZ-6873)

Several new high-precision timestamp fields have been included in several MySQL replication events. For example, in MySQL 8.0.1, there are to microsecond-resolution timestamps added to GTID events that specify the timestamp when the transaction was committed on the immediate primary and when committed on the original primary.

Debezium 2.5 now takes advantage of these values and will use these for the ts_ms fields if they’re available, falling back to the second-based precision if they are not or if you’re using a version of MySQL prior to 8.0.1 (DBZ-7183).

In PostgreSQL 16, you can now define replication slots on a stand-by instance. This enables a plethora of new options, including the ability to perform change data capture from a replica rather than your production system for load distribution, particularly in a very active database.

Debezium 2.5 now supports connecting to a stand-by PostgreSQL 16 server and streaming changes (DBZ-7181).

In previous builds of Debezium, there was a single Oracle streaming metrics bean that exposed all metrics options that spanned across all three streaming adapters. This often lead to some confusion about what metrics are applicable to which streaming adapter so we wanted to define a clear distinction in this case.

With Debezium 2.5, the Oracle streaming metrics beans have been split into three different implementations, one for each adapter type. For observability stacks, this change should be completely transparent unless you were previously gathering a metric for one adapter type while using another. In this case you’ll find that metric is no longer available.

Specifically for LogMiner users, several metrics have been renamed and the old metrics have been deprecated. While you will still be able to use the old metric names in Debezium 2.5, these are scheduled for removal in a future 2.7+ build. The metrics that were deprecated and renamed are as follows:

Debezium 2.5 aligns LOB behavior in snapshot and streaming. When lob.enabled is set to false, the unavailable value placeholder will be explicitly included during snapshot to match the behavior of streaming.

Debezium for SQL Server works by reading the changes captured by the database in what are called capture instances. These instances can come and go based on a user’s needs, and it can be difficult to know if Debezium has concluded its own capture process for a given capture instance.

Debezium 2.5 remedies this problem by emitting a new notification aggregate called Capture Instance, allowing any observer to realize when a capture instance is no longer in use by Debezium. This new notification includes a variety of connector details including the connector’s name along with the start, stop, and commit LSN values. (DBZ-7043)

SQL Serer 2019 introduced the ability to specify column-specific sensitivity classifications to provide better visibility and protections for sensitive data. Unfortunately, the current driver shipped with Debezium 2.4 and earlier does not support this feature. Debezium 2.5 introduces the latest 12.4.2 SQL Server driver so that users can now take advantage of this feature out of the box. (DBZ-7109)

Debezium first introduced the JDBC sink connector in March 2023 as a part of Debezium 2.2. Over the last several months, this connector has seen numerous iterations to improve its stability, feature set, and capabilities. Debezium 2.5 builds atop of those efforts, introducing batch-writes. (DBZ-6317)

In previous versions, the connector worked on each topic event separately; however, the new batch-write support mode will collect the events into buckets and write those changes to the target system using the fewest possible transaction boundaries as possible. This change increases the connector’s throughput capabilities and makes the interactions with the target database far more efficient.

Debezium 2.5 introduces a new JBDC sink feature where users can now specify which fields from the event payload are to be included or excluded from the target database write operation. This feature works just like any other include/exclude combination in the Debezium framework where these two properties are mutually exclusive.

As an example, if we have a simple event payload with the following fields in topic customers:

If we want to avoid writing the address field to the target database and only write the id and name fields to the target table, we can use this new feature to accomplish this. This can be done adding either a field.include.list or field.exclude.list property.

The format of the include/exclude for fields is [<topic-name>:]<field-name>, where the topic-name is optionally and can be omitted if you want to avoid writing the address field for all events. Please see the JDBC sink connector configuration documentation for more details.

The Debezium Server Operator for Kubernetes has been actively improved in this preview release of Debezium 2.5. Several improvements include:

As we continue to improve the Debezium Server Operator for Kubernetes, we’d love to get your feedback.

In previous versions of Debezium, it was not possible to use a service account named differently than the predefined one. This made the process a tad bit cumbersome for users because while you could grant roles and authorization to this predefined account separately, it meant you needed to use this predefined service account rather than one that you may already wish to use.

Debezium 2.5 simplifies this process, allowing you to now use your own, custom service account (DBZ-7111).

Debezium Server Kinesis users will be happy to note that there has been some reliability improvements with the sink adapter with Debezium 2.5. The Kinesis Sink will now automatically retry the delivery of a failed record up to a maximum of 5 attempts before the adapter triggers a failure. This should improve the sink adapter’s delivery reliability and help situations where a batch of changes may overload the sink’s endpoint. (DBZ-7032)

In earlier versions of Debezium Server, users could specify a fixed partition-id to stream all changes to a single partition or provide a static partition-key that will be set on all batch operations, which ultimately lends itself to streaming all changes to the same target partition. There are situations where this may be helpful, but it more often leads to a performance concerns for downstream processing.

Debezium 2.5 adjusts this behavior in order to improve performance. By default, when neither a partitionid or partitionkey is defined, the EventHub sink will send events using a round-robin technique to all available partitions. Events can be forced into a single, fixed partition by specifying a partitionid. Alternatively, the partitionkey can be provided to supply a fixed partition key that will be used to route all events to a specific partition.

If additional partition routing requirements are necessary, you can now combine the PartitionRouting SMT accomplish such tasks. For more details, please see the Event Hubs documentation.

RabbitMQ introduced Streams in version 3.9, which utilizes a fast and efficient protocol that can be combined with AMQP 0.9.1 to support large fan-outs, replay and time travel, and large data sets with extremely high throughput. Debezium 2.5 takes advantage of this new Streams implementation by introducing a new native Streams implementation (DBZ-6703). In order to get started with this new implementation, configure the Debezium Server sink as follows:

Additionally, if you need to pass any other connection parameters to the RabbitMQ connection, you can do so by adding those to the configuration with the prefix debezium.sink.rabbitmqstream.connection. to pass through any config properties.

Please see the Debezium Server RabbitMQ documentation for more details.

The Kafka sink behaviour can now be modified by a custom logic providing alternative implementations for specific functionalities. When the alternative implementations are not available then the default ones are used.

For more details, please see the Apache Kafka Injection points.

Amazon Simple Queue Service (Amazon SQS) is a distributed message queuing service. It supports programmatic sending of messages via web service applications as a way to communicate over the Internet. SQS is intended to provide a highly scalable hosted message queue that resolves issues arising from the common producer–consumer problem or connectivity between producer and consumer.

Debezium 2.5 offers the possibility to send events to Amazon SQS.

Thanks to the contribution from Lars Johansson, Debezium 2.5 introduces a new connector to its portfolio to gather changes from IBM Informix. IBM Informix is an embeddable, high-performance database for integrating SQL, NoSQL, JSON, time-series, and spatial data in one place. It’s designed for analytics at the edge, in the cloud, or on premise.

The IBM Informix connector is bundled just like any of our community lead connectors, it is available on Maven Central or you can download the plug-in archive from our Debezium 2.5 releases page.

The maven artifact coordinates are:

If you would like contribute to the Informix connector, we have added a new repository under the Debezium organization, debezium-connector-informix.

I’d like to thank Lars Johansson for this contribution and his collaboration with the team, kudos!

The community has leveraged the MySQL connector as an alternative to capture changes from MariaDB for quite some time now; however that compatibility was primarily best-case effort.

The Debezium 2.5 release stream aims to bring MariaDB to the forefront as a first-class connector by taking a very clear and methodological approach to incrementally check, validate, and eventually support MariaDB at the same capacity that we do MySQL. Our goal and hope is that we can do this within the scope of the MySQL connector proper; however, there is still quite a bit of ongoing investigation around GTID support that may influence the path forward.

This first preview build of Debezium 2.5 has taken the first step, we’ve verified that the code works against a single MariaDB database deployment, the test suite passes and we’ve addressed any changes needed with the Binlog client to support that deployment. Our next steps is to look into GTID support, which MariaDB supports but using an approach that isn’t compatible with MySQL.

Stay tuned for future builds as we continue to expand on this and we certainly welcome any early feedback.

Both MySQL and MariaDB support what is called Global Transaction Identifiers or GTIDs. These are used in replication to uniquely identify transaction(s) uniquely across a cluster. The implementation details between MySQL and MariaDB differ significantly and in earlier versions of Debezium, we only supported GTID with MySQL.

With Debezium 2.5, we are taking another step forward by introducing GTID support for MariaDB as part of the MySQL connector offering. In order to take advantage of this behavior, you will need to use the MariaDB driver rather than the MySQL driver by using a JDBC connection prefixed as jdbc:mariadb rather than jdbc:mysql. By doing this, you can now take full advantage of working with MariaDB and GTID just like MySQL (DBZ-1482).