An edge case was reported in DBZ-5996 where if a temporal column used ZonedTimestamp and if the column’s value had 0 micro or nanoseconds, rather than emitting the value as 2023-01-19T12:30:00.123000Z, the value would be emitted in a truncated way as 2023-01-19T12:30:00.123Z. This could lead to other issues with converters used in the event pipeline when the output from that column could be formatted inconsistently.

In order to remedy the edge case, the ZonedTimestamp implementation will now pad the fraction-based seconds value of the column’s value to the length/scale of the source database column. Using the example above of a TIMESTAMP(6) MySQL column type, the emitted value will now properly reflect a value of 2023-01-19T12:30:00.123000Z.

While this change in behavior is likely to have minimal impact to most users, we wanted to bring attention to it in the event that you’ve perhaps used other means to handle this edge case in your pipelines. If you have, you should be able to rely on Debezium to emit the value consistently, even when the fraction-based seconds is 0.

Debezium previously sanitized topic and schema names by using an underscore (_) to replace non-ASCII characters that would lead to unsupported topic or schema names when using schema registries. However, if this non-ASCII character was the only difference between two similar topics or schema names that otherwise only varied by case, this would lead to other problems.

In order to address this in the most compatible way, Debezium now uses a strategy-based approach to map characters uniquely. As a side effect of this change, the sanitize.field.names configuration property has been retired and replaced by this new strategy-based approach.

Each connector supports two configuration properties to control this behavior:

These two connector configuration properties support three modes:

This now allows you to pick the most appropriate strategy based on your table or collection naming convention.

All Debezium change events related to inserts, updates, and deletes contain a source info block in the event’s payload. For the Oracle connector, this block contains a special field called ssn that represents the SQL sequence number for this change.

It has been identified that there were corner cases where the value sourced from the database for this field could exceed the maximum value of 2,147,483,647, or the maximum value of an INT32 data type. To fix this corner case, we’ve changed the data type from INT32 to INT64, which allows up to a maximum value of 9,223,372,036,854,775,807.

This change should be entirely non-invasive, but we wanted to bring attention to this should you have pipelines that could be storing this value in a sink system or if you are using a schema registry.

Debezium Server is a standalone Quarkus-based runtime for Debezium source connectors enabling the integration with various platforms like EventHubs, PubSub, Pulsar, Redis, and Kafka, to name a few. With this release, we have moved the code related to Debezium Server to its own GitHub repository.

This change was required in order to support building Debezium Server to include connectors that are not part of the main Debezium repository, connectors such as Db2, Google Spanner, Cassandra 4, and Vitess. Therefore, this means that starting with this release, Debezium Server now ships with all connectors (excluding Cassandra 3) by default.

Debezium intends to sunset publishing container images to docker.io in June 2023. Some may be aware of recent policy changes at Docker around the reduction of their free organization plans, a plan that is used by a number of open-source projects including Debezium.

While Docker walked back their decision, this does raise a question about whether this could happen in the future. Debezium has been dual publishing container artifacts to docker.io and quay.io for quite some time, and we plan to continue doing so throughout this upcoming quarter with the preview releases of Debezium 2.3.

However, effective with the release of Debezium 2.3.0.Final at the end of June 2023, Debezium will cease publishing container artifacts to docker.io and will only be publishing container images moving forward to quay.io.

Jolokia is a JMX-HTTP bridge that provides an alternative to using JSR-160 to gather metrics. It is an agent based approach that improves traditional JMX by introducing unique features like bulk requests and fine-grained security policies.

With Debezium 2.2, the debezium/connect image now ships with Jolokia, but this agent isn’t enabled by default. In order to enable Jolokia support, the container must be started with ENABLE_JOLOKIA set to true. By default, Jolokia will bind to port 8778 when enabled.

In the event that a different port is required, Jolokia will need to be enabled differently. For example, in order to enable Jolokia using port 9779, do not set the ENABLE_JOLOKIA but instead configure the KAFKA_OPTS environment variable as follows:

By specifying the above environment variable, Jolokia’s JMX-HTTP bridge will be available on port 9779 of the container.

In previous releases of Debezium, the connector start-up phase used a fail-fast strategy. Simply put, this meant that if we couldn’t connect, authenticate, or performs any of the start-up phase steps required by the connector, the connector would enter a FAILED state.

One specific problem area for users is if the connector gracefully starts, runs for a period of time, and then eventually encounters some fatal error. If the error is related to a resource that wasn’t accessed during the connector’s start-up lifecycle, the connector would typically gracefully restart just fine. However, the situation is different if the problem was related to the database’s availability and the database was still unavailable during the connector’s start-up phase. In this situation, the connector would fail-fast, and would enter a FAILED state, requiring manual intervention.

The fail-fast approach served Debezium well over the years, but in a world where a resource can come and go without warning, it became clear that changes were needed to improve Debezium’s reliability and resiliency. While the Kafka Connect’s retry/back-off framework has helped in this regard, that doesn’t address the concerns with start-up resources being unavailable with how the code is currently written.

Debezium 2.2 changes this landscape, shifting how we integrate with Kafka Connect’s source connector API slightly. Instead of accessing potentially unavailable resources during the start-up lifecycle, we moved that access to a later phase in the connector’s lifecycle. In effect, the Debezium start-up code is executed lazily that accesses potentially unavailable resources, which allows us to take advantage of the Kafka Connect retry/back-off framework even during our start-up code. In short, if the database is still unavailable during the connector’s start-up, the connector will continue to retry/back-off if Kafka Connect retries are enabled. Only once the maximum number of retry attempts has been reached or a non-retriable error occurs will the connector task enter a FAILED state.

We hope this brings more reliability and resiliency for the Debezium experience, improving how errors are handled in an ever-changing landscape, and provides a solid foundation to manage connector lifecycles.

We have heard from the community on several occasions that it would great to have an out-of-the-box way to determine what values have changed in a Debezium change event. The new single message transform (SMT) ExtractChangedRecordState aims to deliver on this request by adding metadata to the event identifying which fields changed or were unchanged.

In order to get started with this new transformation, configure it as part of your connector configuration:

This transformation can be configured to disclose either what fields changed by setting header.changed, what fields are unchanged by setting header.unchanged, or both by setting both properties as shown above. The transformation will add a new header with the specified name, and it’s value will include a collection of field names based on whether you’ve configured changes, non-changes, or both.

The ExtractNewRecordState single message transformation is extremely useful in situations where you need to consume the Debezium change event in a flattened format. This SMT has been changed in this release to add the ability to drop fields from the payload and the message key of the event.

This new feature introduces three new configuration properties for the transformation:

These new configuration options allow for some exciting ways to manipulate change events. For example, to emit events with only changed fields, pairing the ExtractNewRecordState with the new ExtractChangedRecordState transformation makes this extremely simple and straightforward. An example configuration to only emit changed columns would look like the following:

The above configuration will explicitly not include unchanged fields from the event’s payload value. If a field in the key did not change, it will be unaffected because drop.fields.from.key was left as its default of false. And finally, if a field in the event’s payload is to be dropped because it did not change, but it’s not optional, it will continue to be included in the transformation’s output event to comply with schema compatibility.

Debezium’s relational database initial snapshot process has always been single-threaded. This limitation primarily stems from the complexities of ensuring data consistency across multiple transactions.

Starting in Debezium 2.2, we’re adding a new and initially optional way to utilize multiple threads to perform consistent database snapshot for a connector. This implementation uses these multiple threads to execute table-level snapshots in parallel.

In order to take advantage of this new feature, specify snapshot.max.threads in your connector’s configuration and when this property has a value greater than 1, parallel snapshots will be used.

In the example above, if the connector needs to snapshot more than 4 tables, there will be at most 4 tables being snapshot in parallel. When one thread finishes processing a table, it will get a new table to snapshot from the queue and the process continues until all tables have been snapshot.

Debezium’s incremental snapshot feature has been a tremendous success. It provides an efficient way to perform a consist snapshot of data that can be resumed, which is critical when the snapshot consists of large volumes of data.

However, incremental snapshots do have specific requirements that must be met before the feature can be used. One of those requirements is all tables being snapshot must use a primary key. You may ask, why does a table have no primary key, and we aren’t going to debate that here today; however, suffice to say this occurs more often than you may think.

With Debezium 2.2, incremental snapshots can be performed on key-less tables as long as there is one column that is unique and can be considered a "surrogate key" for incremental snapshot purposes.

To provide the surrogate key column data in an incremental snapshot signal, the signal’s payload must include the new surrogate key attribute, surrogate-key.

In the above example, an incremental snapshot will be started for table public.mytab and the incremental snapshot will use the customer_ref column as the primary key for generating the snapshot windows.

However, the surrogate key feature isn’t just applicable for tables with no primary keys. There are a series of advantages when using this feature with tables that have primary keys:

Quarkus is a Kubernetes Native Java stack that combines the best Java libraries to create fast, low footprint applications. The Debezium Server runtime is based on Quarkus as well as part of Debezium UI. Additionally, the Debezium Outbox extension is also based on the Quarkus platform.

The upgrade to Quarkus 3 introduces a number of improvements, including using the latest stable releases of a plethora of Java libraries, including the migration from Java EE to Jakarta EE. If you are not familiar with this migration, previously most Java EE platform classes were bundled in the package javax.*. Over the past year or two, more applications have started the move from JavaEE or J2EE to Jakarta EE, and Quarkus 3 marks this transition era. Overall, the only real change is that classes that previously resided in javax.* now are placed in jakarta.*.

If your application makes use of the Debezium Quarkus Outbox extension, be aware that in order to use Debezium 2.2 with Quarkus, you will need to migrate to Quarkus 3. This also means that if you want to take advantage of the Outbox extension for Reactive data sources, you will be required to use Quarkus 3 as well.

Finally, if you are developing or maintaining sink adapters for Debezium Server, you will also need to make adjustments to using the new Jakarta EE annotations rather than the older Java EE annotations.

The Debezium 2.2 release ushers in a new era for Debezium which has had a longstanding focus purely on providing a set of source connectors for relational and non-relational databases. This release alters that landscape, introducing a new JDBC sink connector implementation.

The Debezium JDBC sink connector is quite different from other vendor implementations in that it is capable of ingesting change events emitted by Debezium connectors without the need for event flattening. This has the potential to reduce the processing footprint in your pipeline, simplifies the pipeline’s configuration, and allows Debezium’s JDBC sink connector to take advantage of numerous Debezium source connector features such as column type propagation and much more.

Getting started with the Debezium JDBC sink connector is quite simple, lets take a look at an example.

Let’s say we have a Kafka topic called orders that contains Debezium change events that were created without using the ExtractNewRecordState transformation from MySQL. A simple configuration to ingest these change events into a PostgreSQL database might look this the following:

In this example, we’ve specified a series of connection.* properties that define the connection string and credentials for accessing the destination PostgreSQL database. Additionally, records will use UPSERT semantics when writing to the destination database, choosing to use an insert if the record doesn’t exist or updating the record if it does. We have also enabled schema evolution and specified that a table’s key columns should be derived from the event’s primary key.

The JDBC sink connector presently has support for the following relational databases:

We do intend to add additional dialects in the future, and if there one you’d like to see, please get in touch with us either on our mailing list, in chat, or opening a Jira enhancement.

The Debezium for Oracle connector normally manages what is called a flush table, which is an internal table used to manage the flush cycles used by the Oracle Log Writer Buffer (LGWR) process. This flushing process requires that the user account the connector uses to have permission to create and write to this table. Logical stand-by databases often have more restrictive rules about data manipulation and may even be read-only, therefore, writing to the database is unfavorable or even not permissible.

To support an Oracle read-only logical stand-by database, we introduced a flag to disable the creation and management of this flush table. This feature can be used with both Oracle Standalone and Oracle RAC installations, and is currently considered incubating, meaning its subject to change in the future.

In order to enable Oracle read-only logical stand-by support, add the following connector option:

In a future version, we plan to add support for an Oracle read-only physical stand-by database.

Google’s Cloud Spanner platform supports a PostgreSQL interface, which combines the scalability and reliability of the Google Spanner platform with the familiarity and portability of PostgreSQL. When operating Google Spanner with this PostgreSQL interface, metadata of columns and tables is different than when using the standard GoogleSQL dialect.

This release extends the Debezium Spanner connector support not only for the GoogleSQL dialect but also for users that use the Spanner PostgreSQL dialect feature. This means regardless of which dialect your spanner environment relies on, you will be able to capture change events from Spanner using the Debezium Spanner connector seamlessly.

Infinispan is an in-memory, distributed data store that offers flexible deployment options with robust capabilities to store, manage, and process data. Infinispan is based on the notion of a key-value store that allows storing any data type. In order to integrate Debezium Server with Infinispan, the Debezium Server application.properties must be modified to include the following entries:

The above configuration specifies that the sink type to be used is infinispan, which enables the use of the Infinispan module. The following is a description of each of the properties shown above:

For more information on using Debezium Server with Infinispan, see the documentation.

Debezium 2.2 introduces a new sink adapter to the Debezium Server portfolio, allowing Debezium users to send change events to RabbitMQ. The following configuration shows a simple example of how easy it is to configure:

The debezium.sink.rabbitmq.connection.* properties are required while the latter two properties for routingKey and ackTimeout are optional or have preset defaults that should be sufficient for most use cases.

Apache RocketMQ is a cloud-native messaging, eventing, and streaming real-time data processing platform that covers cloud-edge-device collaboration scenarios. In order to integrate Debezium Server with RocketMQ, the Debezium Server application.properties must be modified to include the following entries:

The above configuration specifies that the sink type to be used is rocketmq, which enables the use of the RocketMQ module. The following is a description of each of the properties shown above:

For more information on using Debezium Server with RocketMQ, see the documentation.

In prior versions of the Debezium Server Pulsar sink, the adapter leveraged the send() method to deliver messages in a synchronous way. While this works for sending one-off messages, this has the potential to introduce connector latency as the method waits an acknowledgement of send operation sequentially. Since the Debezium Server sink adapters are provided a collection of events to deliver, the synchronous nature just does not perform well.

Starting Debezium 2.2, the Pulsar sink will now use sendAsync() to asynchronously deliver the batch of events to Pulsar, netting a substantial increase in overall throughput. While each event within the batch is delivered asynchronously, the adapter will only proceed to the next batch once the current batch is acknowledged in entirety.

The outbox pattern is an approach that many microservices leverage to share data across microservice boundaries. We introduced the Debezium Outbox Quarkus Extension in Debezium 1.1 back in early 2020, and it has allowed Quarkus users to leverage the outbox pattern with ease using Debezium.

Thanks to Ingmar Fjolla, Debezium 2.2 includes a new reactive-based implementation of the Debezium Outbox Quarkus Extension. This new implementation is based on Vert.x and Hibernate Reactive, providing a fully asynchronous solution to the outbox pattern using Debezium.

This new extension is included in the Quarkus 3 platform released later this month. However if you want to get started with it today, you can easily drop it directly into your project’s configuration using the following coordinates:

Debezium provides a Storage API framework that enables connectors to store offset and schema history state in a variety of persistence datastores. Moreover, the framework enables contributors to extend the API by adding new storage implementations with ease. Currently, the Storage API framework supports the local FileSystem, a Kafka Topic, or Redis datastores.

With Debezium 2.2, we’re pleased to add Amazon S3 buckets as part of that framework, allowing the schema history to be persisted to an S3 bucket. An example connector configuration using S3 might look like the following:

Debezium’s new storage API has been a huge success over this past year. We initially started with our original file and Kafka based implementations for offset and schema history storage, but that has since grown to support storing schema history on other platforms such as Amazon S3 and Redis.

This release continues to expand on this by adding a new schema history storage implementation for Rocket MQ. In order to get started with storing your schema history into Rocket MQ, the debezium-storage-rocketmq dependency must first be on the classpath and accessible by the connector runtime.

Once the dependency exists, the only remaining step will be configuring the schema history connector configuration. The following example shows basic usage of the Rocket MQ schema history: