A year ago we began this incredible journey to create a new, modern user interface for Debezium Server that aimed to ease the deployment of Debezium on Kubernetes. We are excited to announce that Debezium 3.1 will be the first official release of this years-long effort.

The new Debezium Platform provides a modern pipeline-based approach to designing source and sink configurations, transformation chains, and more within seconds. You can install the Debezium Platform using helm as follows:

For more details on how to deploy using helm, see the README.md.

In addition, this release specifically adds some finishing touches to the user interface, which includes new search/list-view toggles, display applying transforms and editing of connector pipelines, and lastly experienced-user smart editors during configuration of a pipeline.

The following videos show how to use these new features:

A new snapshot.locking.mode has been added to Debezium for MySQL Percona users, to reduce the amount of locking that occurs during the snapshot. The new mode, minimal_percona_no_table_locks, provides the same semantics as minimal_percona, but additionally omits applying table-level locks (DBZ-8717). This provides an alternative for some environments where table-locks are not permitted.

Debezium has added several new fields to the source information block for Oracle change events (DBZ-8740), which include:

These new fields are optional, so schema registry users should find these changes are backward compatible.

When the Vitess value converter emits a date column that is set to a zero date value, depending on the optionality of the column, the field may be emitted as null or as the unix epoch. This creates an issue for consumer applications, as it’s impossible for them to differentiate when the epoch value is provided whether it’s a true epoch value or the sentinel value because the column is zero date in the source.

Debezium 3.1 introduces a new configuration property for Vitess users, override.datetime.to.nullable.

The default, false, continues to emit the date column using the old behavior, where if the column is not null, the unix epoch will be used rather than null. This means that consumers will continue to be unable to differentiate between the two use cases.

When set to true, all date and datetime columns are set as optional, meaning they can be serialized with null, regardless of how the column’s optionality may be set in the source database. This means that if a zero date is set in the source system, the connector will always use null to represent this use case and will no longer use epoch-based values unless the field is populated with an actual non-zero date value.

In Debezium 3.1.0.Alpha2, we introduced a change to emit Vitess binary-collated text, enum, and set column types as character-based field types in change events (DBZ-8679). This unfortunately only covered a subset of column types, and in this release we’ve expanded on that to include tinytext, mediumtext, and longtext types (DBZ-8694).

Debezium’s CloudEvents support has been updated to include support for the traceparent attribute, which provides the ability to integrate with OpenTelemetry to pass the trace details as part of the event (DBZ-8669).

By setting the opentelemetry.tracing.attributes.enabled configuration property to true along with including the traceparent:header as part of the metadata.source, this information will be made available to the CloudEvents converter.

You can customize the way that the conver populates the fields by changing the defaults and specifying the fields' values in the appropriate headers. For example:

You can find other examples in Debezium’s CloudEvents documentation.

You can now access some schema details inside your TinyGo programs using the WASM transformation (DBZ-8737). Two new methods have been added to support time, GetSchemaName and GetSchemaType.

Debezium’s kafka and connect images are all derived from a single common image called connect-base. By default, this base image installs Apicurio, Jolkia, and OpenTelemetry dependencies. This is great for testing purposes, but if you wish to use Debezium’s images as a basis for your own, you may prefer to omit these dependencies if they’re not necessary for your environment.

The connect-base image can now be conditioned to omit any one of these dependencies (DBZ-8709). The OTL_ENABLED, APICURIO_ENABLED, and JOLOKIA_ENABLED environment variables can be set to no to omit those dependencies when building your images, creating a smaller image footprint.

In order to improve throughput and capacity with Google PubSub, we have introduced the ability to specify several new configuration properties for PubSub to support concurrency and compression (DBZ-8715). These new configuration properties can be used in any existing PubSub configuration.

When working with the PubSub sink, the pubsub.address is often not sufficient for production systems where you may need to interact with location-specific (aka region) endpoints. To address this concern, Debezium 3.1 introduces a new configuration property, pubsub.region (DBZ-8735).

The new pubsub.region property allows specifying the Google Cloud region to connect, i.e. us-central1 or asia-northeast1. When specified, Debezium will use the location-specific endpoint for PubSub in the format <region>-pubsub.googleapis.com:443. This permits connecting to the location-specific endpoint instead of the global endpoint.

In Debezium 3.1, we have changed how you can route events using configuration. This new approach uses a strategy-based design, that retains old behaviors and introduces the new key-based routing mechanism (DBZ-8752).

First and foremost, the rabbitmq.routingKeyFromTopicName is deprecated and will be removed in a future release. This functionality has been folded into the new rabbitmq.routingKey.source configuration property, and it can be set one to one of the following values: