Debezium will now begin to track metrics based on the individual create, update, and delete operations performed per relational table. For some connectors such as PostgreSQL and Oracle, these new detailed metrics also track the truncate operations performed per relational table. This can be quite useful for situations where you need to detect specific mutation patterns or where you may want to integrate analytics or observability stacks where this detailed information could be valuable to identifying problems.

For users upgrading to Debezium 3, these new metrics are captured automatically. They are exposed using a map-based pattern of Map<String, Long> where the key is the table name and the value is the number of events observed. The new metrics names are NumberOfCreateEventsSeen, NumberOfDeleteEventsSeen, NumberOfUpdateEventsSeen, and NumberOfTruncateEventsSeen (DBZ-8035).

When the PostgreSQL connector is first deployed, one of its first tasks is to create a replication slot in the database if it doesn’t already exist. The replication slot is pivotal to how the connector works and facilitates the capture and dispatch of changes to Debezium. Unfortunately, there are some database operations that will block the creation of replication slots, such as in-progress transactions, forcing the connector to block indefinitely while waiting for the transaction to conclude. For short-lived transactions, this isn’t generally a concern; however, for long-running transactions that’s an entirely different situation.

In order to improve this experience, a new internal option was added, internal.create.slot.command.timeout, which defaults to 90 seconds. If the creation of the replication slot does not complete within 90 seconds, it will retry up to slot.max.retries. Once the retries are exhausted, the connector will throw an unrecoverable error (DBZ-8073).

The pgvector extension introduces vector search functionality for PostgreSQL. There are three data types this extension introduces: vector, halfvec, and sparsevec.

In Debezium 3, all three data types will be streamed like any other data type. Each data type is emitted based on the following semantic mappings:

There is no additional configuration required after enabling the pgvector extension in your database. Please see the documentation for more details on the semantic mappings (DBZ-8121).

Debezium 3 introduces as new Oracle connector transaction buffer implementation, based on Ehcache to provide off-heap storage of transaction processing and event data. This new implementation adds to the existing Java Heap, Infinispan Embedded, and Infinispan Remote buffer types.

To begin taking advantage of the Ehcache implementation, the log.mining.buffer.type must be set to ehcache. By default, the buffer type is memory to use the JVM’s heap for optimal performance.

In order to for the Ehcache library to start successfully, several additional configurations must be provided to explicitly configure the caches maintained by the cache manager. These new configuration options are:

Debezium creates the Ehcache configuration using XML, so each of these configurations provide XML snippets.

The global configuration is optional, and allows you to provide details about persistence and other Ehcache attributes, excluding specifying <cache> or <default-serializers> tags, which are handled separately. The other individual cache configurations are meant to supply the inner XML bits of a <cache> configuration tag, excluding its <key-type> and <value-type>, which are managed directly by Debezium.

In this example, Ehcache will maintain a combination of heap and off-heap storage for the caches, maintaining at most 256 entries in heap at all times and flushing to disk. The disk caches will be stored at the relative path ./data. This implies that you will need a persistent storage volume available when using disk-based caches.

This is a new feature and is experimental, so we would love your feedback on how we can improve this (DBZ-7758).

PostgreSQL is unique in that you can implement the Outbox pattern without creating an outbox table, by writing logical messages directly into the WAL using pg_logical_emit_message. The unfortunate part is that this data is then sent to Kafka as a series of bytes, which may not always be ideal for consumers who may be looking for structured messages.

Debezium 3 introduces a new PostgreSQL-specific transform called DecodeLogicalDecodingMessageContent. This transform is specifically meant to convert the pg_logical_emit_message event bytes to a structured event payload that consumer applications are capable of understanding.

Given the following configuration:

The event’s value of an event written using pg_logical_emit_message before the transform would be:

After applying the transformation, the event’s value now looks like:

So you can safely implement the Outbox pattern without the physical outbox table! (DBZ-8103).