Just in time for the holidays, it’s my pleasure to announce the fifth maintenance release for Debezium 3, 3.0.5.Final. This release introduces several minor improvements, bugfixes, and some breaking changes. Let’s take a few moments and dive into the details…
Breaking Changes
The team aims to avoid any potential breaking changes between minor releases; however, such changes are sometimes inevitable.
- Kafka Signal Source
-
Debezium was reprocessing Kafka-based signals on connector restarts, which could introduce unpredictable behavior with unintended side effects. As a result, this feature has been removed, and if a connector stops, then the signal must be re-sent (DBZ-7856).
- Change Event Source Info Block
-
The snapshot enumeration in the
source
information block in Debezium’s change events was extended to include all possible cases (DBZ-8496).
New features and improvements
Debezium 3.0.5.Final introduces a couple of improvements and features, lets take a look at each individually.
Core
Allow ad-hoc blocking snapshots on unknown tables
While we recommend using the default configurations for schema management for all tables, we understand that isn’t always possible for every environment. A common question we often see is how does one snapshot newly added tables, particularly if the connector is configured using store.only.captured.tables.ddl
as true
.
In this update, we have simplified the process where you can update your connector configuration with the new table, and once the connector task has restarted, sending an ad-hoc blocking snapshot signal will be permitted even on tables that are unknown but match the capture filters (DBZ-4903).
Snapshot dispatch failure handling improvements
When processing change events, users can control how the connector reacts to specific event handling failures using event.processing.failure.handling.mode
; however, this has traditionally only been applicable for streaming changes.
There are some corner cases where emitting snapshot events could also raise failures. To improve this experience, the event.processing.failure.handling.mode
now influences failures when dispatching snapshot events, too (DBZ-8433).
Connector startup configuration logging improved
When a Debezium source connector starts, it logs all connector configurations. This is very helpful for both the team when diagnosing problems but also for users to identify that their configurations are being understood properly.
In this update, the logging for the connector configuration has changed slightly. You will notice rather than placing each configuration property as a new log entry, each key/value will be appended together separated by new lines, and added to the log as a single entry. This makes it easier to identify these log lines and for tooling to exclude such log entries easier if needed (DBZ-8472).
Postgres
Support for PostgreSQL 17 fail-over replication slots
PostgreSQL 17 introduces a long awaited feature called fail-over for replication slots.
When the replication slot is created, a new property can be specified to enable the replication slot to be created and available on the fail-over replica. The synchronization of the fail-over replication slot can be performed manually by calling pg_sync_replication_slots()
or automatically by enabling the sync_replication_slots
feature.
When automatic synchronization is enabled, this allows Debezium to consume from that slot immediately on fail-over to the replica and not miss any events.
To take advantage of this new feature, your connector must be configured with slot.failover
set to true
. Debezium must be connecting to PostgreSQL 17+ and the primary on the cluster, otherwise no fail-over replication slot will be created (DBZ-8412).
Oracle
New metrics to track partial rollback events
A partial rollback event is something unique to Oracle where a user performs an operation, it gets recorded as being performed in the transaction logs, but then a validation check forces the operation to be undone and rolled back. One of the most notable ways to trigger partial rollbacks are with constraint violations.
The new JMX metric, NumberOfPartialRollbackCount
, tracks the frequency of this event sequence in the transaction logs (DBZ-8491).
The increase in this metric should be rare, and having a few occurrences throughout the day should pose no major problem. When you notice this metric increases frequently in a small window of time, this means you could have a poorly written script where a job is relying on constraint violations for some logic. While the connector can handle these use cases, it’s important to understand this creates unnecessary transaction log activity that can directly impact the speed and latency of streaming changes. |
Skip GoldenGate replication markers
When using Debezium to capture changes from an Oracle database that interacts with Oracle GoldenGate, you may have observed some situations where the low watermark in the offsets did not advance across restarts. This was especially the case when setting lob.enabled
to true
.
In this update, the Debezium Oracle connector will now skip GoldenGate’s replication marker events, which will prevent those synthetic transactions from cluttering the transaction buffer and forcing users to rely on transaction retention (DBZ-8533).
Vitess
Unparseable DDL handling improvements
Several new improvements have been added to Vitess to make the parsing and handling of DDL changes more fault-tolerant (DBZ-8479). These improvements include:
-
Only parsing DDL events if that specific table is tracked in the schema history
-
Strip comments to mimic parse failures
-
Make table and type fault tolerance to gracefully handle edge cases
K8s Operator
Support JDBC offset/history configurations DBZ-8501
We have improved the Kubernetes Operator by introducing support for the JDBC storage module, allowing it to be configured via the CRD.
In the offset schema reference, a new property section called jdbc
has been added, which describes the JDBC offset backing store properties. In the schema history reference, a new property section called jdbc
has been added, which describes the JDBC schema history store properties.
This allows users to store offsets and schema history details for a connector deployed using the Debezium Operator in a JDBC data store easily (DBZ-8501).
Other fixes
In total there were 43 issues resolved in Debezium 3.0.5.Final. The list of changes can also be found in our release notes.
Here are some noteworthy changes:
-
Create smoke test to make sure Debezium Server container image works DBZ-3226
-
Error with debezium.sink.pulsar.client.serviceUrl and debezium-server DBZ-3720
-
MySQL regression - Defaults store.only.captured.tables.ddl to true DBZ-6709
-
ExtractNewRecordState value of optional null field which has default value DBZ-7094
-
DebeziumException: No column '' where ' found in table DBZ-8034
-
Align MySQL and MariaDB grammars with upstream versions DBZ-8270
-
MySQL Connector Does Not Act On
CREATE DATABASE
Records In The Binlog DBZ-8291 -
Vgtid doesn’t contain multiple shard GTIDs when multiple tasks are used DBZ-8432
-
Support MongoDB 8.0 DBZ-8451
-
Update description of
message.key.columns
and format admonitions in PG doc DBZ-8455 -
Object ID cache may fail with concurrent modification exception DBZ-8465
-
Add Basic validation in UI to check for form completion before submitting. DBZ-8474
-
Use schema evolution tool to manage the conductor database DBZ-8486
-
Oracle gathers and logs object attributes for views unnecessarily DBZ-8492
-
ReselectColumnPostProcessor can throw ORA-01003 "no statement parsed" when using fallback non-flashback area query DBZ-8493
-
Oracle DDL ALTER TABLE ADD CONSTRAINT fails to be parsed DBZ-8494
-
Edit Source/Destination on adding new configuration properties its removing old once DBZ-8495
-
Invalid property name in JDBC Schema History DBZ-8500
-
Fix the URL in Pipeline log page DBZ-8502
-
Failed to start LogMiner mining session due to "Required Start SCN" error message DBZ-8503
-
Oracle data pump TEMPLATE_TABLE clause not supported DBZ-8504
-
Postgres alpine images require lang/llvm 19 for build DBZ-8505
-
Update Quarkus Outbox Extension to Quarkus 3.17.3 DBZ-8506
-
Merge conductor and stage into single platform repository DBZ-8508
-
TimezoneConverter include.list should be respected if set DBZ-8514
-
Missing log classes debezium-platform-conductor DBZ-8515
-
Debezium Server fails to start when using the sink Kinesis DBZ-8517
-
Skip GoldenGate REPLICATION MARKER events DBZ-8533
A big thank you to all the contributors from the community who worked diligently on this release: Andrei Leibovski, Anisha Mohanty, Chris Cranford, Fan Yang, Giovanni Panice, Gunnar Morling, Gustavo Lira, Jakub Cechacek, Jean Vintache, Jiri Pechanec, Jose Carvajal Hilario, Juanma Barea, Mario Fiore Vitale, Martin Walsh, Nathan Smit, Ole-Martin Bratteng, Robert Roldan, Thomas Thornton, and Vojtech Juranek!
About Debezium
Debezium is an open source distributed platform that turns your existing databases into event streams, so applications can see and respond almost instantly to each committed row-level change in the databases. Debezium is built on top of Kafka and provides Kafka Connect compatible connectors that monitor specific database management systems. Debezium records the history of data changes in Kafka logs, so your application can be stopped and restarted at any time and can easily consume all of the events it missed while it was not running, ensuring that all events are processed correctly and completely. Debezium is open source under the Apache License, Version 2.0.
Get involved
We hope you find Debezium interesting and useful, and want to give it a try. Follow us on Twitter @debezium, chat with us on Zulip, or join our mailing list to talk with the community. All of the code is open source on GitHub, so build the code locally and help us improve ours existing connectors and add even more connectors. If you find problems or have ideas how we can improve Debezium, please let us know or log an issue.