While the team has been hard at work on Debezium 3.1, we’re pleased to announce a new maintenance update for Debezium 3.0.8.Final.
While our maintenance releases often focus primarily on stability improvements, we are happy to share that we’ve also included some new features in this maintenance release to improve the overall user experience.
Breaking changes
With any new major release of software, there is often several breaking changes; however, we generally prefer to avoid these with maintenance releases if possible. Unfortunately, Debezium 3.0.8.Final is one of those exceptions where we have introduced some breaking changes.
Sparse vector logical type renamed
The PostgreSQL extension vector (aka pgvector) provides an implementation of a variety of vector data types, including one called sparsevec. A sparse vector is one that stores only the populated key/value entries within the vector, excluding those that aren’t set to minimize the space used to represent the data set.
Debezium 3.0 introduced the SparseVector logical type named io.debezium.data.SparseVector. As we began to evaluate implementations for other relational databases, we determined that the logical name was insufficient when we began to implement other sparse vector types (DBZ-8585).
With Debezium 3.1 and 3.0.8.Final, we have repackaged the io.debezium.data.SparseVector class from the PostgreSQL connector into Debezium’s core package, we’ve also renamed the class to SparseDoubleVector, and changed the logical name to io.debezium.data.SparseDoubleVector to align with the class name changes.
| For those who may have been working with |
Changes to schema history configuration defaults
The documentation for schema.history.internal.store.only.captured.databases.ddl provided an incorrect default value. While this is not a code-specific breaking change, you should take a moment and reevaluate whether your deployment’s configuration depends on the different default value or not (DBZ-8558).
Potential Vitess data loss
The Debezium for Vitess connector had a rare but critical data loss bug that has existed since it was first introduced five years ago. If a primary key update is the last operation in a transaction, records may be lost. This bug affects all prior versions. It is highly recommended that users perform an immediate update to 3.0.8.Final or 3.1.0.Alpha2 or later to remedy this potential data loss (DBZ-8594).
Reselect column post processor behavior changed for Oracle
The ReselectColumnsPostProcessor behavior has changed and Oracle LOB columns will be reselected regardless of the lob.enabled configuration property’s value. This change enables users who may not want to mine LOB columns while streaming to still populate the LOB column using the column reselection process as an alternative (DBZ-8653).
New features and improvements
THe upgrade to Debezium 3.0.8.Final introduces some new features for several components.
SSL connectivity with MariaDB
Users may have found that SSL connectivity was not working in prior versions of Debezium when capturing changes from MariaDB. We are pleased to report that this has been fixed, and users should be able to use MariaDB with SSL connections (DBZ-8482).
While the same configuration property, database.ssl.mode can be used, MariaDB uses different values to represent the negotiation steps taken between the driver and database. The following table outlines the MariaDB settings.
| SSL Mode | Description / Purpose |
|---|---|
| Specifies that the JDBC driver should not attempt to negotiate an SSL connection with the database server. If the database server requires a secure connection, the connection will be refused. |
| Specifies that the JDBC driver should use SSL/TLS for data encryption purposes, and that the certificate and hostname won’t be verified. |
| Works identify to the trust mode, but also performs certificate verification, but does not verify the hostname. |
| Works just like |
| Debezium defaults to using MariaDB 11.4 or later defaults to using SSL connections, so you will need to set an appropriate |
Other changes
The following are some noteworthy changes in 3.0.8.Final:
-
Align MySQL and MariaDB grammars with upstream versions (DBZ-8270)
-
JDBC sink connector doesn’t delete rows from a postgres db table (DBZ-8287)
-
MariaDB adapter fails on an ALTER USER statement (DBZ-8436)
-
Expressions cause SQL parser exception in Percona SEQUENCE_TABLE function (DBZ-8559)
-
MongoDataConverter does not recognize nested empty array (DBZ-8572)
-
ORA-65040 occurs on log switches when log.mining.restart.connection is enabled and connection defaults to PDB rather than CDB$ROOT (DBZ-8577)
-
Reduced record buffer doesn’t handle RECORD_VALUE with primary key fields (DBZ-8593)
-
Events for tables with generated columns fail when using hybrid mining strategy (DBZ-8597)
-
ANTLR DDL Parsing error (DBZ-8600)
-
Update Debezium Server and Operator to Quarkus 3.15.3 LTS (DBZ-8601)
-
Remove misleading log entry about undo change failure (DBZ-8645)
-
Oracle metric OldestScnAgeInMilliseconds does not account for database timezone (DBZ-8646)
-
Using RECORD_VALUE with a DELETE event causes NullPointerException (DBZ-8648)
-
Batch size calculation is incorrectly using min-batch-size (DBZ-8652)
-
Oracle performance drop when transaction contains many constraint violations (DBZ-8665)
-
Skip empty transactions with commit with redo thread equal to 0 (DBZ-8681)
-
DDL statement couldn’t be parsed: GRANT SENSITIVE_VARIABLES_OBSERVER (DBZ-8685)
-
Column scale is reset when a column’s type is not changed during an ALTER TABLE (DBZ-8700)
-
Alter table modify column fails when using DEFAULT ON NULL clause (DBZ-8720)
-
ExtractChangedRecordState SMT Now Working With Default Values (DBZ-8721)
-
Restart of Oracle RAC node leads to redo thread being inconsistent indefinitely (DBZ-8724)
-
Specifying archive.log.hours with non-zero value generates bad SQL (DBZ-8725)
-
debezium/connect docker image is not available on arm64 (DBZ-8728)
-
Unknown DDL ALTER INDEX <name> IGNORED (DBZ-8741)
-
Possibly broken schema.history.internal.skip.unparseable.ddl for MariaDB (DBZ-8745)
-
Oracle snapshot’s source.ts does not account for database zone differences (DBZ-8749)
In total, 35 issues were resolved in Debezium 3.0.8.Final. The list of changes can also be found in our release notes.
A big thank you to all the contributors from the community who worked diligently on this release: Alvar Viana, Anisha Mohanty, Chris Cranford, Gaurav Miglani, Henk van Dyk, Jakub Cechacek, Jiri Pechanec, Katerina Galieva, Robert Roldan, Sergey Seroshtan, Thomas Thornton, Vojtech Juranek, Yannick Eisenschmidt, Zhongqiang Gong, and حمود سمبول!
About Debezium
Debezium is an open source distributed platform that turns your existing databases into event streams, so applications can see and respond almost instantly to each committed row-level change in the databases. Debezium is built on top of Kafka and provides Kafka Connect compatible connectors that monitor specific database management systems. Debezium records the history of data changes in Kafka logs, so your application can be stopped and restarted at any time and can easily consume all of the events it missed while it was not running, ensuring that all events are processed correctly and completely. Debezium is open source under the Apache License, Version 2.0.
Get involved
We hope you find Debezium interesting and useful, and want to give it a try. Follow us on Twitter @debezium, chat with us on Zulip, or join our mailing list to talk with the community. All of the code is open source on GitHub, so build the code locally and help us improve ours existing connectors and add even more connectors. If you find problems or have ideas how we can improve Debezium, please let us know or log an issue.
