It’s been about three weeks since we released Debezium 2.4, and in that time the team has been diligently working on what comes next in the evolution of Debezium. I am pleased to announce that today we have released Debezium 2.5.0.Alpha1, the first preview release of Debezium’s 2.5 release stream.
This release includes many new exciting features as well as bug fixes, e.g. a brand-new IBM Informix connector, a preview support for MariaDB with the MySQL connector, improvements to the Debezium Server operator, Kafka 3.6.0 support, and much more.
Let’s take a closer look at these changes and features that are included in Debezium 2.5.0.Alpha1; as always, you can find the complete list of changes for this release in the release notes. Please take special note of any breaking changes that could affect your upgrade path.
IBM Informix Connector
Thanks to the contribution from Lars Johansson, Debezium 2.5 introduces a new connector to its portfolio to gather changes from IBM Informix. IBM Informix is an embeddable, high-performance database for integrating SQL, NoSQL, JSON, time-series, and spatial data in one place. It’s designed for analytics at the edge, in the cloud, or on premise.
The IBM Informix connector is bundled just like any of our community lead connectors, it is available on Maven Central or you can download the plug-in archive from our Debezium 2.5 releases page.
The maven artifact coordinates are:
<dependency>
<groupId>io.debezium</groupId>
<artifactId>debezium-connector-informix</artifactId>
<version>2.5.0.Alpha1</version>
</dependency>
If you would like contribute to the Informix connector, we have added a new repository under the Debezium organization, debezium-connector-informix.
I’d like to thank Lars Johansson for this contribution and his collaboration with the team, kudos!
MariaDB preview support
The community has leveraged the MySQL connector as an alternative to capture changes from MariaDB for quite some time now; however that compatibility was primarily best-case effort.
The Debezium 2.5 release stream aims to bring MariaDB to the forefront as a first-class connector by taking a very clear and methodological approach to incrementally check, validate, and eventually support MariaDB at the same capacity that we do MySQL. Our goal and hope is that we can do this within the scope of the MySQL connector proper; however, there is still quite a bit of ongoing investigation around GTID support that may influence the path forward.
This first preview build of Debezium 2.5 has taken the first step, we’ve verified that the code works against a single MariaDB database deployment, the test suite passes and we’ve addressed any changes needed with the Binlog client to support that deployment. Our next steps is to look into GTID support, which MariaDB supports but using an approach that isn’t compatible with MySQL.
Stay tuned for future builds as we continue to expand on this and we certainly welcome any early feedback.
Oracle Streaming Metrics Changes
In previous builds of Debezium, there was a single Oracle streaming metrics bean that exposed all metrics options that spanned across all three streaming adapters. This often lead to some confusion about what metrics are applicable to which streaming adapter so we wanted to define a clear distinction in this case.
With Debezium 2.5, the Oracle streaming metrics beans have been split into three different implementations, one for each adapter type. For observability stacks, this change should be completely transparent unless you were previously gathering a metric for one adapter type while using another. In this case you’ll find that metric is no longer available.
Specifically for LogMiner users, several metrics have been renamed and the old metrics have been deprecated. While you will still be able to use the old metric names in Debezium 2.5, these are scheduled for removal in a future 2.7+ build. The metrics that were deprecated and renamed are as follows:
Old/Deprecated Metric | New Metric |
---|---|
CurrentRedoLogFileName | CurrentLogFileNames |
RedoLogStatus | RedoLogStatuses |
SwitchCounter | LogSwitchCounter |
FetchingQueryCount | FetchQueryCount |
HoursToKeepTransactionInBuffer | MillisecondsToKeepTransactionsInBuffer |
TotalProcessingTimeInMilliseconds | TotalBatchProcessingTimeInMilliseconds |
RegisteredDmlCount | TotalChangesCount |
MillisecondsToSleepBetweenMiningQuery | SleepTimeInMilliseconds |
NetworkConnectionProblemsCounter | No replacement |
Debezium Server Operator
The Debezium Server Operator for Kubernetes has been actively improved in this preview release of Debezium 2.5. Several improvements include:
-
Ability to set image pull secrets in the CRDs DBZ-6962
-
Ability to set resource limits in the CRDs DBZ-7052
-
Published OLM bundle scripts to Maven Central DBZ-6995
-
Support OKD/OpenShift catalog in OperatorHub release script DBZ-7010
-
Display name and descriptions metadata available in OLM bundle DBZ-7011
-
New metrics endpoint for gathering metrics DBZ-7053
As we continue to improve the Debezium Server Operator for Kubernetes, we’d love to get your feedback.
Community connectors
Other fixes
In addition, there were quite a number of stability and bug fixes that made it into this release. These include the following:
-
Adding Debezium Server example using MySQL and GCP PubSub DBZ-4471
-
Refactor ElapsedTimeStrategy DBZ-6778
-
Multiple debezium:offsets Redis clients DBZ-6952
-
Wrong case-behavior for non-avro column name in sink connector DBZ-6958
-
Handle properly bytea field for jdbc sink to postgresql DBZ-6967
-
Debezium jdbc sink process truncate event failure DBZ-6970
-
Single quote replication includes escaped quotes for N(CHAR/VARCHAR) columns DBZ-6975
-
Provide configuration option to exclude extension attributes from a CloudEvent DBZ-6982
-
Debezium jdbc sink should throw not supporting schema change topic exception DBZ-6990
-
Debezium doesn’t compile with JDK 21 DBZ-6992
-
OLM bundle version for GA releases is invalid DBZ-6994
-
Further refactoring to correct downstream rendering of incremental snapshots topics DBZ-6997
-
ALTER TABLE fails when adding multiple columns to JDBC sink target DBZ-6999
-
Invalid Link to zulip chat in CSV metadata DBZ-7000
-
Make sure to terminate the task once connectivity is lost to either the rebalance or sync topic DBZ-7001
-
Missing .metadata.annotations.repository field in CSV metadata DBZ-7003
-
Single quote replication and loss of data DBZ-7006
-
Remove deprecated embedded engine code DBZ-7013
-
Enable replication slot advance check DBZ-7015
-
Add configuration option to CloudEventsConverter to retrieve id and type from headers DBZ-7016
-
Oracle connector: Payload size over 76020 bytes are getting truncated DBZ-7018
-
Use optional schema for Timezone Converter tests DBZ-7020
-
DDL statement couldn’t be parsed DBZ-7030
-
Blocking ad-hoc snapshot is not really blocking for MySQL DBZ-7035
-
Fake ROTATE event on connection restart cleans metadata DBZ-7037
-
Consolidate resource labels and annotations DBZ-7064
Altogether, XX issues were fixed for this release. A big thank you to all the contributors from the community who worked on this release: Anisha Mohanty, Bertrand Paquet, Bob Roldan, Chris Cranford, David Remy, Don Seiler, Gunnar Morling, Harvey Yue, Jakub Cechacek, Jiri Novotny, Jiri Pechanec, Mario Fiore Vitale, Mark Ducommun, Nancy Xu, Ondrej Babec, Praveen Burgu, Rajendra Dangwal, René Kerner, Robert Roldan, Roman Kudryashov, Sergey Eizner, Shuran Zhang, Thomas Thornton, Vojtech Juranek, and sean!
What’s next?
With the holiday season quickly approaching for many of us, you can expect that the release cadence for Debezium 2.5 to be shorter than usual. We intend to release often like always, but expect that we will likely only do one more alpha build before we began to transition into beta and release candidate builds. Our goal is to deliver Debezium 2.5.0.Final by mid-December, just in time for the holiday break.
While this condensed timeline doesn’t give us lots of room, we have a lot still planned to include for Debezium 2.5. There is still lots of work to do on the MariaDB preview front, supporting GTID and multiple topology deployments. We’re also working on improving the parallelization experience with Debezium Engine, batch support for the JDBC sink connector, MongoDB improvements around large BSON documents, and much more. You can find all the details for our continued plans for Debezium 2.5 on our roadmap.
Lastly, I’d like to remind everyone about the Debezium community meeting. I will be distributing details about the next meeting in the coming weeks, and I would urge folks to be on the look-out and try and stop by our virtual event in early December. It’s a great way to meet the engineers working on Debezium, ask questions in an AMA style format, and to get insights not only into what all is part of Debezium 2.5, but also what lies ahead with Debezium 2.6 and 2.7 for early next year!
As always, please be sure to get in touch with us on the mailing list or Zulip chat if you have questions or feedback. Until next time, stay warm out there!
About Debezium
Debezium is an open source distributed platform that turns your existing databases into event streams, so applications can see and respond almost instantly to each committed row-level change in the databases. Debezium is built on top of Kafka and provides Kafka Connect compatible connectors that monitor specific database management systems. Debezium records the history of data changes in Kafka logs, so your application can be stopped and restarted at any time and can easily consume all of the events it missed while it was not running, ensuring that all events are processed correctly and completely. Debezium is open source under the Apache License, Version 2.0.
Get involved
We hope you find Debezium interesting and useful, and want to give it a try. Follow us on Twitter @debezium, chat with us on Zulip, or join our mailing list to talk with the community. All of the code is open source on GitHub, so build the code locally and help us improve ours existing connectors and add even more connectors. If you find problems or have ideas how we can improve Debezium, please let us know or log an issue.