The Debezium team is excited to announce the first release candidate of Deebzium 2.2, Debezium 2.2.0.CR1.
This release primarily focuses on stability improvements and bug fixes; however, there are a number of new features and breaking changes. In this release, Debezium migrated to Quarkus 3.0.0.Final, there are performance improvements to Debezium Server Pulsar sink, Jolokia can be enabled inside Debezium’s Kafka Connect container image, incubating support for incremental snapshots on MongoDB multi-replica and sharded clusters, and the deprecation usage of Docker Hub for images.
Let’s take a moment and dive into several of these and what it means moving forward!
Upgrade to Quarkus 3
Quarkus is a Kubernetes Native Java stack that combines the best Java libraries to create fast, low footprint applications. The Debezium Server runtime is based on Quarkus as well as part of Debezium UI. Additionally, the Debezium Outbox extension is also based on the Quarkus platform.
The upgrade to Quarkus 3 introduces a number of improvements, including using the latest stable releases of a plethora of Java libraries, including the migration from Java EE to Jakarta EE. If you are not familiar with this migration, previously most Java EE platform classes were bundled in the package
javax.*. Over the past year or two, more applications have started the move from JavaEE or J2EE to Jakarta EE, and Quarkus 3 marks this transition era. Overall, the only real change is that classes that previously resided in
javax.* now are placed in
If your application makes use of the Debezium Quarkus Outbox extension, be aware that in order to use Debezium 2.2 with Quarkus, you will need to migrate to Quarkus 3. This also means that if you want to take advantage of the Outbox extension for Reactive data sources, you will be required to use Quarkus 3 as well.
Finally, if you are developing or maintaining sink adapters for Debezium Server, you will also need to make adjustments to using the new Jakarta EE annotations rather than the older Java EE annotations.
Debezium Server Pulsar Changes
In prior versions of the Debezium Server Pulsar sink, the adapter leveraged the
send() method to deliver messages in a synchronous way. While this works for sending one-off messages, this has the potential to introduce connector latency as the method waits an acknowledgement of send operation sequentially. Since the Debezium Server sink adapters are provided a collection of events to deliver, the synchronous nature just does not perform well.
Starting Debezium 2.2, the Pulsar sink will now use
sendAsync() to asynchronously deliver the batch of events to Pulsar, netting a substantial increase in overall throughput. While each event within the batch is delivered asynchronously, the adapter will only proceed to the next batch once the current batch is acknowledged in entirety.
Jolokia is a JMX-HTTP bridge that provides an alternative to using JSR-160 to gather metrics. It is an agent based approach that improves traditional JMX by introducing unique features like bulk requests and fine-grained security policies.
With Debezium 2.2, the
debezium/connect image now ships with Jolokia, but this agent isn’t enabled by default. In order to enable Jolokia support, the container must be started with
ENABLE_JOLOKIA set to
true. By default, Jolokia will bind to port 8778 when enabled.
In the event that a different port is required, Jolokia will need to be enabled differently. For example, in order to enable Jolokia using port 9779, do not set the
ENABLE_JOLOKIA but instead configure the
KAFKA_OPTS environment variable as follows:
-e KAFKA_OPTS="-javaagent:$(ls "$KAFKA_HOME"/libs/jolokia-jvm-*.jar)=port=9779,host=*"
By specifying the above environment variable, Jolokia’s JMX-HTTP bridge will be available on port 9779 of the container.
Do not forget to add the Jolokia port to the container’s list of exposed ports when starting.
Deprecation of Docker Hub
Docker recently announced a reduction in their free organization account offerings, which is leveraged by a number of open-source communities, including Debezium. Unfortunately, Debezium does not qualify according to their rules.
Despite that Docker chose to walk back their decision, the Debezium team believes that we need a solid path forward to guarantee the availability of our images to our community without disruption. Debezium has been dual-publishing to both
quay.io for quite some time, and we will continue this for Debezium 2.2.0.Final and for all Debezium 2.3 preview releases.
We plan to publish a blog post going into more detail in the coming days with more details. In the meantime, all users should plan to migrate to
quay.io as soon as possible to avoid disruption of fetching newer versions of Debezium.
There were quite a number of other improvements, bug fixes, and stability changes in this release, some noteworthy are:
Upgrade dependencies (Quarkus, etc) of Debezium UI DBZ-4109
Failed retriable operations are retried infinitely DBZ-4488
UI- Add the UI to configure the additional properties for a connector DBZ-5365
Capture events in order across mongodb shards DBZ-5590
DDL events not stored in schema history topic for excluded tables DBZ-6070
Oracle path used current batchSize to calculate end scn is wrong, need to use min batch size DBZ-6155
Upgrade UI build to use Debezium 2.2 or latest DBZ-6173
Oracle-Connector dbz##user needs more rights DBZ-6198
Make quay.io primary image repository DBZ-6216
Multiplatform build of example-postres fails DBZ-6258
Add protoc version property to postgres connector pom.xml DBZ-6261
Pass through configurations for kafka topics/configuration DBZ-6262
Postgres connector doesn’t need logical WAL level when snapshotting only DBZ-6265
Update config properties in RHEL deployment instructions DBZ-6266
MySQL connector doesn’t need to query binlog when snapshotting only DBZ-6271
Table names with spaces are not correctly deserialized when using an Infinispan cache as the transaction buffer DBZ-6273
Infinispan cache configuration used by Oracle tests are not compatible with Infinispan 14.0.2 DBZ-6274
Transaction buffer state can become corrupted when using Infinispan cache with LOBs DBZ-6275
Enable the docker tag to be configurable in the Spanner connector DBZ-6302
Upgrade MySQL JDBC driver to 8.0.32 DBZ-6304
Allow specifying docker image reference in MongoDB testcontainers implementation DBZ-6305
Use MongoDbContainer instead of MongoDBContainer test containers class in ConnectorConfiguration class DBZ-6306
DDL statement couldn’t be parsed - Oracle connector 2.1.3.Final DBZ-6314
Unparsable DDL statements (MySQL/MariaDB) DBZ-6316
Remove outdated information about SYS user accounts with Oracle DBZ-6318
Cassandra 3 cannot be built using JDK20 DBZ-6320
Bundle Jolokia with Debezium connect image DBZ-6323
Altogether, 34 issues were fixed for this release. A big thank you to all the contributors from the community who worked on this release: Anisha Mohanty, Bob Roldan, Chris Cranford, Harvey Yue, Jacob Gminder, Jiri Pechanec, Jochen Schalanda, Mario Fiore Vitale, Mark Bereznitsky, Ondrej Babec, Pengwei Dou, Robert Roldan, and Vojtech Juranek!
Outlook & What’s Next?
We are approaching the end of the Debezium 2.2 release cycle, with a final release expected this week. If there are any outstanding bugs or issues, please get in touch with us so that can be addressed prior to the final release.
The Debezium team is shifting their focus on Debezium 2.3. The Debezium 2.3 release will be a much more condensed and focused release, as our goal is to release it in late June. The Debezium roadmap has been updated and the following features are planned for this quarter:
Support configurable signaling channels
Support exactly once delivery semantics (phase 1)
Kubernetes operator for Debezium Server
Oracle OpenLogReplicator adapter proof-of-concept / incubating implementation
Debezium UI improvements
We would like to hear your feedback or suggestions, so if you have anything you’d like to share be sure to get in touch with us on the mailing list or our chat.
Until next time, let the changes continue to stream…
Debezium is an open source distributed platform that turns your existing databases into event streams, so applications can see and respond almost instantly to each committed row-level change in the databases. Debezium is built on top of Kafka and provides Kafka Connect compatible connectors that monitor specific database management systems. Debezium records the history of data changes in Kafka logs, so your application can be stopped and restarted at any time and can easily consume all of the events it missed while it was not running, ensuring that all events are processed correctly and completely. Debezium is open source under the Apache License, Version 2.0.
We hope you find Debezium interesting and useful, and want to give it a try. Follow us on Twitter @debezium, chat with us on Zulip, or join our mailing list to talk with the community. All of the code is open source on GitHub, so build the code locally and help us improve ours existing connectors and add even more connectors. If you find problems or have ideas how we can improve Debezium, please let us know or log an issue.