Featured Posts

Tags
apache kafka apache kafka apicurio avro aws batch caassandra camel cassandra cdc channels community community stories connectors containers cqrs custom datalake db2 ddd debezium debezium server debezium ui deduplication discussion docker elasticsearch event sourcing exactly once semantics example examples features fedora flink hiring ibmi iceberg informix integration introduction jaeger jdbc json kafka kafka streams kafka streams kogito ksql kubernetes lakehouse machine learning mariadb microservices mongo mongodb mysql news newsletter notifications online learning operator oracle outbox performance postgres presentation production quarkus questdb rds releases schema scylla secrets sentry serialization signaling smt snapshots spanner spark sql sqlserver tensorflow testcontainers tests time series timescaledb topics tracing transactions ui vagrant vitess website

We are happy to announce the first pre-release of Debezium 3, 3.0.0.Alpha1. This release, albeit smaller than our normal pre-releases, is highly focused on a few key points, such as testing the release process with Java 17/21; however, it also includes several new features. Let’s take a moment and talk about the upcoming breaking changes in-depth and the new features you will find.

Breaking changes

With any new major release of software, there is often several breaking changes. The Debezium 3.0.0.Alpha1 release is no exception, so lets discuss the major changes you should be aware about.

Java and Maven requirements changed

This release also shifts the Java requirements needed to both build and run Debezium. In addition, this release also requires a later version of Maven for building Debezium from source.

If you intend to build Debezium from source for any reason, all Debezium projects require using Java 21 to build and Maven 3.9.8. This includes the main repository, community-led repositories, Debezium Server, Operator, etc. If you attempt to build Debezium from source using a version of Java that is less-than Java 21, the build will report a Java version error, instructing you to use Java 21 or higher.

All Debezium connectors require a baseline of Java 17 for runtime. This means if you are not building from source but simply deploying the connectors on runtimes such as Kafka Connect, then Java 17 is sufficient.

If you are using Debezium Server, Debezium Operator, or the Debezium Quarkus Outbox Extension, these components all require Java 21 for both runtime and build-time.

Please see the following chart from an at-a-glance view of the Java and Maven requirements

Table 1. Requirements by component
Component Java (Runtime) Java (Building) Maven (Building)

Debezium Server

Java 21+

Java 21+

3.9.8+

Debezium Operator

Java 21+

Java 21+

3.9.8+

Debezium Quarkus Outbox Extension

Java 21+

Java 21+

3.9.8+

Debezium Connectors
Embedded Engine or Kafka Connect

Java 17+

Java 21+

3.9.8+

New features and improvements

The upgrade to Debezium 3.0.0.Alpha1 introduces several new features and improvements in several components:

MongoDB

MongoDB Sink Connector

Debezium introduced its first sink-based connector in Debezium 2.2, just over a year ago and we’re pleased to announce the inclusion of another sink-based connector for MongoDB as a part of Debezium 3.

Unlike the JDBC sink relational connector that requires an additional plug-in to be installed to use it, the MongoDB sink connector is bundled alongside the MongoDB source connector in the same artifact. So if you have already installed or use the MongoDB source connector and are using Debezium 3 or later, you also have the MongoDB sink connector.

The configuration to get started with the MongoDB is quite straightforward, here’s an example:

{
  "connector.class": "io.debezium.connector.mongodb.MongoDbSinkConnector",
  "connection.string": "...",
  "topics": "topic1,topic2",
  "sink.database": "targetdb",
}

The connection.string and sink.database configuration properties are mandatory. These define the details for connecting to the target MongoDB database and the name of the target database where the changes will be written.

Additionally, the topics configuration property is mandatory by Kafka Connect, and it describes a comma-separated list of regular expressions for the topics that the sink connector will observe.

Documentation for this connector is still a work-in-progress, so if you have any questions or problems, don’t hesitate to reach out to the team on our Zulip chat or Mailing List.

Other changes

  • Debezium postgres jdbc sink not handling infinity values DBZ-7920

  • JdbcSinkTask doesn’t clear offsets on stop DBZ-7946

  • ibmi as400 connector config isn’t prefixed with "database." DBZ-7955

  • Duplicate downstream annotation comments incorrectly refer to Db2 connector DBZ-7968

  • Issue with Hybrid mode and DDL change DBZ-7991

  • Extends process of finding Bundle path DBZ-7992

  • Incorrect offset/history property mapping generatated DBZ-8007

  • Debezium Server Operator on minikube with java.lang.NullPointerException': java.lang.NullPointerException DBZ-8019

  • ORA-65090: operation only allowed in a container database when connecting to a non-CDB database DBZ-8023

  • Added type to Prometheus JMX exporter DBZ-8036

  • Add kafka.producer metrics to debezium-server jmx exporter config DBZ-8037

  • Support FLOAT32 type in debezium-connector-spanner DBZ-8043

In total, 20 issues were resolved in Debezium 3.0.0.Alpha1. The list of changes can also be found in our release notes.

A big thank you to all the contributors from the community who worked diligently on this release: Andreas Martens, Aravind, Chris Cranford, Jakub Cechacek, Jiri Pechanec, Mario Fiore Vitale, Ondrej Babec, René Kerner, Ricardo Rosales, Robert Roldan, Ryan van Huuksloot, and Shuran Zhang!

What’s next & Outlook

With Debezium 2.7.0.Final released, we’re actively looking at the reports from the community and addressing any critical bugs or regressions that were not identified throughout development. If you believe you’ve spotted a bug or regression, please log a Jira issue so that these can be tracked and addressed. Of course, contributions for these are much appreciated, if you have the bandwidth.

While this pre-release is smaller than we generally prefer, it’s focus was primarily on the inclusion of the MongoDB sink connector and the move to Java 17/21. There are still many features in the pipeline that will be coming in a future build of Debezium, including but not limited to:

  • Kafka 3.1+ as baseline

  • New off-heap Oracle cache implementations based on EhCache & Hazelcast

  • Exactly-once semantics support for other relational connectors

  • and more

This list represents a quick glance view at just what’s at the top of our queue, and is subject to change. If you’d like to get involved in the conversation about Debezium 3.0 and the next evolution of the project, contact us on the mailing list or Zulip chat. As always, please review our road map for more details.

For now, that’s it from our side, enjoy the summer and be safe! Until next time…​

Chris Cranford

Chris is a software engineer at Red Hat. He previously was a member of the Hibernate ORM team and now works on Debezium. He lives in North Carolina just a few hours from Red Hat towers.

   

About Debezium

Debezium is an open source distributed platform that turns your existing databases into event streams, so applications can see and respond almost instantly to each committed row-level change in the databases. Debezium is built on top of Kafka and provides Kafka Connect compatible connectors that monitor specific database management systems. Debezium records the history of data changes in Kafka logs, so your application can be stopped and restarted at any time and can easily consume all of the events it missed while it was not running, ensuring that all events are processed correctly and completely. Debezium is open source under the Apache License, Version 2.0.

Get involved

We hope you find Debezium interesting and useful, and want to give it a try. Follow us on Twitter @debezium, chat with us on Zulip, or join our mailing list to talk with the community. All of the code is open source on GitHub, so build the code locally and help us improve ours existing connectors and add even more connectors. If you find problems or have ideas how we can improve Debezium, please let us know or log an issue.