Featured Posts

Tags
apache kafka apache kafka apicurio avro aws batch caassandra camel cassandra cdc channels community community stories connectors containers cqrs custom datalake db2 ddd debezium debezium server debezium ui deduplication discussion docker elasticsearch event sourcing exactly once semantics example examples features fedora flink hiring ibmi iceberg informix integration introduction jaeger jdbc json kafka kafka streams kafka streams kogito ksql kubernetes lakehouse machine learning mariadb microservices mongo mongodb mysql news newsletter notifications online learning operator oracle outbox performance postgres presentation production quarkus questdb rds releases schema scylla secrets sentry serialization signaling smt snapshots spanner spark sql sqlserver tensorflow testcontainers tests time series timescaledb topics tracing transactions vagrant vitess website

I’m very happy to announce the release of Debezium 1.2.0.Final!

Over the last three months, the community has resolved nearly 200 issues. Key features of this release include:

  • New Kafka Connect single message transforms (SMTs) for content-based event routing and filtering; Upgrade to Apache Kafka 2.5

  • Schema change topics for the Debezium connectors for SQL Server, Db2 and Oracle

  • Support for SMTs and message converters in the Debezium embedded engine

  • Debezium Server, a brand-new runtime which allows to propagate data change events to a range of messaging infrastructures like Amazon Kinesis, Google Cloud Pub/Sub, and Apache Pulsar

  • A new column masking mode "consistent hashing", allowing to anonymize column values while still keeping them correlatable

  • New metrics for the MongoDB connector

  • Improved re-connect capability for the SQL Server connector

Debezium Server

Should I pick one feature I’m the most excited about, it would be Debezium Server. It allows even more users to benefit from open-source change data capture with Debezium, no matter which messaging infrastructure they are on. Being able to propagate Debezium data change events via managed services such as Kinesis and Pub/Sub was a feature request we’ve received again and again from the community, and it’s amazing to see that this feature eventually becomes a reality.

Debezium Server will also be a great foundation for exposing Debezium as a native Knative Eventing event source, and you can expect to see more exciting developments in this area very soon. With Debezium Server, there’s now three ways for running Debezium:

  • Via Kafka Connect, using its fantastic eco-system of connectors to set up low-latency data streaming pipelines with Apache Kafka

  • With Debezium Server, sending data change events to a growing number of messaging platforms like Kinesis

  • Through the Debezium Engine, embedded as a library into custom JVM-based applications, e.g. addressing use cases like updating embedded caches

I’m really excited about all the opportunties which this brings to the Debezium community. Debezium Server is powered by the innovative Quarkus framework, which also opens up many interesting technical possibilities. E.g. we could explore running Debezium connectors as native binaries via GraalVM, resulting in a largely reduced memory consumption, which could make this a very interesting deployment option for cloud environments.

A Big Thank You to the Community!

Please refer to the original announcements (Alpha1, Beta1, Beta2, CR1) to learn more about all the new features of Debezium 1.2. You can find the complete list of addressed issues, upgrade procedures, and notes on any backward compatibility changes in the release notes.

An open-source project would be nothing without its community of users in contributors; thanks a lot to the following people from the community who worked on the Debezium 1.2 release:

Aaron Brady, Alexander Iskuskov, Alexander Schwartz, Anders Engström, Anton Kondratev, Andy Teijelo Pérez, Balázs Németh, Bingqin Zhou, Braden Groom, Brandon Brown, cobolbaby, David Jerome, Dave Cumberland, Emmanuel Brard, Ed Laur, Fabian Aussems, Fatih Güçlü Akkaya, Fándly Gergő, Grant Cooksey, Hossein Torabi, Ivan Klass, Ivan Trusov, Jan-Hendrik Dolling, Jeremy Finzel, Joy Gao, Juan Antonio Pedraza, Justin Hiza, Jos Huiting, John Psoroulas, Kewei Shang, Linh Nguyen Hoang, Luis Garcés-Erice, Lukas Krejci, lyidataminr, Max Kaplan, Nayana Hettiarachchi, Peter Urbanetz, René Kerner, Robert B. Hanviriyapunt, Siddhant Agnihotry, and Xuan Shen

This brings the total number of contributors to the main Debezium code repositories to over 200!

Outlook

Let’s close this post with an outlook to the next things to come. We’ll stick to our quarterly release cadence, i.e. you can expect Debezium 1.3 to be out by the end of September, with preview releases in between every three weeks.

We’re currently updating the roadmap for the next release, and your input and feedback on this will be very welcomed! The things we’ve planned so far include more flexible options for snapshotting (re-snapshotting specific tables, filter changes, parallelized snapshotting, etc.), moving Db2 connector out of incubating state, and an exploration of what it’d take to officially support MariaDB.

Onwards and upwards!

Gunnar Morling

Gunnar is a software engineer at Decodable and an open-source enthusiast by heart. He has been the project lead of Debezium over many years. Gunnar has created open-source projects like kcctl, JfrUnit, and MapStruct, and is the spec lead for Bean Validation 2.0 (JSR 380). He’s based in Hamburg, Germany.

   

About Debezium

Debezium is an open source distributed platform that turns your existing databases into event streams, so applications can see and respond almost instantly to each committed row-level change in the databases. Debezium is built on top of Kafka and provides Kafka Connect compatible connectors that monitor specific database management systems. Debezium records the history of data changes in Kafka logs, so your application can be stopped and restarted at any time and can easily consume all of the events it missed while it was not running, ensuring that all events are processed correctly and completely. Debezium is open source under the Apache License, Version 2.0.

Get involved

We hope you find Debezium interesting and useful, and want to give it a try. Follow us on Twitter @debezium, chat with us on Zulip, or join our mailing list to talk with the community. All of the code is open source on GitHub, so build the code locally and help us improve ours existing connectors and add even more connectors. If you find problems or have ideas how we can improve Debezium, please let us know or log an issue.