Featured Posts

Tags
apache kafka apache kafka apicurio avro aws batch caassandra camel cassandra cdc channels community community stories connectors containers cqrs custom datalake db2 ddd debezium debezium server debezium ui deduplication discussion docker elasticsearch event sourcing exactly once semantics example examples features fedora flink hiring ibmi iceberg informix integration introduction jaeger jdbc json kafka kafka streams kafka streams kogito ksql kubernetes lakehouse machine learning mariadb microservices mongo mongodb mysql news newsletter notifications online learning operator oracle outbox performance postgres presentation production quarkus questdb rds releases schema scylla secrets sentry serialization signaling smt snapshots spanner spark sql sqlserver tensorflow testcontainers tests time series timescaledb topics tracing transactions vagrant vitess website

It’s my great pleasure to announce the release of Debezium 1.8.0.Final!

Besides a strong focus on the Debezium connector for MongoDB (more on that below), the 1.8 release brings support for Postgres' logical decoding messages, support for configuring SMTs and topic creation settings in the Debezium UI, and much more.

Overall, the community has fixed 242 issues for this release. A big thank you to everyone who helped to make this release happen on time, sticking to our quarterly release cadence!

Improvements to the Debezium Connector for MongoDB

The team has made a strong push to bring multiple new features and improvements to the connector for MongoDB. It has now a brand-new capturing implementation based on MongoDB Change Streams, which allows for some very exciting new functionalities. More specifically, the connector now

  • Supports and has been tested with all the latest versions up to 5.0

  • Can optionally emit the complete document state for update events (by means of the Change Streams capability of reading back the entire document affected by change)

  • Provides support for incremental snapshots, as already known from the other Debezium connectors (more details on that in a separate blog post)

  • Helps you to implement the outbox pattern for microservices data exchange by means of an event routing SMT, specifically tailored to the event format emitted by this connector

Further Improvements

Besides the work on the MongoDB connector, many improvements and feature additions have been made to the other connectors. Amongst other things,

  • The names of transaction metadata topics are configurable

  • The Debezium UI has been further built out (see here and here for demos of this)

  • The Debezium connector for Postgres now supports logical decoding messages, as emitted using the pg_logical_emit_message() function

  • There’s a new snapshot mode SCHEMA_ONLY_RECOVERY for the Debezium connector for Oracle

  • The Debezium connector for Oracle supports TRUNCATE events and the binary.handling.mode option for controlling how BLOB data is exported

  • There’s support for remote Infinispan caches for buffering large Oracle transactions

  • The Debezium connector for MySQL now can export table comments; it also supports heartbeat action queries and schema changes while an incremental snapshot is running; in addition, it received many improvements to its DDL parser and character set handling

  • The Debezium connector for Vitess supports transaction metadata events, has an improved source struct, and supports re-sharding operations in a more flexible way

Please take a look at the original release announcements (Alpha1, Alpha2, Beta1, and CR1) as well as the 1.8 release notes in order to learn more about these and other new features of this release.

Many thanks to all the folks from the Debezium community which contributed code changes to this release:

Abhishek Hodavdekar, Alexander Schwartz, Andrey Yegorov, Anisha Mohanty, Ashique Ansari, Biel Garau Estarellas, Bob Roldan, Camile Sing, Chris Cranford, Christopher Burch, Claus Guttesen, Daniel PETISME, Fabian Martinez, Gunnar Morling, Guy Korland, Harvey Yue, Hossein Torabi, Hussain Ansari, Jacob Gminder, Jakub Cechacek, Jiabao Sun, Jiri Novotny, Jiri Pechanec, Jose Luis, Juan Fiallo, Judah Rand, Katerina Galieva, Lairen Hightower, Laurent Broudoux, 陆杰, Mario Mueller, Mark Drilling, Mike Kamornikov, Plugaru Tudor, René Kerner, Robert Roldan, Sergei Morozov, Shichao An, Sungho Hwang, Thiago Dantas, Tom Billiet, Ünal Sürmeli, Vadzim Ramanenka, Vivek Wassan, Vojtech Juranek, Willie Zhu, Yang Wu, and Zongwen Li!

What’s Next?

With another release shipped on schedule, it’s time for a break and take a rest over the upcoming holidays. We’ll be back to business in early January, with the planning for the 1.9 release being the first activity.

Please let us know about any requirements and feature requests you may have. One area we’d like to focus on for the next release is performance benchmarking and subsequentially applying performance improvements based on that. It also looks like there will be new community-led Debezium connector for a distributed NoSQL store; stay tuned for the details around this super-exciting development!

Later in the year, you also can expect the release of Debezium 2.0, where we’ll focus on cleaning up some inconsistencies and removing some deprecated features such as wal2json support in the Debezium connector for Postgres.

For now, we wish everybody a happy holiday season, and, if you’re into it, Merry Christmas! Please note the core team will be on PTO mostly for the coming weeks, so replies to emails, chat messages, issue reports, and pull requests will be slower than usual. Upwards and onwards!

Gunnar Morling

Gunnar is a software engineer at Decodable and an open-source enthusiast by heart. He has been the project lead of Debezium over many years. Gunnar has created open-source projects like kcctl, JfrUnit, and MapStruct, and is the spec lead for Bean Validation 2.0 (JSR 380). He’s based in Hamburg, Germany.

   

About Debezium

Debezium is an open source distributed platform that turns your existing databases into event streams, so applications can see and respond almost instantly to each committed row-level change in the databases. Debezium is built on top of Kafka and provides Kafka Connect compatible connectors that monitor specific database management systems. Debezium records the history of data changes in Kafka logs, so your application can be stopped and restarted at any time and can easily consume all of the events it missed while it was not running, ensuring that all events are processed correctly and completely. Debezium is open source under the Apache License, Version 2.0.

Get involved

We hope you find Debezium interesting and useful, and want to give it a try. Follow us on Twitter @debezium, chat with us on Zulip, or join our mailing list to talk with the community. All of the code is open source on GitHub, so build the code locally and help us improve ours existing connectors and add even more connectors. If you find problems or have ideas how we can improve Debezium, please let us know or log an issue.