Featured Posts

Tags
analytics announcement announcement discussion survey apache kafka apache kafka apicurio avro aws batch caassandra camel cassandra cdc channels community community stories connectors containers cqrs custom datalake db2 ddd debezium debezium server debezium ui deduplication discussion docker elasticsearch event sourcing exactly once semantics example examples features fedora flink hiring ibmi iceberg images informix integration introduction jaeger jdbc json kafka kafka streams kafka streams kogito ksql kubernetes lakehouse machine learning mariadb metrics microservices mongo mongodb monitoring mysql news newsletter notifications online learning operator oracle outbox performance postgres presentation production quarkus questdb rds releases schema scylla secrets sentry serialization signaling smt snapshots spanner spark sql sqlserver tensorflow testcontainers tests time series timescaledb topics tracing transactions ui vagrant vitess website

I am very happy to announce the release of Debezium 1.7.0.CR1!

For this release, we’ve reworked how column filters are handled during snapshotting, the Debezium container images have been updated to use Fedora 34 as their base, there’s support for MySQL INVISIBLE columns, and much more.

Column Filtering During Snapshotting

While the different Debezium connectors already had the capability to exclude specific columns of the captured tables from change events, these filters were only applied when processing the data within the connectors. For initial snapshots, a more efficient approach has been implemented now: tailored SQL SELECT statements will be executed for fetching only the actually included columns (DBZ-2525). This allows for significant performance gains when for instance excluding large BLOB columns from change events.

Updated Container Image Base

The Debezium container images for Apache Kafka, Kafka Connect, and Apache ZooKeeper are based on the Fedora 34 minimal container base image (DBZ-3939). This change became necessary as the previously used base image (derived from CentOS 7) was not maintained any longer. While this change will be transparent for most users of Debezium, some adjustments may be required for those users who derive their own custom images from the Debezium ones, e.g. when installing further packages using the operating system’s package manager. Please refer to the release notes for more details.

Further Fixes

As we’re approaching the 1.7 Final release, most changes have been centered around bug fixing and maturing the code base. Some of the resolved issues include:

  • Support for INVISIBLE columns as available since MySQL 8.0.23 (DBZ-3623); we’ve used that occassion to also update the Debezium example image for MySQL to version 8.0 (DBZ-3936)

  • The SQL Server allows for the usage of custom connection factories (DBZ-4001)

  • Several fixes to DML and DDL parsing for MySQL (DBZ-3969, DBZ-3984) and Oracle (DBZ-3892, DBZ-3962)

Altogether, 47 issues have been fixed for this release. A big thank you to all contributors: Anisha Mohanty, Ashmeet Lamba, Bob Roldan, Camile Sing, Chris Cranford, Dhrubajyoti G, Gunnar Morling, Harvey Yue, Hussain Ansari, Indra Shukla, Jakub Cechacek, Jiabao Sun, Jiri Novotny, Jiri Pechanec, Katerina Galieva, René Kerner, and Yuan Zhang.

Going forward, we’re planning to do another CR (candidate release) in a few days, followed by Debezium 1.7.0.Final at the end of the month. We’ll primarily focus on bug fixing and some asorted performance optimizations. There’ll also be some exciting improvements to the Debezium UI, which should be wrapped up for the 1.7 Final release: support for the configuration of single message transforms (SMTs), as well as the ability to configure topic creation settings.

In parallel, we’re working on the roadmap for Debezium 1.8, planned to be released by the end of the year. Please reach out in the comments below or on the mailing list if you’d like to raise specific feature requests for this release.

Gunnar Morling

Gunnar is a software engineer at Decodable and an open-source enthusiast by heart. He has been the project lead of Debezium over many years. Gunnar has created open-source projects like kcctl, JfrUnit, and MapStruct, and is the spec lead for Bean Validation 2.0 (JSR 380). He’s based in Hamburg, Germany.

   

About Debezium

Debezium is an open source distributed platform that turns your existing databases into event streams, so applications can see and respond almost instantly to each committed row-level change in the databases. Debezium is built on top of Kafka and provides Kafka Connect compatible connectors that monitor specific database management systems. Debezium records the history of data changes in Kafka logs, so your application can be stopped and restarted at any time and can easily consume all of the events it missed while it was not running, ensuring that all events are processed correctly and completely. Debezium is open source under the Apache License, Version 2.0.

Get involved

We hope you find Debezium interesting and useful, and want to give it a try. Follow us on Twitter @debezium, chat with us on Zulip, or join our mailing list to talk with the community. All of the code is open source on GitHub, so build the code locally and help us improve ours existing connectors and add even more connectors. If you find problems or have ideas how we can improve Debezium, please let us know or log an issue.