Featured Posts

Tags
apache kafka apache kafka apicurio avro aws batch caassandra camel cassandra cdc channels community community stories connectors containers cqrs custom datalake db2 ddd debezium debezium server debezium ui deduplication discussion docker elasticsearch event sourcing exactly once semantics example examples features fedora flink hiring ibmi iceberg informix integration introduction jaeger jdbc json kafka kafka streams kafka streams kogito ksql kubernetes lakehouse machine learning mariadb microservices mongo mongodb mysql news newsletter notifications online learning operator oracle outbox performance postgres presentation production quarkus questdb rds releases schema scylla secrets sentry serialization signaling smt snapshots spanner spark sql sqlserver tensorflow testcontainers tests time series timescaledb topics tracing transactions vagrant vitess website

It’s my pleasure to announce the release of Debezium 1.6.0.Beta2!

This release adds support for Pravega to Debezium Server, expands the snapshotting options of the Debezium Oracle connector, and provides a range of bug fixes and other improvements across different Debezium connectors.

Pravega Support for Debezium Server

With Debezium Server, we’re on a mission to bring open-source change data capture to all the users, no matter which data streaming platform or commit log they are using. So we are very happy to receive a contribution which adds support for Pravega to Debezium Server. A Cloud Native Computing Foundation (CNCF) sandbox and Apache 2.0 licensed open-source project, Pravega describes itself as a "storage abstraction for continuously generated and unbounded data".

The Debezium Server Pravega sink adapter offers two modes: non-transactional and transactional. The non-transactional mode individually writes each event in a Debezium batch to Pravega. The transactional mode writes the Debezium batch to a Pravega transaction that commits when the batch is completed.

To learn more about using Debezium with Pravega, please refer to the documentation. Many thanks to Derek Moore for this fantastic contribution!

Oracle Snapshotting Improvements

The Debezium connector for Oracle received two improvements related to snapshotting:

  • Support for the snapshot.include.collection.list option (DBZ-3062); this allows to create an initial snapshot only for a subset of all those tables captured by the connector

  • New option snapshot.locking.mode which provides control over the locking behavior when the connector captures the schema of the tables (DBZ-3557); in particular, this allows to disable locking completely, which is very useful if you can guarantee that no DDL changes are happening while the connector is taking the (schema) snapshot

In addition, there’s several bug fixes for this connector, including a few ones related to DDL and DML parsing (DBZ-3545, DBZ-3549, DBZ-3554, DBZ-3606), handling of RAC installations (DBZ-3563, DBZ-3599), and more efficient handling of LOB columns (DBZ-3556).

Further Improvements and Bugfixes

The Debezium connector for SQL Server saw two performance-related improvments (3486, DBZ-3515). The schemas of change events from the Postgres connector contain default values now, based on the source column definition (DBZ-2790). This comes in handy for instance when deriving downstream table schemas from a change event stream.

Other fixes include correct identification of primary members in MongoDB replica sets (DBZ-3522), support for the JSON function in the MySQL connector’s DDL parser (DBZ-3559), and the upgrade of the Debezium Quarkus extension for implementing the outbox pattern to Quarkus 2.0 (DBZ-3550).

Overall, 48 issues have been addressed in Debezium 1.6.0.Beta2. We’re deeply grateful to all the community members contributing to this release:

Anisha Mohanty, Bingqin Zhou, Bob Roldan, Chris Collingwood, Chris Cranford, Derek Moore, Eric Slep, Gilles Vaquez, Gunnar Morling, Jackey Zhang, Jakub Cechacek, Jiri Novotny, Jiri Pechanec, Kevin Pullin, Patrick Chu, Robert Roldan, Sara Fonseca, Sergei Morozov, Stefan Miklosovic, and Vadzim Ramanenka!

With Beta2 through the door, we’re entering the stabilization phase for the 1.6 release cycle. You can expect one or two CRs (candidate releases), before the final release, which is planned for the end of the month, barring any unforeseen complications of cause. Besides some more bug fixes and documentation improvements we’re also intending to upgrade to Apache Kafka 2.8, which will allow you to take a sneak peak at using Debezium with ZooKeeper-less Kafka!

In parallel, we’re going to work on the roadmap for Debezium 1.7 (due by the end of September). Please get in touch via the mailing list if you have specific feature requests for this release!

Gunnar Morling

Gunnar is a software engineer at Decodable and an open-source enthusiast by heart. He has been the project lead of Debezium over many years. Gunnar has created open-source projects like kcctl, JfrUnit, and MapStruct, and is the spec lead for Bean Validation 2.0 (JSR 380). He’s based in Hamburg, Germany.

   

About Debezium

Debezium is an open source distributed platform that turns your existing databases into event streams, so applications can see and respond almost instantly to each committed row-level change in the databases. Debezium is built on top of Kafka and provides Kafka Connect compatible connectors that monitor specific database management systems. Debezium records the history of data changes in Kafka logs, so your application can be stopped and restarted at any time and can easily consume all of the events it missed while it was not running, ensuring that all events are processed correctly and completely. Debezium is open source under the Apache License, Version 2.0.

Get involved

We hope you find Debezium interesting and useful, and want to give it a try. Follow us on Twitter @debezium, chat with us on Zulip, or join our mailing list to talk with the community. All of the code is open source on GitHub, so build the code locally and help us improve ours existing connectors and add even more connectors. If you find problems or have ideas how we can improve Debezium, please let us know or log an issue.