Featured Posts

Tags
analytics announcement announcement discussion survey apache kafka apache kafka apicurio avro aws batch caassandra camel cassandra cdc channels community community stories connectors containers cqrs custom datalake db2 ddd debezium debezium server debezium ui deduplication discussion docker elasticsearch event sourcing exactly once semantics example examples features fedora flink hiring ibmi iceberg images informix integration introduction jaeger jdbc json kafka kafka streams kafka streams kogito ksql kubernetes lakehouse machine learning mariadb metrics microservices mongo mongodb monitoring mysql news newsletter notifications online learning operator oracle outbox performance postgres presentation production quarkus questdb rds releases schema scylla secrets sentry serialization signaling smt snapshots spanner spark sql sqlserver tensorflow testcontainers tests time series timescaledb topics tracing transactions ui vagrant vitess website

I’m very happy to announce the release of Debezium 1.5.0.Beta1!

This release adds a brand-new component — the web-based Debezium UI --, transaction metadata support for the MySQL connector, a large number of improvements to the LogMiner-based capture implementation for the Debezium Oracle connector, support for Vitess 9.0, and much more. Let’s explore some of the new features in the following.

Debezium UI

The different Debezium connectors provide great power and flexibility for setting up and running change data capture sources for a range of databases. But this flexibility also comes at a cost: getting started with the connectors can take some time for understanding all the different options and their semantics. Another critical aspect is operating the connectors, i.e. gaining insight into their current status and metrics, being able to react to connector failures, and more etc.

Based on this feedback from the community, we have explored over the last few months how a graphical user interface could help with these matters. The initial proof-of-concept looked very promising, so we decided to move forward and make the UI an official component of the Debezium project. Still under active development, you already can try out the UI today (available as a container image on Docker Hub) and use it to set up Debezium connectors in your Kafka Connect clusters.

We’ll follow up with more details on the Debezium UI in a separate blog post within the next few days, discussing its current status, the roadmap for this component, and more.

Improved LogMiner-based CDC Implementation

Continuing our current focus on the LogMiner-based CDC implementation for Oracle, we’ve fixed a substantial number of issues for this connector. Amongst them are:

  • Drastically improved DML parsing performance (DBZ-3078); a new hand-written parser for the LogMiner DML statements allows for better throughput of this connector, the existing external parser implementation will be removed very soon

  • Support for capturing changes from multiple schemas (DBZ-3009)

  • Support for column filtering (DBZ-3167)

  • Correct transaction metadata (DBZ-3090)

  • Several bug fixes related to log file switching and similar (DBZ-2754, DBZ-3001, DBZ-3153, etc.)

Vitess Connector

Led by community member Kewei Shang, the Debezium connector for Vitess now supports Vitess 9.0 (DBZ-3100). The connector also can capture changes from JSON and ENUM columns (DBZ-3115, DBZ-3124), and it implements the configuration validation API of Kafka Connect (DBZ-3117).

Other Features

Further fixes and improvements in this release including the following:

  • The Debezium MySQL connector can expose metadata about transaction boundaries (DBZ-3114); this is one of the first benefits we obtain by rebasing this connector onto the common Debezium connector framework, as discussed in the 1.5.0.Alpha1 release announcement

  • The Debezium connector for Postgres is tested and validated against PG 13 (DBZ-3022)

  • Ability to customize offsets when using the Debezium embedded API (DBZ-2897)

  • Support for CREATE OR REPLACE INDEX DDL when using the MySQL connector with MariaDB (DBZ-3067)

  • Infinite timestamp values supported with Postgres (DBZ-2614)

Altogether, a grand total of 78 issues have been addressed for this release.

As always, a big thanks you to all the community members who contributed: Adhaam Ehab, Ahmed Eljami, Bingqin Zhou, Fahim Farook, Ganesh Ramasubramanian, Harvey Yue, Kevin Pullin, Kewei Shang, Michael Cizmar, Nitin Agarwal, Russell Ballard, Shane Paul, Thomas Prelle, Thomas Thornton, and Yilong Chang.

Gunnar Morling

Gunnar is a software engineer at Decodable and an open-source enthusiast by heart. He has been the project lead of Debezium over many years. Gunnar has created open-source projects like kcctl, JfrUnit, and MapStruct, and is the spec lead for Bean Validation 2.0 (JSR 380). He’s based in Hamburg, Germany.

   

About Debezium

Debezium is an open source distributed platform that turns your existing databases into event streams, so applications can see and respond almost instantly to each committed row-level change in the databases. Debezium is built on top of Kafka and provides Kafka Connect compatible connectors that monitor specific database management systems. Debezium records the history of data changes in Kafka logs, so your application can be stopped and restarted at any time and can easily consume all of the events it missed while it was not running, ensuring that all events are processed correctly and completely. Debezium is open source under the Apache License, Version 2.0.

Get involved

We hope you find Debezium interesting and useful, and want to give it a try. Follow us on Twitter @debezium, chat with us on Zulip, or join our mailing list to talk with the community. All of the code is open source on GitHub, so build the code locally and help us improve ours existing connectors and add even more connectors. If you find problems or have ideas how we can improve Debezium, please let us know or log an issue.