Featured Posts

Tags
apache kafka apache kafka apicurio avro aws batch caassandra camel cassandra cdc channels community community stories connectors containers cqrs custom datalake db2 ddd debezium debezium server debezium ui deduplication discussion docker elasticsearch event sourcing exactly once semantics example examples features fedora flink hiring ibmi iceberg informix integration introduction jaeger jdbc json kafka kafka streams kafka streams kogito ksql kubernetes lakehouse machine learning mariadb microservices mongo mongodb mysql news newsletter notifications online learning operator oracle outbox performance postgres presentation production quarkus questdb rds releases schema scylla secrets sentry serialization signaling smt snapshots spanner spark sql sqlserver tensorflow testcontainers tests time series timescaledb topics tracing transactions vagrant vitess website

The Debezium UI team is pleased to announce support for Single Message Transformations (SMTs) in the Debezium UI!

Our goal with the Debezium graphical user interface is to allow users to set up and operate connectors more easily. To that end, we have added support for Kafka Connect single message transformations to the UI. Read futher for more information, and for a video demo of the new feature!

Single Message Transformations (SMTs)

Connectors can be configured with transformations to make lightweight per message modifications. Common SMT use cases include format conversions (e.g. different date formats and number types), message filtering and routing, handling of "tombstone" events, encryption/decryption, etc.

Debezium provides several single message transformations (SMTs) that you can use to either modify records before they are sent to Apache Kafka (by applying them to the Debezium connectors), or when they are read from Kafka by a sink connector. For instance we provide SMTs for extracting only the "after" part from change events and propagate that one in a flat row format and SMTs for routing the events from an outbox table.

To learn more about the SMTs coming with Debezium, please refer to the reference documentation. And thanks to the support for SMTs in the Debezium UI, setting them up is easier than ever; For a short demo of this feature in action, see the following video:

Fun fact: this video is the very first entry to our brand-new Debezium YouTube channel! We recommend you subscribe to the channel to never miss any new videos.

Trying It Out Yourself

We have created a self-contained example UI demo, which is included under debezium-examples on Github. The UI demo includes a Docker Compose file which brings up several sources with data as well as the UI. Please refer to the README file for more details on running the Debezium UI demo.

To learn more about the Debezium UI, please refer to the reference documentation.

Next Steps

We plan to continue with improvements and new features for the UI in the coming releases. Some items under consideration:

  • Incorporation of more Debezium connector types, such as the ones for SQL Server and Oracle

  • Addition and improvement of connector metrics and monitoring

  • Add capability for viewing and editing connector properties after creation

  • …​And more!

We’d also be very happy to learn about your requirements and feedback on the Debezium UI. Please let us know in the comments below, or send a message to our mailing list.

A big thank you to the team who have contributed in many ways: Ashique Ansari, Indra Shukla, René Kerner and Gunnar Morling!

Mark Drilling

Mark is a software developer at Red Hat. He has spent most of his career at Red Hat working on various Data Virtualization projects, and now works on Debezium. He lives in O'Fallon, Missouri USA.

 

About Debezium

Debezium is an open source distributed platform that turns your existing databases into event streams, so applications can see and respond almost instantly to each committed row-level change in the databases. Debezium is built on top of Kafka and provides Kafka Connect compatible connectors that monitor specific database management systems. Debezium records the history of data changes in Kafka logs, so your application can be stopped and restarted at any time and can easily consume all of the events it missed while it was not running, ensuring that all events are processed correctly and completely. Debezium is open source under the Apache License, Version 2.0.

Get involved

We hope you find Debezium interesting and useful, and want to give it a try. Follow us on Twitter @debezium, chat with us on Zulip, or join our mailing list to talk with the community. All of the code is open source on GitHub, so build the code locally and help us improve ours existing connectors and add even more connectors. If you find problems or have ideas how we can improve Debezium, please let us know or log an issue.