Featured Posts

Tags
apache kafka apache kafka apicurio avro aws batch caassandra camel cassandra cdc channels community community stories connectors containers cqrs custom datalake db2 ddd debezium debezium server debezium ui deduplication discussion docker elasticsearch event sourcing exactly once semantics example examples features fedora flink hiring ibmi iceberg informix integration introduction jaeger jdbc json kafka kafka streams kafka streams kogito ksql kubernetes lakehouse machine learning mariadb microservices mongo mongodb mysql news newsletter notifications online learning operator oracle outbox performance postgres presentation production quarkus questdb rds releases schema scylla secrets sentry serialization signaling smt snapshots spanner spark sql sqlserver tensorflow testcontainers tests time series timescaledb topics tracing transactions vagrant vitess website

We are pleased to announce the first official release of the Debezium graphical user interface!

As announced a few months back, our team has been working on a Debezium UI proof-of-concept. The goal of the PoC was to explore ways in which a graphical UI could facilitate the getting started and operational experience of Debezium users.

Debezium is very flexible - each connector can be configured and fine-tuned in a variety of ways. It provides metrics which give the user insight into the state of the running Debezium connectors, allowing the customer to safely operate CDC pipelines in huge installations with thousands of connectors. This flexibility, however, comes with a learning curve for the user to understand all of the different settings and options.

To that end, we have produced a UI which will allow users to set up and operate connectors more easily. The UI is now available as part of the Debezium releases for our community!

Let’s take a closer look at some features of the UI in the following.

UI Connector List

The main page of the UI displays all the registered connectors. Some of the highlights of the main page are as follows:

  • Kafka connect cluster can be selected via the dropdown in the header.

  • Connector table shows each connector with it’s type (MySQL, PostgreSQL, MongoDB), connector status and connector tasks.

  • A connector row can be expanded to show more details, as shown below with the 'testPostgres' connector. Metrics are shown in the expansion area (Note: this feature is still under development and not functional yet). Connector tasks are shown, with ability to Restart the task if desired.

  • The kebab menu at the right of each connector row provides actions which allow the user to Pause, Resume, Restart or Delete the connector.

UI Create Connector Wizard

The user can create a connector by clicking on the Create a connector button on the main page. The first two steps of the wizard are required, but the remaining steps are optional. Each step will validate the user entries and provide feedback if there are problems. After completing steps 1 and 2 successfully, the user can proceed to the final page to review and create the connector.

Create Connector - Connector type (required)

Choose the type of connector in step 1. Currently the MongoDB, MySQL and PostgreSQL connector types are supported. Addition of more connector types is currently in progress.

Create Connector - Properties (required)

The basic connection properties for the selected connector are entered in step 2, and the properties must be validated before proceeding. Advanced connection properties are also provided in a separate section of this step. Upon successful validation, the user may proceed to the next steps (Additional properties) - or they can elect to bypass the additional properties and proceed directly to Review.

Create Connector - Additional properties (optional)

The Additional properties are optional and can be summarized as follows:

  • Filter definition - entry of regular expressions which define the filters for inclusion/exclusion of the items that will be included for CDC. The included items are displayed as the filters are entered and applied.

  • Data options - Snapshot and Mapping properties (optional). The defaults can be viewed and changed if desired.

  • Runtime options - Engine and Heartbeat properties (optional). The defaults can be viewed and changed if desired.

Create Connector - Review

The Review step provides a summary of the configuration that will be used to create the connector. If happy with the selections, click 'Finish' to create the connector. If the properties need adjustment, navigate back to the earlier steps.

Design Considerations

The UI is implemented as a Quarkus-based web application. The backend is configured with the URL(s) of one or more Kafka Connect clusters and provides a REST interface for the frontend. The frontend user interface uses ReactJS as the primary technology, utilizing Patternfly react components and design patterns.

As with everything in Debezium, the UI is fully open source (Apache License Version 2.0). You can find the UI Source Code under the Debezium organization on Github.

Trying It Out Yourself

Debezium UI Container Image

The Debezium UI container image is available for running the UI. E.g. run the following comand to start the UI and connect it to an existing Kafka Connect instance via Docker (where KAFKA_CONNECT_URI supplies a comma-separated list of the available Kafka Connect URI(s)):

$ docker run -it --rm --name debezium-ui -p 8080:8080 -e KAFKA_CONNECT_URI=http://connect:8083 debezium/debezium-ui:1.7

The UI connects to Kafka Connect via REST, so you need to make sure that the latter is reachable, e.g. by running both components on the same Docker network.

Currently, the UI connects to un-authenticated Kafka Connect instances. Also, there’s no authorization or authentication implemented in the UI itself yet. Until that is the case, you should secure the components e.g. with your own proxy for authorization, if needed.

Self-contained example

We have also created a self-contained example UI demo, which is included under debezium-examples on Github. The UI demo includes a Docker Compose file which brings up several sources with data as well as the UI. Please refer to the README file for more details on running the Debezium UI demo.

To learn more about the Debezium UI, please refer to the reference documentation

Next Steps

We plan to continue with improvements and new features for the UI in the coming releases. Some items under consideration:

  • Incorporation of more Debezium connector types, such as the ones for SQL Server and Oracle

  • Add capability to configure topic creation settings and single message transformations

  • Addition and improvement of connector metrics and monitoring

  • Add capability for viewing and editing connector properties after creation

  • …​And more!

We’d also be very happy to learn about your requirements and feedback on the Debezium UI. Please let us know in the comments below, or send a message to our mailing list.

A big thank you to the team who have contributed in many ways: Ashique Ansari, Indra Shukla, June Zhang, Na Ding, René Kerner and Gunnar Morling!

Mark Drilling

Mark is a software developer at Red Hat. He has spent most of his career at Red Hat working on various Data Virtualization projects, and now works on Debezium. He lives in O'Fallon, Missouri USA.

 

About Debezium

Debezium is an open source distributed platform that turns your existing databases into event streams, so applications can see and respond almost instantly to each committed row-level change in the databases. Debezium is built on top of Kafka and provides Kafka Connect compatible connectors that monitor specific database management systems. Debezium records the history of data changes in Kafka logs, so your application can be stopped and restarted at any time and can easily consume all of the events it missed while it was not running, ensuring that all events are processed correctly and completely. Debezium is open source under the Apache License, Version 2.0.

Get involved

We hope you find Debezium interesting and useful, and want to give it a try. Follow us on Twitter @debezium, chat with us on Zulip, or join our mailing list to talk with the community. All of the code is open source on GitHub, so build the code locally and help us improve ours existing connectors and add even more connectors. If you find problems or have ideas how we can improve Debezium, please let us know or log an issue.