Hello everyone, Jakub here. You may have noticed that there wasn’t much happening around Debezium UI lately. This, however, would be only partially true. We own you an explanation in this regard, so please bear with me.
Let’s start with the status of the current UI project. It became increasing clear that while UI for Debezium is an important part of our vision, developing a UI strictly tied to Kafka Connect is not the right direction. It was thus decided to freeze the development of the current web UI project. There were two main reasons for this decision:
-
The team doesn’t have the bandwidth (nor the desire) to develop a general purpose Kafka connector UI that would be on par with some of the existing offerings (e.g. Lenses). So our UI would only ever be able to work with Debezium connectors specifically.
-
Debezium Server is gaining popularity and we would like to offer a user friendly way of managing this flavour of Debezium as well. Unfortunately, the deployment model of Debezium Server is very different from deploying Debezium as a connector.
What’s Next?
As the previous paragraph mentioned the current UI, it’s probably not a suprise that we did not give up on the idea. Part of the decision was to explore a possibility of creating a new UI which, while still focused exclusively on Debezium, would gradually support multiple deployment models. The idea is to start with Debezium Server deployed on Kubernetes, followed by bare metal deployment of Debezium Server. At some point in the future we would like to cycle back to Kafka Connect and support also the deployment and operation of Debezium as Kafka Connect connectors.
Designing the New UI
One of the goals we defined for ourselves when designing the new platform was to simplify the understanding of what Debezium does. For this reason, we are going to abstract away from the underlying deployment model. Instead of focusing on servers and connectors, we want to allow our users to think about Debezium in terms of data. Thus the new generation of the UI should operate with data pipelines — you define the source, the destination, and any data transformations.
These can then be composed into a deployable pipeline. Depending on the target environment (Kafka Connect, Kubernetes, bare metal), an appropriate deployment will be used.
When pipeline is deployed to a Kubernetes or a bare metal machine
-
The Source is the configuration of the Debezium Source connector
-
Transformations are the usual SMTs
-
The Destination coresponds to the configuration of Debezium Server’s Sink.
When pipeline is deployed to Kafka Connect
-
The Source is the configuration of the Debezium Source connector
-
Transformations are the usual SMTs
-
The destination is always a Kafka topic
In the future, we could also allow our JDBC and MongoDB sink connectors to be used as destinations, allowing for truly end-to-end defined pipelines when Kafka Connect is targeted.
Where Do We Stand Right Now
We are at the begining of this journey. Nevetheless, there is a rough PoC prototype ready for you to examine. As already briefly mentioned, initially the plan is to target only Kubernetes as supported deployment environment, more specifically Debezium Server on Kubernetes. This choice allows us to take an advantage of our Debezium Operator.
We intend to introduce configuration and feature support gradually. Thus the configuration is deliberately opinionated and simplistic — currently all pipelines can only use ephemeral storage. In this early version, you must also set all configuration properties as simple key-value pairs. We intend to change this in the future, and as a next step, you can look forward to auto-completion of property names, and ultimately, fully specialized forms will be rendered.
You can review the current status of the codebase on GitHub:
We need your help!
At the end of this short report, we would like to ask you to try this early PoC and let us know your opinion. We would especially appreciate if you could let as know your answers to the following questions:
-
What features are a must for you?
-
What features would you consider nice to have?
-
What is your visual impression?
Thank you and expect to hear from us soon with more updates to come!
About Debezium
Debezium is an open source distributed platform that turns your existing databases into event streams, so applications can see and respond almost instantly to each committed row-level change in the databases. Debezium is built on top of Kafka and provides Kafka Connect compatible connectors that monitor specific database management systems. Debezium records the history of data changes in Kafka logs, so your application can be stopped and restarted at any time and can easily consume all of the events it missed while it was not running, ensuring that all events are processed correctly and completely. Debezium is open source under the Apache License, Version 2.0.
Get involved
We hope you find Debezium interesting and useful, and want to give it a try. Follow us on Twitter @debezium, chat with us on Zulip, or join our mailing list to talk with the community. All of the code is open source on GitHub, so build the code locally and help us improve ours existing connectors and add even more connectors. If you find problems or have ideas how we can improve Debezium, please let us know or log an issue.
