Featured Posts

Tags
analytics announcement announcement discussion survey apache kafka apache kafka apicurio avro aws batch caassandra camel cassandra cdc channels community community stories connectors containers cqrs custom datalake db2 ddd debezium debezium server debezium ui deduplication discussion docker elasticsearch event sourcing exactly once semantics example examples features fedora flink hiring ibmi iceberg images informix integration introduction jaeger jdbc json kafka kafka streams kafka streams kogito ksql kubernetes lakehouse machine learning mariadb metrics microservices mongo mongodb monitoring mysql news newsletter notifications online learning operator oracle outbox performance postgres presentation production quarkus questdb rds releases schema scylla secrets sentry serialization signaling smt snapshots spanner spark sql sqlserver tensorflow testcontainers tests time series timescaledb topics tracing transactions ui vagrant vitess website

The Debezium UI team continues to add support for more features, allowing users to more easily configure connectors. In this article, we’ll describe and demonstrate the UI support for topic automatic creation. Read further for more information, including a video demo!

Topic Auto-creation

When you start a Debezium connector, the topics for the captured events are created by the Kafka broker based on a default, possibly customized, broker configuration (if auto.create.topics.enable = true). But often when you use Debezium and Kafka in a production environment, you might choose to disable Kafka’s topic auto creation capability (auto.create.topics.enable = false), or you want the connector topics to be configured differently from the default. In this case you’ll need to create topics for Debezium’s captured data sources upfront.

Beginning with Kafka 2.6.0, Kafka Connect provides means of customizing the settings of specififc topics created by source connectors such as Debezium (KIP-158). If Kafka Connect topic creation is enabled (topic.creation.enable = true), the Debezium UI now allows you to configure connector topics using the UI.

Kafka Connect Topic Creation

Kafka Connect topic creation works with groups. There is a default group, which is used when there is no other group defined that matches the topic.

You can also define multiple custom topic groups, each with it’s own configuration. Each group can specify its configuration parameters to customize how the matched topics of the group will be created. The custom groups will fall back to the default group settings for the required replication.factor and partitions properties. If the configuration for a custom topic group leaves other properties undefined, the values specified in the default group are not applied.

To find more detail about topic auto-creation with Debezium, please refer to the reference documentation. You can also refer to this blog post for a full example. Watch the following video for a quick demo of topic creation in the Debezium UI:

Self-contained Example

You can try out topic auto-creation (and more) with our self-contained example UI demo - which is included under debezium-examples on GitHub. The UI demo includes a Docker Compose file which brings up several sources with data as well as the UI. Please refer to the README file for more details on running the Debezium UI demo.

To learn more about the Debezium UI, please refer to the reference documentation.

More coming soon!

Stay tuned for further improvements and new features in the UI in the coming releases. Support for SQL Server and Oracle connectors are coming soon!

A big thank you to the team who have contributed in many ways: Ashique Ansari, Indra Shukla, René Kerner and Gunnar Morling!

Mark Drilling

Mark is a software developer at Red Hat. He has spent most of his career at Red Hat working on various Data Virtualization projects, and now works on Debezium. He lives in O'Fallon, Missouri USA.

 

About Debezium

Debezium is an open source distributed platform that turns your existing databases into event streams, so applications can see and respond almost instantly to each committed row-level change in the databases. Debezium is built on top of Kafka and provides Kafka Connect compatible connectors that monitor specific database management systems. Debezium records the history of data changes in Kafka logs, so your application can be stopped and restarted at any time and can easily consume all of the events it missed while it was not running, ensuring that all events are processed correctly and completely. Debezium is open source under the Apache License, Version 2.0.

Get involved

We hope you find Debezium interesting and useful, and want to give it a try. Follow us on Twitter @debezium, chat with us on Zulip, or join our mailing list to talk with the community. All of the code is open source on GitHub, so build the code locally and help us improve ours existing connectors and add even more connectors. If you find problems or have ideas how we can improve Debezium, please let us know or log an issue.