Since introducing the Debezium Management Platform (Debezium Platform), our goal has been to make it easy to build CDC data pipelines so you can focus on how your data flows from sources to destinations. Many users already run Debezium connector on Kafka Connect or with Debezium Server.
So to further ease the onboarding and jump start the process, we introduced the ability to reuse the configuration of any existing Debezium connector on Kafka Connect or any Debezium Server configuration in the Debezium Platform.
Using Smart editor
You can now use the smart editor to Auto format the Debezium connector configuration on Kafka Connect or Debezium Server configuration to create the data pipeline or its component i.e source, Destination and Transforms.
There are two ways to use this:
-
In your regular resource creation flow, toggle between the traditional form-based UI to the Smart Editor.
-
Alternatively, create the resource via Smart editor directly from the catalog page, bypassing the traditional flow of selecting the type first and then filling out the form.
Source
You can create a source using either a configuration of a Debezium connector on Kafka Connect or a Debezium Server configuration.
Configure source using Debezium connector on Kafka Connect
To reuse an existing Debezium connector configuration, you can upload a file that contains the configuration, or you can copy and paste the configuration into the Smart Editor. Stage UI intelligently recognizes the format of the configuration that you provide, and prompts you to reformat it for use by the platform.
For example, consider the following JSON for specifying the configuration of a Debezium MySQL connector on Kafka Connect:
{
"name": "inventory-connector",
"config": {
"connector.class": "io.debezium.connector.mysql.MySqlConnector",
"tasks.max": "1",
"database.hostname": "mysql",
"database.port": "3306",
"database.user": "debezium",
"database.password": "dbz",
"database.server.id": "184054",
"topic.prefix": "dbserver1",
"database.include.list": "inventory"
}
}When you add this JSON to the Smart Editor, it detects that it is formatted for Kafka Connect and displays an alert.
The Smart Editor toolbar then displays an Auto format option, as shown in the preceding figure. The Auto format option automatically converts the provided JSON into the required Debezium Platform JSON format.
Optionally, you can then edit the JSON, for example to update the value of the name field or populate the description field.
The following image shows the JSON that results after you use the Auto format option to convert the configuration for a MySQL data source.
Configure a source using Debezium server configuration
Similarly, you can provide Server configuration in the Smart Editor to automatically extract the source configuration and reformat it for use in the Debezium platform. For more information, see Configure destination using Debezium server configuration.
Destination
You can use the debezium server configuration directly to create a destination
Configure a destination using Debezium server configuration
You can directly use a Debezium Server configuration by either uploading it, or pasting it into the Smart Editor. The Stage UI intelligently recognizes the configuration type and prompts you to reformat it for use by the platform. The editor automatically extracts the Debezium server sink configuration and converts it into the JSON format that the platform supports.
For example, consider the following properties from a Debezium Server configuration:
# ...
debezium.sink.type=pubsub
debezium.sink.pubsub.project.id=debezium-tutorial-local
debezium.sink.pubsub.address=pubsub:8085
debezium.source.connector.class=io.debezium.connector.mysql.MySqlConnector
debezium.source.database.hostname=mysql
debezium.source.database.server.id=223344
debezium.source.database.port=3306
debezium.source.database.user=debezium
debezium.source.database.password=dbz
debezium.source.schema.history.internal=io.debezium.storage.file.history.FileSchemaHistory
debezium.source.schema.history.internal.file.filename=data/schema.dat
debezium.source.offset.storage.file.filename=data/offsets.dat
debezium.source.offset.flush.interval.ms=0
debezium.source.topic.prefix=tutorial
debezium.source.database.include.list=inventory
debezium.source.table.include.list=inventory.customers
# ..When you add the preceding properties file to the Smart Editor, it confirms that it recognizes the file as a Debezium Server configuration.
The Smart Editor toolbar displays an Auto format option, as shown in the preceding image. Select this option to automatically extract the sink configuration and convert the sink properties into the JSON format that the Debezium Platform supports.
The following image shows the JSON that results after the editor automatically formats the publish-subscribe sink properties that you added. Edit the JSON to update the value of the name field, and optionally populate the description field.
Create a pipeline with Debezium server configuration
In the Pipeline Designer, you create a data pipeline by bringing all the pieces together: select a Source, optional Transforms, and a Destination, either by choosing existing resources or creating new ones. If you already have a Debezium Server configuration, you may not want to create each component individually.
You can now upload the Debezium Server configuration directly into the Pipeline Designer to create the complete data pipeline in one step. The UI analyzes the configuration and extracts the relevant settings for each resource to sequentially create the Source, Destination, and Transforms.
For a short demo of this feature in action, see the following video:
| This video is part of our Debezium YouTube channel! We recommend you subscribe to the channel to never miss any new videos. |
More coming soon!
Stay tuned for further improvements and new feature announcements in upcoming releases.
You can leave a comment below or please feel free to get in touch with us on our mailing list or in our Zulip chat.
About Debezium
Debezium is an open source distributed platform that turns your existing databases into event streams, so applications can see and respond almost instantly to each committed row-level change in the databases. Debezium is built on top of Kafka and provides Kafka Connect compatible connectors that monitor specific database management systems. Debezium records the history of data changes in Kafka logs, so your application can be stopped and restarted at any time and can easily consume all of the events it missed while it was not running, ensuring that all events are processed correctly and completely. Debezium is open source under the Apache License, Version 2.0.
Get involved
We hope you find Debezium interesting and useful, and want to give it a try. Follow us on Twitter @debezium, chat with us on Zulip, or join our mailing list to talk with the community. All of the code is open source on GitHub, so build the code locally and help us improve ours existing connectors and add even more connectors. If you find problems or have ideas how we can improve Debezium, please let us know or log an issue.
