Welcome to this series of articles dedicated to signaling and notifications in Debezium! This post serves as the first installment in the series, where we will introduce the signaling and notification features offered by Debezium and discuss the available channels for interacting with the platform.
In the subsequent parts of this series, we will delve deeper into customizing signaling channels and explore additional topics such as JMX signaling and notifications.
In today’s interconnected software applications and systems, seamlessly integrating with other products is crucial for building robust and efficient solutions. One of the key challenges in building integrated software ecosystems is establishing effective communication channels between different components and services.
Debezium introduces a comprehensive signaling and notification system providing seamless integration with any third-party solution. Combining the signal and notification systems, you can now orchestrate complex yet efficient pipelines to stay informed about the status and progress of the state managed by Debezium.
The new signaling system has been re-designed to be extensible, providing a simplified approach to adding new, custom signal channels on top of the various implementations available. Let’s explore the different signals and notification channels, covering their functionality and describing their benefits.
Signaling in Debezium
Signaling in Debezium refers to the mechanism through which users can trigger specific actions within the product. It allows users to interact with Debezium and control some behaviors. One notable use case of signaling is the execution of incremental snapshots, which permits to start an ad-hoc snapshot of database.
Debezium provides different signaling channels through which users can send signals to the platform. Let’s explore the available signaling channels:
- Signaling through Database Tables
-
In earlier versions of Debezium, signaling required a dedicated database table where you would insert specific records to trigger Debezium to execute a task. For example, an inserted row could initiate an incremental snapshot, prompting the connector to run a snapshot operation based on the given signal criteria.
- Signaling through Kafka Topics
-
In earlier versions of Debezium, you could send signals via a configured Kafka topic; however, this was only available for MySQL using read-only access and global transaction identifiers (GTID) behavior. With Debezium 2.3, the Kafka signal channel is available to all connectors.
This enhancement provides a simplified integration approach and a unified and consistent approach for signals across all supported Debezium databases. You can send signals to a specific Kafka topic, and Debezium will consume and process that signal as though it originated from the signal table itself.
Using a Kafka topic for signals provides several advantages. First, it aligns with event-driven design, making it a natural fit with change data capture and Debezium. Additionally, it provides a secure way to send signals to Debezium without necessarily providing the user with direct access to the underlying database.
| Even when using the Kafka signal approach, the incremental snapshot feature still requires the presence and use of the signaling table to manage some bookkeeping needed for the incremental snapshot process. You can only omit the signal table when using MySQL in a read-only way with global transaction identifiers (GTIDs) enabled. |
- Signaling through File
-
You can trigger signals from the file system thanks to a contribution by the Debezium user community. This approach is an excellent alternative when not relying on messaging infrastructures like Kafka combined with Debezium Server or the embedded engine or when you cannot use the database for signals.
- Signaling through Java Management Extensions (JMX)
-
With this channel, you can send signals calling the signal operation exposed through a dedicated MBean. You can do it by connecting to the MBeanServer with your preferred client. A dedicated post about this feature will follow.
- Signaling through custom channel
-
We also redesigned the signaling mechanism to be extensible. You can provide your implementations and use it all via configuration. A dedicated post about this feature will follow.
Notification in Debezium:
Notifications focus on retrieving status and progress information about Debezium and its internal processes. It provides insights into the execution and completion of tasks such as initial snapshots or incremental snapshots. Debezium generates notifications to keep users informed about the progress of these tasks, facilitating monitoring and decision-making based on real-time information. By leveraging the notification functionality, users can track the state of Debezium and take appropriate actions based on the received notifications.
Debezium provides a variety of ways to obtain notifications. Let’s explore the out-of-the-box channels:
- Notification through Sink channel
-
This implementation sends notifications through the Connect API to a configured topic. Users can enable this channel and specify the topic name. By publishing notifications to the specified topic, Debezium allows users to consume and process notifications in a way that suits their needs. Since
Sinkis agnostic (i.e. not necessarily Kafka), this integrates seamlessly with Debezium Server to deliver notifications to any of its sink adapters. - Notification through logs
-
This channel appends notifications directly to the end of the connector’s log. This approach provides convenient access for monitoring, debugging, and analyzing notification details.
- Notifications through Java Management Extensions (JMX)
-
This notification channel exposes a dedicated Debezium MBean with several attributes that contain a list of generated notifications. This channel combines common industry standard monitoring techniques like JMX to react and consume Debezium notifications. A dedicated post will follow to discuss this in more detail.
- Notification through custom channel
-
Debezium’s notification mechanism is extensible, allowing users to implement custom channels to deliver notifications using means that best fit their needs. We will cover how to provide a custom notification channel in a dedicated post.
Conclusion
Signals and Notifications are now foundational features in Debezium. Signaling empowers users to interact with Debezium and trigger actions, while notification provides valuable information about Debezium’s state and progress.
In previous versions of Debezium, initiating an incremental snapshot was only possible using a signaling table. Users had to configure a specific table as a signaling mechanism to trigger the incremental snapshot. However, for MySQL with GTIDs enabled, it was possible to utilize Kafka for signaling purposes.
With the release of Debezium version 2.3, we have introduced significant improvements to both the signal and notification subsystems. We’ve unified several behaviors across connectors and made the entire system extensible, aiming to simplify both custom and future contributed implementations. We hope this enhances the overall experience of working with Debezium and provides a way to integrate Debezium with other third-party applications and tools seamlessly. These new and improved features allow you to maximize the capabilities of the Debezium change data capture platform in limitless ways.
Stay tuned for Part 2, where we will discuss how to customize signaling and notification channels in Debezium.
About Debezium
Debezium is an open source distributed platform that turns your existing databases into event streams, so applications can see and respond almost instantly to each committed row-level change in the databases. Debezium is built on top of Kafka and provides Kafka Connect compatible connectors that monitor specific database management systems. Debezium records the history of data changes in Kafka logs, so your application can be stopped and restarted at any time and can easily consume all of the events it missed while it was not running, ensuring that all events are processed correctly and completely. Debezium is open source under the Apache License, Version 2.0.
Get involved
We hope you find Debezium interesting and useful, and want to give it a try. Follow us on Twitter @debezium, chat with us on Zulip, or join our mailing list to talk with the community. All of the code is open source on GitHub, so build the code locally and help us improve ours existing connectors and add even more connectors. If you find problems or have ideas how we can improve Debezium, please let us know or log an issue.
