You are viewing documentation for the current development version of Debezium.
If you want to view the latest stable version of this page, please go here.

Sending signals to a Debezium connector

This feature is currently in incubating state, i.e. exact semantics, configuration options etc. may change in future revisions, based on the feedback we receive. Please let us know if you encounter any problems while using this extension.

Overview

Sometimes it is necessary for the user to modify a connector behaviour or trigger a one-time action (e.g. snapshot of a single table). To fulfill such needs Debezium provides a mechanism how to send a signal to a Debezium connector to trigger an action. As it might be necessary to synchronize such an action with the dataflow the signal is implemented as a write to a data collection.

Signalling is disabled by default. The name of the signalling data collection must be set via connector configuration parameter to enable the function. The signalling data collection must be explictly added among captured data collections, i.e., included in the table.include.list parameter.

Table 1. Connector configuration
Parameter Description

signal.data.collection

Fully-qualified name of data collection.
The name format is connector specific, e.g. "some_schema.debezium_signals" in case of the Debezium Postgres connector.
For SQL Server it should be formatted as "some_database.some_schema.debezium_signals".

Data collection structure

The signalling data collection must conform to a required format: It must contain three fields; the naming of the fields is arbitrary and only order of the field is important. It is recommended but not mandatory to use the field names as defined in the table below:

Table 2. Signalling data collection structure
Column Type Description

id

string
(required)

A unique identifier of this signal instance.
It can be used for logging, debugging or deduplication. Usually an UUID string.

type

string
(required)

The type of the signal to be sent.
There are signals common for all connectors or specific to a subset of connectors.

data

string
(optional)

JSON formatted parameters that are passed to a signal action.
Each signal has its own expected set of data.

Such table could be created with a DDL command similar to:

CREATE TABLE debezium_signal (id VARCHAR(42) PRIMARY KEY, type VARCHAR(32) NOT NULL, data VARCHAR(2048) NULL);
Table 3. Example of a signal record
Column Value

id

924e3ff8-2245-43ca-ba77-2af9af02fa07

type

log

data

{"message": "Signal message at offset {}"}

Signal Actions

These signals are common to all connectors

Logging

The type of the logging action is log. It will print a provided message to the log optionally including streaming position.

Table 4. Action parameters
Name Description

message

The string printed to the log.
If a placeholder {} is added to the message it will be replaced with streaming coordinates.

Table 5. Example of a logging record
Column Value

id

924e3ff8-2245-43ca-ba77-2af9af02fa07

type

log

data

{"message": "Signal message at offset {}"}

Triggering an Incremental Snapshot

Available for MySQL, PostgresSQL, Oracle, SQL Server and Db2 connectors

The type of the action is execute-snapshot. It will start an incremental snapshot operation. The operation first reads the first and last primary key values and use those as start and end point for each table.

Optional Connector Configuration

incremental.snapshot.chunk.size: Integer value. The number of rows collected at each fetch operation on the database.

Table 6. Action parameters
Name Description

data-collections

The list of data collections to be snapshoted.
Formatting should be:

  • MySQL: "some_database.some_table"

  • PostgresSQL and Db2: "some_schema.some_table"

  • SQL Server: "some_database.some_schema.some_table"

  • Oracle: "some_schema.some_table"

Table 7. Example of executing an incremental snapshot
Column Value

id

d139b9b7-7777-4547-917d-e1775ea61d41

type

execute-snapshot

data

{"data-collections": ["public.MyFirstTable", "public.MySecondTable"]}