Decode Logical Decoding Message Content

Table of Contents

The DecodeLogicalDecodingMessageContent SMT converts the binary content of a PostgreSQL logical decoding message to a structured form. You can use the SMT on its own, or in conjunction with another SMT, such as the Outbox Event Router.

Example

You configure a connector to use the DecodeLogicalDecodingMessageContent SMT by adding the SMT to the Kafka Connect configuration for the connector, as in the following example:

"connector.class": "io.debezium.connector.postgresql.PostgresConnector",
...
"transforms": "decodeLogicalDecodingMessageContent",
"transforms.decodeLogicalDecodingMessageContent.type": "io.debezium.connector.postgresql.transforms.DecodeLogicalDecodingMessageContent",
"key.converter": "org.apache.kafka.connect.json.JsonConverter",
"key.converter.schemas.enable": false,
"value.converter": "org.apache.kafka.connect.json.JsonConverter",
"value.converter.schemas.enable": "false",
...

The following example shows key and value of an event record before and after the transformation is applied.

Example 1. Effect of applying the DecodeLogicalDecodingMessageContent SMT
Key before the SMT processes the record
{
	"prefix": "test-prefix"
}
Value before the SMT processes the record
{
	"op": "m",
	"ts_ms": 1723115240065,
	"source": {
		"version": "3.0.0-SNAPSHOT",
		"connector": "postgresql",
		"name": "connector-name",
		"ts_ms": 1723115239782,
		"snapshot": "false",
		"db": "source-db",
		"sequence": "[\"26997744\",\"26997904\"]",
		"ts_us": 1723115239782690,
		"ts_ns": 1723115239782690000,
		"schema": "",
		"table": "",
		"txId": 756,
		"lsn": 26997904,
		"xmin": null
	},
	"message": {
		"prefix": "test-prefix",
		"content": "eyJpZCI6IDEsICJpdGVtIjogIkRlYmV6aXVtIGluIEFjdGlvbiIsICJzdGF0dXMiOiAiRU5URVJFRCIsICJxdWFudGl0eSI6IDIsICJ0b3RhbFByaWNlIjogMzkuOTh9"
	}
}
Key after the SMT processes the record

null

Value after the SMT processes the record
{
	"op": "c",
	"ts_ms": 1723115415729,
	"source": {
		"version": "3.0.0-SNAPSHOT",
		"connector": "postgresql",
		"name": "connector-name",
		"ts_ms": 1723115415640,
		"snapshot": "false",
		"db": "source-db",
		"sequence": "[\"26717416\",\"26717576\"]",
		"ts_us": 1723115415640161,
		"ts_ns": 1723115415640161000,
		"schema": "",
		"table": "",
		"txId": 745,
		"lsn": 26717576,
		"xmin": null
	},
	"after": {
		"id": 1,
		"item": "Debezium in Action",
		"status": "ENTERED",
		"quantity": 2,
		"totalPrice": 39.98
	}
}

In the preceding example, the SMT applies the following changes to the original event record:

  1. Removes the key that contained the prefix field in the original logical decoding message ("prefix": "test-prefix")

  2. Converts the value of the op field from an m (message) to a c (create), effectively changing the type of the event from a message to an INSERT.

  3. Replaces the message field with an after field that contains the decoded content of a logical decoding message.

After the SMT applies these changes, the record can be more easily processed by other SMTs, such as the Outbox Event Router.

Configuration options

The following table lists the configuration options that you can use with the DecodeLogicalDecodingMessageContent SMT.

Table 1. DecodeLogicalDecodingMessageContent SMT configuration options

Property

Type

Default

Description

fields.null.include

boolean

false

Specifies how the decoding process handles fields that have null values in the source message. By default, the transformation removes fields that have null values.