Timezone Converter
When Debezium emits event records, the timezone values for timestamp fields in the record can vary, depending on the type and configuration of the data source.
To maintain data consistency and precision within data processing pipelines and applications, you can use the Timezone Converter SMT to ensure that event records use a consistent timezone to represent timestamp data.
The SMT converts the value of the specified field to the target timezone by using the converted.timezone configuration option.
You can specify the target timezone as a geographic timezone, for example, America/New_York, or as a UTC offset, such as +02:00.
It is assumed that the fields of the record are in UTC.
Along with the specified timezone, the SMT also provides configuration options to include or exclude specific fields from timezone conversion using the include.list and exclude.list configuration options.
The SMT supports all Debezium and Kafka Connect temporal and non-temporal types.
|
To comply with daylight savings time, you must specify a geographic timezone in the |
|
The |
The SMT also allows conversion of event metadata fields in the source information block, such as ts_ms to the target timezone. In order to convert the metadata fields, you must include the source prefix in the fieldname of the include.list or exclude.list configuration option.
|
If the schema for timestamp fields in the source information block, like |
Example: Basic configuration
Configure the TimezoneConverter SMT in the Kafka Connect configuration for a connector to convert the time-based fields in an event record to a target timezone.
For example, to convert all timestamp fields in an event record from UTC to the Pacific/Easter timezone, add the following lines to your connector configuration:
transforms=convertTimezone
transforms.convertTimezone.type=io.debezium.transforms.TimezoneConverter
transforms.convertTimezone.converted.timezone=Pacific/EasterEffect of applying the TimezoneConverter SMT
The following examples show how the TimezoneConverter transformation modifies the timestamp fields in an event record.
The first example shows a Debezium event record that is not processed by the transformation; the record retains its original timestamp values.
The next example shows the same event record after the transformation is applied.
Per the configuration specified in the basic configuration example, the SMT converts the original UTC values of timestamp fields in the source message to Pacific/Easter timezone values.
TimezoneConverter transformationThe value of the created_at field shows the UTC time.
{
"before": null,
"after": {
"id": 1,
"first_name": "Anne",
"last_name": "Kretchmar",
"email": "annek@noanswer.org",
"created_at": "2011-01-11T16:40:30.123456789+00:00"
},
"source": {
"version": "2.7.0.Alpha1",
"connector": "postgresql",
"name": "PostgreSQL_server",
"ts_ms": 1559033904863,
"ts_us": 1559033904863000,
"ts_ns": 1559033904863000000,
"snapshot": true,
"db": "postgres",
"sequence": "[\"24023119\",\"24023128\"]",
"schema": "public",
"table": "customers",
"txId": 555,
"lsn": 24023128,
"xmin": null
},
"op": "c",
"ts_ms": 1559033904863,
"ts_us": 1559033904863875,
"ts_ns": 1559033904863875124
}TimezoneConverter transformationThe SMT converts the original UTC value of the created_at field to the time in the target Pacific/Easter timezone that is specified in the Basic configuration example.
The SMT also adds an event_timestamp field.
{
"before": null,
"after": {
"id": 1,
"first_name": "Anne",
"last_name": "Kretchmar",
"email": "annek@noanswer.org",
"created_at": "2011-01-11T11:40:30.123456789-05:00"
},
"source": {
"version": "2.7.0.Alpha1",
"connector": "postgresql",
"name": "PostgreSQL_server",
"ts_ms": 1559033904863,
"ts_us": 1559033904863752,
"ts_ns": 1559033904863752000,
"snapshot": true,
"db": "postgres",
"sequence": "[\"24023119\",\"24023128\"]",
"schema": "public",
"table": "customers",
"txId": 555,
"lsn": 24023128,
"xmin": null,
"id": 100
},
"op": "c",
"ts_ms": 1559033904863,
"ts_us": 1559033904863971,
"ts_ns": 1559033904863971541,
"event_timestamp": 1626102708861
}Example: Advanced configuration
Instead of converting all timestamp fields in an event record, you can configure the SMT to convert specific fields only.
The following example shows how you might use the include.list option in the SMT configuration to convert only the created_at, and updated_at timestamp fields in an event record.
The following configuration uses a fixed offset, rather than a geographic timezone designator, to convert the time from UTC to +05:30.
transforms=convertTimezone
transforms.convertTimezone.type=io.debezium.transforms.TimezoneConverter
transforms.convertTimezone.converted.timezone=+05:30
transforms.convertTimezone.include.list=source:customers:created_at,customers:updated_atIn some cases, you might want to exclude specific timestamp fields from timezone conversion.
For example, to exclude the updated_at timestamp field in an event record from timezone conversion, use the exclude.list configuration option as in the following example:
transforms=convertTimezone
transforms.convertTimezone.type=io.debezium.transforms.TimezoneConverter
transforms.convertTimezone.converted.timezone=+05:30
transforms.convertTimezone.exclude.list=source:customers:updated_atConfiguration options
The following table lists the configuration options for the TimezoneConverter SMT.
Property |
Description |
Type |
Importance |
A string that specifies the target timezone to which the timestamp fields should be converted.
The target timezone can be specified as a geographic timezone, such as, |
string |
high |
|
A comma-separated list of rules that specify the fields that the SMT includes for timezone conversion. Specify rules by using one of the following formats:
|
list |
medium |
|
A comma-separated list of rules that specify the fields to exclude from timezone conversion. Specify rules by using one of the following formats:
|
list |
medium |