Release early, release often! After the 1.1 Beta1 and 1.0.1 Final releases earlier this week, I’m today happy to share the news about the release of Debezium 1.1.0.Beta2!
The main addition in Beta2 is support for integration tests of your change data capture (CDC) set-up using Testcontainers. In addition, the Quarkus extension for implementing the outbox pattern as well as the SMT for extracting the
after state of change events have been re-worked and offer more configuration flexibility now.
When setting up CDC pipelines, you should test your configuration thoroughly: the source database must be correctly configured (e.g. think binlog mode of MySQL), connectors must use the right credentials, filters and more.
Automation is king, and thus we’re very excited about the new support for writing CDC integration tests using Testcontainers (DBZ-1722). With just a few lines of code, you can set up all the required services using Linux containers, deploy a Debezium connector and run assertions against the emitted change data events.
We’re planning to publish a blog post dedicated to this topic very soon, in the mean time refer to to the documentation to learn more.
More Configuration Options for the Quarkus Outbox Pattern Extension
The Quarkus extension for implementing the outbox pattern comes in handy if a service needs to update its own database as well as send events to external consumers: by writing events into an "outbox" table and capturing them from there using Debezium, unsafe "dual writes" to a database and Apache Kafka are avoided.
To offer more flexibility, the Quarkus extension allows now for fully flexible customization of the outbox table’s column types DBZ-1711. E.g. you can set the option
JSONB NOT NULL, in order to use a Postgres
JSONB column for the outbox table’s payload column.
After State Extraction
The SMT for extracting the
after state of change events allows for a more flexible propagation of specific event attributes now: using the new
add.headers options, any top-level attribute (
ts_ms) as well as
transaction attributes can be propagated into the outgoing record and/or as header of the Kafka record:
transforms=unwrap,... transforms.unwrap.type=io.debezium.transforms.ExtractNewRecordState transforms.unwrap.add.fields=table,lsn transforms.unwrap.add.headers=op,source.ts_ms
add.source.fields options have been deprecated and will be removed in a future Debezium version.
Further Changes and Bugfixes
Overall 13 issues have been addressed for the Debezium 1.1 Beta2 release.
Most notably, a known Kafka Connect issue has been mitigated which may cause missed change events when the Postgres or Oracle connectors are stopped (DBZ-1766), and the MongoDB SMT for extracting the new document state handles identifiers with characters un-supported by Apache Avro, e.g.
$ref, gracefully (DBZ-1767).
Gunnar is a software engineer at Decodable and an open-source enthusiast by heart. He has been the project lead of Debezium over many years. Gunnar has created open-source projects like kcctl, JfrUnit, and MapStruct, and is the spec lead for Bean Validation 2.0 (JSR 380). He’s based in Hamburg, Germany.
Debezium is an open source distributed platform that turns your existing databases into event streams, so applications can see and respond almost instantly to each committed row-level change in the databases. Debezium is built on top of Kafka and provides Kafka Connect compatible connectors that monitor specific database management systems. Debezium records the history of data changes in Kafka logs, so your application can be stopped and restarted at any time and can easily consume all of the events it missed while it was not running, ensuring that all events are processed correctly and completely. Debezium is open source under the Apache License, Version 2.0.
We hope you find Debezium interesting and useful, and want to give it a try. Follow us on Twitter @debezium, chat with us on Zulip, or join our mailing list to talk with the community. All of the code is open source on GitHub, so build the code locally and help us improve ours existing connectors and add even more connectors. If you find problems or have ideas how we can improve Debezium, please let us know or log an issue.