Release early, release often! After the 1.1 Beta1 and 1.0.1 Final releases earlier this week, I’m today happy to share the news about the release of Debezium 1.1.0.Beta2!
The main addition in Beta2 is support for integration tests of your change data capture (CDC) set-up using Testcontainers. In addition, the Quarkus extension for implementing the outbox pattern as well as the SMT for extracting the after state of change events have been re-worked and offer more configuration flexibility now.
Testcontainers Support
When setting up CDC pipelines, you should test your configuration thoroughly: the source database must be correctly configured (e.g. think binlog mode of MySQL), connectors must use the right credentials, filters and more.
Automation is king, and thus we’re very excited about the new support for writing CDC integration tests using Testcontainers (DBZ-1722). With just a few lines of code, you can set up all the required services using Linux containers, deploy a Debezium connector and run assertions against the emitted change data events.
We’re planning to publish a blog post dedicated to this topic very soon, in the mean time refer to to the documentation to learn more.
More Configuration Options for the Quarkus Outbox Pattern Extension
The Quarkus extension for implementing the outbox pattern comes in handy if a service needs to update its own database as well as send events to external consumers: by writing events into an "outbox" table and capturing them from there using Debezium, unsafe "dual writes" to a database and Apache Kafka are avoided.
To offer more flexibility, the Quarkus extension allows now for fully flexible customization of the outbox table’s column types DBZ-1711. E.g. you can set the option quarkus.debezium-outbox.payload.column-definition to JSONB NOT NULL, in order to use a Postgres JSONB column for the outbox table’s payload column.
More Flexible After State Extraction
The SMT for extracting the after state of change events allows for a more flexible propagation of specific event attributes now: using the new add.fields and add.headers options, any top-level attribute (op, ts_ms) as well as source and transaction attributes can be propagated into the outgoing record and/or as header of the Kafka record:
transforms=unwrap,...
transforms.unwrap.type=io.debezium.transforms.ExtractNewRecordState
transforms.unwrap.add.fields=table,lsn
transforms.unwrap.add.headers=op,source.ts_msThe existing operation.header and add.source.fields options have been deprecated and will be removed in a future Debezium version.
Further Changes and Bugfixes
Overall 13 issues have been addressed for the Debezium 1.1 Beta2 release.
Most notably, a known Kafka Connect issue has been mitigated which may cause missed change events when the Postgres or Oracle connectors are stopped (DBZ-1766), and the MongoDB SMT for extracting the new document state handles identifiers with characters un-supported by Apache Avro, e.g. $ref, gracefully (DBZ-1767).
A big shout-out to all the contributors of this release: Alex Soto, Daan Roosen, John Psoroulas, Matthias Wessendorf, Melissa Winstanley, as well as Sergei Egorov for his advice on the Testcontainers work.
Gunnar Morling
Gunnar is a software engineer at Decodable and an open-source enthusiast by heart. He has been the project lead of Debezium over many years. Gunnar has created open-source projects like kcctl, JfrUnit, and MapStruct, and is the spec lead for Bean Validation 2.0 (JSR 380). He’s based in Hamburg, Germany.
About Debezium
Debezium is an open source distributed platform that turns your existing databases into event streams, so applications can see and respond almost instantly to each committed row-level change in the databases. Debezium is built on top of Kafka and provides Kafka Connect compatible connectors that monitor specific database management systems. Debezium records the history of data changes in Kafka logs, so your application can be stopped and restarted at any time and can easily consume all of the events it missed while it was not running, ensuring that all events are processed correctly and completely. Debezium is open source under the Apache License, Version 2.0.
Get involved
We hope you find Debezium interesting and useful, and want to give it a try. Follow us on Twitter @debezium, chat with us on Zulip, or join our mailing list to talk with the community. All of the code is open source on GitHub, so build the code locally and help us improve ours existing connectors and add even more connectors. If you find problems or have ideas how we can improve Debezium, please let us know or log an issue.
