Featured Posts

Tags
apache kafka apache kafka apicurio avro aws batch caassandra camel cassandra cdc channels community community stories connectors containers cqrs custom datalake db2 ddd debezium debezium server debezium ui deduplication discussion docker elasticsearch event sourcing exactly once semantics example examples features fedora flink hiring ibmi iceberg informix integration introduction jaeger jdbc json kafka kafka streams kafka streams kogito ksql kubernetes lakehouse machine learning mariadb microservices mongo mongodb mysql news newsletter notifications online learning operator oracle outbox performance postgres presentation production quarkus questdb rds releases schema scylla secrets sentry serialization signaling smt snapshots spanner spark sql sqlserver tensorflow testcontainers tests time series timescaledb topics tracing transactions vagrant vitess website

I’m very happy to announce the release of Debezium 1.3.0.CR1!

As we approach the final stretch of Debezium 1.3 Final, we took this opportunity to add delegate converter support for the ByteBufferConverter and introduce a debezium-scripting module. In addition, there’s also a range of bug fixes and quite a bit of documentation polish; overall, not less than 15 issues have been resolved for this release.

ByteBufferConverter improvements

The ByteBufferConverter is a converter that is used with the Outbox event router SMT to serialize an existing Avro payload column. In a recent report (DBZ-2396), the ByteBufferConverter was unable to serialize events emitted from a connector that was configured to emit heartbeat, transaction metadata, or schema change events. In order to improve the converter’s compatibility when these events are emitted, the ByteBufferConverter can now be configured to delegate event serialization to an additional converter. This delegation is necessary so that heartbeat, transaction metadata, and schema change events (if applicable) can be serialized.

In order to use the Outbox event router SMT and the ByteBufferConverter with these event types, the connector configuration must be changed to reflect the delegate converter and its configurable options.

As an example to use the Apache Kafka JsonConverter as a delegate with schemas disabled, the following configuration would need to be included in the connector:

value.converter=io.debezium.converters.ByteBufferConverter
value.converter.delegate.converter.type=org.apache.kafka.connect.json.JsonConverter
value.converter.delegate.converter.type.schemas.enable=false

For more information about using the ByteBufferConverter, please see the Using Avro as the payload format section in the Outbox event router documentation.

Scripting module

In this release, the SMTs for content-based routing and filtering that both use JSR 223 scripting engines have been moved out of debezium-core and into a separate artifact debezium-scripting (DBZ-2549). Any connector that previous used these SMTs requires that the new artifact be added to the plug-in directories for those connector(s).

When using the Debezium container image for Kafka Connect, set the environment variable ENABLE_DEBEZIUM_SCRIPTING to true to enable this feature. This change is done to allow the scripting functionality to be available only in environments with an apppropriately secured Kafka Connect configuration interface.

Misc. Features and Bug Fixes

In addition, the community has completed the work on some other features and fixes, too:

  • Catch up streaming before snapshot may duplicate messages upon resuming streaming DBZ-2550

  • Fix Quarkus datasource configuration for Quarkus 1.9 DBZ-2558

  • Implement connection retry support for Oracle DBZ-2531

As always, please refer to the release notes for the complete list of resolved issues as well as procedures for upgrading from earlier Debezium versions.

Thank you so much to Guillaume Smet and Grant Cooksey for their contributions to this release.

Outlook

Barring any unforeseen regressions and bug reports, Debezium 1.3 Final should be out next week. Until then, we’ll focus on some more polishing. The community-lead work towards a Debezium connector for Vitess is also making good progress, with an initial release of this new connector planned with Debezium 1.4 Alpha1 in late October.

Chris Cranford

Chris is a software engineer at Red Hat. He previously was a member of the Hibernate ORM team and now works on Debezium. He lives in North Carolina just a few hours from Red Hat towers.

   

About Debezium

Debezium is an open source distributed platform that turns your existing databases into event streams, so applications can see and respond almost instantly to each committed row-level change in the databases. Debezium is built on top of Kafka and provides Kafka Connect compatible connectors that monitor specific database management systems. Debezium records the history of data changes in Kafka logs, so your application can be stopped and restarted at any time and can easily consume all of the events it missed while it was not running, ensuring that all events are processed correctly and completely. Debezium is open source under the Apache License, Version 2.0.

Get involved

We hope you find Debezium interesting and useful, and want to give it a try. Follow us on Twitter @debezium, chat with us on Zulip, or join our mailing list to talk with the community. All of the code is open source on GitHub, so build the code locally and help us improve ours existing connectors and add even more connectors. If you find problems or have ideas how we can improve Debezium, please let us know or log an issue.