Featured Posts

Tags
apache kafka apache kafka apicurio avro aws batch caassandra camel cassandra cdc channels community community stories connectors containers cqrs custom datalake db2 ddd debezium debezium server debezium ui deduplication discussion docker elasticsearch event sourcing exactly once semantics example examples features fedora flink hiring ibmi iceberg informix integration introduction jaeger jdbc json kafka kafka streams kafka streams kogito ksql kubernetes lakehouse machine learning mariadb microservices mongo mongodb mysql news newsletter notifications online learning operator oracle outbox performance postgres presentation production quarkus questdb rds releases schema scylla secrets sentry serialization signaling smt snapshots spanner spark sql sqlserver tensorflow testcontainers tests time series timescaledb topics tracing transactions vagrant vitess website

It’s my pleasure to announce the second release of the Debezium 1.8 series, 1.8.0.Alpha2!

With the holiday season just around the corner, the team’s release schedule remains steadfast. While Debezium 1.8.0.Alpha2 delivers quite a lot of bugfixes and minor changes, there are a few notable changes:

  • MySQL support for heartbeat action queries

  • Configurable transaction topic name

In addition, the latest 1.2 tag of the debezium/tooling image is available. The newest version includes all the latest tools, including kcctl, a super simple, cuddly CLI for Apache Kafka Connect.

This release does include several breaking changes. Please see the release notes for details on what changed and how to upgrade.

MySQL heartbeat action query support

A heartbeat action query can be enabled by supplying a heartbeat.action.query configuration option in the connector’s configuration. This property is meant to supply a SQL statement that the connector will execute periodically.

The initial implementation of the heartbeat action query was specifically for PostgreSQL to handle dealing with WAL growth under specific conditions. But a heartbeat action query has many uses and is entirely connector or even user driven.

For example, you may want to notify downstream consumers that your MySQL topology has changed by supplying consumers with an event with the GTID. The following configuration shows how to capture changes from the heartbeat action query table that can then be consumed easily by your CDC pipeline.

"table.include.list": "gtid_history",
"heartbeat.action.query": "INSERT INTO gtid_history( select * from mysql.gtid_executed )"

Configurable transaction topic names

The Debezium transaction metadata topic had previously used a relatively non-configurable naming convention of <database.server.name>.transaction. While it was possible to manipulate the topic name using a single message transform (SMT) as a workaround, we felt that allowing this to be a bit more flexible in Debezium proper made sense.

A new configuration option, transaction.topic.prefix, has been introduced that allows the connector configuration to adjust the naming of the transaction metadata topic. The configuration option value specifies what will be used as a direct replacement for the `<database.server.name>~ portion of the topic name. If this configuration option is not supplied, the prior topic naming behavior will continue to be used; requiring no changes for existing connector deployments.

Other Fixes

There were quite a number of bugfixes and stability changes in this release, some noteworthy are:

  • Invalid default value error on captured table DDL with default value DBZ-3710

  • Incremental snapshot doesn’t work without primary key DBZ-4107

  • Signal based incremental snapshot is failing if database name contains dash DBZ-4244

Altogether, 45 issues were fixed for this release. A big thank you to all the contributors from the community who worked on this release: Abhishek Hodavdekar, Alexander Schwartz, Andrey Yegorov, Harvey Yue, Hussain Ansari, Jiabao Sun, Jiri Novotny, Jose Luis, Juan Fiallo, Katerina Galieva, Mike Kamornikov, and Sergei Morozov!

Debezium 1.7

In addition to this release, we also released Debezium 1.7.1.Final, a bugfix update for the 1.7 series. The 1.7.1.Final release includes many of the bugfixes in the 1.8 series that have been done since 1.7.0.Final. For more information on what changed in 1.7.1.Final, please see the release notes.

Outlook

The holiday season is upon us, but we intend to stick to our release cadence as closely as possible. If you haven’t already taken an opportunity, we’d love your feedback on the open discussion regarding Debezium 2.0 on the mailing list. In the meantime, you can expect the first beta release of 1.8 in a couple of weeks.

Chris Cranford

Chris is a software engineer at Red Hat. He previously was a member of the Hibernate ORM team and now works on Debezium. He lives in North Carolina just a few hours from Red Hat towers.

   

About Debezium

Debezium is an open source distributed platform that turns your existing databases into event streams, so applications can see and respond almost instantly to each committed row-level change in the databases. Debezium is built on top of Kafka and provides Kafka Connect compatible connectors that monitor specific database management systems. Debezium records the history of data changes in Kafka logs, so your application can be stopped and restarted at any time and can easily consume all of the events it missed while it was not running, ensuring that all events are processed correctly and completely. Debezium is open source under the Apache License, Version 2.0.

Get involved

We hope you find Debezium interesting and useful, and want to give it a try. Follow us on Twitter @debezium, chat with us on Zulip, or join our mailing list to talk with the community. All of the code is open source on GitHub, so build the code locally and help us improve ours existing connectors and add even more connectors. If you find problems or have ideas how we can improve Debezium, please let us know or log an issue.