Featured Posts

Tags
apache kafka apache kafka apicurio avro aws batch caassandra camel cassandra cdc channels community community stories connectors containers cqrs custom datalake db2 ddd debezium debezium server debezium ui deduplication discussion docker elasticsearch event sourcing exactly once semantics example examples features fedora flink hiring ibmi iceberg informix integration introduction jaeger jdbc json kafka kafka streams kafka streams kogito ksql kubernetes lakehouse machine learning mariadb microservices mongo mongodb mysql news newsletter notifications online learning operator oracle outbox performance postgres presentation production quarkus questdb rds releases schema scylla secrets sentry serialization signaling smt snapshots spanner spark sql sqlserver tensorflow testcontainers tests time series timescaledb topics tracing transactions vagrant vitess website

While we remain on track with the upcoming Debezium 2.6 release, we are continuously looking at improvements and fixes that are being made that ensures that older releases continue to provide the best possible experience. With that, the team is pleased to announce the first maintenance release of Debezium 2.5, Debezium 2.5.1.Final. Let’s dive into what this release includes and what you should be aware of while upgrading…​

Breaking changes

The team aims to avoid any potential breaking changes between minor releases; however, such changes are sometimes inevitable.

Re-select Columns Post Processor

  • The re-select columns post processor used the key based on the message.key.columns when building the query. This is not correct for most tables with primary keys. The default behavior has changed and the table primary key is used by default. A new configuration option was introduced to allow the user to choose between whether to use the primary key or the generated key, reselect.use.event.key (DBZ-7358).

Improvements and changes

MongoDB post-image changes

The MongoDB connector’s event payload can be configured to include the full document that was changed in an update. The connector previously made an opinionated choice about how the full document would be fetched as part of the change stream; however, this behavior was not consistent with our expectations in all use cases.

Debezium 2.5+ introduces a new configuration option, capture.mode.full.update.type, allowing the connector to explicitly control how the change stream’s full document lookup should be handled (DBZ-7299). The default value for this option is lookup, meaning that the database will make a separate look-up to fetch the full document. If you are working with MongoDB 6+, you can also elect to use post_image to rely on MongoDB change stream’s post-image support.

Other changes

Altogether, 21 issues were fixed in this release:

  • Connector is getting stopped while processing bulk update(50k) records in debezium server 2.0.1.Final DBZ-6955

  • Empty object sent to GCP Pub/Sub after DELETE event DBZ-7098

  • Debezium-ddl-parser crashes on parsing MySQL DDL statement (sub-query with UNION) DBZ-7259

  • Oracle DDL parsing error in PARTITION REFERENCE DBZ-7266

  • PostgreSQL ad-hoc blocking snapshots fail when snapshot mode is "never" DBZ-7311

  • Ad-hoc blocking snapshot dies with "invalid snapshot identifier" immediately after connector creation DBZ-7312

  • Specifying a table include list with spaces between elements cause LogMiner queries to miss matches DBZ-7315

  • Debezium heartbeat.action.query does not start before writing to WAL: part 2 DBZ-7316

  • Oracle connector is occasionally unable to find SCN DBZ-7345

  • Initial snapshot notifications should use full identifier. DBZ-7347

  • Debezium fails after table split operation DBZ-7360

  • MSSQL wrong default values in db schema for varchar, nvarchar, char columns DBZ-7374

  • Replace additional role binding definition in kubernetes.yml with @RBACRule DBZ-7381

  • Kinesis Sink Exception on PutRecord DBZ-7417

  • ParsingException (MariaDB Only): alterSpec drop foreign key with 'tablename.' prefix DBZ-7420

A big thank you to all the contributors from the community who worked on this release: Anisha Mohanty, Breno Moreira, Chris Cranford, Ilyas Ahsan, Jakub Cechacek, James Johnston, Jiri Pechanec, Mario Fiore Vitale, Mickael Maison, Peter Hamer, Robert Roldan, V K, and leoloel!

Outlook & What’s next?

The team remains dedicated toward Debezium 2.6 over the next few months. However, in that time, you can expect that we’ll continue to review regressions and bugfixes, and provide maintenance releases for Debezium 2.5.

In the meantime, if you’re interested in what the team has in store for Debezium 2.6 and beyond throughout 2024, we encourage you to review our road map, which includes details about our path to Debezium 3.0 later this year. If you have any suggestions or ideas, please feel free to get in touch with us on our mailing list or in our Zulip chat.

We have two upcoming talks later this week at Fosdem 2024 in Brussels. Ondrej Babec will be speaking about performance testing and why it’s important and Vojtech Juranek will use Debezium to feed AI models in real-time. If you plan to attend, be sure to check out these talks and meet and greet with our Debezium engineers.

Until next time…​

Chris Cranford

Chris is a software engineer at Red Hat. He previously was a member of the Hibernate ORM team and now works on Debezium. He lives in North Carolina just a few hours from Red Hat towers.

   

About Debezium

Debezium is an open source distributed platform that turns your existing databases into event streams, so applications can see and respond almost instantly to each committed row-level change in the databases. Debezium is built on top of Kafka and provides Kafka Connect compatible connectors that monitor specific database management systems. Debezium records the history of data changes in Kafka logs, so your application can be stopped and restarted at any time and can easily consume all of the events it missed while it was not running, ensuring that all events are processed correctly and completely. Debezium is open source under the Apache License, Version 2.0.

Get involved

We hope you find Debezium interesting and useful, and want to give it a try. Follow us on Twitter @debezium, chat with us on Zulip, or join our mailing list to talk with the community. All of the code is open source on GitHub, so build the code locally and help us improve ours existing connectors and add even more connectors. If you find problems or have ideas how we can improve Debezium, please let us know or log an issue.