Featured Posts

Tags
apache kafka apache kafka apicurio avro aws batch caassandra camel cassandra cdc channels community community stories connectors containers cqrs custom datalake db2 ddd debezium debezium server debezium ui deduplication discussion docker elasticsearch event sourcing exactly once semantics example examples features fedora flink hiring ibmi iceberg informix integration introduction jaeger jdbc json kafka kafka streams kafka streams kogito ksql kubernetes lakehouse machine learning mariadb microservices mongo mongodb mysql news newsletter notifications online learning operator oracle outbox performance postgres presentation production quarkus questdb rds releases schema scylla secrets sentry serialization signaling smt snapshots spanner spark sql sqlserver tensorflow testcontainers tests time series timescaledb topics tracing transactions vagrant vitess website

I’m excited to announce the release of Debezium 1.4.0.Alpha2!

This second pass of the 1.4 release line provides a few useful new features:

  • New API hook for the PostgreSQL Snapshotter interface

  • Field renaming using ExtractNewRecordState SMT’s add.fields and add.headers configurations

Overall, the community fixed 37 issues for this release. Let’s take a closer look at some of the highlights.

PostgreSQL Snapshotter completion hook

The PostgreSQL Snapshotter API is a contract that allows for the customization of the snapshot process. This API was introduced in 0.9.3.Final and has continued to evolve in the releases since.

A new backward compatible completion hook has been added:

void snapshotCompleted()

This new hook is called by the snapshot process when the snapshot has concluded, allowing implementations to clean-up any resources it may have allocated prior streaming changes.

ExtractNewRecordState SMT field renaming support

One of the features of the ExtractNewRecordState SMT is that the transformation can retain parts of the original message in the transformed message’s header or payload. This release extends this feature to allow specifying a new name to be used for the field when added to the message header or payload.

For example, to add the source database’s event timestamp to the message header using the new renaming feature, the SMT configuration would be:

transforms=unwrap
transforms.unwrap.type=io.debezium.transforms.ExtractNewRecordState
transforms.unwrap.add.headers=source.ts_ms:timestamp

The format of the add.headers and add.fields configuration options have been improved to support a comma-separated list of fields with the syntax <OLD_FIELD>[:NEW_FIELD]. The above emitted message’s headers would now contain __timestamp rather than the default __source.ts_ms field.

This syntax improvement remains backward compatible.

Bugfixes

Also a number of bugs were fixed, e.g.:

  • Oracle throw "no snapshot found based on specified time" when running flashback query DBZ-1446

  • Exception when PK definition precedes column definition DBZ-2580

  • Patroni can’t stop PostgreSQL when Debezium is streaming DBZ-2617

  • ChangeRecord informations don’t connect with the TableSchema DBZ-2679

  • MySQL connector fails on a zero date DBZ-2682

  • Oracle LogMiner doesn’t support partition tables DBZ-2683

  • DB2 doesn’t start reliably in OCP DBZ-2693

  • Dropped columns cause NPE in SqlServerConnector DBZ-2716

  • Timestamp default value in 'yyyy-mm-dd' format fails MySQL connector DBZ-2726

  • Connection timeout on write should retry DBZ-2727

  • No viable alternative at input error on "min" column DBZ-2738

  • SQLServer CI error in SqlServerConnectorIT.whenCaptureInstanceExcludesColumnsAndColumnsRenamedExpectNoErrors:1473 DBZ-2747

  • debezium-connector-db2: DB2 SQL Error: SQLCODE=-206 on DB2 for z/OS DBZ-2755

  • no viable alternative at input 'alter table order drop CONSTRAINT' DBZ-2760

  • Tests are failing on macos DBZ-2762

A big thank you to all the contributors from the community who worked on this release: Alexander Iskuskov, Alisa Houskova, Anisha Mohanty, Andrey Ignatenko, Arik Cohen, Babur Duisenov, Grant Cooksey, Ganesh Ramasubramanian, Hoa Le, Kewei Shang, Mans Singh, Mohamed Pudukulathan, Peter Urbanetz, Ramesh Reddy, Sergei Morozov, and Vadzim Ramanenka.

Chris Cranford

Chris is a software engineer at Red Hat. He previously was a member of the Hibernate ORM team and now works on Debezium. He lives in North Carolina just a few hours from Red Hat towers.

   

About Debezium

Debezium is an open source distributed platform that turns your existing databases into event streams, so applications can see and respond almost instantly to each committed row-level change in the databases. Debezium is built on top of Kafka and provides Kafka Connect compatible connectors that monitor specific database management systems. Debezium records the history of data changes in Kafka logs, so your application can be stopped and restarted at any time and can easily consume all of the events it missed while it was not running, ensuring that all events are processed correctly and completely. Debezium is open source under the Apache License, Version 2.0.

Get involved

We hope you find Debezium interesting and useful, and want to give it a try. Follow us on Twitter @debezium, chat with us on Zulip, or join our mailing list to talk with the community. All of the code is open source on GitHub, so build the code locally and help us improve ours existing connectors and add even more connectors. If you find problems or have ideas how we can improve Debezium, please let us know or log an issue.