Featured Posts

Tags
apache kafka apache kafka apicurio avro aws batch caassandra camel cassandra cdc channels community community stories connectors containers cqrs custom datalake db2 ddd debezium debezium server debezium ui deduplication discussion docker elasticsearch event sourcing exactly once semantics example examples features fedora flink hiring ibmi iceberg informix integration introduction jaeger jdbc json kafka kafka streams kafka streams kogito ksql kubernetes lakehouse machine learning mariadb microservices mongo mongodb mysql news newsletter notifications online learning operator oracle outbox performance postgres presentation production quarkus questdb rds releases schema scylla secrets sentry serialization signaling smt snapshots spanner spark sql sqlserver tensorflow testcontainers tests time series timescaledb topics tracing transactions vagrant vitess website

I’m pleased to announce the immediate availability of Debezium 2.6.1.Final. This release is the first maintenance release that focuses on addressing several critical stability issues with the 2.6.0.Final release, including classpath loading problems with Debezium Server, MongoDB silently stops gathering changes, and a race condition with the Oracle Infinispan buffer implementation.

Let’s take a few moments and dive into these and more…​

New features and improvements

Debezium 2.6.1.Final also introduces many improvements and features, lets take a look at each individually.

Oracle RAW data type to STRING converter

Debezium treats a RAW column type as a series of bytes, and therefore, change events that contain RAW columns use a schema type of BYTES. This is a reasonable default because Debezium does not know the application’s purpose for using a RAW column, and it is packaging the data from the column as-is. However, this may not be ideal for consumer applications that would prefer the data emitted as a STRING type rather than as BYTES.

To help users, we’ve introduced a new RawToStringConverter, which automatically emits RAW columns as STRING based types rather than the default of BYTES (DBZ-7753).

To configure the converter, simply add the following configuration:

converters=raw-to-string
converters.raw-to-string.type=io.debezium.connector.oracle.converters.RawToStringConverter
converters.raw-to-string.selector=.*.MY_TABLE.MY_RAW_COLUMN

The selector property allows the converter to specifically target one or multiply RAW columns across one or more tables by providing a comma-separated list of regular expressions.

The selector property is optional, and if omitted, the converter applies to ALL RAW columns.

Fixes and stability improvements

Debezium Server snapshotting fixed

If you attempted to perform or re-execute the snapshot phase with Debezium Server in 2.6.0.Final, the process would fail with an error about acquiring a lock. This was quickly identified as it severely affected the ability to use any Debezium Connector under Debezium Server based on 2.6.0.Final.

We have adjusted how we resolve the components needed for the new Snapshotter API so that when multiple connectors exist on the same class path, the resolution appropriate picks the right component. In addition, we’ve also improved the Debezium Server test suite to look specifically for these types of issues moving forward to avoid the potential release of such a build that has similar problems.

Debezium Server users should move to 2.6.1.Final and not use 2.6.0.Final.

Other fixes

  • Incremental Snapshot: read duplicate data when database has 1000 tables DBZ-7716

  • SQLServerConnectorIT.shouldNotStreamWhenUsingSnapshotModeInitialOnly check an old log message DBZ-7729

  • Snapshot fails with an error of invalid lock DBZ-7732

  • Column CON_ID queried on V$THREAD is not available in Oracle 11 DBZ-7737

  • Redis NOAUTH Authentication Error when DB index is specified DBZ-7740

  • Getting oldest transaction in Oracle buffer can cause NoSuchElementException with Infinispan DBZ-7741

  • The MySQL Debezium connector is not doing the snapshot after the reset. DBZ-7743

  • MongoDb connector doesn’t work with Load Balanced cluster DBZ-7744

  • Pod Security Context not set from template DBZ-7749

  • Apply MySQL binlog client version 0.29.1 - bugfix: read long value when deserializing gtid transaction’s length DBZ-7757

  • Change streaming exceptions are swallowed by BufferedChangeStreamCursor DBZ-7759

  • Sql-Server connector fails after initial start / processed record on subsequent starts DBZ-7765

  • Valid resume token is considered invalid which leads to new snapshot with some snapshot modes DBZ-7770

In total there were 14 issues resolved in Debezium 2.6.1.Final. The list of changes can also be found in our release notes.

A big thank you to all the contributors from the community who worked diligently on this release: Andrey Pustovetov, Anisha Mohanty, Chris Cranford, Chris Recalis, Jakub Cechacek, Jiri Pechanec, Lourens Naudé, Mario Fiore Vitale, Robert Roldan, and Selman Genç!

Chris Cranford

Chris is a software engineer at Red Hat. He previously was a member of the Hibernate ORM team and now works on Debezium. He lives in North Carolina just a few hours from Red Hat towers.

   

About Debezium

Debezium is an open source distributed platform that turns your existing databases into event streams, so applications can see and respond almost instantly to each committed row-level change in the databases. Debezium is built on top of Kafka and provides Kafka Connect compatible connectors that monitor specific database management systems. Debezium records the history of data changes in Kafka logs, so your application can be stopped and restarted at any time and can easily consume all of the events it missed while it was not running, ensuring that all events are processed correctly and completely. Debezium is open source under the Apache License, Version 2.0.

Get involved

We hope you find Debezium interesting and useful, and want to give it a try. Follow us on Twitter @debezium, chat with us on Zulip, or join our mailing list to talk with the community. All of the code is open source on GitHub, so build the code locally and help us improve ours existing connectors and add even more connectors. If you find problems or have ideas how we can improve Debezium, please let us know or log an issue.