Featured Posts

Tags
apache kafka apache kafka apicurio avro aws batch caassandra camel cassandra cdc channels community community stories connectors containers cqrs custom datalake db2 ddd debezium debezium server debezium ui deduplication discussion docker elasticsearch event sourcing exactly once semantics example examples features fedora flink hiring ibmi iceberg informix integration introduction jaeger jdbc json kafka kafka streams kafka streams kogito ksql kubernetes lakehouse machine learning mariadb microservices mongo mongodb mysql news newsletter notifications online learning operator oracle outbox performance postgres presentation production quarkus questdb rds releases schema scylla secrets sentry serialization signaling smt snapshots spanner spark sql sqlserver tensorflow testcontainers tests time series timescaledb topics tracing transactions vagrant vitess website

It’s my pleasure to announce the second release of the Debezium 1.9 series, 1.9.0.Alpha2!

This release includes support for Oracle 21c, improvements around Redis for Debezium Server, configuring the kafka.query.timeout.ms option, and a number of bug fixes around DDL parsers, build infrastructure, etc.

Overall, the community fixed 51 issues for this release. Let’s take a closer look at some of the highlights.

Support for Oracle 21c

The Debezium Oracle connector has been tested with the latest release of Oracle 21c, 21.3.0.0, and is compatible. If you use either the LogMiner or the Xstreams adapter, you should now be able to use Oracle’s latest flagship version and stream change events without any changes. If you are on Oracle 12 or Oracle 19 and perform a database upgrade, your connector configuration should require no changes and remain compatible.

Configuring kafka.query.timeout.ms

When using the Kafka Admin Client and issuing API calls, the default timeout is 3 seconds. The new kafka.query.timeout.ms field can be used to provide a custom timeout to the Kafka Admin Client to avoid possible timeout problems in environments that may use TLS or SSL encryption or where network latency causes an unexpected timeout.

Thanks to the great work done by community member, Snigdhajyoti Ghosh.

Improvements in Redis for Debezium Server

We have three new fields in the Redis support for Debezium Server

  • redis.retry.initial.delay.ms

  • redis.retry.max.delay.ms

  • batch.size

Redis allows specifying a maximum memory limit using the maxmemory configuration; however, if this field is not configured then Redis will continue to allocate memory. If all memory is consumed, an OutOfMemory exception occurs. The Redis sink now uses redis.retry.initial.delay.ms and redis.retry.max.delay.ms to set an initial and max-retry delay to be more resilient to this and connection-related issues. If you have or are experiencing such exceptions, we urge you to try these new settings to improve the sink’s resilience and experience.

Pipeline-based transactions can substantially increase Redis queries. In order to leverage pipeline-based transactions, the batch.size configuration option can be specified which will allow Redis to write batches of change records rather than each record one by one.

Thanks to Yossi Shirizli, for these amazing improvements.

Other fixes

Some notable bug fixes and upgrades are:

  • Suspected inconsistent documentation for 'Ad-hoc read-only Incremental snapshot' DBZ-4171

  • Oracle Logminer: snapshot→stream switch misses DB changes in ongoing transactions DBZ-4367

  • DDL parsing issue: ALTER TABLE …​ MODIFY PARTITION …​ DBZ-4649

  • OracleSchemaMigrationIT fails with Xstream adapter DBZ-4703

  • Migrating UI from webpack-dev-server v3 to v4 DBZ-4642

  • Upgrade postgres driver to version 42.3.2 DBZ-4658

  • Upgrade to Quarkus 2.7.0.Final DBZ-4677

  • Update shared UG deployment file for use with downstream OCP Install Guide DBZ-4700

  • Indicate ROWID is not supported by XStream DBZ-4702

  • Incremental snapshots does not honor column case sensitivity DBZ-4584

  • Build trigger issues DBZ-4672

  • Cannot expand JSON payload with nested arrays of objects DBZ-4704

We will also be backporting the critical bugfixes to the 1.8 branch and will release Debezium 1.8.1.Final later this week.

A big thank you to all the contributors from the community who worked on this release: Andrei Isac, Anisha Mohanty, Bob Roldan, Chris Cranford, David Haglund, Dominique Chanet, Farid Uyar, Gunnar Morling, Harvey Yue, Ismail Simsek, Jason Schweier, Jiri Novotny, Jiri Pechanec, Mark Drilling, Nathan Smit, Paweł Malon, René Kerner, Shichao An, Snigdhajyoti Ghosh, Vojtech Juranek, and Yossi Shirizli!

Anisha Mohanty

Anisha is a Software Engineer at Red Hat. Currently working with the Debezium Team. She lives in Bangalore, India.

   

About Debezium

Debezium is an open source distributed platform that turns your existing databases into event streams, so applications can see and respond almost instantly to each committed row-level change in the databases. Debezium is built on top of Kafka and provides Kafka Connect compatible connectors that monitor specific database management systems. Debezium records the history of data changes in Kafka logs, so your application can be stopped and restarted at any time and can easily consume all of the events it missed while it was not running, ensuring that all events are processed correctly and completely. Debezium is open source under the Apache License, Version 2.0.

Get involved

We hope you find Debezium interesting and useful, and want to give it a try. Follow us on Twitter @debezium, chat with us on Zulip, or join our mailing list to talk with the community. All of the code is open source on GitHub, so build the code locally and help us improve ours existing connectors and add even more connectors. If you find problems or have ideas how we can improve Debezium, please let us know or log an issue.