Featured Posts

Tags
apache kafka apache kafka apicurio avro aws batch caassandra camel cassandra cdc channels community community stories connectors containers cqrs custom datalake db2 ddd debezium debezium server debezium ui deduplication discussion docker elasticsearch event sourcing exactly once semantics example examples features fedora flink hiring ibmi iceberg informix integration introduction jaeger jdbc json kafka kafka streams kafka streams kogito ksql kubernetes lakehouse machine learning mariadb microservices mongo mongodb mysql news newsletter notifications online learning operator oracle outbox performance postgres presentation production quarkus questdb rds releases schema scylla secrets sentry serialization signaling smt snapshots spanner spark sql sqlserver tensorflow testcontainers tests time series timescaledb topics tracing transactions vagrant vitess website

It has been several weeks since we released the first installment of Debezium 2.3, and I’m excited to announce the next iteration of Debezium 2.3 with 2.3.1.Final. As with any micro-release, the focus is on stability and bug fixes, as well as adjustments to our documentation; however there are some changes that are noteworthy that I would like to take a few moments to highlight.

New Features

While micro-releases generally do not include new features, there are situations where it makes sense to backport something that is new, particularly when the change is localized, adequately tested, and we’re confident that it will provide a strong benefit for the community utilizing stable releases. There are four major new features:

Error Handling improvements

In earlier releases, a connector configuration property errors.max.retries existed only for the SQL Server connector to control error handling behavior on specific connection-released failures. In short, this property allowed a connection-based failure to be retried up to the configured number of times before Debezium would hard fault. With 2.3.1.Final, we have unified this configuration option and its now available for all connectors.

This setting is useful when using Debezium particularly in non-Kafka Connect environments or where you may have configured Kafka Connect to not allow retry/back-off but you want to have that type of behavior in your Debezium source connector deployments. (DBZ-6573)

JDBC sink should not ingest schema change topics

The JDBC sink connector was never designed to ingest events from the schema change topic; however, the connector should be resilient and provide you with an adequate message rather than an exception indicating that there may be a misconfiguration.

Starting with this micro-release, the JDBC sink connector will analyze the event in more detail and if we determine that the event is a schema change, the connector will provide a descriptive warning stating the problem and request that the configuration be adjusted to avoid ingesting the schema changes topic. We hope this clarity will solve situations where an overlooked configuration error leads to a confusing exception and instead provides a direct remedy. (DBZ-6655)

MongoDB SRV support

MongoDB SRV support is a recent addition to the MongoDB connector’s feature set; however, as a part of DBZ-6701, we identified a situation where Debezium Server was unable to handle a MongoDB SRV connection string. We identified that the problem is related to the fact that Quarkus disables JNDI by default, and this creates an issue with the MongoDB driver with how it handles SRV connection strings.

With this micro-release, this problem is now resolved in Debezium 2.3. We specifically enable JNDI when building Debezium Server and this now allows the MongoDB SRV connection strings to work properly. A huge thanks to the community for helping diagnose this bug with the team!

Cassandra no longer skips batches with DELETE

The Cassandra connector experienced a rather critical regression with batch operations. Lets assume the following was observed from the Cassandra commit logs:

BEGIN BATCH
DELETE FROM ...
INSERT INTO ...
INSERT INTO ...
APPLY BATCH

The issue (DBZ-6576) was this particular batch was being interpreted incorrectly and as a a result, the two following INSERT operations were being omitted from the connector’s event stream. Thanks to the work by the community, a patch was applied and included in Debezium 2.3.1.Final where this is no longer the case and the Debezium change event stream will contain the DELETE as well as all other events that follow.

Other changes

Debezium 2.3.1.Final also includes quite a number of bug fixes and stability improvements, see below:

  • Oracle unsupported DDL statement - drop multiple partitions DBZ-6585

  • Only Struct objects supported for [Header field insertion], found: null DBZ-6588

  • MySQL parser cannot parse CAST AS dec DBZ-6590

  • Refactor retry handling in Redis schema history DBZ-6594

  • Excessive Log Message 'Marking Processed Record for Topic' DBZ-6597

  • Oracle DDL parser does not properly detect end of statement when comments obfuscate the semicolon DBZ-6599

  • Fixed DataCollections for table scan completion notificaiton DBZ-6605

  • Oracle connector is not recoverable if ORA-01327 is wrapped by another JDBC or Oracle exception DBZ-6610

  • Fatal error when parsing Mysql (Percona 5.7.39-42) procedure DBZ-6613

  • MySQL ALTER USER with RETAIN CURRENT PASSWORD fails with parsing exception DBZ-6622

  • Inaccurate documentation regarding additional-condition DBZ-6628

  • Oracle connection SQLRecoverableExceptions are not retried by default DBZ-6633

  • When Debezium Mongodb connector encounter authentication or under privilege errors, the connection between debezium and mongodb keeps going up. DBZ-6643

  • Cannot delete non-null interval value DBZ-6648

  • ConcurrentModificationException thrown in Debezium 2.3 DBZ-6650

  • Dbz crashes on parsing Mysql Procedure Code (Statement Labels) DBZ-6651

  • Vitess: Connector fails if table name is a mysql reserved word DBZ-6656

  • Retriable operations are retried infinitely since error handlers are not reused DBZ-6670

  • NotificationIT with Oracle xstream fails randomly DBZ-6672

  • Flaky Oracle test: shouldCaptureChangesForTransactionsAcrossSnapshotBoundaryWithoutReemittingDDLChanges DBZ-6673

  • Oracle DDL parser does not support column visibility on ALTER TABLE DBZ-6677

  • MongoDB SRV protocol not working in Debezium Server DBZ-6701

  • Add tzdata-java to UI installation Dockerfile DBZ-6713

Please refer to the release notes to learn more about all fixed bugs, update procedures, etc.

Many thanks to the following individuals from the community who contributed to Debezium 2.3.1.Final: Andrei Isac, Anil Dasari, Anisha Mohanty, Ant Kutschera, Artur Gukasian, Balint Bene, Bob Roldan, Chris Cranford, David Beck, Harvey Yue, Indra Shukla, Jakub Cechacek, Jiri Pechanec, Katerina Galieva, Mario Fiore Vitale, Ondrej Babec, René Kerner, Seo Jae-kwon, Stefan Miklosovic, and Vojtech Juranek!

Outlook and what’s next?

Debezium 2.3 will continue to receive maintenance updates throughout this quarter with the next being early/middle August and likely additional follow-ups late August and September, all barring on community feedback on regressions.

Presently the team is actively working on Debezium 2.4 in parallel, and I would expect to see Alpha2 most likely at the early part of August. We have lots of new features planned for Debezium 2.4, so I urge you to take a look at our road map for more details.

I will be making a formal announcement next week about the new, upcoming Debezium community event. This will be hosted by the Debezium team to provide a space where the team, contributors, and community can openly collaborate. I’ll have more details next week, so be on the look out for this!

And finally, Current 2023 (aka Kafka Summit) is nearing. The event this year is being hosted in San Jose, California on September 26th and 27th. If you’re planning to attend, please drop me an email. I would enjoy an opportunity to talk with the community and gather your feedback about Debezium and how we can improve moving forward!

Until then…​

Chris Cranford

Chris is a software engineer at Red Hat. He previously was a member of the Hibernate ORM team and now works on Debezium. He lives in North Carolina just a few hours from Red Hat towers.

   

About Debezium

Debezium is an open source distributed platform that turns your existing databases into event streams, so applications can see and respond almost instantly to each committed row-level change in the databases. Debezium is built on top of Kafka and provides Kafka Connect compatible connectors that monitor specific database management systems. Debezium records the history of data changes in Kafka logs, so your application can be stopped and restarted at any time and can easily consume all of the events it missed while it was not running, ensuring that all events are processed correctly and completely. Debezium is open source under the Apache License, Version 2.0.

Get involved

We hope you find Debezium interesting and useful, and want to give it a try. Follow us on Twitter @debezium, chat with us on Zulip, or join our mailing list to talk with the community. All of the code is open source on GitHub, so build the code locally and help us improve ours existing connectors and add even more connectors. If you find problems or have ideas how we can improve Debezium, please let us know or log an issue.