It has been several weeks since we released the first installment of Debezium 2.3, and I’m excited to announce the next iteration of Debezium 2.3 with 2.3.1.Final. As with any micro-release, the focus is on stability and bug fixes, as well as adjustments to our documentation; however there are some changes that are noteworthy that I would like to take a few moments to highlight.
While micro-releases generally do not include new features, there are situations where it makes sense to backport something that is new, particularly when the change is localized, adequately tested, and we’re confident that it will provide a strong benefit for the community utilizing stable releases. There are four major new features:
Error Handling improvements
In earlier releases, a connector configuration property
errors.max.retries existed only for the SQL Server connector to control error handling behavior on specific connection-released failures. In short, this property allowed a connection-based failure to be retried up to the configured number of times before Debezium would hard fault. With 2.3.1.Final, we have unified this configuration option and its now available for all connectors.
This setting is useful when using Debezium particularly in non-Kafka Connect environments or where you may have configured Kafka Connect to not allow retry/back-off but you want to have that type of behavior in your Debezium source connector deployments. (DBZ-6573)
JDBC sink should not ingest schema change topics
The JDBC sink connector was never designed to ingest events from the schema change topic; however, the connector should be resilient and provide you with an adequate message rather than an exception indicating that there may be a misconfiguration.
Starting with this micro-release, the JDBC sink connector will analyze the event in more detail and if we determine that the event is a schema change, the connector will provide a descriptive warning stating the problem and request that the configuration be adjusted to avoid ingesting the schema changes topic. We hope this clarity will solve situations where an overlooked configuration error leads to a confusing exception and instead provides a direct remedy. (DBZ-6655)
MongoDB SRV support
MongoDB SRV support is a recent addition to the MongoDB connector’s feature set; however, as a part of DBZ-6701, we identified a situation where Debezium Server was unable to handle a MongoDB SRV connection string. We identified that the problem is related to the fact that Quarkus disables JNDI by default, and this creates an issue with the MongoDB driver with how it handles SRV connection strings.
With this micro-release, this problem is now resolved in Debezium 2.3. We specifically enable JNDI when building Debezium Server and this now allows the MongoDB SRV connection strings to work properly. A huge thanks to the community for helping diagnose this bug with the team!
Cassandra no longer skips batches with DELETE
The Cassandra connector experienced a rather critical regression with batch operations. Lets assume the following was observed from the Cassandra commit logs:
BEGIN BATCH DELETE FROM ... INSERT INTO ... INSERT INTO ... APPLY BATCH
The issue (DBZ-6576) was this particular batch was being interpreted incorrectly and as a a result, the two following INSERT operations were being omitted from the connector’s event stream. Thanks to the work by the community, a patch was applied and included in Debezium 2.3.1.Final where this is no longer the case and the Debezium change event stream will contain the
DELETE as well as all other events that follow.
Debezium 2.3.1.Final also includes quite a number of bug fixes and stability improvements, see below:
Oracle unsupported DDL statement - drop multiple partitions DBZ-6585
Only Struct objects supported for [Header field insertion], found: null DBZ-6588
MySQL parser cannot parse CAST AS dec DBZ-6590
Refactor retry handling in Redis schema history DBZ-6594
Excessive Log Message 'Marking Processed Record for Topic' DBZ-6597
Oracle DDL parser does not properly detect end of statement when comments obfuscate the semicolon DBZ-6599
Fixed DataCollections for table scan completion notificaiton DBZ-6605
Oracle connector is not recoverable if ORA-01327 is wrapped by another JDBC or Oracle exception DBZ-6610
Fatal error when parsing Mysql (Percona 5.7.39-42) procedure DBZ-6613
MySQL ALTER USER with RETAIN CURRENT PASSWORD fails with parsing exception DBZ-6622
Inaccurate documentation regarding additional-condition DBZ-6628
Oracle connection SQLRecoverableExceptions are not retried by default DBZ-6633
When Debezium Mongodb connector encounter authentication or under privilege errors, the connection between debezium and mongodb keeps going up. DBZ-6643
Cannot delete non-null interval value DBZ-6648
ConcurrentModificationException thrown in Debezium 2.3 DBZ-6650
Dbz crashes on parsing Mysql Procedure Code (Statement Labels) DBZ-6651
Vitess: Connector fails if table name is a mysql reserved word DBZ-6656
Retriable operations are retried infinitely since error handlers are not reused DBZ-6670
NotificationIT with Oracle xstream fails randomly DBZ-6672
Flaky Oracle test: shouldCaptureChangesForTransactionsAcrossSnapshotBoundaryWithoutReemittingDDLChanges DBZ-6673
Oracle DDL parser does not support column visibility on ALTER TABLE DBZ-6677
MongoDB SRV protocol not working in Debezium Server DBZ-6701
Add tzdata-java to UI installation Dockerfile DBZ-6713
Please refer to the release notes to learn more about all fixed bugs, update procedures, etc.
Many thanks to the following individuals from the community who contributed to Debezium 2.3.1.Final: Andrei Isac, Anil Dasari, Anisha Mohanty, Ant Kutschera, Artur Gukasian, Balint Bene, Bob Roldan, Chris Cranford, David Beck, Harvey Yue, Indra Shukla, Jakub Cechacek, Jiri Pechanec, Katerina Galieva, Mario Fiore Vitale, Ondrej Babec, René Kerner, Seo Jae-kwon, Stefan Miklosovic, and Vojtech Juranek!
Outlook and what’s next?
Debezium 2.3 will continue to receive maintenance updates throughout this quarter with the next being early/middle August and likely additional follow-ups late August and September, all barring on community feedback on regressions.
Presently the team is actively working on Debezium 2.4 in parallel, and I would expect to see Alpha2 most likely at the early part of August. We have lots of new features planned for Debezium 2.4, so I urge you to take a look at our road map for more details.
I will be making a formal announcement next week about the new, upcoming Debezium community event. This will be hosted by the Debezium team to provide a space where the team, contributors, and community can openly collaborate. I’ll have more details next week, so be on the look out for this!
And finally, Current 2023 (aka Kafka Summit) is nearing. The event this year is being hosted in San Jose, California on September 26th and 27th. If you’re planning to attend, please drop me an email. I would enjoy an opportunity to talk with the community and gather your feedback about Debezium and how we can improve moving forward!
Debezium is an open source distributed platform that turns your existing databases into event streams, so applications can see and respond almost instantly to each committed row-level change in the databases. Debezium is built on top of Kafka and provides Kafka Connect compatible connectors that monitor specific database management systems. Debezium records the history of data changes in Kafka logs, so your application can be stopped and restarted at any time and can easily consume all of the events it missed while it was not running, ensuring that all events are processed correctly and completely. Debezium is open source under the Apache License, Version 2.0.
We hope you find Debezium interesting and useful, and want to give it a try. Follow us on Twitter @debezium, chat with us on Zulip, or join our mailing list to talk with the community. All of the code is open source on GitHub, so build the code locally and help us improve ours existing connectors and add even more connectors. If you find problems or have ideas how we can improve Debezium, please let us know or log an issue.