An edge case was reported in DBZ-5996 where if a temporal column used ZonedTimestamp and if the column’s value had 0 micro or nanoseconds, rather than emitting the value as 2023-01-19T12:30:00.123000Z, the value would be emitted in a truncated way as 2023-01-19T12:30:00.123Z. This could lead to other issues with converters used in the event pipeline when the output from that column could be formatted inconsistently.

In order to remedy the edge case, the ZonedTimestamp implementation will now pad the fraction-based seconds value of the column’s value to the length/scale of the source database column. Using the example above of a TIMESTAMP(6) MySQL column type, the emitted value will now properly reflect a value of 2023-01-19T12:30:00.123000Z.

While this change in behavior is likely to have minimal impact to most users, we wanted to bring attention to it in the event that you’ve perhaps used other means to handle this edge case in your pipelines. If you have, you should be able to rely on Debezium to emit the value consistently, even when the fraction-based seconds is 0.

The Debezium for Oracle connector normally manages what is called a flush table, which is an internal table used to manage the flush cycles used by the Oracle Log Writer Buffer (LGWR) process. This flushing process requires that the user account the connector uses to have permission to create and write to this table. Logical stand-by databases often have more restrictive rules about data manipulation and may even be read-only, therefore, writing to the database is unfavorable or even not permissible.

To support an Oracle read-only logical stand-by database, we introduced a flag to disable the creation and management of this flush table. This feature can be used with both Oracle Standalone and Oracle RAC installations, and is currently considered incubating, meaning its subject to change in the future.

In order to enable Oracle read-only logical stand-by support, add the following connector option:

In a future version, we plan to add support for an Oracle read-only physical stand-by database.

Debezium provides a Storage API framework that enables connectors to store offset and schema history state in a variety of persistence datastores. Moreover, the framework enables contributors to extend the API by adding new storage implementations with ease. Currently, the Storage API framework supports the local FileSystem, a Kafka Topic, or Redis datastores.

With Debezium 2.2, we’re pleased to add Amazon S3 buckets as part of that framework, allowing the schema history to be persisted to an S3 bucket. An example connector configuration using S3 might look like the following:

In previous releases of Debezium, the connector start-up phase used a fail-fast strategy. Simply put, this meant that if we couldn’t connect, authenticate, or performs any of the start-up phase steps required by the connector, the connector would enter a FAILED state.

One specific problem area for users is if the connector gracefully starts, runs for a period of time, and then eventually encounters some fatal error. If the error is related to a resource that wasn’t accessed during the connector’s start-up lifecycle, the connector would typically gracefully restart just fine. However, the situation is different if the problem was related to the database’s availability and the database was still unavailable during the connector’s start-up phase. In this situation, the connector would fail-fast, and would enter a FAILED state, requiring manual intervention.

The fail-fast approach served Debezium well over the years, but in a world where a resource can come and go without warning, it became clear that changes were needed to improve Debezium’s reliability and resiliency. While the Kafka Connect’s retry/back-off framework has helped in this regard, that doesn’t address the concerns with start-up resources being unavailable with how the code is currently written.

Debezium 2.2 changes this landscape, shifting how we integrate with Kafka Connect’s source connector API slightly. Instead of accessing potentially unavailable resources during the start-up lifecycle, we moved that access to a later phase in the connector’s lifecycle. In effect, the Debezium start-up code is executed lazily that accesses potentially unavailable resources, which allows us to take advantage of the Kafka Connect retry/back-off framework even during our start-up code. In short, if the database is still unavailable during the connector’s start-up, the connector will continue to retry/back-off if Kafka Connect retries are enabled. Only once the maximum number of retry attempts has been reached or a non-retriable error occurs will the connector task enter a FAILED state.

We hope this brings more reliability and resiliency for the Debezium experience, improving how errors are handled in an ever-changing landscape, and provides a solid foundation to manage connector lifecycles.

Debezium Server is a Quarkus-based framework that allows executing a Debezium connector from the command line, without Kafka or Kafka Connect, allowing the delivery of Debezium change events to any destination framework. With Debezium 2.2, two new sink connectors have been added to Debezium Server to support sending change events to Apache RocketMQ and to Infinispan.

There were quite a number of bugfixes and stability changes in this release, some noteworthy are:

Altogether, 42 issues were fixed for this release. A big thank you to all the contributors from the community who worked on this release: Akshansh Jain, Gabor, Anil Dasari, Animesh Kumar, Anisha Mohanty, Bob Roldan, Chris Cranford, Erdinç Taşkın, Govinda Sakhare, Harvey Yue, Hossein Torabi, Indra Shukla, Jakub Zalas, Jeremy Ford, Jiri Pechanec, Jochen Schalanda, Luca Scannapieco, Mario Fiore Vitale, Mark Lambert, Rajendra Dangwal, Sun Xiao Jian, Vojtech Juranek, Yohei Yoshimuta, and yohei yoshimuta!

As the road to Debezium 2.2 is just starting, this initial release covers quite a lot of the features we’ve outlined our recent 2023 road map update. However, there are still a number of features that are still in active development, which include:

As we continue development on Debezium 2.2 and bugfixes to Debezium 2.1, we would love to hear your feedback or suggestions, whether it’s regarding our road map, the changes in this release, or something you’d like to see that we haven’t mentioned. Be sure to get in touch with us on the mailing list or our chat if there is. Or if you just want to stop by and give us a "Hello", we’d welcome that too.

Until next time…​