It’s my pleasure to announce the first release of the Debezium 1.7 series, 1.7.0.Alpha1!
With the summer in a full-swing, this release brings additional improvements to the Debezium Oracle connector but also to the others as well.
Alternative Oracle Connector LogMiner processors
The Debezium connector for Oracle uses an in-memory buffer to cache in-progress transaction changes until the transaction is either committed or rolled back. This cache can require a significant memory footprint depending on the number of events in the transaction, how many concurrent transactions are in-progress, as well as the data itself that represents each event such as large character or binary objects. This can be tough to manage for some environments that have ongoing long-running transactions.
This release introduces a significant refactoring to the connector processing code that allows for varied modes of processing LogMiner change events. Out-of-the-box the memory-based implementation remains the default, but we’ve included a new implementation based on the Infinispan distributed data store. This implementation makes it possible to track any number of in-progress transactions regardless of their size or duration.
To take advantage of the new Infinispan implementation, the following configuration options must be provided:
As this is early work, the new Infinispan buffer mode currently has a few limitations:
The Infinispan configuration is not exposed outside to the user for tuning
Only file-based cache store is supported; if you work with multi-node Kafka Connect clusters, a networked filesystem must be used for the cache store in order to support rebalancing of connector tasks in the Connect cluster
This should not stop you from giving it a test drive and send as much feedback as possible to us for further improvements. We expect to remove the above restrictions in the next release.
The major feature of the 1.6 release has got also new improvements. The Debezium connector for Oracle now supports incremental snapshots in the same way as the other connectors, so Oracle users now can easily add new tables to the capture list and get them snapshotted on the fly.
For the MySQL connector, there is a new incremental snapshot mode that could be used for databases that do not allow writing to the signal table. Kate Galieva invented a method based on GTID lists that could be used to provide watermarking without writing to the database. Thank you for the nice contribution!
Altogether, 49 issues were fixed for this release. A big thank you goes out to all the community members who contributed: Blake Peno, Alfusainey Jallow, Bingqin Zhou, Hossein Torabi, Katerina Galieva, Kyley Jex, Martín Pérez, Naveen Kumar KR, Patrick Chu, Pavel Strashkin, Raphael Auv, Sergei Morozov, Thiago Dantas, Tin Nguyen, Tommy Karlsson, WenChao Ke, and yangsanity.
For the upcoming 1.7 preview releases, we’re planning to focus on completing the follow-up task for the Oracle LogMiner processor and provide the support for JDBC based process too. We will explore incremental snapshotting support for MongoDB and do research on MariaDB support.
Release of 1.6.1.Final
Debezium is an open source distributed platform that turns your existing databases into event streams, so applications can see and respond almost instantly to each committed row-level change in the databases. Debezium is built on top of Kafka and provides Kafka Connect compatible connectors that monitor specific database management systems. Debezium records the history of data changes in Kafka logs, so your application can be stopped and restarted at any time and can easily consume all of the events it missed while it was not running, ensuring that all events are processed correctly and completely. Debezium is open source under the Apache License, Version 2.0.
We hope you find Debezium interesting and useful, and want to give it a try. Follow us on Twitter @debezium, chat with us on Zulip, or join our mailing list to talk with the community. All of the code is open source on GitHub, so build the code locally and help us improve ours existing connectors and add even more connectors. If you find problems or have ideas how we can improve Debezium, please let us know or log an issue.