Today it’s my great pleasure to announce the availability of Debezium 2.1.0.Final!
You might recently noticed that Debezium went a bit silent for the last few weeks. No, we are not going away. In fact the elves in Google worked furiously to bring you a present under a Christmas tree - Debezium Spanner connector.
| Release 2.1.0.Final did not contain a mandatory dependency. This is fixed in 2.1.1.Final hot update. |
After a plenty of intensive effort we woud like to present Nancy Xu as the leading engineer behind the implementation of the Debezium Spanner connector for Google’s Cloud Spanner distributed database. The connector itself is now in incubating state and still not fully feature complete (for example initial snapshots are not supported yet). Yet it is ready for a general use in scenarios where a robust Spanner-to-Kafka streaming implementation is required.
The initial release provides
As exciting these news are, this is not the only new feature available for Debezium. The release brings a nice pack of additional improvements.
-
Vitess connector supports initial snapshotting. This is a completely new feature and means that now the default behaviour for a new Vitess connector instance is to snapshot the current table content and then switch to streaming.
-
Starting Debezium 2.0 we extracted a set of interfaces to provide additional pluggable persistent stores. Redis offset and internal schema history store was converted into module and is now available for generic use.
-
MySQL connector processes
TRUNCATE TABLEcommands. When detected atmessage is emitted into the table topic. This feature is optional and is diabled by default. -
Kafka Connect provides so-called
predicatesthat enable user to apply transfromations conditionally. Debezium Engine and Debezium Server supports the same functionality too and is configured in the same way as kafka Connect. -
PostgreSQL connector is compatible with PostgreSQL 15.
-
Cassandra connector is from the very start a bit odd duckling in the way how the codebase is written and connector deployed. This meant that only Kafka was suported as a destination. The connector was rewritten so it now can run inside Debezium Server and so any supported sink can be used as the destination.
-
Nats JetStream is the new sink provided by Debezium Server.
-
Kafka Connect by default calculates the topic partition number based on the message primary key. With the new
ComputePartitionit is possible to define a list of per-table columns to explicitly calculate and set the partition number. -
PostgreSQL flushes LSN (and thus truncates the WAL) when the message is recorded in Kafka. For scenarios that prefer manual WAL management it is possible to disable this behaviour.
-
MongoDB connector always connected and streamed from primary node in the cluster. This is no longer necessary and non-primary nodes are preferred.
Other fixes & improvements
There were many bugfixes, stability changes, and improvements throughout the development of Debezium 2.1. Altogether, a total of 149 issues were fixed for this release.
A big thank you to all the contributors from the community who worked on this major release: Masazumi Kobayashi, Andrey Pustovetov, Anisha Mohanty, Balázs Sipos, Bob Roldan, Bobby Tiernay, Chris Cranford, Egyed Tamas, Enzo Cappa, Erdinç Taşkın, Gabor Andras, Gunnar Morling, Harvey Yue, Jakub Cechacek, Jeremy Ford, Jiri Novotny, Jiri Pechanec, Keri Harris, Marcelo Avancini, Mario Fiore Vitale, Mark Lambert, Martin Medek, Mike Kamornikov, Nancy Xu, Nir Levy, Ondrej Babec, Poonam Meghnani, Praveen Burgu, Raúl Estrada, Robert Roldan, Sahap Asci, Stefan Miklosovic, Subodh Kant Chaturvedi, Vojtech Juranek, Wu Zhenhua, Xuan Shen, Yohei Yoshimuta, tooptoop4!
What’s next?
So what are our current plans for the first quarter of the next year? Some potential features you can expect include:
-
Initial work on JDBC sink connector
-
Configurable signalling channels
-
JDBC and S3 history and offset storage support
As always, this roadmap is heavily influenced by the community, i.e. you. So if you would like to see any particular items here, please let us know.
Merry Christmas and Happy New Year 2023!
Onwards and Upwards!
About Debezium
Debezium is an open source distributed platform that turns your existing databases into event streams, so applications can see and respond almost instantly to each committed row-level change in the databases. Debezium is built on top of Kafka and provides Kafka Connect compatible connectors that monitor specific database management systems. Debezium records the history of data changes in Kafka logs, so your application can be stopped and restarted at any time and can easily consume all of the events it missed while it was not running, ensuring that all events are processed correctly and completely. Debezium is open source under the Apache License, Version 2.0.
Get involved
We hope you find Debezium interesting and useful, and want to give it a try. Follow us on Twitter @debezium, chat with us on Zulip, or join our mailing list to talk with the community. All of the code is open source on GitHub, so build the code locally and help us improve ours existing connectors and add even more connectors. If you find problems or have ideas how we can improve Debezium, please let us know or log an issue.
