I am excited to share that Debezium 2.0.0.Alpha1 has been released!
This release is the first of several planned pre-releases of Debezium 2.0 over the next five months. Each pre-release plans to focus on strategic changes in the hope that as we move forward, changes can be easily tested and regressions addressed quickly.
In this release, some of the most notable changes include requiring Java 11 to use Debezium or any of its components, the removal of wal2json
support for PostgreSQL and the legacy MySQL connector implementation, as well as some notable features such as improved Debezium Server Google Pub/Sub sink support, and a multitude of bugfixes. Let’s take a look at a few of these.
Java 11 required
We have wanted to make the jump to using Java 11 as a build requirement for quite some time now, and with Debezium 2.0 this is now possible. With Java 11, this enables us to take advantage of new language features, such as the new String
API and Predicate
support changes in the codebase, while also benefiting from many Java performance improvements.
Our very own Vojtech Juranek will be publishing a blog post next week that discusses the switch to Java 11 and 17 in greater detail. I highly recommend giving it a read as it provides a deep dive into the technical background & effort that went into making this possible.
So before migrating to Debezium 2.0, be sure that Java 11 is available.
PostgreSQL wal2json support removed
The PostgreSQL connector has supported several plugins throughout Debezium 1.x, including decoderbufs
, wal2json
, and pgoutput
. PostgreSQL 9.6 recently reached end of life on November 11, 2021. This presented a great opportunity for us to review the supported decoders and to see whether we could streamline those options.
Since pgoutput
is a native decoder supported by all non-EOL versions of PostgreSQL (PG10+), it made sense to remove wal2json
. Reducing the number of decoders to 2 (down from 3), allows us to streamline the code for PostgreSQL, reduces the overall maintenance cost of the connector, and gives us a much more narrow target for overall support.
If you are still using PostgreSQL 9.6 or were using wal2json
previously, you will need to migrate to at least PostgreSQL 10.0 or to decoderbufs
or pgougput
respectively before upgrading to Debezium 2.0.
Legacy MySQL implementation removed
As some of you may or may not know, we implemented the MySQL connector based on the common-connector framework back in Debezium 1.5 (Feb 2021). As a part of that re-write, we introduced the ability for MySQL users to enable the legacy connector behavior using the configuration option internal.implementation
set as legacy
. This legacy implementation was deprecated in favor of the new common-connector framework behavior. With Debezium 2.0, this internal.implementation
configuration option and the legacy connector implementation have been removed.
If your current connector deployment relies on this legacy implementation, you should be aware that by upgrading to Debezium 2.0, the connector will no longer use that older implementation and will use the common-connector implementation only. Feature-wise, both implementations are on-par with one another with one exception: the legacy implementation had experimental support for changing filter configurations. If you have relied on this legacy behavior, be aware that feature is no longer available.
Other fixes & improvements
There are several bugfixes and stability changes in this release, some noteworthy are:
-
Implement Pub/Sub Lite change consumer DBZ-4450
-
Add Google Pub/Sub emulator support DBZ-4491
-
Making Postgres
PSQLException: This connection has been closed.
retriable DBZ-4948 -
Should store event header timestamp in HistoryRecord DBZ-4998
-
Getting java.sql.SQLException: ORA-01291: missing logfile while running with archive log only DBZ-4879
-
Debezium uses wrong LCR format for Oracle 12.1 DBZ-4932
-
NPE caused by io.debezium.connector.oracle.antlr.listener.ColumnDefinitionParserListener.resolveColumnDataType DBZ-4976
-
Outbox Transform does not allow expanded payload with additional fields in the envelope DBZ-4989
-
CLOB with single quotes causes parser exception DBZ-4994
-
Cassandra 3 handler does not process partition deletions correctly DBZ-5022
-
SQL Server in multi-partition mode fails if a new database is added to an existing configuration DBZ-5033
-
Upgrade to Quarkus 2.8.2.Final DBZ-5062
Altogether, 55 issues were fixed for this release.
A big thank you to all the contributors from the community who worked on this release: Wang Min Chao, Andrey Pustovetov, Anisha Mohanty, Brad Morgan, Calin Laurentiu Ilie, Chad Marmon, Chris Cranford, Gunnar Morling, Harvey Yue, Jakub Cechacek, Jan Doms, Jiri Novotny, Jiri Pechanec, Lars Werkman, Martin Medek, Mohammad Yousuf Minhaj Zia, Oren Elias, Paul Tzen, Plugaru Tudor, René Kerner, Sergei Morozov, Stefan Miklosovic, Vojtech Juranek, and Yossi Shirizli!
What’s Next?
We have resolved the runtime problem with Debezium Server in the 1.9.1.Final release, so you can expect a 1.9.2.Final later this week which will also address other bugfixes. You can continue to expect updates to 1.9 in the weeks that follow as bugs are reported and fixes are made to address those.
As we continue our efforts on Debezium 2.0, you can expect a second pre-release in the coming weeks, sticking to our regular 3-week cadence. In this next pre-release, we plan to focus on message schema versioning/naming, connector configuration changes with new pass-thru namespaces, removal of deprecated options, as well as unifying default value handling, just to name a few on the roadmap.
And speaking of Debezium’s roadmap, stay tuned as we’ll have more to share about Debezium 2.0, its future releases of 2.x, all on our roadmap soon!
About Debezium
Debezium is an open source distributed platform that turns your existing databases into event streams, so applications can see and respond almost instantly to each committed row-level change in the databases. Debezium is built on top of Kafka and provides Kafka Connect compatible connectors that monitor specific database management systems. Debezium records the history of data changes in Kafka logs, so your application can be stopped and restarted at any time and can easily consume all of the events it missed while it was not running, ensuring that all events are processed correctly and completely. Debezium is open source under the Apache License, Version 2.0.
Get involved
We hope you find Debezium interesting and useful, and want to give it a try. Follow us on Twitter @debezium, chat with us on Zulip, or join our mailing list to talk with the community. All of the code is open source on GitHub, so build the code locally and help us improve ours existing connectors and add even more connectors. If you find problems or have ideas how we can improve Debezium, please let us know or log an issue.