Subscribe


Debezium 0.9.4.Final Released

It’s my pleasure to announce the release of Debezium 0.9.4.Final!

This is a drop-in replacement for earlier Debezium 0.9.x versions, containing mostly bug fixes and some improvements related to metrics. Overall, 17 issues were resolved.

MySQL Connector Improvements

The Debezium connector for MySQL comes with two new metrics:

  • Whether GTID is enabled for offset tracking or not (DBZ-1221)

  • Number of filtered events (DBZ-1206)

It also supports database connections using TLS 1.2 (DBZ-1208) now.

New Postgres Datatypes

The Postgres connector now allows to capture changes to columns of the CIDR and INET types (DBZ-1189).

Bug Fixes

The fixed bugs include the following:

  • Closing connection after snapshotting (DBZ-1218)

  • Can parse ALTER statement affecting enum column with character set options (DBZ-1203)

  • Avoiding timeout after bootstrapping a new table (DBZ-1207)

Check out the release notes for the complete list of issues fixed in Debezium 0.9.4.

Many thanks to Debezium community members Andrey Pustovetov, Jordan Bragg, Joy Gao, Preethi Sadagopan, Renato Mefi, Sasha Kovryga, Shubham Rawat and Stephen Powis for their contributions to this release!

About Debezium

Debezium is an open source distributed platform that turns your existing databases into event streams, so applications can see and respond almost instantly to each committed row-level change in the databases. Debezium is built on top of Kafka and provides Kafka Connect compatible connectors that monitor specific database management systems. Debezium records the history of data changes in Kafka logs, so your application can be stopped and restarted at any time and can easily consume all of the events it missed while it was not running, ensuring that all events are processed correctly and completely. Debezium is open source under the Apache License, Version 2.0.

Get involved

We hope you find Debezium interesting and useful, and want to give it a try. Follow us on Twitter @debezium, chat with us on Gitter, or join our mailing list to talk with the community. All of the code is open source on GitHub, so build the code locally and help us improve ours existing connectors and add even more connectors. If you find problems or have ideas how we can improve Debezium, please let us know or log an issue.


Debezium 0.9.3.Final Released

The Debezium team is happy to announce the release of Debezium 0.9.3.Final!

This is mostly a bug-fix release and a drop-in replacement for earlier Debezium 0.9.x versions, but there are few significant new features too. Overall, 17 issues were resolved.

Container images will be released with a small delay due to some Docker Hub configuration issues.

New Features

The 0.9.3 release comes with two larger new features:

  • A feature request was made to execute a partial recovery of the replication process after losing the replication slot with the PostgreSQL database, e.g. after failing over to a secondary database host (DBZ-1082). Instead of adding yet another snapshotting mode, we took a step back and decided to make the Postgres snapshotting process more customizable by introducing a service provider interface (SPI). This lets you implement and register your own Java class for controlling the snaphotting process. See the issue description of DBZ-1082 for one possible custom implementation of this SPI, which is based on Postgres' catalog_xmin property and selects all records altered after the last known xmin position. To learn more about the SPI, see the the Snapshotter contract. Note that the feature is still in incubating phase and the SPI should be considered unstable for the time being.

  • Not long ago we published blogpost about implementing the outbox pattern with Debezium for propagating data changes between microservices. Community member Renato Mefi expanded the idea and created a ready-made implementation of the single message transform (SMT) described in the post for routing events from the outbox table to specific topics. This SMT is part of the Debezium core library now (DBZ-1169). Its usage will be described in the documentation soon; for the time being please refer to the EventRouter type and the accompanying configuration class.

Bug fixes

We did a couple of fixes related to the Debezium Postgres connector:

  • A regression that introduced a deadlock in snapshotting process has been fixed (DBZ-1161)

  • The hstore datatype works correctly in snapshot phase (DBZ-1162)

  • The wal2json plug-in processes also empty events (DBZ-1181) as e.g. originating from materialize view updates; this should help to resolve some of the issues where log files in Postgres couldn’t be discarded due to Debezium’s replication slot not advancing.

  • The commit time is propely converted to microseconds (DBZ-1174)

Also the Debezium MySQL connector saw a number of fixes especially in SQL parser:

  • The SERIAL datatype and default value is now supported (DBZ-1185)

  • A specific detail in the MySQL grammar that allows to enumerate table options in ALTER TABLE without comma works (DBZ-1186)

  • A false alarm for empty MySQL password is no longer reported (DBZ-1188)

  • It is no longer necessary to create history topic manually for broker without default topic replication value (DBZ-1179)

It is now possible to process multiple schemas with a single Oracle connector (DBZ-1166).

Check out the release notes for the complete list of issues fixed in Debezium 0.9.3.

Many thanks to Debezium community members Renato Mefi, Shubham Rawat, Addison Higham, Jon Casstevens, Ashar Hassan and Josh Stanfield for their contributions to this release!

About Debezium

Debezium is an open source distributed platform that turns your existing databases into event streams, so applications can see and respond almost instantly to each committed row-level change in the databases. Debezium is built on top of Kafka and provides Kafka Connect compatible connectors that monitor specific database management systems. Debezium records the history of data changes in Kafka logs, so your application can be stopped and restarted at any time and can easily consume all of the events it missed while it was not running, ensuring that all events are processed correctly and completely. Debezium is open source under the Apache License, Version 2.0.

Get involved

We hope you find Debezium interesting and useful, and want to give it a try. Follow us on Twitter @debezium, chat with us on Gitter, or join our mailing list to talk with the community. All of the code is open source on GitHub, so build the code locally and help us improve ours existing connectors and add even more connectors. If you find problems or have ideas how we can improve Debezium, please let us know or log an issue.


Debezium 0.9.2.Final Released

The Debezium team is happy to announce the release of Debezium 0.9.2.Final!

This is mostly a bug-fix release and a drop-in replacement for earlier Debezium 0.9.x versions. Overall, 18 issues were resolved.

A couple of fixes relate to the Debezium Postgres connector:

  • When not using REPLICA IDENTITY FULL, certain data types could trigger exceptions for update or delete events; those are fixed now (DBZ-1141, DBZ-1149)

  • The connector won’t fail any longer when encountering a change to a row with an unaltered TOAST column value (DBZ-1146)

Also the Debezium MySQL connector saw a number of fixes:

  • The connector works correctly now when using GTIDs and ANSI_QUOTES SQL mode (DBZ-1147)

  • The new Antlr-based DDL parsers can handle column names that are key words such as MEDIUM (DBZ-1150)

  • TIME columns with a default value larger than 23:59:59 can be exported now (DBZ-1137)

Another important fix was done in the Debezium connector for SQL Server, where the connector archive deployed to Maven Central accidentally contained all test-scoped and provided-scoped dependencies. This has been resolved now, so the connector archive only contains the actually needed JARs and thus is much smaller (DBZ-1138).

New Features

The 0.9.2 release also comes with two small new features:

  • You can pass arbitrary parameters to the logical decoding plug-in used by the Postgres connector; this can for instance be used with wal2json to limit the number of tables to capture on the server side (DBZ-1130)

  • The MongoDB connector now has the long-awaited snapshotting mode NEVER (DBZ-867), i.e. you can set up a new connector without taking an initial snapshot and instantly beginning streaming changes from the oplog

Version Updates

As of this release, Debezium has been upgraded to Apache Kafka 2.1.1. Amongst others, this release fixes an issue where the Kafka Connect REST API would expose connector credentials also when those were configured via secrets (KAFKA-5117). We’ve also upgraded the binlog client used by the MySQL connector to version 0.19.0 (DBZ-1140), which fixes a bug that had caused exceptions during rebalancing the connector before (DBZ-1132).

Check out the release notes for the complete list of issues fixed in Debezium 0.9.2.

Many thanks to Debezium community members Andrey Pustovetov, Keith Barber, Krizhan Mariampillai and Taylor Rolison for their contributions to this release!

About Debezium

Debezium is an open source distributed platform that turns your existing databases into event streams, so applications can see and respond almost instantly to each committed row-level change in the databases. Debezium is built on top of Kafka and provides Kafka Connect compatible connectors that monitor specific database management systems. Debezium records the history of data changes in Kafka logs, so your application can be stopped and restarted at any time and can easily consume all of the events it missed while it was not running, ensuring that all events are processed correctly and completely. Debezium is open source under the Apache License, Version 2.0.

Get involved

We hope you find Debezium interesting and useful, and want to give it a try. Follow us on Twitter @debezium, chat with us on Gitter, or join our mailing list to talk with the community. All of the code is open source on GitHub, so build the code locally and help us improve ours existing connectors and add even more connectors. If you find problems or have ideas how we can improve Debezium, please let us know or log an issue.


Debezium 0.9.1.Final Released

Quickly following up to last week’s release of Debezium 0.9, it’s my pleasure today to announce the release of Debezium 0.9.1.Final!

This release fixes a couple of bugs which were reported after the 0.9 release. Most importantly, there are two fixes to the new Debezium connector for SQL Server, which deal with correct handling of LSNs after connector restarts (DBZ-1128, DBZ-1131). The connector also uses more reasonable defaults for the selectMethod and fetchSize options of the SQL Server JDBC driver (DBZ-1065), which can help to significantly increase through-put and reduce memory consumption of the connector.

The MySQL connector supports GENERATED columns now with the new Antlr-based DDL parser (DBZ-1123), and for the Postgres connector the handling of primary key column definition changes was improved (DBZ-997).

In terms of new features, there is a new container image provided on Docker Hub now: the debezium/tooling image contains a couple of open-source CLI tools (currently kafkacat, httpie, jq, mycli and pqcli) which greatly help when working with Debezium connectors, Apache Kafka and Kafka Connect on the command line (DBZ-1125). A big thank you to the respective authors these fantastic tools!

CLI tools for working with Debezium

Altogether, 12 issues were resolved in this release. Please refer to the release notes to learn more about all fixed bugs, update procedures etc.

Thanks a lot to community members Ivan Lorenz and Tomaz Lemos Fernandes for their contributions to this release!

About Debezium

Debezium is an open source distributed platform that turns your existing databases into event streams, so applications can see and respond almost instantly to each committed row-level change in the databases. Debezium is built on top of Kafka and provides Kafka Connect compatible connectors that monitor specific database management systems. Debezium records the history of data changes in Kafka logs, so your application can be stopped and restarted at any time and can easily consume all of the events it missed while it was not running, ensuring that all events are processed correctly and completely. Debezium is open source under the Apache License, Version 2.0.

Get involved

We hope you find Debezium interesting and useful, and want to give it a try. Follow us on Twitter @debezium, chat with us on Gitter, or join our mailing list to talk with the community. All of the code is open source on GitHub, so build the code locally and help us improve ours existing connectors and add even more connectors. If you find problems or have ideas how we can improve Debezium, please let us know or log an issue.


Debezium 0.9.0.Final Released

I’m delighted to announce the release of Debezium 0.9 Final!

This release only adds a small number of changes since last week’s CR1 release; most prominently there’s some more metrics for the SQL Server connector (lag behind master, number of transactions etc.) and two bug fixes related to the handling of partitioned tables in MySQL (DBZ-1113) and Postgres (DBZ-1118).

Having been in the works for six months after the initial Alpha release, Debezium 0.9 comes with a brand new connector for SQL Server, lots of new features and improvements for the existing connectors, updates to the latest versions of Apache Kafka and the supported databases as well as a wide range of bug fixes.

Some key features of the release besides the aforementioned CDC connector for SQL Server are:

  • Initial snapshotting for the Oracle connector (which remains to be a "tech preview" at this point)

  • Brand-new metrics for the SQL Server and Oracle connectors and extended metrics for the MySQL connector

  • Field filtering and renaming for MongoDB

  • A new handler interface for the embedded engine

  • Lots of improvements around the "event flattening" SMT for MongoDB

  • More detailed source info in CDC events and optional metadata such as a column’s source type

  • Option to delay snapshots for a given time

  • Support for HSTORE columns in Postgres

  • Incubating support for picking up changes to the whitelist/blacklist configuration of the MySQL connector

As a teaser on the connector metrics support, here’s a screenshot of Java Mission Control displaying the SQL Server connector metrics:

Monitoring the Debezium SQL Server connector

The list above is far from being exhaustive; please take a look at the preview release announcements (Alpha1, Alpha2, Beta1, Beta2 and CR 1) as well as the full list of a whopping 176 fixed issues in JIRA.

It’s hard to say which of the changes and new features I’m most excited about, but one thing surely sticking out is the tremendous amount of community work on this release. Not less than 34 different members of Debezium’s outstanding community have contributed to this release. A huge and massive "Thank You!" to all of you:

When upgrading from earlier Debezium releases, please make sure to read the information regarding update procedures and breaking changes in the release notes. One relevant change to the users of the Debezium connector for MySQL is that our new Antlr-based DDL parser is used by default now. After lots of honing we felt it’s time for using the new parser by default now. While the existing parser can still be used as a fallback as of Debezium 0.9, it will be phased out in 0.10.

Next Steps

After some drinks to celebrate this release, the plan is to do a 0.9.1 release rather quickly (probably in two weeks from now), providing improvements and potential bug fixes to the features and changes done in 0.9. We’ll also begin the work on Debezium 0.10, stay tuned for the details on that!

For further plans beyond that, take a look at our road map. Any suggestions and ideas are very welcomed on mailing list or in the comments below.

If you’re just about to begin using Debezium for streaming changes out of your database, you might be interested in join us for the upcoming webinar on February 7th. After a quick overview, you’ll see Debezium in action, as it streams changes to a browser-based dashboard and more. You can also find lots of resources around Debezium and change data capture such as blog posts and presentations in our curated list of online resources.

About Debezium

Debezium is an open source distributed platform that turns your existing databases into event streams, so applications can see and respond almost instantly to each committed row-level change in the databases. Debezium is built on top of Kafka and provides Kafka Connect compatible connectors that monitor specific database management systems. Debezium records the history of data changes in Kafka logs, so your application can be stopped and restarted at any time and can easily consume all of the events it missed while it was not running, ensuring that all events are processed correctly and completely. Debezium is open source under the Apache License, Version 2.0.

Get involved

We hope you find Debezium interesting and useful, and want to give it a try. Follow us on Twitter @debezium, chat with us on Gitter, or join our mailing list to talk with the community. All of the code is open source on GitHub, so build the code locally and help us improve ours existing connectors and add even more connectors. If you find problems or have ideas how we can improve Debezium, please let us know or log an issue.


back to top