Subscribe


Debezium 0.6.2 Is Released

We are accelerating! Three weeks after the 0.6.1 release, the Debezium team is bringing Debezium 0.6.2 to you!

This release revolves mostly around bug fixes, but there are a few new features, too. Let’s take a closer look at some of the changes.

PostgreSQL Connector

The big news for the Postgres connector is that Debezium now runs against PostgreSQL 10 thanks to a contribution from Scofield Xu. As a part of this change we are providing a Docker Image with PostgreSQL 10, too, and we have set up a daily run of our integration tests against it.

If you are building Postgres yourself using the Debezium logical decoding plug-in, you can save quite some megabytes if you don’t need the PostGIS geometric extension: thanks to the work by Danila Kiver, it’s now possible to omit that extension.

MySQL Connector

We’ve received multiple reports related to parsing MySQL DDL statements, e.g. there were a few specific invocations of the ALTER TABLE statement which weren’t handled correctly. Those as well as a few other parser bugs have been fixed.

If you work with the TIMESTAMP column type and your Kafka Connect server isn’t using UTC as timezone, then the fix for DBZ-260 is applying to you. In that case, the ISO 8601 formatted String emitted by Debezium would have, incorrectly, contained the UTC date and time plus the zone offset (as per the time zone the Kafka Connect server is located in) before. Whereas now it will contain the date and time adjusted to the zone offset. This may require adjustments to to downstream consumers if they were relying on the previous, incorrect behavior.

DBZ-217 gives you more flexibility for handling corrupt events encountered in the MySQL binlog. By default, the connector will stop at the problematic event in such case. But you now also have the option to just log the event and its position and continue the processing after it.

Another nice improvement for the MySQL connector is a much reduced CPU load after the snapshot has been completed, when using the "snapshot only" mode (DBZ-396).

MongoDB Connector

This connector received an important fix applying when more than one thread is used to performing the initial snapshot (DBZ-438). Before, it could happen that single messages got lost during snapshotting which is fixed now.

Examples and Docker Images

We have expanded our examples repository with an Avro example, which may be interesting to you if you’d like to not work with JSON messages but rather the compact Avro binary format and the Confluent schema registry.

As a part of our release process we are now creating micro tags for our Docker images for every released version. While tags in the format x.y.z are fixed in time, tags in the format x.y are rolling updates and always point to the latest micro release of that image.

Please see the full change log for more details and the complete list of fixed issues.

What’s next?

The Debezium 0.7 release is planned to be out in two to three weeks from now.

It will contain the move to Apache Kafka 1.0.0 and bring support for the wal2json logical decoding plug-in for Postgres. This will eventually allow to use the Debezium Postgres connector on Amazon RDS (once the correct wal2json version is available there).

In parallel, the work around handling updates to the whitelist configuration of the MySQL connector continues (it may be ready for 0.7.0), and so does the work on the Oracle connector (which will be shipping in a future release).

If you’d like to contribute, please let us know. We’re happy about any help and will work with you to get you started quickly. Check out the details below on how to get in touch.

About Debezium

Debezium is an open source distributed platform that turns your existing databases into event streams, so applications can see and respond almost instantly to each committed row-level change in the databases. Debezium is built on top of Kafka and provides Kafka Connect compatible connectors that monitor specific database management systems. Debezium records the history of data changes in Kafka logs, so your application can be stopped and restarted at any time and can easily consume all of the events it missed while it was not running, ensuring that all events are processed correctly and completely. Debezium is open source under the Apache License, Version 2.0.

Get involved

We hope you find Debezium interesting and useful, and want to give it a try. Follow us on Twitter @debezium, chat with us on Gitter, or join our mailing list to talk with the community. All of the code is open source on GitHub, so build the code locally and help us improve ours existing connectors and add even more connectors. If you find problems or have ideas how we can improve Debezium, please let us know or log an issue.


Debezium 0.6.1 Is Released

Just shy of a month after the 0.6.0 release, I’m happy to announce the release of Debezium 0.6.1!

This release contains several bugfixes, dependency upgrades and a new option for controlling how BIGINT UNSIGNED columns are conveyed. We also expanded the set of Docker images and Docker Compose files accompanying our tutorial, so you can run it now with all the databases we support.

Let’s take a closer look at some of the changes.

New connector option for controlling BIGINT UNSIGNED representation

BIGINT UNSIGNED columns from MySQL databases have been represented using Kafka Connect’s Decimal type until now. This type allows to represent all possible values of such columns, but its based on a byte array, so it can be a bit cumbersome to handle for consumers. Therefore we added a new option named bigint.unsigned.handling.mode to the MySQL connector that allows to represent such columns using long.

For the very most cases that’s the preferable option, only if your column contains values larger than 2^63 (which MySQL doesn’t recommend due to potential value losses when performing calculations), you should stick to the Decimal representation.

Using long will be the default as of Debezium 0.7, for the 0.6.x timeline we decided to go with the previous behavior (i.e. using Decimal) for the sake of backwards compatibility.

Thanks a lot to Ben Williams who contributed this feature!

New example Docker images and Docker Compose files

In the Debezium examples repository we now provide Docker Compose files which let you run the tutorial with all the three databases we currently support, MySQL, Postgres and MongoDB.

Just choose the Compose file for your preferred database and get a all the required components (ZooKeeper, Apache Kafka, Kafka Connect and the database) running within a few seconds.

We’ve also deployed Docker images for Postgres and MongoDB to the Debezium organization on Docker Hub, so you got some data to play with.

Version upgrades

We’ve upgraded our images from Kafka 0.11.0.0 to 0.11.0.1. Also the binlog client library used by the MySQL connector was upgraded from 0.9.0 to 0.13.0.

Bugfixes

Finally, several bugs were fixed in 0.6.1. E.g. you can now name a column column in MySQL (DBZ-408), generated DROP TEMP TABLE statements won’t flood the DB history topic (DBZ-295) and we’ve fixed a case where the Postgres connector would stop working due to an internal error but fail to report though via the task/connector status (DBZ-380).

Please see the full change log for more details and the complete list of fixed issues.

What’s next?

The work on Debezium 0.7 has already begun and we’ve merged the first set of changes. You can expect to see support for using the wal2json logical decoding plug-in with the Postgres connector, which will finally allow it to use Debezium with Postgres on Amazon RDS! We’ve also started our explorations of providing a connector for Oracle (DBZ-20) and hope to report some progress here soon.

While the work on Debezium 0.7 continues, you will likely continue to see one or more 0.6.x bugfix releases. We’ve automated the release process as much as possible, making it a breeze to ship a new release and getting fixes into your hands quickly.

If you’d like to contribute, please let us know. We’re happy about any help and will work with you to get you started quickly. Check out the details below on how to get in touch.

About Debezium

Debezium is an open source distributed platform that turns your existing databases into event streams, so applications can see and respond almost instantly to each committed row-level change in the databases. Debezium is built on top of Kafka and provides Kafka Connect compatible connectors that monitor specific database management systems. Debezium records the history of data changes in Kafka logs, so your application can be stopped and restarted at any time and can easily consume all of the events it missed while it was not running, ensuring that all events are processed correctly and completely. Debezium is open source under the Apache License, Version 2.0.

Get involved

We hope you find Debezium interesting and useful, and want to give it a try. Follow us on Twitter @debezium, chat with us on Gitter, or join our mailing list to talk with the community. All of the code is open source on GitHub, so build the code locally and help us improve ours existing connectors and add even more connectors. If you find problems or have ideas how we can improve Debezium, please let us know or log an issue.


Debezium 0.6 Is Out

What’s better than getting Java 9? Getting Java 9 and a new version of Debezium at the same time! So it’s with great happiness that I’m announcing the release of Debezium 0.6 today.

What’s in it?

Debezium is now built against and tested with Apache Kafka 0.11.0. Also the Debezium Docker images have been updated do that version (DBZ-305). You should make sure to read the Kafka update guide when upgrading from an earlier version.

To improve integration with existing Kafka sink connectors such as the JDBC sink connector or the Elasticsearch connector, Debezium provides a new single message transformation (DBZ-226). This SMT converts Debezium’s CDC event structure into a more conventional structure commonly used in other sink and non-CDC source connectors where the message represents the state of the inserted or updated row, or null in the case of a deleted row. This lets your for instance capture the changes from a table in MySQL and update a corresponding table in a Postgres database accordingly. We’ll provide a complete example showing the usage of that new SMT in the next few days.

If you are doing the Debezium tutorial, you will like the new Docker Compose set-up provided in the examples repo (DBZ-127). This lets you start all the required Docker containers with a single command.

New connector features

Now let’s take a look at some of the changes around the specific Debezium connectors. The MySQL connector has seen multiple improvements, e.g.:

  • Snapshot consistency wasn’t guaranteed before in some corner cases (DBZ-210); that’s fixed now

  • DEC and FIXED types supported in the DDL parser (DBZ-359; thanks to Liu Hanlin!)

  • UNION clause supported for ALTER TABLE (DBZ-346)

For the MongoDB connector, the way of serializing ids into the key payload of CDC events has changed (DBZ-306). The new format allows to read back ids into the correct type. We also took the opportunity and made the id field name consistent with the other connectors, i.e. it’s "id" now. Note: that change may break existing consumers, so some work on your end may be required, depending on the implementation of your consumer. The details are discussed in the release notes and the format of message keys is described in depth in the connector documentation. Kudos to Hans-Peter Grahsl who contributed on this feature!

Another nice improvement for this connector is support for SSL connections (DBZ-343).

Finally, the Postgres connector learned some new tricks, too:

  • Support for variable-width numeric columns (DBZ-318)

  • Views won’t stop the connector any more (DBZ-319)

  • Warnings and notifications emitted by the server are correctly forwarded to the log (DBZ-279)

Please refer to the changelog for an overview of all the 20 issues fixed in Debezium 0.6.0.

What’s next?

High on our agenda is exploring support for Oracle (DBZ-20). We are also looking into using another logical decoding plug-in (wal2json) for the Postgres connector, which would enable to use Debezium with Postgres instances running on Amazon RDS. Another feature being worked on by community member Moira Tagle is support for updates to the table.whitelist for existing connector instances. Finally, we’ve planned to test and adapt the existing MySQL connector for providing CDC functionality to MariaDB.

Debezium 0.7 with one or more out of those features as well as hopefully some others will be released later this year. We’ll likely also do further 0.6.x releases with bug fixes as required.

You’d like to contribute? That’s great - let us know and we’ll get you started. Check out the details below on how to get in touch.

About Debezium

Debezium is an open source distributed platform that turns your existing databases into event streams, so applications can see and respond almost instantly to each committed row-level change in the databases. Debezium is built on top of Kafka and provides Kafka Connect compatible connectors that monitor specific database management systems. Debezium records the history of data changes in Kafka logs, so your application can be stopped and restarted at any time and can easily consume all of the events it missed while it was not running, ensuring that all events are processed correctly and completely. Debezium is open source under the Apache License, Version 2.0.

Get involved

We hope you find Debezium interesting and useful, and want to give it a try. Follow us on Twitter @debezium, chat with us on Gitter, or join our mailing list to talk with the community. All of the code is open source on GitHub, so build the code locally and help us improve ours existing connectors and add even more connectors. If you find problems or have ideas how we can improve Debezium, please let us know or log an issue.


Debezium 0.5.2 Is Out

I’m very happy to announce the release of Debezium 0.5.2!

As the previous release, the 0.5.2 release fixes several bugs in the MySQL, Postgres and MongoDB connectors. But there are also several new features and options:

  • The decimal.handling.mode option already known from the MySQL connector is now also supported for PostgreSQL (DBZ-337). It lets you control how NUMERIC and DECIMAL columns are represented in change events (either using Kafka’s Decimal type or as double).

  • The MongoDB connector supports the options database.whitelist and database.blacklist now (DBZ-302)

  • The PostgreSQL connector can deal with array-typed columns as well as with quoted identifiers for tables, schemas etc. (DBZ-297, DBZ-298)

  • The Debezium Docker images run on Red Hat’s OpenShift cloud environment (DBZ-267)

Speaking about the Docker images, we’ve set up nightly tags for the Debezium images on Docker Hub, allowing you to grab the latest improvements even before an official release has been cut. The connector archives are also deployed to the Sonatype OSS Maven repository.

Finally, we’ve spent some time to extend the documentation on some things not covered before:

  • Avro Serialization describes how to use the use the Avro converter and the Confluent Schema Registry instead of the JSON converter instead of the default JSON converter for serializing change events, resulting in much smaller message sizes; The Avro converter itself has also been added to the Debezium Docker image for Kafka Connect, so you can use it right away

  • Topic Routing describes how to use Debezium’s ByLogicalTableRouter single message transformation (SMT) for routing the change events from multiple tables into a single topic, which for instance is very useful when working with sharded tables

Please refer to the changelog for an overview of all the 19 issues fixed in Debezium 0.5.2.

The following people from the community have sent in pull requests for this release: Emrul Islam, Eric S. Kreiser, Mario Mueller, Matteo Capitanio, Omar Al-Safi and Satyajit Vegesna.

Thanks a lot to you and everyone else in the community for contributing to Debezium via feature requests, bug reports, discussions and questions!

What’s next

The next version of Debezium will be 0.6 (planned for September). This release is planned to bring the upgrade to Kafka 0.11. We’ll also look into an SMT for transforming the change events emitted by Debezium into a flat representation, which for instance will be very useful in conjunction with the JDBC sink connector.

While 0.6 is planned to be more of a "stabilization release", 0.7 should bring a long-awaited major feature: we’ve planned to explore support for Oracle and hopefully will do an initial release of a Debezium connector for that database.

In other words, exciting times are ahead! If you’d like to get involved, let us know. Check out the details below on how to get in touch.

About Debezium

Debezium is an open source distributed platform that turns your existing databases into event streams, so applications can see and respond almost instantly to each committed row-level change in the databases. Debezium is built on top of Kafka and provides Kafka Connect compatible connectors that monitor specific database management systems. Debezium records the history of data changes in Kafka logs, so your application can be stopped and restarted at any time and can easily consume all of the events it missed while it was not running, ensuring that all events are processed correctly and completely. Debezium is open source under the Apache License, Version 2.0.

Get involved

We hope you find Debezium interesting and useful, and want to give it a try. Follow us on Twitter @debezium, chat with us on Gitter, or join our mailing list to talk with the community. All of the code is open source on GitHub, so build the code locally and help us improve ours existing connectors and add even more connectors. If you find problems or have ideas how we can improve Debezium, please let us know or log an issue.


Debezium 0.5.1 Released

It’s my pleasure to announce the release of Debezium 0.5.1!

This release fixes several bugs in the MySQL, Postgres and MongoDB connectors. There’s also support for some new datatypes: POINT on MySQL (DBZ-222) and TSTZRANGE on Postgres (DBZ-280). This release is a drop-in replacement for 0.5.0, upgrading is recommended to all users.

Note that in the — rather unlikely — case that you happened to enable Debezium for all the system tables of MySQL, any configured table filters will be applied to these system tables now, too (DBZ-242). This may require an adjustment of your filters if you indeed wanted to capture all system tables but only selected non-system tables.

Please refer to the changelog for an overview of all the 29 issues fixed in Debezium 0.5.1.

The Docker image containing Kafka Connect and all the Debezium 0.5.x connectors as well as the image containing Postgres and the Debezium logical decoding plug-in have been updated to 0.5.1, too.

As Debezium continues to evolve and grow, the number of people contributing to the project is also going up. The following people have sent in pull requests for this release: Aaron Rosenberg, Alexander Kukushkin, Brendan Maguire, Duncan Sands, David Leibovic, Jiri Pechanec, nacivida, Omar Al-Safi, Randall Hauch and Tom Bentley.

Thanks a lot to you and everyone else in the community contributing via feature requests, bug reports, discussions and questions!

What’s next

We’ve planned to do further bug fix releases for the 0.5.x line. Specifically, we’ll release a fix for DBZ-217 shortly, which is about the MySQL connector stumbling when getting across a corrupt event in the binlog.

In parallel we’re looking into Debezium connectors for SQL Server and Oracle. While we cannot promise anything yet in terms of when these will be ready to be published, we hope to have at least one of them ready some time soon. Stay tuned and get involved!

About Debezium

Debezium is an open source distributed platform that turns your existing databases into event streams, so applications can see and respond almost instantly to each committed row-level change in the databases. Debezium is built on top of Kafka and provides Kafka Connect compatible connectors that monitor specific database management systems. Debezium records the history of data changes in Kafka logs, so your application can be stopped and restarted at any time and can easily consume all of the events it missed while it was not running, ensuring that all events are processed correctly and completely. Debezium is open source under the Apache License, Version 2.0.

Get involved

We hope you find Debezium interesting and useful, and want to give it a try. Follow us on Twitter @debezium, chat with us on Gitter, or join our mailing list to talk with the community. All of the code is open source on GitHub, so build the code locally and help us improve ours existing connectors and add even more connectors. If you find problems or have ideas how we can improve Debezium, please let us know or log an issue.


back to top