Debezium Blog

We’re happy to announce that Debezium 0.3.5 is now available for use with Kafka Connect 0.10.0.1. This release contains several fixes for the MySQL connector and adds the ability to use with multi-master MySQL servers as sources. See the release notes for specifics on these changes. We’ve also updated the Debezium Docker images labelled 0.3 and latest, which we use in our tutorial.

One of the fixes is signficant, and so we strongly urge all users to upgrade to this release from all earlier versions. In prior versions, the MySQL connector may stop without completing all updates in a transaction, and when the connector restarts it starts with the next transaction and therefore might fail to capture some of the change events in the earlier transaction. This release fixes this issue so that when restarting it will always pick up where it left off, even if that point is in the middle of a transaction. Note that this fix only takes affect once a connector is upgraded and restarted. See the issue for more details.

Thanks to Akshath, Anton, Chris, and others for their help with the release, issues, discussions, contributions, and questions!

We’re happy to announce that Debezium 0.3.4 is now available for use with Kafka Connect 0.10.0.1. This release contains several new features for the MySQL connector: support for MySQL’s JSON datatype, a new snapshot mode called schema_only, and JMX metrics. Also, the Debezium Docker images for Zookeeper, Kafka, and Kafka Connect have all been updated to allow optionally expose JMX metrics in these services. And, one backward-incompatible fix was made to the change event’s ts_sec field. See the release notes for specifics.

We’ve also updated the Debezium Docker images labelled 0.3 and latest, which we use in our tutorial.

Thanks to Akshath, Chris, Vitalii, Dennis, Prannoy, and others for their help with the release, issues, discussions, contributions, and questions!

MySQL 5.7 introduced a new data type for storing and working with JSON data. Clients can define tables with columns using the new JSON datatype, and they can store and read JSON data using SQL statements and new built-in JSON functions to construct JSON data from other relational columns, introspect the structure of JSON values, and search within and manipulate JSON data. It possible to define generated columns on tables whose values are computed from the JSON value in another column of the same table, and to then define indexes with those generated columns. Overall, this is really a very powerful feature in MySQL.

Debezium’s MySQL connector will support the JSON datatype starting with the upcoming 0.3.4 release. JSON document, array, and scalar values will appear in change events as strings with io.debezium.data.json for the schema name. This will make it natural for consumers to work with JSON data. BTW, this is the same semantic schema type used by the MongoDB connector to represent JSON data.

This sounds straightforward, and we hope it is. But implementing this required a fair amount of work. That’s because although MySQL exposes JSON data as strings to client applications, internally it stores all JSON data in a special binary form that allows the MySQL engine to efficiently access the JSON data in queries, JSON functions and generated columns. All JSON data appears in the binlog in this binary form as well, which meant that we had to parse the binary form ourselves if we wanted to extract the more useful string representation. Writing and testing this parser took a bit of time and effort, and ultimately we donated it to the excellent MySQL binlog client library that the connector uses internally to read the binlog events.

We’re happy to announce that Debezium 0.3.3 is now available for use with Kafka Connect 0.10.0.1. This release contains a handful of bug fixes and minor improvements for the MySQL connector, including better handling of BIT(n) values, ENUM and SET values, and GTID sets, This release also improves the log messages output by the MySQL connectors to better represent the ongoing activity when consuming the changes from the source database. See the release notes for specifics.

We’ve also updated the Debezium Docker images labelled 0.3 and latest, which we use in our tutorial. We’ve also updated the tutorial to use the latest Docker installations on Linux, Windows, and OS X.

Thanks to Akshath, Chris, Randy, Prannoy, Umang, Horia, and others for their help with the release, issues, discussions, contributions, and questions!

We’re happy to announce that Debezium 0.3.2 is now available for use with Kafka Connect 0.10.0.1. This release contains a handful of bug fixes and minor improvements for the MySQL connector and MongoDB connector. The MySQL connector better handles BIT(n) values and zero-value date and timestamp values. This release also improves the log messages output by the MySQL and MongoDB connectors to better represent the ongoing activity when consuming the changes from the source database. See the release notes for specifics.

We’ve also updated the Debezium Docker images labelled 0.3 and latest, which we use in our tutorial. We’ve also updated the tutorial to use the latest Docker installations on Linux, Windows, and OS X.

Thanks to Akshath, Colum, Emmanuel, Konstantin, Randy, RenZhu, Umang, and others for their help with the release, issues, discussions, contributions, and questions!