Subscribe


Debezium 0.7.2 Is Released

It’s my pleasure to announce the release of Debezium 0.7.2!

Amongst the new features there’s support for geo-spatial types, a new snapshotting mode for recovering a lost DB history topic for the MySQL connector, and a message transformation for converting MongoDB change events into a structure which can be consumed by many more sink connectors. And of course we fixed a whole lot of bugs, too.

Debezium 0.7.2 is a drop-in replacement for previous 0.7.x versions. When upgrading from versions earlier than 0.7.0, please check out the release notes of all 0.7.x releases to learn about any steps potentially required for upgrading.

A big thank you goes out to our fantastic community members for their hard work on this release: Andrey Pustovetov, Denis Mikhaylov, Peter Goransson, Robert Coup, Sairam Polavarapu and Tom Bentley.

Now let’s take a closer look at some of new features.

MySQL Connector

The biggest change of the MySQL connector is support for geo-spatial column types such as GEOMETRY, POLYGON, MULTIPOINT etc.

There are two new logical field types — io.debezium.data.geometry.Geometry and io.debezium.data.geometry.Geography — for representing geo-spatial columns in change data messages. These types represent geo-spatial data via WKB ("well-known binary") and SRID (coordinate reference system identifier), allowing downstream consumers to interpret the change events using any existing library with support for parsing WKB. A blog post with more details on this will follow soon.

The new snapshotting mode schema_only_recovery comes in handy when for some reason you lost (parts of) the DB history topic used by the MySQL connector. It’s also useful if you’d like to compact that topic by re-creating it. Please refer to the connector documentation for the details of this mode, esp. when it’s safe (and when not) to make use of it.

Another new feature related to managing the size of the DB history topic is the option to control whether to include all DDL events or only those pertaining to tables captured as per the whitelist/blacklist configuration. Again, check out the connector docs to learn more about the specifics of that setting.

Finally, we fixed a few shortcomings of the MySQL DDL parser (DBZ-524, DBZ-530).

PostgreSQL Connector

Similar to the MySQL connector, there’s largely improved support for geo-spatial columns in Postgres now. More specifically, PostGIS column types can be represented in change data events now. Thanks a lot for Robert Coup who contributed this feature!

Also the support for Postgres array columns has been expanded, e.g. we now support to track changes to VARCHAR and DATE array columns. Note that the connector doesn’t yet work with geo-spatial array columns (should you ever have those), but this should be added soon, too.

If you’d like to include just a subset of the rows of a captured table in snapshots, you may like the ability to specify dedicated SELECT statements to do so. For instance this can be used to exclude any logically deleted records — which you can recognize based on some flag in that table — from the snapshot.

A few bugs in this connector where reported and fixed by community members, too, e.g. the connector can be correctly paused now (thanks, Andrey Pustovetov), and we fixed an issue which could potentially have committed an incorrect offset to Kafka Connect (thanks, Thon Mekathikom).

MongoDB Connector

If you’ve ever compared the structures of change events emitted by the Debezium RDBMS connectors (MySQL, Postgres) and the MongoDB connector, you’ll know that the message structure of the latter is a bit different than the others. Due to the schemaless nature of MongoDB, the change events essentially contain a String with a JSON representation of the applied insert or patch. This structure cannot be consumed by existing sink connectors, such as the Confluent connectors for JDBC or Elasticsearch.

This gets possible now by means of a newly added single message transformation (SMT), which parses these JSON strings and creates a structured Kafka Connect record from it (thanks, Sairam Polavarapu!). When applying this SMT to the JDBC sink connector, you can now stream data changes from MongoDB to any supported relational database.

Note that this SMT is work-in-progress, details of its emitted message structure may still change. Also there are some inherent limitations to what can be achieved with it, if you e.g. have arrays in your MongoDB documents, the record created by this SMT will be structured accordingly, but many sink connectors cannot process such structure.

We have some ideas for further development here, e.g. there could be an option for flattening out (non-array) nested structures, so that e.g. { "address" { "street" : "..." } } would be represented as address_street, which then could be consumed by sink connectors expecting a flat structure.

The new SMT is described in detail in our docs.

What’s next?

Please see the full change log for more details and the complete list of issues fixed in Debezium 0.7.2.

The 0.7.3 release is scheduled for February 14th.

We’ll focus on some more bug fixes, also we’re working on having Debezium regulary emit heartbeat messages to a dedicated topic. This will be practical for diagnostic purposes but also help to regularly trigger commits of the offset in Kafka Connect. That’s beneficial in certain situations when capturing tables which only very infrequently change.

We’ve also worked out a roadmap describing our ideas for future work on Debezium, going beyond the next bugfix releases. While nothing is cast in stone, this is our idea of the features to add in the coming months. If you miss anything important on this roadmap, please tell us either in the comments below or send a message to our Google group. Looking forward to your feedback!

About Debezium

Debezium is an open source distributed platform that turns your existing databases into event streams, so applications can see and respond almost instantly to each committed row-level change in the databases. Debezium is built on top of Kafka and provides Kafka Connect compatible connectors that monitor specific database management systems. Debezium records the history of data changes in Kafka logs, so your application can be stopped and restarted at any time and can easily consume all of the events it missed while it was not running, ensuring that all events are processed correctly and completely. Debezium is open source under the Apache License, Version 2.0.

Get involved

We hope you find Debezium interesting and useful, and want to give it a try. Follow us on Twitter @debezium, chat with us on Gitter, or join our mailing list to talk with the community. All of the code is open source on GitHub, so build the code locally and help us improve ours existing connectors and add even more connectors. If you find problems or have ideas how we can improve Debezium, please let us know or log an issue.


Debezium 0.7.1 Is Released

Just last few days before Christmas we are releasing Debezium 0.7.1! This is a bugfix release that fixes few annoying issues that were found during first rounds of use of Debezium 0.7 by our community. All issues relate to either newly provided wal2json support or reduced risk of internal race condition improvement.

Robert Coup has found a performance regression in situations when 0.7.0 was used with old version of Protobuf decoder.

Suraj Savita (and others) has found an issue when our code failed to correctly detect it runs with Amazon RDS wal2json plug-in. We are outsmarted by the JDBC driver internals and included a distinct plugin decoder name wal2json_rds that bypasses detection routine and by default expects it runs against Amazon RDS instance. This mode should be used only with RDS instances.

We have also gathered feedback from first tries to run with Amazon RDS and included a short section in our documentation on this topic.

About Debezium

Debezium is an open source distributed platform that turns your existing databases into event streams, so applications can see and respond almost instantly to each committed row-level change in the databases. Debezium is built on top of Kafka and provides Kafka Connect compatible connectors that monitor specific database management systems. Debezium records the history of data changes in Kafka logs, so your application can be stopped and restarted at any time and can easily consume all of the events it missed while it was not running, ensuring that all events are processed correctly and completely. Debezium is open source under the Apache License, Version 2.0.

Get involved

We hope you find Debezium interesting and useful, and want to give it a try. Follow us on Twitter @debezium, chat with us on Gitter, or join our mailing list to talk with the community. All of the code is open source on GitHub, so build the code locally and help us improve ours existing connectors and add even more connectors. If you find problems or have ideas how we can improve Debezium, please let us know or log an issue.


Debezium 0.7.0 Is Released

It’s not Christmas yet, but we already got a present for you: Debezium 0.7.0 is here, full of new features as well as many bug fixes! A big thank you goes out to all the community members who contributed to this release. It is very encouraging for us to see not only more and more issues and feature requests being reported, but also pull requests coming in.

Note that this release comes with a small number of changes to the default mappings for some data types. We try to avoid this sort of changes as far as possible, but in some cases it is required, e.g. if the previous mapping could have caused potential value losses. Please see below for the details and also make sure to check out the full change log which describes these changes in detail.

Now let’s take a closer look at some of new features.

Based on Apache Kafka 1.0

A few weeks ago the Apache Kafka team has released version 1.0.0. This was an important milestone for the Kafka community, and we now can happily declare that Debezium is built against and runs on that Apache Kafka version. Our Docker images were also promoted to contain Apache Kafka and Kafka Connect 1.0.0.

PostgreSQL Connector

The big news for the PostgreSQL connector is that it now supports the wal2json logical decoding plugin as an alternative to the existing DecoderBufs plug-in. This means that you now can use Debezium to stream changes out of PostgreSQL on Amazon RDS, as wal2json is the logical decoding plugin used in this environment. Many thanks to Robert Coup who significantly contributed to this feature.

Working on this plug-in, we noticed that there was a potential race condition when it comes to applying changes to the schema of captured tables. In that case it could have happened that a number of messages pertaining to data changes done before the schema change were emitted using the new schema. With the exception of a few corner cases (which are described here), this has been addressed when using Debezium’s own DecoderBufs plug-in. So it’s highly recommended to upgrade the DecoderBufs plug-in to the new version before upgrading the Debezium connector. We’ve also worked closely with the author of the wal2json plug-in (big thanks for the quick help!) to prevent the issue when using the wal2json plug-in.

While the Debezium Docker images for Postgres already come with the latest version of DecoderBufs and wal2json, RDS for now is still using an older version of wal2json. Until this has been updated, special attention must be paid when applying schema changes to captured tables. Please see the changelog for a in-depth description of this issue and ways to mitigate it.

There are new daily running CI jobs that verify that the wal2json plugin passes our test suite. For the foreseeable future we’ll support both, wal2json as well as the existing DecoderBufs plug-in. The latter should be more efficient due to the usage of the Protocol Buffers binary format, whereas the former comes in handy for RDS or other cloud environments where you don’t have control over the installed logical decoding plug-ins, but wal2json is available.

In other news on the Postgres connector, Andrey Pustovetov discovered and proposed a fix for a multi-threading bug that could have put the connector into an undefined state if a rebalance in the Connect cluster was triggered during snapshotting. Thanks, Andrey!

MySQL Connector

In the MySQL connector we’ve fixed two issues which affect the default mapping of certain column types.

Following up to the new BIGINT UNSIGNED mapping introduced in Debezium 0.6.1, this type is now encoded as int64 in Debezium messages by default as it is easier for (polyglot) clients to work with. This is a reasonable mapping for the vast majority of cases. Only when using values > 2^63, you should switch it back to the Decimal logical type which is a bit more cumbersome to handle, though. This should be a rare situation, as MySQL advices against using unsigned values > 2^63 due to potential value losses when performing DB-side calculations. Please see the connector documentation for the details.

Rene Kerner has improved the support for the MySQL TIME type. MySQL allows to store values larger than 23:59:59 in such columns, and the type int32 which was previously used for TIME(0-3) columns isn’t enough to convey the entire possible value range. Therefore all TIME columns in MySQL are by default represented as int64 now, using the io.debezium.time.MicroTime logical type, i.e. the value represents micro-seconds. If needed, you can switch to the previous mapping by setting time.precision.mode to adaptive, but you should only do so if you’re sure that you only ever will have values that fit into int32. This option is only kept for a transitioning period and will be removed in a future release.

Recently we got a report that MySQL’s binlog can contain ROLLBACK statements and thus transactions that are actually not committed. Of course no data change messages should be emitted in this situation. This e.g. can be the case when temporary tables are dropped. So we introduced a look-ahead buffer functionality that reads the binlog by transaction and excludes those that were rolled back. This feature should be considered incubating and is disabled by default for the time being. We’d like to gather your feedback on this, so if you’d benefit from this feature, please give it a try and let us know if you run into any issues. For further details please refer to the binlog.buffer.size setting in the MySQL connector docs.

Andras Istvan Nagy came with the idea and implemented a way for explicitly selecting the rows from each table that will be part of the snapshotting process. This can for instance be very useful if you work with soft deletes and would like to exclude all logically deleted records from snapshotting.

Please see the full change log for more details and the complete list of fixed issues.

What’s next?

The Debezium 0.7.1 release is planned to be out roughly two weeks after Christmas.

It will contain a new SMT that will unwind MongoDB change events into a regular JSON consumable by sink connectors.

A big overhaul of GEOMETRY types is in progress. When completed, all GEOMETRY types will be supported by both MySQL and PostgreSQL connectors and they will be available in standard WKB format for easy consumption by polyglot clients.

There is ongoing work for the MySQL connector to allow dynamic update of table.whitelist option. This will allow the user to re-configure the set of tables captured without need to re-create connector.

If you’d like to contribute, please let us know. We’re happy about any help and will work with you to get you started quickly. Check out the details below on how to get in touch.

About Debezium

Debezium is an open source distributed platform that turns your existing databases into event streams, so applications can see and respond almost instantly to each committed row-level change in the databases. Debezium is built on top of Kafka and provides Kafka Connect compatible connectors that monitor specific database management systems. Debezium records the history of data changes in Kafka logs, so your application can be stopped and restarted at any time and can easily consume all of the events it missed while it was not running, ensuring that all events are processed correctly and completely. Debezium is open source under the Apache License, Version 2.0.

Get involved

We hope you find Debezium interesting and useful, and want to give it a try. Follow us on Twitter @debezium, chat with us on Gitter, or join our mailing list to talk with the community. All of the code is open source on GitHub, so build the code locally and help us improve ours existing connectors and add even more connectors. If you find problems or have ideas how we can improve Debezium, please let us know or log an issue.


Debezium 0.6.2 Is Released

We are accelerating! Three weeks after the 0.6.1 release, the Debezium team is bringing Debezium 0.6.2 to you!

This release revolves mostly around bug fixes, but there are a few new features, too. Let’s take a closer look at some of the changes.

PostgreSQL Connector

The big news for the Postgres connector is that Debezium now runs against PostgreSQL 10 thanks to a contribution from Scofield Xu. As a part of this change we are providing a Docker Image with PostgreSQL 10, too, and we have set up a daily run of our integration tests against it.

If you are building Postgres yourself using the Debezium logical decoding plug-in, you can save quite some megabytes if you don’t need the PostGIS geometric extension: thanks to the work by Danila Kiver, it’s now possible to omit that extension.

MySQL Connector

We’ve received multiple reports related to parsing MySQL DDL statements, e.g. there were a few specific invocations of the ALTER TABLE statement which weren’t handled correctly. Those as well as a few other parser bugs have been fixed.

If you work with the TIMESTAMP column type and your Kafka Connect server isn’t using UTC as timezone, then the fix for DBZ-260 is applying to you. In that case, the ISO 8601 formatted String emitted by Debezium would have, incorrectly, contained the UTC date and time plus the zone offset (as per the time zone the Kafka Connect server is located in) before. Whereas now it will contain the date and time adjusted to the zone offset. This may require adjustments to to downstream consumers if they were relying on the previous, incorrect behavior.

DBZ-217 gives you more flexibility for handling corrupt events encountered in the MySQL binlog. By default, the connector will stop at the problematic event in such case. But you now also have the option to just log the event and its position and continue the processing after it.

Another nice improvement for the MySQL connector is a much reduced CPU load after the snapshot has been completed, when using the "snapshot only" mode (DBZ-396).

MongoDB Connector

This connector received an important fix applying when more than one thread is used to performing the initial snapshot (DBZ-438). Before, it could happen that single messages got lost during snapshotting which is fixed now.

Examples and Docker Images

We have expanded our examples repository with an Avro example, which may be interesting to you if you’d like to not work with JSON messages but rather the compact Avro binary format and the Confluent schema registry.

As a part of our release process we are now creating micro tags for our Docker images for every released version. While tags in the format x.y.z are fixed in time, tags in the format x.y are rolling updates and always point to the latest micro release of that image.

Please see the full change log for more details and the complete list of fixed issues.

What’s next?

The Debezium 0.7 release is planned to be out in two to three weeks from now.

It will contain the move to Apache Kafka 1.0.0 and bring support for the wal2json logical decoding plug-in for Postgres. This will eventually allow to use the Debezium Postgres connector on Amazon RDS (once the correct wal2json version is available there).

In parallel, the work around handling updates to the whitelist configuration of the MySQL connector continues (it may be ready for 0.7.0), and so does the work on the Oracle connector (which will be shipping in a future release).

If you’d like to contribute, please let us know. We’re happy about any help and will work with you to get you started quickly. Check out the details below on how to get in touch.

About Debezium

Debezium is an open source distributed platform that turns your existing databases into event streams, so applications can see and respond almost instantly to each committed row-level change in the databases. Debezium is built on top of Kafka and provides Kafka Connect compatible connectors that monitor specific database management systems. Debezium records the history of data changes in Kafka logs, so your application can be stopped and restarted at any time and can easily consume all of the events it missed while it was not running, ensuring that all events are processed correctly and completely. Debezium is open source under the Apache License, Version 2.0.

Get involved

We hope you find Debezium interesting and useful, and want to give it a try. Follow us on Twitter @debezium, chat with us on Gitter, or join our mailing list to talk with the community. All of the code is open source on GitHub, so build the code locally and help us improve ours existing connectors and add even more connectors. If you find problems or have ideas how we can improve Debezium, please let us know or log an issue.


Debezium 0.6.1 Is Released

Just shy of a month after the 0.6.0 release, I’m happy to announce the release of Debezium 0.6.1!

This release contains several bugfixes, dependency upgrades and a new option for controlling how BIGINT UNSIGNED columns are conveyed. We also expanded the set of Docker images and Docker Compose files accompanying our tutorial, so you can run it now with all the databases we support.

Let’s take a closer look at some of the changes.

New connector option for controlling BIGINT UNSIGNED representation

BIGINT UNSIGNED columns from MySQL databases have been represented using Kafka Connect’s Decimal type until now. This type allows to represent all possible values of such columns, but its based on a byte array, so it can be a bit cumbersome to handle for consumers. Therefore we added a new option named bigint.unsigned.handling.mode to the MySQL connector that allows to represent such columns using long.

For the very most cases that’s the preferable option, only if your column contains values larger than 2^63 (which MySQL doesn’t recommend due to potential value losses when performing calculations), you should stick to the Decimal representation.

Using long will be the default as of Debezium 0.7, for the 0.6.x timeline we decided to go with the previous behavior (i.e. using Decimal) for the sake of backwards compatibility.

Thanks a lot to Ben Williams who contributed this feature!

New example Docker images and Docker Compose files

In the Debezium examples repository we now provide Docker Compose files which let you run the tutorial with all the three databases we currently support, MySQL, Postgres and MongoDB.

Just choose the Compose file for your preferred database and get a all the required components (ZooKeeper, Apache Kafka, Kafka Connect and the database) running within a few seconds.

We’ve also deployed Docker images for Postgres and MongoDB to the Debezium organization on Docker Hub, so you got some data to play with.

Version upgrades

We’ve upgraded our images from Kafka 0.11.0.0 to 0.11.0.1. Also the binlog client library used by the MySQL connector was upgraded from 0.9.0 to 0.13.0.

Bugfixes

Finally, several bugs were fixed in 0.6.1. E.g. you can now name a column column in MySQL (DBZ-408), generated DROP TEMP TABLE statements won’t flood the DB history topic (DBZ-295) and we’ve fixed a case where the Postgres connector would stop working due to an internal error but fail to report though via the task/connector status (DBZ-380).

Please see the full change log for more details and the complete list of fixed issues.

What’s next?

The work on Debezium 0.7 has already begun and we’ve merged the first set of changes. You can expect to see support for using the wal2json logical decoding plug-in with the Postgres connector, which will finally allow it to use Debezium with Postgres on Amazon RDS! We’ve also started our explorations of providing a connector for Oracle (DBZ-20) and hope to report some progress here soon.

While the work on Debezium 0.7 continues, you will likely continue to see one or more 0.6.x bugfix releases. We’ve automated the release process as much as possible, making it a breeze to ship a new release and getting fixes into your hands quickly.

If you’d like to contribute, please let us know. We’re happy about any help and will work with you to get you started quickly. Check out the details below on how to get in touch.

About Debezium

Debezium is an open source distributed platform that turns your existing databases into event streams, so applications can see and respond almost instantly to each committed row-level change in the databases. Debezium is built on top of Kafka and provides Kafka Connect compatible connectors that monitor specific database management systems. Debezium records the history of data changes in Kafka logs, so your application can be stopped and restarted at any time and can easily consume all of the events it missed while it was not running, ensuring that all events are processed correctly and completely. Debezium is open source under the Apache License, Version 2.0.

Get involved

We hope you find Debezium interesting and useful, and want to give it a try. Follow us on Twitter @debezium, chat with us on Gitter, or join our mailing list to talk with the community. All of the code is open source on GitHub, so build the code locally and help us improve ours existing connectors and add even more connectors. If you find problems or have ideas how we can improve Debezium, please let us know or log an issue.


back to top