Subscribe


PostgreSQL support added to Debezium

With the recent Debezium release, we’re happy to announce that a new PostgreSQL connector has been added alongside the already existing MySQL and MongoDB connectors.

Tip

Make sure you read the connector documentation for an in-depth look at the different configuration options.

Getting started

The fastest way to check out the new connector is using Debezium’s Postgres docker image which is based on a vanilla Postgres docker image on top of which it compiles and installs a PostgreSQL logical decoding plugin and sets up the necessary permissions for streaming changes locally (on localhost)

Once you fire up the Docker machine with the database server, starting up and configuring the connector to stream changes from that machine is exactly the same as described in detail by the Debezium tutorial. The only obvious difference is that instead of the MySQL machine and MySQL connector configuration you need to use the PostgreSQL machine and the PostgreSQL connector configuration parameters.

Using the connector in your own environment

Unlike the Mongo and MySQL connectors, getting the PostgreSQL connector up and running is a bit more complicated due to the fact that it requires a server-side logical decoding plugin running in the PostgreSQL server.

In general, there are three major steps involved in getting the connector running in your environment:

  1. Compiling and installing the logical decoding plugin into your own server

  2. Setting up the PostgreSQL server with appropriate replication permissions

  3. Starting the Kafka Connect, Broker and Zookeeper machines

For steps 1 and 2 you can check out our PostgreSQL Docker image together with the sources for the logical decoding plugin

For step 3 you can either use Debezium’s Kafka Docker images or perform a similar setup locally. The Debezium tutorial and the the connector documentation are great resources for helping out with this task.


Debezium 0.4.0 Released

We’re happy to announce that Debezium 0.4.0 is now available for use with Kafka Connect 0.10.1.1. This release introduces a new PostgreSQL connector, and contains over a dozen fixes combined for the MongoDB connector and MySQL connector, including preliminar support for Amazon RDS and Amazon Aurora (MySQL compatibility). See the release notes for specifics on these changes.

We’ve also created Debezium Docker images labelled 0.4 and latest, which we use in our tutorial.

Thanks to Horia, Chris, Akshath, Ramesh, Matthias, Anton, Sagi, barton, and others for their help with this release, issues, discussions, contributions, and questions!

What’s next

We’ll continue to improve the MongoDB, MySQL, and PostgreSQL connectors and pushing out 0.4.x releases. We’re also going to work on a few new connectors, though we’ll likely increase the minor version with each new connector. Stay tuned and get involved!

About Debezium

Debezium is an open source distributed platform that turns your existing databases into event streams, so applications can see and respond almost instantly to each committed row-level change in the databases. Debezium is built on top of Kafka and provides Kafka Connect compatible connectors that monitor specific database management systems. Debezium records the history of data changes in Kafka logs, so your application can be stopped and restarted at any time and can easily consume all of the events it missed while it was not running, ensuring that all events are processed correctly and completely. Debezium is open source under the Apache License, Version 2.0.

Get involved

We hope you find Debezium interesting and useful, and want to give it a try. Follow us on Twitter @debezium, chat with us on Gitter, or join our mailing list to talk with the community. All of the code is open source on GitHub, so build the code locally and help us improve ours existing connectors and add even more connectors. If you find problems or have ideas how we can improve Debezium, please let us know or log an issue.


Debezium 0.3.6 Released

We’re happy to announce that Debezium 0.3.6 is now available for use with Kafka Connect 0.10.0.1. This release contains over a dozen fixes combined for the MySQL connector and MongoDB connectors. See the release notes for specifics on these changes.

We’ve also updated the Debezium Docker images labelled 0.3 and latest, which we use in our tutorial.

Thanks to Farid, RenZhu, Dongjun, Anton, Chris, Dennis, Sharaf, Rodrigo, Tim, and others for their help with this release, issues, discussions, contributions, and questions!

What’s next

We’ll continue to improve the MongoDB and MySQL connectors, and we also have a great PostgreSQL connector that is nearly ready to be released. With the new connector we’ll switch release numbers to 0.4.x and plan to stop issuing 0.3.x releases. Stay tuned for this next 0.4.0 release!

About Debezium

Debezium is an open source distributed platform that turns your existing databases into event streams, so applications can see and respond almost instantly to each committed row-level change in the databases. Debezium is built on top of Kafka and provides Kafka Connect compatible connectors that monitor specific database management systems. Debezium records the history of data changes in Kafka logs, so your application can be stopped and restarted at any time and can easily consume all of the events it missed while it was not running, ensuring that all events are processed correctly and completely. Debezium is open source under the Apache License, Version 2.0.

Get involved

We hope you find Debezium interesting and useful, and want to give it a try. Follow us on Twitter @debezium, chat with us on Gitter, or join our mailing list to talk with the community. All of the code is open source on GitHub, so build the code locally and help us improve the MySQL connector and add more connectors. If you find problems or have ideas how we can improve Debezium, please let us know or log an issue.


Debezium 0.3.5 Released

We’re happy to announce that Debezium 0.3.5 is now available for use with Kafka Connect 0.10.0.1. This release contains several fixes for the MySQL connector and adds the ability to use with multi-master MySQL servers as sources. See the release notes for specifics on these changes. We’ve also updated the Debezium Docker images labelled 0.3 and latest, which we use in our tutorial.

One of the fixes is signficant, and so we strongly urge all users to upgrade to this release from all earlier versions. In prior versions, the MySQL connector may stop without completing all updates in a transaction, and when the connector restarts it starts with the next transaction and therefore might fail to capture some of the change events in the earlier transaction. This release fixes this issue so that when restarting it will always pick up where it left off, even if that point is in the middle of a transaction. Note that this fix only takes affect once a connector is upgraded and restarted. See the issue for more details.

Thanks to Akshath, Anton, Chris, and others for their help with the release, issues, discussions, contributions, and questions!

About Debezium

Debezium is an open source distributed platform that turns your existing databases into event streams, so applications can see and respond almost instantly to each committed row-level change in the databases. Debezium is built on top of Kafka and provides Kafka Connect compatible connectors that monitor specific database management systems. Debezium records the history of data changes in Kafka logs, so your application can be stopped and restarted at any time and can easily consume all of the events it missed while it was not running, ensuring that all events are processed correctly and completely. Debezium is open source under the Apache License, Version 2.0.

Get involved

We hope you find Debezium interesting and useful, and want to give it a try. Follow us on Twitter @debezium, chat with us on Gitter, or join our mailing list to talk with the community. All of the code is open source on GitHub, so build the code locally and help us improve the MySQL connector and add more connectors. If you find problems or have ideas how we can improve Debezium, please let us know or log an issue.


Debezium 0.3.4 Released

We’re happy to announce that Debezium 0.3.4 is now available for use with Kafka Connect 0.10.0.1. This release contains several new features for the MySQL connector: support for MySQL’s JSON datatype, a new snapshot mode called schema_only, and JMX metrics. Also, the Debezium Docker images for Zookeeper, Kafka, and Kafka Connect have all been updated to allow optionally expose JMX metrics in these services. And, one backward-incompatible fix was made to the change event’s ts_sec field. See the release notes for specifics.

We’ve also updated the Debezium Docker images labelled 0.3 and latest, which we use in our tutorial.

Thanks to Akshath, Chris, Vitalii, Dennis, Prannoy, and others for their help with the release, issues, discussions, contributions, and questions!

About Debezium

Debezium is an open source distributed platform that turns your existing databases into event streams, so applications can see and respond almost instantly to each committed row-level change in the databases. Debezium is built on top of Kafka and provides Kafka Connect compatible connectors that monitor specific database management systems. Debezium records the history of data changes in Kafka logs, so your application can be stopped and restarted at any time and can easily consume all of the events it missed while it was not running, ensuring that all events are processed correctly and completely. Debezium is open source under the Apache License, Version 2.0.

Get involved

We hope you find Debezium interesting and useful, and want to give it a try. Follow us on Twitter @debezium, chat with us on Gitter, or join our mailing list to talk with the community. All of the code is open source on GitHub, so build the code locally and help us improve the MySQL connector and add more connectors. If you find problems or have ideas how we can improve Debezium, please let us know or log an issue.


back to top