Subscribe


Debezium 0.5.1 Released

It’s my pleasure to announce the release of Debezium 0.5.1!

This release fixes several bugs in the MySQL, Postgres and MongoDB connectors. There’s also support for some new datatypes: POINT on MySQL (DBZ-222) and TSTZRANGE on Postgres (DBZ-280). This release is a drop-in replacement for 0.5.0, upgrading is recommended to all users.

Note that in the — rather unlikely — case that you happened to enable Debezium for all the system tables of MySQL, any configured table filters will be applied to these system tables now, too (DBZ-242). This may require an adjustment of your filters if you indeed wanted to capture all system tables but only selected non-system tables.

Please refer to the changelog for an overview of all the 29 issues fixed in Debezium 0.5.1.

The Docker image containing Kafka Connect and all the Debezium 0.5.x connectors as well as the image containing Postgres and the Debezium logical decoding plug-in have been updated to 0.5.1, too.

As Debezium continues to evolve and grow, the number of people contributing to the project is also going up. The following people have sent in pull requests for this release: Aaron Rosenberg, Alexander Kukushkin, Brendan Maguire, Duncan Sands, David Leibovic, Jiri Pechanec, nacivida, Omar Al-Safi, Randall Hauch and Tom Bentley.

Thanks a lot to you and everyone else in the community contributing via feature requests, bug reports, discussions and questions!

What’s next

We’ve planned to do further bug fix releases for the 0.5.x line. Specifically, we’ll release a fix for DBZ-217 shortly, which is about the MySQL connector stumbling when getting across a corrupt event in the binlog.

In parallel we’re looking into Debezium connectors for SQL Server and Oracle. While we cannot promise anything yet in terms of when these will be ready to be published, we hope to have at least one of them ready some time soon. Stay tuned and get involved!

About Debezium

Debezium is an open source distributed platform that turns your existing databases into event streams, so applications can see and respond almost instantly to each committed row-level change in the databases. Debezium is built on top of Kafka and provides Kafka Connect compatible connectors that monitor specific database management systems. Debezium records the history of data changes in Kafka logs, so your application can be stopped and restarted at any time and can easily consume all of the events it missed while it was not running, ensuring that all events are processed correctly and completely. Debezium is open source under the Apache License, Version 2.0.

Get involved

We hope you find Debezium interesting and useful, and want to give it a try. Follow us on Twitter @debezium, chat with us on Gitter, or join our mailing list to talk with the community. All of the code is open source on GitHub, so build the code locally and help us improve ours existing connectors and add even more connectors. If you find problems or have ideas how we can improve Debezium, please let us know or log an issue.


Hello Debezium!

When I first learned about the Debezium project last year, I was very excited about it right away.

I could see how this project would be very useful for many people out there and I was very impressed by the professional way it was set up: a solid architecture for change data capture based on Apache Kafka, a strong focus on robustness and correctness also in the case of failures, the overall idea of creating a diverse eco-system of CDC connectors. All that based on the principles of open source, combined with extensive documentation from day one, a friendly and welcoming web site and a great getting-started experience.

So you can imagine that I was more than enthusiastic about the opportunity to take over the role of Debezium’s project lead. Debezium and CDC have close links to some data-centric projects I’ve been previously working on and also tie in with ideas I’ve been pursuing around CQRS, even sourcing and denormalization. As core member of the Hibernate team at Red Hat, I’ve implemented the initial Elasticsearch support for Hibernate Search (which deals with full-text index updates via JPA/Hibernate). I’ve also contributed to Hibernate OGM - a project which connects JPA and the world of NoSQL. One of the plans for OGM is to create a declarative denormalization engine for creating read models optimized for specific use cases. It will be very interesting to see how this plays together with the capabilities provided by Debezium.

Currently I am serving as the lead of the Bean Validation 2.0 specification (JSR 380) as well as its reference implementation Hibernate Validator. Two other projects close to my heart are MapStruct - a code generator for bean-to-bean mappings - and ModiTect, which is tooling for Java 9 modules and their descriptors. In general, I’m a strong believer into the idea of open source and I just love it to work with folks from all over the world to create useful tools and libraries.

Joining the Debezium community and working on change data capture is a great next step. There are so many things to do: connectors for Oracle, SQL Server and Cassandra, but also things like an entity join processor which would allow to step from row-level events to more aggregated business-level events (e.g. for updating a combined search index for an order and its order lines) or tooling for managing and visualizing histories of event schema changes.

One thing I’d like to emphasize is that the project’s direction generally isn’t going to change very much. Red Hat is fully committed to maintaining and evolving the project together with you, the Debezium community. The ride really has just begun!

Finally, let me say a huge thank you to Randall for his excellent work! You’ve been a true role model for going from an idea over pitching it - within Red Hat as well as within the wider community - to building a steadily growing and evolving project. It’s stating the obvious, but it wouldn’t be for Debezium without you. Thanks for everything and looking forward very much to working with you and the community on this great project!

Onwards,

--Gunnar


Debezium Evolving

Just before I started the Debezium project in early 2016, Martin Kleppmann gave several presentations about turning the database inside out and how his Bottled Water project demonstrated the importantance that change data capture can play in using Kafka for stream processing. Then Kafka Connect was announced, and at that point it seemed obvious to me that Kafka Connect was the foundation upon which practical and reusable change data capture can be built. As these techniques and technologies were becoming more important to Red Hat, I was given the opportunity to start a new open source project and community around building great CDC connectors for a variety of databases management systems.

Over the past few years, we have created Kafka Connect connectors for MySQL, then MongoDB, and most recently PostgreSQL. Each were initially limited and had a number of problems and issues, but over time more and more people have tried the connectors, asked questions, answered questions, mentioned Debezium on Twitter, tested connectors in their own environments, reported problems, fixed bugs, discussed limitations and potential new features, implemented enhancements and new features, improved the documentation, and wrote blog posts. Simply put, people with similar needs and interests have worked together and have formed a community. Additional connectors for Oracle and SQL Server are in the works, but could use some help to move things along more quickly.

It’s really exciting to see how far we’ve come and how the Debezium community continues to evolve and grow. And it’s perhaps as good a time as any to hand the reigns over to someone else. In fact, after nearly 10 wonderful years at Red Hat, I’m making a bigger change and as of today am part of Confluent’s engineering team, where I expect to play a more active role in the broader Kafka community and more directly with Kafka Connect and Kafka Streams. I definitely plan to stay involved in the Debezium community, but will no longer be leading the project. That role will instead be filled by Gunnar Morling, who’s recently joined the Debezium community but has extensive experience in open source, the Hibernate community, and the Bean Validation specification effort. Gunnar is a great guy and an excellent developer, and will be an excellent lead for the Debezium community.

Will the Debezium project change? To some degree it will always continue to evolve just as it has from the very beginning, and that’s a healthy thing. But a lot is staying the same. Red Hat remains committed to the Debezium project, and will continue its sponsorship and community-oriented governance that has worked so well from the beginning. And just as importantly, we the community are still here and will continue building the best open source CDC connectors.

So keep up the great work, and look for and take advantage of opportunities to become more involved in Debezium. Please give a warm welcome to Gunnar by introducing yourself in the developer and / or user chat rooms and mention how you’re using Debezium and what the Debezium community means to you.


Debezium 0.5.0 Released

We’re happy to announce that Debezium 0.5.0 is now available for use with Kafka Connect 0.10.2.0. This release also includes a few fixes for the MySQL connector. See the release notes for specifics on these changes, and be sure to check out the Kafka documentation for compatibility with the version of the Kafka broker that you are using.

Kafka Connect 0.10.2.0 comes with a significant new feature called Single Message Transforms, and you can now use them with Debezium connectors. SMTs allow you to modify the messages produced by Debezium connectors and any oher Kafka Connect source connectors, before those messages are written to Kafka. SMTs can also be used with Kafka Connect sink connectors to modify the messages before the sink connectors processes them. You can use SMTs to filter out or mask specific fields, add new fields, modify existing fields, change the topic and/or topic partition to which the messages are written, and even more. And you can even chain multiple SMTs together.

Kafka Connect comes with a number of built-in SMTs that you can simply configure and use, but you can also create your own SMT implementations to do more complex and interesting things. For example, although Debezium connectors normally map all of the changes in each table (or collection) to separate topics, you can write a custom SMT that uses a completely different mapping between tables and topics and even add fields to message keys and/or values. Using your new SMT is also very easy - simply put it on the Kafka Connect classpath and update the connector configuration to use it.

We’ve also added Debezium Docker images labelled 0.5 and latest, which we use in our tutorial.

Thanks to Sanjay and everyone in the community for their help with this release, issues, discussions, contributions, and questions!

What’s next

We’ll continue to improve the MongoDB, MySQL, and PostgreSQL connectors and pushing out 0.5.x releases with fixes. And we’re still working on connectors for SQL Server and Oracle. Stay tuned and get involved!

About Debezium

Debezium is an open source distributed platform that turns your existing databases into event streams, so applications can see and respond almost instantly to each committed row-level change in the databases. Debezium is built on top of Kafka and provides Kafka Connect compatible connectors that monitor specific database management systems. Debezium records the history of data changes in Kafka logs, so your application can be stopped and restarted at any time and can easily consume all of the events it missed while it was not running, ensuring that all events are processed correctly and completely. Debezium is open source under the Apache License, Version 2.0.

Get involved

We hope you find Debezium interesting and useful, and want to give it a try. Follow us on Twitter @debezium, chat with us on Gitter, or join our mailing list to talk with the community. All of the code is open source on GitHub, so build the code locally and help us improve ours existing connectors and add even more connectors. If you find problems or have ideas how we can improve Debezium, please let us know or log an issue.


Debezium 0.4.1 Released

We’re happy to announce that Debezium 0.4.1 is now available for use with Kafka Connect 0.10.1.1. This release includes several fixes for the MongoDB connector and MySQL connector, including improved support for Amazon RDS and Amazon Aurora (MySQL compatibility). See the release notes for specifics on these changes.

We’ve also updated the Debezium Docker images labelled 0.4 and latest, which we use in our tutorial.

Thanks to Jan, Horia, David, Josh, Johan, Sanjay, Saulius, and everyone in the community for their help with this release, issues, discussions, contributions, and questions!

What’s next

Kafka 0.10.2.0 is out, so we plan to release 0.5.0 next week with all of the changes/fixes in 0.4.1 but with support for Kafka 0.10.2.0. We’ll then continue to improve the MongoDB, MySQL, and PostgreSQL connectors and pushing out 0.5.x releases. Stay tuned and get involved!

About Debezium

Debezium is an open source distributed platform that turns your existing databases into event streams, so applications can see and respond almost instantly to each committed row-level change in the databases. Debezium is built on top of Kafka and provides Kafka Connect compatible connectors that monitor specific database management systems. Debezium records the history of data changes in Kafka logs, so your application can be stopped and restarted at any time and can easily consume all of the events it missed while it was not running, ensuring that all events are processed correctly and completely. Debezium is open source under the Apache License, Version 2.0.

Get involved

We hope you find Debezium interesting and useful, and want to give it a try. Follow us on Twitter @debezium, chat with us on Gitter, or join our mailing list to talk with the community. All of the code is open source on GitHub, so build the code locally and help us improve ours existing connectors and add even more connectors. If you find problems or have ideas how we can improve Debezium, please let us know or log an issue.


back to top