Featured Posts

Tags
analytics announcement announcement discussion survey apache kafka apache kafka apicurio avro aws batch caassandra camel cassandra cdc channels community community stories connectors containers cqrs custom datalake db2 ddd debezium debezium server debezium ui deduplication discussion docker elasticsearch event sourcing exactly once semantics example examples features fedora flink hiring ibmi iceberg images informix integration introduction jaeger jdbc json kafka kafka streams kafka streams kogito ksql kubernetes lakehouse machine learning mariadb metrics microservices mongo mongodb monitoring mysql news newsletter notifications online learning operator oracle outbox performance postgres presentation production quarkus questdb rds releases schema scylla secrets sentry serialization signaling smt snapshots spanner spark sql sqlserver tensorflow testcontainers tests time series timescaledb topics tracing transactions ui vagrant vitess website

Debezium Blog

Although Debezium makes it easy to capture database changes and record them in Kafka, one of the more important decisions you have to make is how those change events will be serialized in Kafka. Every message in Kafka has a key and a value, and to Kafka these are opaque byte arrays. But when you set up Kafka Connect, you have to say how the Debezium event keys and values should be serialized to a binary form, and your consumers will also have to deserialize them back into a usable form.

Debezium event keys and values are both structured, so JSON is certainly a reasonable option — it’s flexible, ubiquitous, and language agnostic, but on the other hand it’s quite verbose. One alternative is Avro, which is also flexible and language agnostic, but also faster and results in smaller binary representations. Using Avro requires a bit more setup effort on your part and some additional software, but the advantages are often worth it.

Read More    

We’re happy to announce that Debezium 0.3.1 is now available for use with Kafka Connect 0.10.0.1. This release contains an updated MySQL connector with a handful of bug fixes and two significant but backward-compatible changes. First, the MySQL connector now supports using secure connections to MySQL, adding to the existing ability to connect securely to Kafka. Second, the MySQL connector is able to capture MySQL string values using the proper character sets so that any values stored in the database can be captured correctly in events. See our release notes for details of these changes and for upgrading recommendations.

We’ve also updated the Debezium Docker images labelled 0.3 and latest, which we use in our tutorial.

Thanks to Chris, Akshath, barten, and and others for their help with the release, issues, discussions, contributions, and questions!

Read More    

After a few weeks delay, Debezium 0.3.0 is now available for use with Kafka Connect 0.10.0.1. This release contains an updated MySQL connector with quite a few bug fixes, and a new MongoDB connector that captures the changes made to a MongoDB replica set or MongoDB sharded cluster. See the documentation for details about how to configure these connectors and how they work.

We’ve also updated the Debezium Docker images (with labels 0.3 and latest) used in our tutorial.

Thanks to Andrew, Bhupinder, Chris, David, Horia, Konstantin, Tony, and others for their help with the release, issues, discussions, contributions, and questions!

Read More    

I’m happy to announce that Debezium 0.2.4 is now available for use with Kafka Connect 0.9.0.1. This release adds more verbose logging during MySQL snapshots, enables taking snapshots of very large MySQL databases, and correct a potential exception during graceful shutdown. See our release notes for details of these changes and for upgrading recommendations.

We’ve also updated the Debezium Docker images (with label 0.2 and latest) used in our tutorial.

Thanks to David and wangshao for their help with the release, issues, discussions, contributions, and questions! Stay tuned for our next release, which will be 0.3 and will have a new MongoDB connector and will support Kafka Connect 0.10.0.1.

Read More    

Change data capture is a hot topic. Debezium’s goal is to make change data capture easy for multiple DBMSes, but admittedly we’re still a young open source project and so far we’ve only released a connector for MySQL with a connector for MongoDB that’s just around the corner. So it’s great to see how others are using and implementing change data capture. In this post, we’ll review Yelp’s approach and see how it is strikingly similar to Debezium’s MySQL connector.

Read More