Subscribe


Debezium 0.10.0.Beta2 Released

It’s my pleasure to announce the release of Debezium 0.10.0.Beta2!

This further stabilizes the 0.10 release line, with lots of bug fixes to the different connectors. 23 issues were fixed for this release; a couple of those relate to the DDL parser of the MySQL connector, e.g. around RENAME INDEX (DBZ-1329), SET NEW in triggers (DBZ-1331) and function definitions with the COLLATE keyword (DBZ-1332).

For the Postgres connector we fixed a potential inconsistency when flushing processed LSNs to the database (DBZ-1347). Also the "include.unknown.datatypes" option works as expected now during snapshotting (DBZ-1335) and the connector won’t stumple upon materialized views during snapshotting any longer (DBZ-1345).

The SQL Server connector will use much less memory in many situations (DBZ-1065) and it’s configurable now whether it should emit tombstone events for deletions or not (DBZ-835). This also was added for the Oracle connector, bringing consistency for this option across all the connectors.

Note that this release can be used with Apache Kafka 2.x, but not with 1.x. This was an unintentional change and compatibility with 1.x will be restored for the Beta3 release (the issue to track is DBZ-1361).

Please refer to the 0.10.0.Beta2 release notes to learn more about all resolved issues and the upgrading procedure.

Many thanks to everybody from the Debezium community who contributed to this release: Cheng Pan, Guillaume Rosauro, Mariusz Strzelecki and Stathis Souris.

About Debezium

Debezium is an open source distributed platform that turns your existing databases into event streams, so applications can see and respond almost instantly to each committed row-level change in the databases. Debezium is built on top of Kafka and provides Kafka Connect compatible connectors that monitor specific database management systems. Debezium records the history of data changes in Kafka logs, so your application can be stopped and restarted at any time and can easily consume all of the events it missed while it was not running, ensuring that all events are processed correctly and completely. Debezium is open source under the Apache License, Version 2.0.

Get involved

We hope you find Debezium interesting and useful, and want to give it a try. Follow us on Twitter @debezium, chat with us on Gitter, or join our mailing list to talk with the community. All of the code is open source on GitHub, so build the code locally and help us improve ours existing connectors and add even more connectors. If you find problems or have ideas how we can improve Debezium, please let us know or log an issue.


Debezium Wears Fedora

The Debezium project strives to provide an easy deployment of connectors, so users can try and run connectors of their choice mostly by getting the right connector archive and unpacking it into the plug-in path of Kafka Connect.

This is true for all connectors but for the Debezium PostgreSQL connector. This connector is specific in the regard that it requires a logical decoding plug-in to be installed inside the PostgreSQL source database(s) themselves. Currently, there are two supported logical plug-ins:

  • postgres-decoderbufs, which uses Protocol Buffers as a very compact transport format and which is maintained by the Debezium community

  • JSON-based, which is based on JSON and which is maintained by its own upstream community

These plug-ins can be consumed and deployed in two ways; the easiest one is to use one of our pre-made Postgres container images, which contain both plug-ins and are already configured as required. If you are using containers in your datacenter, and/or if you start a fresh database from scratch, then this can be a great option.

The other approach is building from source. Even if this is usually an easy task, it still brings a barrier to an easy start and requires a non-trivial knowledge of the Linux operating system.

To bridge the gap between those two extremes we’ve created and published an RPM package, available for Fedora 30 and later. By installing this package you will have the necessary binaries deployed, and the only task remaining is to configure PostgreSQL to enable the plug-in. The RPM is based on the latest stable Debezium release, 0.9.5.Final at this point.

Example

Let’s show how the package works. We will use the Vagrant tool as an easy way for firing up a pre-provisioned virtual machine with Fedora. Of course, that’s not a requirement and the same steps apply for any other way of running Fedora.

Create and start virtual machine with Fedora 30:

$ vagrant init fedora/30-cloud-base

A `Vagrantfile` has been placed in this directory. You are now
ready to `vagrant up` your first virtual environment! Please read
the comments in the Vagrantfile as well as documentation on
`vagrantup.com` for more information on using Vagrant.

$ vagrant up

Bringing machine 'default' up with 'virtualbox' provider...
.
.
.
==> default: Machine booted and ready!

Log into the virtual machine:

$ vagrant ssh

Install the PostgreSQL server and Protocol Buffers logical decoding plug-in:

$ sudo dnf -y install postgresql postgres-decoderbufs
.
.
.
Installed:
  postgres-decoderbufs-0.9.5-1.fc30.x86_64              postgresql-11.3-1.fc30.x86_64
  postgis-2.5.1-1.fc30.x86_64                           armadillo-9.400.4-1.fc30.x86_64
  blas-3.8.0-12.fc30.x86_64                             cairo-1.16.0-5.fc30.x86_64
  cups-libs-1:2.2.11-2.fc30.x86_64                      fontconfig-2.13.1-8.fc30.x86_64
  lapack-3.8.0-12.fc30.x86_64                           libgfortran-9.1.1-1.fc30.x86_64
  libpq-11.3-2.fc30.x86_64                              libquadmath-9.1.1-1.fc30.x86_64
  mariadb-connector-c-3.0.10-1.fc30.x86_64              mariadb-connector-c-config-3.0.10-1.fc30.noarch
  nss-3.44.0-2.fc30.x86_64                              nss-softokn-3.44.0-2.fc30.x86_64
  nss-softokn-freebl-3.44.0-2.fc30.x86_64               nss-sysinit-3.44.0-2.fc30.x86_64
  nss-util-3.44.0-2.fc30.x86_64                         poppler-0.73.0-9.fc30.x86_64
  postgresql-server-11.3-1.fc30.x86_64                  proj-5.2.0-2.fc30.x86_64
  proj-datumgrid-1.8-2.fc30.noarch                      uriparser-0.9.3-1.fc30.x86_64
  SuperLU-5.2.1-6.fc30.x86_64                           arpack-3.5.0-6.fc28.x86_64
  atk-2.32.0-1.fc30.x86_64                              avahi-libs-0.7-18.fc30.x86_64
  cfitsio-3.450-3.fc30.x86_64                           dejavu-fonts-common-2.37-1.fc30.noarch
  dejavu-sans-fonts-2.37-1.fc30.noarch                  fontpackages-filesystem-1.44-24.fc30.noarch
  freexl-1.0.5-3.fc30.x86_64                            fribidi-1.0.5-2.fc30.x86_64
  gdal-libs-2.3.2-7.fc30.x86_64                         gdk-pixbuf2-2.38.1-1.fc30.x86_64
  gdk-pixbuf2-modules-2.38.1-1.fc30.x86_64              geos-3.7.1-1.fc30.x86_64
  giflib-5.1.9-1.fc30.x86_64                            graphite2-1.3.10-7.fc30.x86_64
  gtk-update-icon-cache-3.24.8-1.fc30.x86_64            gtk2-2.24.32-4.fc30.x86_64
  harfbuzz-2.3.1-1.fc30.x86_64                          hdf5-1.8.20-6.fc30.x86_64
  hicolor-icon-theme-0.17-5.fc30.noarch                 jasper-libs-2.0.14-8.fc30.x86_64
  jbigkit-libs-2.1-16.fc30.x86_64                       lcms2-2.9-5.fc30.x86_64
  libXcomposite-0.4.4-16.fc30.x86_64                    libXcursor-1.1.15-5.fc30.x86_64
  libXdamage-1.1.4-16.fc30.x86_64                       libXfixes-5.0.3-9.fc30.x86_64
  libXft-2.3.2-12.fc30.x86_64                           libXi-1.7.9-9.fc30.x86_64
  libXinerama-1.1.4-3.fc30.x86_64                       libaec-1.0.4-1.fc30.x86_64
  libdap-3.20.3-1.fc30.x86_64                           libgeotiff-1.4.3-3.fc30.x86_64
  libgta-1.0.9-2.fc30.x86_64                            libjpeg-turbo-2.0.2-1.fc30.x86_64
  libkml-1.3.0-19.fc30.x86_64                           libspatialite-4.3.0a-11.fc30.x86_64
  libtiff-4.0.10-4.fc30.x86_64                          libwebp-1.0.2-2.fc30.x86_64
  netcdf-4.4.1.1-12.fc30.x86_64                         nspr-4.21.0-1.fc30.x86_64
  ogdi-3.2.1-4.fc30.x86_64                              openblas-0.3.5-5.fc30.x86_64
  openblas-openmp-0.3.5-5.fc30.x86_64                   openblas-serial-0.3.5-5.fc30.x86_64
  openblas-threads-0.3.5-5.fc30.x86_64                  openblas-threads64_-0.3.5-5.fc30.x86_64
  openjpeg2-2.3.1-1.fc30.x86_64                         pango-1.43.0-3.fc30.x86_64
  pixman-0.38.0-1.fc30.x86_64                           poppler-data-0.4.9-3.fc30.noarch
  protobuf-c-1.3.1-2.fc30.x86_64                        unixODBC-2.3.7-4.fc30.x86_64
  xerces-c-3.2.2-2.fc30.x86_64

Complete!

Next, initialize the database:

$ sudo /usr/bin/postgresql-setup --initdb

Now enable the plug-in in the database server configuration file /var/lib/pgsql/data/postgresql.conf by adding the following parameters:

# MODULES
shared_preload_libraries = 'decoderbufs'

# REPLICATION
wal_level = logical             # minimal, archive, hot_standby, or logical (change requires restart)
max_wal_senders = 8             # max number of walsender processes (change requires restart)
wal_keep_segments = 4           # in logfile segments, 16MB each; 0 disables
#wal_sender_timeout = 60s       # in milliseconds; 0 disables
max_replication_slots = 4       # max number of replication slots (change requires restart)

Configure the security file /var/lib/pgsql/data/pg_hba.conf for the database user that will be used by Debezium (e.g. debezium) by adding these parameters:

local   replication     debezium                          trust
host    replication     debezium  127.0.0.1/32            trust
host    replication     debezium  ::1/128                 trust

Finally, restart PostgreSQL:

$ sudo systemctl restart postgresql

And that’s it: Now we have a PostgreSQL database, that is ready to stream changes to the Debezium PostgreSQL connector. Of course, the plug-in can also be installed to an already existing database (Postgres versions 9 and later), just by installing the RPM package and setting up the config and security files in the described way.

Outlook: pgoutput

While the decoderbufs plug-in is our recommended choice for a logical decoding plug-in, there are cases where you may not be able to use it. Most specifically, you typically don’t have the flexibility to install custom plug-ins in cloud-based environments such as Amazon RDS.

This is why we’re exploring a third alternative to decoderbufs and wal2sjon right now, which is to leverage Postgres logical replication mechanism. There’s a built-in plug-in, pgoutput based on this, which exists in every Postgres database since version 10. We’re still in the process of exploring the implications (and possible limitations) of using pgoutput, but so far things look promising and it may eventually be a valuable tool to have in the box.

Stay tuned for more details coming soon!

About Debezium

Debezium is an open source distributed platform that turns your existing databases into event streams, so applications can see and respond almost instantly to each committed row-level change in the databases. Debezium is built on top of Kafka and provides Kafka Connect compatible connectors that monitor specific database management systems. Debezium records the history of data changes in Kafka logs, so your application can be stopped and restarted at any time and can easily consume all of the events it missed while it was not running, ensuring that all events are processed correctly and completely. Debezium is open source under the Apache License, Version 2.0.

Get involved

We hope you find Debezium interesting and useful, and want to give it a try. Follow us on Twitter @debezium, chat with us on Gitter, or join our mailing list to talk with the community. All of the code is open source on GitHub, so build the code locally and help us improve ours existing connectors and add even more connectors. If you find problems or have ideas how we can improve Debezium, please let us know or log an issue.


Debezium 0.10.0.Beta1 Released

Another week, another Debezium release — I’m happy to announce the release of Debezium 0.10.0.Beta1!

Besides the upgrade to Apache Kafka 2.2.1 (DBZ-1316), this mostly fixes some bugs, including a regression to the MongoDB connector introduced in the Alpha2 release (DBZ-1317).

A very welcomed usability improvement is that the connectors will log a warning now if not at least one table is actually captured as per the whitelist/blacklist configuration (DBZ-1242). This helps to prevent the accidental exclusion all tables by means of an incorrect filter expression, in which case the connectors "work as intended", but no events are propagated to the message broker.

Please see the release notes for the complete list of issues fixed in this release. Also make sure to examine the upgrade guidelines for 0.10.0.Alpha1 and Alpha2 when upgrading from earlier versions.

Many thanks to community members Cheng Pan and Ching Tsai for their contributions to this release!

About Debezium

Debezium is an open source distributed platform that turns your existing databases into event streams, so applications can see and respond almost instantly to each committed row-level change in the databases. Debezium is built on top of Kafka and provides Kafka Connect compatible connectors that monitor specific database management systems. Debezium records the history of data changes in Kafka logs, so your application can be stopped and restarted at any time and can easily consume all of the events it missed while it was not running, ensuring that all events are processed correctly and completely. Debezium is open source under the Apache License, Version 2.0.

Get involved

We hope you find Debezium interesting and useful, and want to give it a try. Follow us on Twitter @debezium, chat with us on Gitter, or join our mailing list to talk with the community. All of the code is open source on GitHub, so build the code locally and help us improve ours existing connectors and add even more connectors. If you find problems or have ideas how we can improve Debezium, please let us know or log an issue.


Debezium’s Newsletter 01/2019

Welcome to the first edition of the Debezium community newsletter in which we share blog posts, group discussions, as well as StackOverflow questions that are relevant to our user community.

Articles

Gunnar Morling recently attended Kafka Summit in London where he gave a talk on Change Data Streaming Patterns for Microservices With Debezium. You can watch the full presentation here.

Strimzi provides an easy way to run Apache Kafka on Kubernetes or Openshift. This article by Sincy Sebastian shows just how simple it is to replicate change events from MySQL to Elastic Search using Debezium.

Debezium allows replicating data between heterogeneous data stores with ease. This article by Matthew Groves explains how you can replicate data from MySQL to CouchBase.

As the size of data that systems maintain continues to grow, this begins to impact how we capture, compute, and report real-time analytics. This article by Maria Patterson explains how you can use Debezium to stream data from Postgres, perform analytical calculations using KSQL, and then stream those results back to Postgres for consumption.

In a recent article published in Portuguese, Paulo Singaretti illustrates how they use Debezium and Kafka to stream changes from their relational database and then store the change stream results in Google Cloud Services.

This recent blog by Jia Zhai provides a complete tutorial showing how to use Debezium connectors with Apache Pulsar.

Time to upgrade

Debezium version 0.9.5 was just released. If you are using the 0.9 branch you should definitely check out 0.9.5. For details on the bug fixes as well as the enhancements this version includes, check out the release notes.

The Debezium team has also begun active development on the next major version, 0.10. We recently published a blog that provides an overview behind what 0.10 is meant to deliver. If you want details on the bug fixes and enhancements we’ve packed into this release, you can view the issue list.

Feedback

We intend to publish new additions of this newsletter periodically. Should anyone have any suggestions on changes or what could be highlighted here, we welcome that feedback. You can reach out to us via any of our community channels found here.


Debezium 0.10.0.Alpha2 Released

Release early, release often — Less than a week since the Alpha1 we are announcing the release of Debezium 0.10.0.Alpha2!

This is an incremental release that completes some of the tasks started in the Alpha1 release and provides a few bugfixes and also quality improvements in our Docker images.

The change in the logic of the snapshot field has been delivered (DBZ-1295) as outlined in the last announcement. All connectors now provide information which of the records is the last one in the snapshot phase so that downstream consumers can react to this.

Apache ZooKeeper was upgraded to version 3.4.14 to fix a security vulnerability (CVE-2019-0201).

Our regular contributor Renato dived deeply into our image build scripts and enriched (DBZ-1279) them with a Dockerfile linter.

Schema change events include the table name(s) in the metadata describing which tables are affected by the change (DBZ-871).

Bartosz Miedlar has fixed a bug in MySQL ANTLR grammar causing issues with identifiers in backquotes (DBZ-1300).

What’s next?

We hope we will be able to keep the recent release cadence and get lout the first beta version of 0.10 in two weeks.

Stay tuned for more!

About Debezium

Debezium is an open source distributed platform that turns your existing databases into event streams, so applications can see and respond almost instantly to each committed row-level change in the databases. Debezium is built on top of Kafka and provides Kafka Connect compatible connectors that monitor specific database management systems. Debezium records the history of data changes in Kafka logs, so your application can be stopped and restarted at any time and can easily consume all of the events it missed while it was not running, ensuring that all events are processed correctly and completely. Debezium is open source under the Apache License, Version 2.0.

Get involved

We hope you find Debezium interesting and useful, and want to give it a try. Follow us on Twitter @debezium, chat with us on Gitter, or join our mailing list to talk with the community. All of the code is open source on GitHub, so build the code locally and help us improve ours existing connectors and add even more connectors. If you find problems or have ideas how we can improve Debezium, please let us know or log an issue.


back to top