Subscribe


Debezium 0.10.0.Beta4 Released

The temperatures are slowly cooling off after the biggest summer heat, an the Debezium community is happy to announce the release of Debezium 0.10.0.Beta4. In this release we’re happy to share some news we don’t get to share too often: with Apache Cassandra, another database gets added to the list of databases supported by Debezium!

In addition, we finished our efforts for rebasing the existing Postgres connector to Debezium framework structure established for the SQL Server and Oracle connectors. This means more shared coded between these connectors, and in turn reduced maintenance efforts for the development team going forward; but there’s one immediately tangible advantage for you coming with this, too: the Postgres connector now exposes the same metrics you already know from the other connectors.

Finally, the new release contains a range of bugfixes and other useful improvements. Let’s explore some details below.

Incubating Cassandra Connector

If you have been following this blog lately, you’ll have read about the latest addition to the Debezium family in Joy Gao’s excellent posts about the new connector (part 1, part 2).

In case you haven’t read those yet, we’d highly recommend to do so in order to learn more about the challenges encountered when implementing a CDC connector for a distributed datastore such as Cassandra as well as the design decisions made in order to come up with a first "minimal viable product". Joy also did a great talk at QCon last year, which touches on the topic of CDC for Cassandra.

Having been originally developed internally at long-term Debezium user WePay, the WePay team decided to open-source their work, put it under the Debezium umbrella and continue to evolve it there. That’s really great news for the Debezium community! We couldn’t be happier about this contribution and look forward to evolving this new connector together in the open.

At this point the Cassandra connector is in "incubating" state, i.e. its design and implementation are still pretty much in flux, the event structure which it creates may change in future releases etc. Note that, unlike the other Debezium connectors, this one currently is not based on Kafka Connect. Instead, it is implemented as a standalone process running on Cassandra node(s) themselves. Refer to the blog posts linked above for the reasoning behind this design and possible future developments around this. Needless to say, any ideas and contributions in this area will be highly welcomed.

Together with the connector we’ve also provided an initial draft of the connector documentation; this is still work-in-progress and will be amended in the next few days.

Further New Features

The Postgres connector supports the metrics known from SQL Server and Oracle now (DBZ-777). When using the SQL Server connector, it is now ensured that tables are snapshotted in a deterministic order, as defined by the given table whitelist configuration (DBZ-1254).

There have also been two improvements to our SMTs (single message transformations):

  • The SMT for new record state extraction allows to add additional columns for propagating metadata fields from the source block (DBZ-1395, e.g. useful to propagate the transaction into sink tables).

  • The default structure produced by the outbox routing SMT has been further streamlined (DBZ-1385); the message value will now only contain the contents of the configured outbox table payload column. In case you want to re-add the eventType value, you can configure it as an "additional field", which either goes into the message as a header (recommended) or into the message value, which as before will be a nested structure then.

Bugfixes and Other Improvements

Finally, here’s an overview of asorted bugfixes in the 0.10 Beta4 release:

  • The MySQL connector handles GRANT DELETE ON <table> statements correctly (DBZ-1411)

  • Superfluous tables scans are avoided when using the initial_schema_only snapshot strategy with SQL Server (DBZ-1417)

  • The superfluous creation of connections is avoided when obtaining the xmin position of Postgres (DBZ-1381)

  • The new record state extraction SMT handles heartbeat events correctly (DBZ-1430)

Please refer to the 0.10.0.Beta4 release notes for the complete list of addressed issues and the upgrading procedure.

A big thank you goes out to all the contributors from the Debezium community who worked on this release: Joy Gao, Renato Mefi and Guillaume Rosauro!

About Debezium

Debezium is an open source distributed platform that turns your existing databases into event streams, so applications can see and respond almost instantly to each committed row-level change in the databases. Debezium is built on top of Kafka and provides Kafka Connect compatible connectors that monitor specific database management systems. Debezium records the history of data changes in Kafka logs, so your application can be stopped and restarted at any time and can easily consume all of the events it missed while it was not running, ensuring that all events are processed correctly and completely. Debezium is open source under the Apache License, Version 2.0.

Get involved

We hope you find Debezium interesting and useful, and want to give it a try. Follow us on Twitter @debezium, chat with us on Gitter, or join our mailing list to talk with the community. All of the code is open source on GitHub, so build the code locally and help us improve ours existing connectors and add even more connectors. If you find problems or have ideas how we can improve Debezium, please let us know or log an issue.


Debezium 0.10.0.Beta3 Released

The summer is at its peak but Debezium community is not relenting in its effort so the Debezium 0.10.0.Beta3 is released.

This version not only continues in incremental improvements of Debezium but also brings new shiny features.

All of you who are using PostgreSQL 10 and higher as a service offered by different cloud providers definitely felt the complications when you needed to deploy logical decoding plugin necessary to enable streaming. This is no longer necessary. Debezium now supports (DBZ-766) pgoutput replication protocol that is available out-of-the-box since PostgreSQL 10.

There is a set of further minor improvements. The tombstones for deletes are configurable for all connectors now (DBZ-1365). Also tables without primary keys are now supported for all connectors (DBZ-916). This further reduces the gap between old and new connectors capabilities.

There are improvements for heartbeat system. Heartbeat messages now contain the timestamp (DBZ-1363) of when they were created in their body. The new messages are properly skipped by the Outbox router (DBZ-1388). MySQL connector additionally uses heartbeats for BinlogReader (DBZ-1338). MongoDB connector now utilizes heartbeats too (DBZ-1198).

As we now that metrics are very important for keeping Debezium happy in production we have extended the set of supported metrics. A new metric count of events in error (DBZ-1222) is added so it is easy to monitor any non-standards in processing. Database history recovery can take a long time during startup so it is now possible to monitor the progress of it (DBZ-1356).

The other changes include updating of Docker images to use Kafka 2.3.0 (DBZ-1358). PostgreSQL supports lockless snapshotting (DBZ-1238) and Outbox router now process delete messages (DBZ-1320).

We continue with stabilization of the 0.10 release line, with lots of bug fixes to the different connectors.

Multiple defects in MySQL parser have been fixed (DBZ-1398, (DBZ-1397, DBZ-1376) and SAVEPOINT statements are no longer recorded in database history (DBZ-794).

Under certain circumstances, it was possible that PostgreSQL connector lost the first event while switching to streaming from the snapshot (DBZ-1400).

Please refer to the 0.10.0.Beta3 release notes to learn more about all resolved issues and the upgrading procedure.

Many thanks to everybody from the Debezium community who contributed to this release: Addison Higham, Bin Li, Brandon Brown and Renato Mefi.

About Debezium

Debezium is an open source distributed platform that turns your existing databases into event streams, so applications can see and respond almost instantly to each committed row-level change in the databases. Debezium is built on top of Kafka and provides Kafka Connect compatible connectors that monitor specific database management systems. Debezium records the history of data changes in Kafka logs, so your application can be stopped and restarted at any time and can easily consume all of the events it missed while it was not running, ensuring that all events are processed correctly and completely. Debezium is open source under the Apache License, Version 2.0.

Get involved

We hope you find Debezium interesting and useful, and want to give it a try. Follow us on Twitter @debezium, chat with us on Gitter, or join our mailing list to talk with the community. All of the code is open source on GitHub, so build the code locally and help us improve ours existing connectors and add even more connectors. If you find problems or have ideas how we can improve Debezium, please let us know or log an issue.


Debezium 0.10.0.Beta2 Released

It’s my pleasure to announce the release of Debezium 0.10.0.Beta2!

This further stabilizes the 0.10 release line, with lots of bug fixes to the different connectors. 23 issues were fixed for this release; a couple of those relate to the DDL parser of the MySQL connector, e.g. around RENAME INDEX (DBZ-1329), SET NEW in triggers (DBZ-1331) and function definitions with the COLLATE keyword (DBZ-1332).

For the Postgres connector we fixed a potential inconsistency when flushing processed LSNs to the database (DBZ-1347). Also the "include.unknown.datatypes" option works as expected now during snapshotting (DBZ-1335) and the connector won’t stumple upon materialized views during snapshotting any longer (DBZ-1345).

The SQL Server connector will use much less memory in many situations (DBZ-1065) and it’s configurable now whether it should emit tombstone events for deletions or not (DBZ-835). This also was added for the Oracle connector, bringing consistency for this option across all the connectors.

Note that this release can be used with Apache Kafka 2.x, but not with 1.x. This was an unintentional change and compatibility with 1.x will be restored for the Beta3 release (the issue to track is DBZ-1361).

Please refer to the 0.10.0.Beta2 release notes to learn more about all resolved issues and the upgrading procedure.

Many thanks to everybody from the Debezium community who contributed to this release: Cheng Pan, Guillaume Rosauro, Mariusz Strzelecki and Stathis Souris.

About Debezium

Debezium is an open source distributed platform that turns your existing databases into event streams, so applications can see and respond almost instantly to each committed row-level change in the databases. Debezium is built on top of Kafka and provides Kafka Connect compatible connectors that monitor specific database management systems. Debezium records the history of data changes in Kafka logs, so your application can be stopped and restarted at any time and can easily consume all of the events it missed while it was not running, ensuring that all events are processed correctly and completely. Debezium is open source under the Apache License, Version 2.0.

Get involved

We hope you find Debezium interesting and useful, and want to give it a try. Follow us on Twitter @debezium, chat with us on Gitter, or join our mailing list to talk with the community. All of the code is open source on GitHub, so build the code locally and help us improve ours existing connectors and add even more connectors. If you find problems or have ideas how we can improve Debezium, please let us know or log an issue.


Debezium Wears Fedora

The Debezium project strives to provide an easy deployment of connectors, so users can try and run connectors of their choice mostly by getting the right connector archive and unpacking it into the plug-in path of Kafka Connect.

This is true for all connectors but for the Debezium PostgreSQL connector. This connector is specific in the regard that it requires a logical decoding plug-in to be installed inside the PostgreSQL source database(s) themselves. Currently, there are two supported logical plug-ins:

  • postgres-decoderbufs, which uses Protocol Buffers as a very compact transport format and which is maintained by the Debezium community

  • JSON-based, which is based on JSON and which is maintained by its own upstream community

These plug-ins can be consumed and deployed in two ways; the easiest one is to use one of our pre-made Postgres container images, which contain both plug-ins and are already configured as required. If you are using containers in your datacenter, and/or if you start a fresh database from scratch, then this can be a great option.

The other approach is building from source. Even if this is usually an easy task, it still brings a barrier to an easy start and requires a non-trivial knowledge of the Linux operating system.

To bridge the gap between those two extremes we’ve created and published an RPM package, available for Fedora 30 and later. By installing this package you will have the necessary binaries deployed, and the only task remaining is to configure PostgreSQL to enable the plug-in. The RPM is based on the latest stable Debezium release, 0.9.5.Final at this point.

Example

Let’s show how the package works. We will use the Vagrant tool as an easy way for firing up a pre-provisioned virtual machine with Fedora. Of course, that’s not a requirement and the same steps apply for any other way of running Fedora.

Create and start virtual machine with Fedora 30:

$ vagrant init fedora/30-cloud-base

A `Vagrantfile` has been placed in this directory. You are now
ready to `vagrant up` your first virtual environment! Please read
the comments in the Vagrantfile as well as documentation on
`vagrantup.com` for more information on using Vagrant.

$ vagrant up

Bringing machine 'default' up with 'virtualbox' provider...
.
.
.
==> default: Machine booted and ready!

Log into the virtual machine:

$ vagrant ssh

Install the PostgreSQL server and Protocol Buffers logical decoding plug-in:

$ sudo dnf -y install postgresql postgres-decoderbufs
.
.
.
Installed:
  postgres-decoderbufs-0.9.5-1.fc30.x86_64              postgresql-11.3-1.fc30.x86_64
  postgis-2.5.1-1.fc30.x86_64                           armadillo-9.400.4-1.fc30.x86_64
  blas-3.8.0-12.fc30.x86_64                             cairo-1.16.0-5.fc30.x86_64
  cups-libs-1:2.2.11-2.fc30.x86_64                      fontconfig-2.13.1-8.fc30.x86_64
  lapack-3.8.0-12.fc30.x86_64                           libgfortran-9.1.1-1.fc30.x86_64
  libpq-11.3-2.fc30.x86_64                              libquadmath-9.1.1-1.fc30.x86_64
  mariadb-connector-c-3.0.10-1.fc30.x86_64              mariadb-connector-c-config-3.0.10-1.fc30.noarch
  nss-3.44.0-2.fc30.x86_64                              nss-softokn-3.44.0-2.fc30.x86_64
  nss-softokn-freebl-3.44.0-2.fc30.x86_64               nss-sysinit-3.44.0-2.fc30.x86_64
  nss-util-3.44.0-2.fc30.x86_64                         poppler-0.73.0-9.fc30.x86_64
  postgresql-server-11.3-1.fc30.x86_64                  proj-5.2.0-2.fc30.x86_64
  proj-datumgrid-1.8-2.fc30.noarch                      uriparser-0.9.3-1.fc30.x86_64
  SuperLU-5.2.1-6.fc30.x86_64                           arpack-3.5.0-6.fc28.x86_64
  atk-2.32.0-1.fc30.x86_64                              avahi-libs-0.7-18.fc30.x86_64
  cfitsio-3.450-3.fc30.x86_64                           dejavu-fonts-common-2.37-1.fc30.noarch
  dejavu-sans-fonts-2.37-1.fc30.noarch                  fontpackages-filesystem-1.44-24.fc30.noarch
  freexl-1.0.5-3.fc30.x86_64                            fribidi-1.0.5-2.fc30.x86_64
  gdal-libs-2.3.2-7.fc30.x86_64                         gdk-pixbuf2-2.38.1-1.fc30.x86_64
  gdk-pixbuf2-modules-2.38.1-1.fc30.x86_64              geos-3.7.1-1.fc30.x86_64
  giflib-5.1.9-1.fc30.x86_64                            graphite2-1.3.10-7.fc30.x86_64
  gtk-update-icon-cache-3.24.8-1.fc30.x86_64            gtk2-2.24.32-4.fc30.x86_64
  harfbuzz-2.3.1-1.fc30.x86_64                          hdf5-1.8.20-6.fc30.x86_64
  hicolor-icon-theme-0.17-5.fc30.noarch                 jasper-libs-2.0.14-8.fc30.x86_64
  jbigkit-libs-2.1-16.fc30.x86_64                       lcms2-2.9-5.fc30.x86_64
  libXcomposite-0.4.4-16.fc30.x86_64                    libXcursor-1.1.15-5.fc30.x86_64
  libXdamage-1.1.4-16.fc30.x86_64                       libXfixes-5.0.3-9.fc30.x86_64
  libXft-2.3.2-12.fc30.x86_64                           libXi-1.7.9-9.fc30.x86_64
  libXinerama-1.1.4-3.fc30.x86_64                       libaec-1.0.4-1.fc30.x86_64
  libdap-3.20.3-1.fc30.x86_64                           libgeotiff-1.4.3-3.fc30.x86_64
  libgta-1.0.9-2.fc30.x86_64                            libjpeg-turbo-2.0.2-1.fc30.x86_64
  libkml-1.3.0-19.fc30.x86_64                           libspatialite-4.3.0a-11.fc30.x86_64
  libtiff-4.0.10-4.fc30.x86_64                          libwebp-1.0.2-2.fc30.x86_64
  netcdf-4.4.1.1-12.fc30.x86_64                         nspr-4.21.0-1.fc30.x86_64
  ogdi-3.2.1-4.fc30.x86_64                              openblas-0.3.5-5.fc30.x86_64
  openblas-openmp-0.3.5-5.fc30.x86_64                   openblas-serial-0.3.5-5.fc30.x86_64
  openblas-threads-0.3.5-5.fc30.x86_64                  openblas-threads64_-0.3.5-5.fc30.x86_64
  openjpeg2-2.3.1-1.fc30.x86_64                         pango-1.43.0-3.fc30.x86_64
  pixman-0.38.0-1.fc30.x86_64                           poppler-data-0.4.9-3.fc30.noarch
  protobuf-c-1.3.1-2.fc30.x86_64                        unixODBC-2.3.7-4.fc30.x86_64
  xerces-c-3.2.2-2.fc30.x86_64

Complete!

Next, initialize the database:

$ sudo /usr/bin/postgresql-setup --initdb

Now enable the plug-in in the database server configuration file /var/lib/pgsql/data/postgresql.conf by adding the following parameters:

# MODULES
shared_preload_libraries = 'decoderbufs'

# REPLICATION
wal_level = logical             # minimal, archive, hot_standby, or logical (change requires restart)
max_wal_senders = 8             # max number of walsender processes (change requires restart)
wal_keep_segments = 4           # in logfile segments, 16MB each; 0 disables
#wal_sender_timeout = 60s       # in milliseconds; 0 disables
max_replication_slots = 4       # max number of replication slots (change requires restart)

Configure the security file /var/lib/pgsql/data/pg_hba.conf for the database user that will be used by Debezium (e.g. debezium) by adding these parameters:

local   replication     debezium                          trust
host    replication     debezium  127.0.0.1/32            trust
host    replication     debezium  ::1/128                 trust

Finally, restart PostgreSQL:

$ sudo systemctl restart postgresql

And that’s it: Now we have a PostgreSQL database, that is ready to stream changes to the Debezium PostgreSQL connector. Of course, the plug-in can also be installed to an already existing database (Postgres versions 9 and later), just by installing the RPM package and setting up the config and security files in the described way.

Outlook: pgoutput

While the decoderbufs plug-in is our recommended choice for a logical decoding plug-in, there are cases where you may not be able to use it. Most specifically, you typically don’t have the flexibility to install custom plug-ins in cloud-based environments such as Amazon RDS.

This is why we’re exploring a third alternative to decoderbufs and wal2sjon right now, which is to leverage Postgres logical replication mechanism. There’s a built-in plug-in, pgoutput based on this, which exists in every Postgres database since version 10. We’re still in the process of exploring the implications (and possible limitations) of using pgoutput, but so far things look promising and it may eventually be a valuable tool to have in the box.

Stay tuned for more details coming soon!

About Debezium

Debezium is an open source distributed platform that turns your existing databases into event streams, so applications can see and respond almost instantly to each committed row-level change in the databases. Debezium is built on top of Kafka and provides Kafka Connect compatible connectors that monitor specific database management systems. Debezium records the history of data changes in Kafka logs, so your application can be stopped and restarted at any time and can easily consume all of the events it missed while it was not running, ensuring that all events are processed correctly and completely. Debezium is open source under the Apache License, Version 2.0.

Get involved

We hope you find Debezium interesting and useful, and want to give it a try. Follow us on Twitter @debezium, chat with us on Gitter, or join our mailing list to talk with the community. All of the code is open source on GitHub, so build the code locally and help us improve ours existing connectors and add even more connectors. If you find problems or have ideas how we can improve Debezium, please let us know or log an issue.


Debezium 0.10.0.Beta1 Released

Another week, another Debezium release — I’m happy to announce the release of Debezium 0.10.0.Beta1!

Besides the upgrade to Apache Kafka 2.2.1 (DBZ-1316), this mostly fixes some bugs, including a regression to the MongoDB connector introduced in the Alpha2 release (DBZ-1317).

A very welcomed usability improvement is that the connectors will log a warning now if not at least one table is actually captured as per the whitelist/blacklist configuration (DBZ-1242). This helps to prevent the accidental exclusion all tables by means of an incorrect filter expression, in which case the connectors "work as intended", but no events are propagated to the message broker.

Please see the release notes for the complete list of issues fixed in this release. Also make sure to examine the upgrade guidelines for 0.10.0.Alpha1 and Alpha2 when upgrading from earlier versions.

Many thanks to community members Cheng Pan and Ching Tsai for their contributions to this release!

About Debezium

Debezium is an open source distributed platform that turns your existing databases into event streams, so applications can see and respond almost instantly to each committed row-level change in the databases. Debezium is built on top of Kafka and provides Kafka Connect compatible connectors that monitor specific database management systems. Debezium records the history of data changes in Kafka logs, so your application can be stopped and restarted at any time and can easily consume all of the events it missed while it was not running, ensuring that all events are processed correctly and completely. Debezium is open source under the Apache License, Version 2.0.

Get involved

We hope you find Debezium interesting and useful, and want to give it a try. Follow us on Twitter @debezium, chat with us on Gitter, or join our mailing list to talk with the community. All of the code is open source on GitHub, so build the code locally and help us improve ours existing connectors and add even more connectors. If you find problems or have ideas how we can improve Debezium, please let us know or log an issue.


back to top