Featured Posts

Tags
apache kafka apache kafka apicurio avro aws batch caassandra camel cassandra cdc channels community community stories connectors containers cqrs custom datalake db2 ddd debezium debezium server debezium ui deduplication discussion docker elasticsearch event sourcing exactly once semantics example examples features fedora flink hiring ibmi iceberg informix integration introduction jaeger jdbc json kafka kafka streams kafka streams kogito ksql kubernetes lakehouse machine learning mariadb microservices mongo mongodb mysql news newsletter notifications online learning operator oracle outbox performance postgres presentation production quarkus questdb rds releases schema scylla secrets sentry serialization signaling smt snapshots spanner spark sql sqlserver tensorflow testcontainers tests time series timescaledb topics tracing transactions vagrant vitess website

The Debezium project strives to provide an easy deployment of connectors, so users can try and run connectors of their choice mostly by getting the right connector archive and unpacking it into the plug-in path of Kafka Connect.

This is true for all connectors but for the Debezium PostgreSQL connector. This connector is specific in the regard that it requires a logical decoding plug-in to be installed inside the PostgreSQL source database(s) themselves. Currently, there are two supported logical plug-ins:

  • postgres-decoderbufs, which uses Protocol Buffers as a very compact transport format and which is maintained by the Debezium community

  • JSON-based, which is based on JSON and which is maintained by its own upstream community

These plug-ins can be consumed and deployed in two ways; the easiest one is to use one of our pre-made Postgres container images, which contain both plug-ins and are already configured as required. If you are using containers in your datacenter, and/or if you start a fresh database from scratch, then this can be a great option.

The other approach is building from source. Even if this is usually an easy task, it still brings a barrier to an easy start and requires a non-trivial knowledge of the Linux operating system.

To bridge the gap between those two extremes we’ve created and published an RPM package, available for Fedora 30 and later. By installing this package you will have the necessary binaries deployed, and the only task remaining is to configure PostgreSQL to enable the plug-in. The RPM is based on the latest stable Debezium release, 0.9.5.Final at this point.

Example

Let’s show how the package works. We will use the Vagrant tool as an easy way for firing up a pre-provisioned virtual machine with Fedora. Of course, that’s not a requirement and the same steps apply for any other way of running Fedora.

Create and start virtual machine with Fedora 30:

$ vagrant init fedora/30-cloud-base

A `Vagrantfile` has been placed in this directory. You are now
ready to `vagrant up` your first virtual environment! Please read
the comments in the Vagrantfile as well as documentation on
`vagrantup.com` for more information on using Vagrant.

$ vagrant up

Bringing machine 'default' up with 'virtualbox' provider...
.
.
.
==> default: Machine booted and ready!

Log into the virtual machine:

$ vagrant ssh

Install the PostgreSQL server and Protocol Buffers logical decoding plug-in:

$ sudo dnf -y install postgresql postgres-decoderbufs
.
.
.
Installed:
  postgres-decoderbufs-0.9.5-1.fc30.x86_64              postgresql-11.3-1.fc30.x86_64
  postgis-2.5.1-1.fc30.x86_64                           armadillo-9.400.4-1.fc30.x86_64
  blas-3.8.0-12.fc30.x86_64                             cairo-1.16.0-5.fc30.x86_64
  cups-libs-1:2.2.11-2.fc30.x86_64                      fontconfig-2.13.1-8.fc30.x86_64
  lapack-3.8.0-12.fc30.x86_64                           libgfortran-9.1.1-1.fc30.x86_64
  libpq-11.3-2.fc30.x86_64                              libquadmath-9.1.1-1.fc30.x86_64
  mariadb-connector-c-3.0.10-1.fc30.x86_64              mariadb-connector-c-config-3.0.10-1.fc30.noarch
  nss-3.44.0-2.fc30.x86_64                              nss-softokn-3.44.0-2.fc30.x86_64
  nss-softokn-freebl-3.44.0-2.fc30.x86_64               nss-sysinit-3.44.0-2.fc30.x86_64
  nss-util-3.44.0-2.fc30.x86_64                         poppler-0.73.0-9.fc30.x86_64
  postgresql-server-11.3-1.fc30.x86_64                  proj-5.2.0-2.fc30.x86_64
  proj-datumgrid-1.8-2.fc30.noarch                      uriparser-0.9.3-1.fc30.x86_64
  SuperLU-5.2.1-6.fc30.x86_64                           arpack-3.5.0-6.fc28.x86_64
  atk-2.32.0-1.fc30.x86_64                              avahi-libs-0.7-18.fc30.x86_64
  cfitsio-3.450-3.fc30.x86_64                           dejavu-fonts-common-2.37-1.fc30.noarch
  dejavu-sans-fonts-2.37-1.fc30.noarch                  fontpackages-filesystem-1.44-24.fc30.noarch
  freexl-1.0.5-3.fc30.x86_64                            fribidi-1.0.5-2.fc30.x86_64
  gdal-libs-2.3.2-7.fc30.x86_64                         gdk-pixbuf2-2.38.1-1.fc30.x86_64
  gdk-pixbuf2-modules-2.38.1-1.fc30.x86_64              geos-3.7.1-1.fc30.x86_64
  giflib-5.1.9-1.fc30.x86_64                            graphite2-1.3.10-7.fc30.x86_64
  gtk-update-icon-cache-3.24.8-1.fc30.x86_64            gtk2-2.24.32-4.fc30.x86_64
  harfbuzz-2.3.1-1.fc30.x86_64                          hdf5-1.8.20-6.fc30.x86_64
  hicolor-icon-theme-0.17-5.fc30.noarch                 jasper-libs-2.0.14-8.fc30.x86_64
  jbigkit-libs-2.1-16.fc30.x86_64                       lcms2-2.9-5.fc30.x86_64
  libXcomposite-0.4.4-16.fc30.x86_64                    libXcursor-1.1.15-5.fc30.x86_64
  libXdamage-1.1.4-16.fc30.x86_64                       libXfixes-5.0.3-9.fc30.x86_64
  libXft-2.3.2-12.fc30.x86_64                           libXi-1.7.9-9.fc30.x86_64
  libXinerama-1.1.4-3.fc30.x86_64                       libaec-1.0.4-1.fc30.x86_64
  libdap-3.20.3-1.fc30.x86_64                           libgeotiff-1.4.3-3.fc30.x86_64
  libgta-1.0.9-2.fc30.x86_64                            libjpeg-turbo-2.0.2-1.fc30.x86_64
  libkml-1.3.0-19.fc30.x86_64                           libspatialite-4.3.0a-11.fc30.x86_64
  libtiff-4.0.10-4.fc30.x86_64                          libwebp-1.0.2-2.fc30.x86_64
  netcdf-4.4.1.1-12.fc30.x86_64                         nspr-4.21.0-1.fc30.x86_64
  ogdi-3.2.1-4.fc30.x86_64                              openblas-0.3.5-5.fc30.x86_64
  openblas-openmp-0.3.5-5.fc30.x86_64                   openblas-serial-0.3.5-5.fc30.x86_64
  openblas-threads-0.3.5-5.fc30.x86_64                  openblas-threads64_-0.3.5-5.fc30.x86_64
  openjpeg2-2.3.1-1.fc30.x86_64                         pango-1.43.0-3.fc30.x86_64
  pixman-0.38.0-1.fc30.x86_64                           poppler-data-0.4.9-3.fc30.noarch
  protobuf-c-1.3.1-2.fc30.x86_64                        unixODBC-2.3.7-4.fc30.x86_64
  xerces-c-3.2.2-2.fc30.x86_64

Complete!

Next, initialize the database:

$ sudo /usr/bin/postgresql-setup --initdb

Now enable the plug-in in the database server configuration file /var/lib/pgsql/data/postgresql.conf by adding the following parameters:

# MODULES
shared_preload_libraries = 'decoderbufs'

# REPLICATION
wal_level = logical             # minimal, archive, hot_standby, or logical (change requires restart)
max_wal_senders = 8             # max number of walsender processes (change requires restart)
wal_keep_segments = 4           # in logfile segments, 16MB each; 0 disables
#wal_sender_timeout = 60s       # in milliseconds; 0 disables
max_replication_slots = 4       # max number of replication slots (change requires restart)

Configure the security file /var/lib/pgsql/data/pg_hba.conf for the database user that will be used by Debezium (e.g. debezium) by adding these parameters:

local   replication     debezium                          trust
host    replication     debezium  127.0.0.1/32            trust
host    replication     debezium  ::1/128                 trust

Finally, restart PostgreSQL:

$ sudo systemctl restart postgresql

And that’s it: Now we have a PostgreSQL database, that is ready to stream changes to the Debezium PostgreSQL connector. Of course, the plug-in can also be installed to an already existing database (Postgres versions 9 and later), just by installing the RPM package and setting up the config and security files in the described way.

Outlook: pgoutput

While the decoderbufs plug-in is our recommended choice for a logical decoding plug-in, there are cases where you may not be able to use it. Most specifically, you typically don’t have the flexibility to install custom plug-ins in cloud-based environments such as Amazon RDS.

This is why we’re exploring a third alternative to decoderbufs and wal2sjon right now, which is to leverage Postgres logical replication mechanism. There’s a built-in plug-in, pgoutput based on this, which exists in every Postgres database since version 10. We’re still in the process of exploring the implications (and possible limitations) of using pgoutput, but so far things look promising and it may eventually be a valuable tool to have in the box.

Stay tuned for more details coming soon!

Jiri Pechanec

Jiri is a software developer (and a former quality engineer) at Red Hat. He spent most of his career with Java and system integration projects and tasks. He lives near Brno, Czech Republic.

   

About Debezium

Debezium is an open source distributed platform that turns your existing databases into event streams, so applications can see and respond almost instantly to each committed row-level change in the databases. Debezium is built on top of Kafka and provides Kafka Connect compatible connectors that monitor specific database management systems. Debezium records the history of data changes in Kafka logs, so your application can be stopped and restarted at any time and can easily consume all of the events it missed while it was not running, ensuring that all events are processed correctly and completely. Debezium is open source under the Apache License, Version 2.0.

Get involved

We hope you find Debezium interesting and useful, and want to give it a try. Follow us on Twitter @debezium, chat with us on Zulip, or join our mailing list to talk with the community. All of the code is open source on GitHub, so build the code locally and help us improve ours existing connectors and add even more connectors. If you find problems or have ideas how we can improve Debezium, please let us know or log an issue.