The Debezium project strives to provide an easy deployment of connectors, so users can try and run connectors of their choice mostly by getting the right connector archive and unpacking it into the plug-in path of Kafka Connect.
This is true for all connectors but for the Debezium PostgreSQL connector. This connector is specific in the regard that it requires a logical decoding plug-in to be installed inside the PostgreSQL source database(s) themselves. Currently, there are two supported logical plug-ins:
These plug-ins can be consumed and deployed in two ways; the easiest one is to use one of our pre-made Postgres container images, which contain both plug-ins and are already configured as required. If you are using containers in your datacenter, and/or if you start a fresh database from scratch, then this can be a great option.
The other approach is building from source. Even if this is usually an easy task, it still brings a barrier to an easy start and requires a non-trivial knowledge of the Linux operating system.
To bridge the gap between those two extremes we’ve created and published an RPM package, available for Fedora 30 and later. By installing this package you will have the necessary binaries deployed, and the only task remaining is to configure PostgreSQL to enable the plug-in. The RPM is based on the latest stable Debezium release, 0.9.5.Final at this point.
Let’s show how the package works. We will use the Vagrant tool as an easy way for firing up a pre-provisioned virtual machine with Fedora. Of course, that’s not a requirement and the same steps apply for any other way of running Fedora.
Create and start virtual machine with Fedora 30:
$ vagrant init fedora/30-cloud-base A `Vagrantfile` has been placed in this directory. You are now ready to `vagrant up` your first virtual environment! Please read the comments in the Vagrantfile as well as documentation on `vagrantup.com` for more information on using Vagrant. $ vagrant up Bringing machine 'default' up with 'virtualbox' provider... . . . ==> default: Machine booted and ready!
Log into the virtual machine:
$ vagrant ssh
Install the PostgreSQL server and Protocol Buffers logical decoding plug-in:
$ sudo dnf -y install postgresql postgres-decoderbufs . . . Installed: postgres-decoderbufs-0.9.5-1.fc30.x86_64 postgresql-11.3-1.fc30.x86_64 postgis-2.5.1-1.fc30.x86_64 armadillo-9.400.4-1.fc30.x86_64 blas-3.8.0-12.fc30.x86_64 cairo-1.16.0-5.fc30.x86_64 cups-libs-1:2.2.11-2.fc30.x86_64 fontconfig-2.13.1-8.fc30.x86_64 lapack-3.8.0-12.fc30.x86_64 libgfortran-9.1.1-1.fc30.x86_64 libpq-11.3-2.fc30.x86_64 libquadmath-9.1.1-1.fc30.x86_64 mariadb-connector-c-3.0.10-1.fc30.x86_64 mariadb-connector-c-config-3.0.10-1.fc30.noarch nss-3.44.0-2.fc30.x86_64 nss-softokn-3.44.0-2.fc30.x86_64 nss-softokn-freebl-3.44.0-2.fc30.x86_64 nss-sysinit-3.44.0-2.fc30.x86_64 nss-util-3.44.0-2.fc30.x86_64 poppler-0.73.0-9.fc30.x86_64 postgresql-server-11.3-1.fc30.x86_64 proj-5.2.0-2.fc30.x86_64 proj-datumgrid-1.8-2.fc30.noarch uriparser-0.9.3-1.fc30.x86_64 SuperLU-5.2.1-6.fc30.x86_64 arpack-3.5.0-6.fc28.x86_64 atk-2.32.0-1.fc30.x86_64 avahi-libs-0.7-18.fc30.x86_64 cfitsio-3.450-3.fc30.x86_64 dejavu-fonts-common-2.37-1.fc30.noarch dejavu-sans-fonts-2.37-1.fc30.noarch fontpackages-filesystem-1.44-24.fc30.noarch freexl-1.0.5-3.fc30.x86_64 fribidi-1.0.5-2.fc30.x86_64 gdal-libs-2.3.2-7.fc30.x86_64 gdk-pixbuf2-2.38.1-1.fc30.x86_64 gdk-pixbuf2-modules-2.38.1-1.fc30.x86_64 geos-3.7.1-1.fc30.x86_64 giflib-5.1.9-1.fc30.x86_64 graphite2-1.3.10-7.fc30.x86_64 gtk-update-icon-cache-3.24.8-1.fc30.x86_64 gtk2-2.24.32-4.fc30.x86_64 harfbuzz-2.3.1-1.fc30.x86_64 hdf5-1.8.20-6.fc30.x86_64 hicolor-icon-theme-0.17-5.fc30.noarch jasper-libs-2.0.14-8.fc30.x86_64 jbigkit-libs-2.1-16.fc30.x86_64 lcms2-2.9-5.fc30.x86_64 libXcomposite-0.4.4-16.fc30.x86_64 libXcursor-1.1.15-5.fc30.x86_64 libXdamage-1.1.4-16.fc30.x86_64 libXfixes-5.0.3-9.fc30.x86_64 libXft-2.3.2-12.fc30.x86_64 libXi-1.7.9-9.fc30.x86_64 libXinerama-1.1.4-3.fc30.x86_64 libaec-1.0.4-1.fc30.x86_64 libdap-3.20.3-1.fc30.x86_64 libgeotiff-1.4.3-3.fc30.x86_64 libgta-1.0.9-2.fc30.x86_64 libjpeg-turbo-2.0.2-1.fc30.x86_64 libkml-1.3.0-19.fc30.x86_64 libspatialite-4.3.0a-11.fc30.x86_64 libtiff-4.0.10-4.fc30.x86_64 libwebp-1.0.2-2.fc30.x86_64 netcdf-184.108.40.206-12.fc30.x86_64 nspr-4.21.0-1.fc30.x86_64 ogdi-3.2.1-4.fc30.x86_64 openblas-0.3.5-5.fc30.x86_64 openblas-openmp-0.3.5-5.fc30.x86_64 openblas-serial-0.3.5-5.fc30.x86_64 openblas-threads-0.3.5-5.fc30.x86_64 openblas-threads64_-0.3.5-5.fc30.x86_64 openjpeg2-2.3.1-1.fc30.x86_64 pango-1.43.0-3.fc30.x86_64 pixman-0.38.0-1.fc30.x86_64 poppler-data-0.4.9-3.fc30.noarch protobuf-c-1.3.1-2.fc30.x86_64 unixODBC-2.3.7-4.fc30.x86_64 xerces-c-3.2.2-2.fc30.x86_64 Complete!
Next, initialize the database:
$ sudo /usr/bin/postgresql-setup --initdb
Now enable the plug-in in the database server configuration file
/var/lib/pgsql/data/postgresql.conf by adding the following parameters:
# MODULES shared_preload_libraries = 'decoderbufs' # REPLICATION wal_level = logical # minimal, archive, hot_standby, or logical (change requires restart) max_wal_senders = 8 # max number of walsender processes (change requires restart) wal_keep_segments = 4 # in logfile segments, 16MB each; 0 disables #wal_sender_timeout = 60s # in milliseconds; 0 disables max_replication_slots = 4 # max number of replication slots (change requires restart)
Configure the security file
/var/lib/pgsql/data/pg_hba.conf for the database user that will be used by Debezium (e.g.
debezium) by adding these parameters:
local replication debezium trust host replication debezium 127.0.0.1/32 trust host replication debezium ::1/128 trust
Finally, restart PostgreSQL:
$ sudo systemctl restart postgresql
And that’s it: Now we have a PostgreSQL database, that is ready to stream changes to the Debezium PostgreSQL connector. Of course, the plug-in can also be installed to an already existing database (Postgres versions 9 and later), just by installing the RPM package and setting up the config and security files in the described way.
While the decoderbufs plug-in is our recommended choice for a logical decoding plug-in, there are cases where you may not be able to use it. Most specifically, you typically don’t have the flexibility to install custom plug-ins in cloud-based environments such as Amazon RDS.
This is why we’re exploring a third alternative to decoderbufs and wal2sjon right now, which is to leverage Postgres logical replication mechanism. There’s a built-in plug-in, pgoutput based on this, which exists in every Postgres database since version 10. We’re still in the process of exploring the implications (and possible limitations) of using pgoutput, but so far things look promising and it may eventually be a valuable tool to have in the box.
Stay tuned for more details coming soon!
Debezium is an open source distributed platform that turns your existing databases into event streams, so applications can see and respond almost instantly to each committed row-level change in the databases. Debezium is built on top of Kafka and provides Kafka Connect compatible connectors that monitor specific database management systems. Debezium records the history of data changes in Kafka logs, so your application can be stopped and restarted at any time and can easily consume all of the events it missed while it was not running, ensuring that all events are processed correctly and completely. Debezium is open source under the Apache License, Version 2.0.
We hope you find Debezium interesting and useful, and want to give it a try. Follow us on Twitter @debezium, chat with us on Zulip, or join our mailing list to talk with the community. All of the code is open source on GitHub, so build the code locally and help us improve ours existing connectors and add even more connectors. If you find problems or have ideas how we can improve Debezium, please let us know or log an issue.