Deploying the Debezium 1.0 Developer Preview with AMQ Streams

The following describes how to set up the Debezium 1.0.0 Developer Preview connectors for change data capture on Red Hat’s OpenShift container platform.

This and other documentation pages mention Debezium 1.0.0 Developer Preview, please note that this version corresponds to version 0.10.0.Beta2 of upstream Debezium project.

Apache Kafka Deployment

For setting up Apache Kafka and Apache Kafka Connect on OpenShift, Red Hat AMQ Streams can be used, which offers "Kafka as a Service". It consists of an enterprise grade operator and images that bring Kafka to OpenShift.

Follow these steps to prepare your Kafka cluster:

We now have a working Kafka cluster running within OpenShift with Kafka Connect s2i.

After a while and depending on your Kafka configuration, these pods should be running (assuming you used "debezium-kafka-cluster" as the AMQ Streams Kafka cluster name):

$ oc get pods

NAME                                                      READY     STATUS      RESTARTS   AGE
debezium-kafka-cluster-entity-operator-7b6b9d4c5f-k7b92   3/3       Running     0          5m
debezium-kafka-cluster-kafka-0                            2/2       Running     0          4m
debezium-kafka-cluster-zookeeper-0                        2/2       Running     0          4m
strimzi-cluster-operator-97cd5cf7b-l58bq                  1/1       Running     0          6m

In addition to running pods you should have a DeploymentConfig associated with your Connect s2i.

Instead of utilising the KafkaConnectS2I you can create a new Dockerfile based on AMQ’s Kafka image, including the required Debezium connectors. The precise instruction can also be found in AMQ Streams documentation. In this case the last item in the list above can be omitted.

Debezium Deployment

To deploy a Kafka Cluster with the Debezium Developer Preview connectors we need to

  • Download the connector archives

  • Download the associated database drivers

  • Prepare the plugin directory structure

  • Start the OpenShift s2i build to deploy the final Kafka Connect cluster

Downloading Database Drivers

The Debezium Developer Preview comes without database drivers included. The table below links to driver dependencies which need to be included in a directory of each repsective connector prior to triggering the s2i build.

Connector Driver

MySql

PostgreSQL

SQL Server

MongoDB

Preparing Directory Structure

Download the connector archives from Red Hat Developer Portal, extract the archives, and download the required driver denendencies. Once complete you should end with the following directory structure:

$ tree ./plugins
plugins
├── debezium-connector-mongodb
│   ├── CHANGELOG.md
│   ├── CONTRIBUTE.md
│   ├── COPYRIGHT.txt
│   ├── LICENSE-3rd-PARTIES.txt
│   ├── LICENSE.txt
│   ├── README.md
│   ├── bson-3.9.1.jar
│   ├── debezium-connector-mongodb-0.10.0.Beta1-redhat-00001.jar
│   ├── debezium-core-0.10.0.Beta1-redhat-00001.jar
│   ├── mongodb-driver-3.9.1.jar
│   └── mongodb-driver-core-3.9.1.jar
├── debezium-connector-mysql
│   ├── CHANGELOG.md
│   ├── CONTRIBUTE.md
│   ├── COPYRIGHT.txt
│   ├── LICENSE-3rd-PARTIES.txt
│   ├── LICENSE.txt
│   ├── README.md
│   ├── antlr4-runtime-4.7.0.redhat-00007.jar
│   ├── debezium-connector-mysql-0.10.0.Beta1-redhat-00001.jar
│   ├── debezium-core-0.10.0.Beta1-redhat-00001.jar
│   ├── debezium-ddl-parser-0.10.0.Beta1-redhat-00001.jar
│   ├── mysql-binlog-connector-java-0.19.1.redhat-00002.jar
│   └── mysql-connector-java-8.0.16.jar
├── debezium-connector-postgres
│   ├── CHANGELOG.md
│   ├── CONTRIBUTE.md
│   ├── COPYRIGHT.txt
│   ├── LICENSE-3rd-PARTIES.txt
│   ├── LICENSE.txt
│   ├── README.md
│   ├── debezium-connector-postgres-0.10.0.Beta1-redhat-00001.jar
│   ├── debezium-core-0.10.0.Beta1-redhat-00001.jar
│   ├── postgresql-42.2.5.jar
│   └── protobuf-java-2.6.1.redhat-1.jar
└── debezium-connector-sqlserver
    ├── CHANGELOG.md
    ├── CONTRIBUTE.md
    ├── COPYRIGHT.txt
    ├── LICENSE-3rd-PARTIES.txt
    ├── LICENSE.txt
    ├── README.md
    ├── debezium-connector-sqlserver-0.10.0.Beta1-redhat-00001.jar
    ├── debezium-core-0.10.0.Beta1-redhat-00001.jar
    └── mssql-jdbc-7.2.2.jre8.jar

Deploying Kafka Connect with the Debezium Developer Preview

With plugin directory structure ready, all we have to do now in order to deploy our Kafka Connect cluster is triggering the s2i build.

# Check the name of our build config
$ oc get buildconfigs

NAME                                     TYPE     FROM     LATEST
debezium-kafka-connect-cluster-connect   Source   Binary   2

# Build and deploy Kafka Connect cluster with Debezium
$ oc start-build debezium-kafka-connect-cluster-connect --from-dir=./plugins

After a while all parts should be up and running:

oc get pods

NAME                                                      READY     STATUS      RESTARTS   AGE
debezium-kafka-cluster-entity-operator-7b6b9d4c5f-k7b92   3/3       Running     0          10m
debezium-kafka-cluster-kafka-0                            2/2       Running     0          9m
debezium-kafka-cluster-zookeeper-0                        2/2       Running     0          9m
debezium-kafka-connect-cluster-connect-2-jw695            1/1       Running     0          1m
debezium-kafka-connect-cluster-connect-2-deploy           0/1       Completed   0          3m
strimzi-cluster-operator-97cd5cf7b-l58bq                  1/1       Running     0          11m

Alternatively, you can go to the "Pods" view of your OpenShift Web Console to confirm all pods are up and running:

openshift amq pods

Verifying the Deployment

Next we are going to verify whether the deployment is correct by emulating the Debezium Tutorial and following the steps in OpenShift Installation

You will want to change the pod names to correspond with your AMQ Streams deployment.

Questions and Resources

In case of any requests or questions related to running the Debezium Developer Preview with AMQ Streams on OpenShift, please let us know by sending an e-mail to debezium-cdc-preview mailing list.