Debezium Operator

Table of Contents

Overview
Deploy Debezium Operator using Helm
Tracking the progress of the operator deployment
DebeziumServer custom resource
DebeziumServer specification
Complete Debezium Server configuration example

Overview

The Debezium Operator is a Kubernetes operator that provides automated deployment and management of Debezium Server instances on Kubernetes and OpenShift platforms. This operator implements the standard Kubernetes operator pattern to transform user-defined DebeziumServer custom resources into running change data capture (CDC) pipelines that stream database changes to various sink systems.

Deploy Debezium Operator using Helm

Helm is the preferred method for installing the operator. The operator is available in the Debezium Helm Chart Repository. You can install the operator to an existing namespace, or to a new namespace.

Enter the following command to create a dedicated namespace:
```
kubectl create namespace debezium
```
Add the Debezium charts repository and install the operator by entering the following command:
```
helm repo add debezium https://charts.debezium.io
helm install my-debezium-operator debezium/debezium-operator --version 3.2.0-final -n debezium
```
Due to a versioning problem, you must currently specify a version number, even if you just want to install the last stable version.

You can find the available versions of the operator in the Debezium Helm Chart Repository.

Tracking the progress of the operator deployment

You can run commands to monitor the progress of the deployment and view the operator log.

Procedure

To monitor the progress of the deployment, run the following command:
```
kubectl get pod -n debezium --watch
```
To monitor the operator log, enter the following command:
```
kubectl logs deployment/debezium-operator -n debezium -f
```

After the operator is running, it watches for new custom resources and creates a Debezium Server instance that is based on those custom resources.

DebeziumServer custom resource

You specify the configuration for a Debezium Server instance by declaring properties in a DebeziumServer custom resource.

Example 1. Basic structure of the Debezium Server custom resource

apiVersion: debezium.io/v1alpha1
kind: DebeziumServer
metadata:
  name: my-debezium-server
spec:
  image: String
  version: String
  runtime:
    environment:
      vars:
        - name: String
          value: String
      from:  # EnvFromSource array
        - sourceName: String
          # add other EnvFromSource fields as needed
    storage:
      data:
        type: persistent | ephemeral
        claimName: String  # required if type is "persistent"
      external:  # Volume array
        - name: String
          # add other Volume fields as needed
    jmx:
      enabled: boolean
      port: int  # defaults to 1099
    templates:
      container:
        resources: ResourceRequirements
        securityContext: SecurityContext
      pod:
        metadata:
          annotations:  # Map<String, String>
            key: value
          labels:  # Map<String, String>
            key: value
        imagePullSecrets:  # List
          - name: String
        affinity: Affinity
        securityContext: PodSecurityContext
  quarkus:
    config:
      # quarkus properties
      format:
        value:
          type: String
          config:
            # format properties
      key:
        type: String
        config:
          # format properties
      header:
        type: String
        config:
          # format properties
  transforms:
    - type: String
      predicate: String
      negate: Boolean
      config:
        # transformation properties
  predicates:
    name:
      type: String
      config:
        # predicate properties
  sink:
    type: String
    config:
      # sink properties
  source:
    class: String
    config:
      # source connector properties

DebeziumServer specification

DebeziumServer schema reference

Table 1. DebeziumServer properties
Property	Type	Default	Description
`spec`	`DebeziumServerSpec`	No default value	The specification of Debezium Server
`status`	`DebeziumServerStatus`	No default value	The status of the Debezium Server instance.

DebeziumServerStatus schema reference

Used in: DebeziumServer

Table 2. DebeziumServerStatus properties
Property	Type	Default	Description
`conditions`	`List<Condition>`	No default value	List of status conditions
`observedGeneration`	Long	0	Latest observed generation

Condition schema reference

Used in: DebeziumServerStatus

Table 3. Condition properties
Property	Type	Default	Description
`status`	String	No default value	The status of the condition, either True, False or Unknown.
`message`	String	No default value	Human-readable message indicating details about the condition’s last transition.
`type`	String	No default value	Unique identifier of a condition.

DebeziumServerSpec schema reference

Used in: DebeziumServer

Table 4. DebeziumServerSpec properties
Property	Type	Default	Description
`image`	String	No default value	Image used for Debezium Server container. This property takes precedence over version.
`version`	String	same as operator	Version of Debezium Server to be used.
`sink`	`Sink`	No default value	Sink configuration.
`source`	`Source`	No default value	Debezium source connector configuration.
`format`	`Format`	No default value	Message output format configuration.
`quarkus`	`Quarkus`	No default value	Quarkus configuration passed to the Debezium Server process.
`runtime`	`Runtime`	No default value	Specifies whether you can modify various aspects of the Debezium Server runtime.
`transforms`	`List<Transformation>`	No default value	Single Message Transformations employed by this instance of Debezium Server.
`predicates`	`Map<String, Predicate>`	No default value	Predicates employed by this instance of Debezium Server.

Sink schema reference

Used in: DebeziumServerSpec

Table 5. Sink properties
Property	Type	Default	Description
`type`	String	No default value	Sink type recognized by this Debezium Server instance.
`config`	Map	No default value	Sink configuration properties.

Source schema reference

Used in: DebeziumServerSpec

Table 6. Source properties
Property	Type	Default	Description
`sourceClass`	String	No default value	Fully qualified name of source connector Java class.
`offset`	`Offset`	No default value	Offset store configuration
`schemaHistory`	`SchemaHistory`	No default value	Schema history store configuration
`config`	Map	No default value	Source connector configuration properties.

Format schema reference

Used in: DebeziumServerSpec

Table 7. Format properties
Property	Type	Default	Description
`key`	`FormatType`	No default value	Message key format configuration.
`value`	`FormatType`	No default value	Message value format configuration.
`header`	`FormatType`	No default value	Message header format configuration.

FormatType schema reference

Used in: Format

Table 8. FormatType properties
Property	Type	Default	Description
`type`	String	json	Format type recognized by Debezium Server.
`config`	Map	No default value	Format configuration properties.

Quarkus schema reference

Used in: DebeziumServerSpec

Table 9. Quarkus properties
Property	Type	Default	Description
`config`	Map	No default value	Quarkus configuration properties.

Runtime schema reference

Used in: DebeziumServerSpec

Table 10. Runtime properties
Property	Type	Default	Description
`api`	`RuntimeApi`	No default value	API configuration
`storage`	`RuntimeStorage`	No default value	Storage configuration
`environment`	`RuntimeEnvironment`	No default value	Additional environment variables used by this Debezium Server instance.
`jmx`	`JmxConfig`	No default value	JMX configuration.
`templates`	`Templates`	No default value	Debezium Server resource templates.
`serviceAccount`	String	No default value	An existing service account used to run the Debezium Server pod
`metrics`	`Metrics`	No default value	Metrics configuration

RuntimeApi schema reference

Used in: Runtime

Table 11. RuntimeApi properties
Property	Type	Default	Description
`enabled`	boolean	No default value	Whether the API should be enabled for this instance of Debezium Server
`port`	int	8080	Port number used by the k8s service exposing the API

RuntimeStorage schema reference

Used in: Runtime

Table 12. RuntimeStorage properties
Property	Type	Default	Description
`data`	`DataStorage`	No default value	File storage configuration used by this instance of Debezium Server.
`external`	`List<Volume>`	No default value	Additional volumes mounted to /debezium/external

RuntimeEnvironment schema reference

Used in: Runtime

Table 13. RuntimeEnvironment properties
Property	Type	Default	Description
`vars`	`List<ContainerEnvVar>`	No default value	Environment variables applied to the container.
`from`	`List<EnvFromSource>`	No default value	Additional environment variables set from ConfigMaps or Secrets in containers.

JmxConfig schema reference

Used in: Runtime

Table 14. JmxConfig properties
Property	Type	Default	Description
`enabled`	boolean	false	Whether JMX should be enabled for this Debezium Server instance.
`port`	int	1099	JMX port.
`authentication`	`JmxAuthentication`	No default value	JMX authentication config.

JmxAuthentication schema reference

Used in: JmxConfig

Table 15. JmxAuthentication properties
Property	Type	Default	Description
`enabled`	boolean	false	Whether JMX authentication should be enabled for this Debezium Server instance.
`secret`	String	No default value	Secret providing credential files
`accessFile`	String	jmxremote.access	JMX access file name and secret key
`passwordFile`	String	jmxremote.password	JMX password file name and secret key

Templates schema reference

Used in: Runtime

Table 16. Templates properties
Property	Type	Default	Description
`container`	`ContainerTemplate`	No default value	Container template
`pod`	`PodTemplate`	No default value	Pod template.
`volumeClaim`	`PersistentVolumeClaimSpec`	No default value	PVC template for data volume if no explicit claim is specified.

Metrics schema reference

Used in: Runtime

Table 17. Metrics properties
Property	Type	Default	Description
`jmxExporter`	`JmxExporter`	No default value	Prometheus JMX exporter configuration

JmxExporter schema reference

Used in: Metrics

Table 18. JmxExporter properties
Property	Type	Default	Description
`enabled`	boolean	No default value	Enables JMX Prometheus exporter
`configFrom`	ConfigMapKeySelector	No default value	Config map key reference which value will be used as configuration file

Transformation schema reference

Used in: DebeziumServerSpec

Table 19. Transformation properties
Property	Type	Default	Description
`type`	String	No default value	Fully qualified name of Java class implementing the transformation.
`config`	Map	No default value	Transformation specific configuration properties.
`predicate`	String	No default value	The name of the predicate to be applied to this transformation.
`negate`	boolean	false	Determines if the result of the applied predicate will be negated.

Predicate schema reference

Used in: DebeziumServerSpec

Table 20. Predicate properties
Property	Type	Default	Description
`type`	String	No default value	Fully qualified name of Java class implementing the predicate.
`config`	Map	No default value	Predicate configuration properties.

ConfigMapOffsetStore schema reference

Used in: Offset

Table 21. ConfigMapOffsetStore properties
Property	Type	Default	Description
`name`	String	No default value	Name of the offset config map
`config`	Map	No default value	Additional store configuration properties.

ContainerEnvVar schema reference

Used in: RuntimeEnvironment

Table 22. ContainerEnvVar properties
Property	Type	Default	Description
`name`	String	No default value	The environment variable name.
`value`	String	No default value	The environment variable value.

ContainerTemplate schema reference

Used in: Templates

Table 23. ContainerTemplate properties
Property	Type	Default	Description
`resources`	`ResourceRequirements`	No default value	CPU and memory resource requirements.
`securityContext`	`SecurityContext`	No default value	Container security context.
`probes`	`Probes`	No default value	Container probes configuration.

CustomStore schema reference

Used in: Offset, SchemaHistory

Table 24. CustomStore properties
Property	Type	Default	Description
`type`	String	No default value	Fully qualified name of Java class implementing the store.
`config`	Map	No default value	Store configuration properties.

DataStorage schema reference

Used in: RuntimeStorage

Table 25. DataStorage properties
Property	Type	Default	Description
`type`	ephemeral,persistent	ephemeral	Storage type.
`claimName`	String	No default value	Name of persistent volume claim for persistent storage.

FileOffsetStore schema reference

Used in: Offset

Table 26. FileOffsetStore properties
Property	Type	Default	Description
`fileName`	String	No default value	Name of the offset file (relative to data root)
`config`	Map	No default value	Additional store configuration properties.

FileSchemaHistoryStore schema reference

Used in: SchemaHistory

Table 27. FileSchemaHistoryStore properties
Property	Type	Default	Description
`fileName`	String	No default value	Name of the offset file (relative to data root)
`config`	Map	No default value	Additional store configuration properties.

InMemoryOffsetStore schema reference

Used in: Offset

Table 28. InMemoryOffsetStore properties
Property	Type	Default	Description
`config`	Map	No default value	Additional store configuration properties.

InMemorySchemaHistoryStore schema reference

Used in: SchemaHistory

Table 29. InMemorySchemaHistoryStore properties
Property	Type	Default	Description
`config`	Map	No default value	Additional store configuration properties.

JdbcOffsetStore schema reference

Used in: Offset

Table 30. JdbcOffsetStore properties
Property	Type	Default	Description
`table`	`JdbcOffsetTableConfig`	No default value	The configuration of the offset table
`url`	String	No default value	JDBC connection URL
`user`	String	No default value	Username used to connect to the storage database
`password`	String	No default value	Password used to connect to the storage database
`retryDelay`	long	No default value	Retry delay on connection failure (in milliseconds)
`maxRetries`	int	No default value	Maximum number of retries on connection failure
`config`	Map	No default value	Additional store configuration properties.

JdbcOffsetTableConfig schema reference

Used in: JdbcOffsetStore

Table 31. JdbcOffsetTableConfig properties
Property	Type	Default	Description
`name`	String	No default value	The name of the offset table
`ddl`	String	No default value	DDL statement to create the offset table
`select`	String	No default value	Statement used to select from the offset table
`insert`	String	No default value	Statement used to insert into the offset table
`delete`	String	No default value	Statement used to update the offset table

JdbcSchemaHistoryStore schema reference

Used in: SchemaHistory

Table 32. JdbcSchemaHistoryStore properties
Property	Type	Default	Description
`table`	`JdbcSchemaHistoryTableConfig`	No default value	The configuration of the offset table
`url`	String	No default value	JDBC connection URL
`user`	String	No default value	Username used to connect to the storage database
`password`	String	No default value	Password used to connect to the storage database
`retryDelay`	long	No default value	Retry delay on connection failure (in milliseconds)
`maxRetries`	int	No default value	Maximum number of retries on connection failure
`config`	Map	No default value	Additional store configuration properties.

JdbcSchemaHistoryTableConfig schema reference

Used in: JdbcSchemaHistoryStore

Table 33. JdbcSchemaHistoryTableConfig properties
Property	Type	Default	Description
`name`	String	No default value	The name of the offset table
`ddl`	String	No default value	DDL statement to create the schema history table
`select`	String	No default value	Statement used to select from the schema history table
`insert`	String	No default value	Statement used to insert into the schema history table
`dataExistsSelect`	String	No default value	Statement used to check existence of some data in the schema history table

KafkaOffsetStore schema reference

Used in: Offset

Table 34. KafkaOffsetStore properties
Property	Type	Default	Description
`props`	Map	No default value	Additional Kafka client properties.
`bootstrapServers`	String	No default value	A list of host/port pairs that the connector uses for establishing an initial connection to the Kafka cluster
`topic`	String	No default value	The name of the Kafka topic where offsets are to be stored
`partitions`	int	No default value	The number of partitions used when creating the offset storage topic
`replicationFactor`	int	No default value	Replication factor used when creating the offset storage topic
`config`	Map	No default value	Additional store configuration properties.

KafkaSchemaHistoryStore schema reference

Used in: SchemaHistory

Table 35. KafkaSchemaHistoryStore properties
Property	Type	Default	Description
`bootstrapServers`	String	No default value	A list of host/port pairs that the connector uses for establishing an initial connection to the Kafka cluster
`topic`	String	No default value	The name of the Kafka topic where offsets are to be stored
`partitions`	int	No default value	The number of partitions used when creating the offset storage topic
`replicationFactor`	int	No default value	Replication factor used when creating the offset storage topic
`config`	Map	No default value	Additional store configuration properties.

MetadataTemplate schema reference

Used in: PodTemplate

Table 36. MetadataTemplate properties
Property	Type	Default	Description
`labels`	Map<String, String>	No default value	Labels added to the Kubernetes resource
`annotations`	Map<String, String>	No default value	Annotations added to the Kubernetes resource

Offset schema reference

Used in: Source

Table 37. Offset properties
Property	Type	Default	Description
`file`	`FileOffsetStore`	No default value	File backed offset store configuration
`memory`	`InMemoryOffsetStore`	No default value	Memory backed offset store configuration
`redis`	`RedisOffsetStore`	No default value	Redis backed offset store configuration
`kafka`	`KafkaOffsetStore`	No default value	Kafka backing store configuration
`jdbc`	`JdbcOffsetStore`	No default value	JDBC backing store configuration
`configMap`	`ConfigMapOffsetStore`	No default value	Config map backed offset store configuration
`store`	`CustomStore`	No default value	Arbitrary offset store configuration
`flushMs`	long	60000	Interval at which to try commiting offsets

PodTemplate schema reference

Used in: Templates

Table 38. PodTemplate properties
Property	Type	Default	Description
`metadata`	`MetadataTemplate`	No default value	Metadata applied to the resource.
`imagePullSecrets`	`List<LocalObjectReference>`	No default value	List of local references to secrets used for pulling any of the images used by this Pod.
`affinity`	`Affinity`	No default value	Pod affinity rules
`securityContext`	`PodSecurityContext`	No default value	Pod-level security attributes and container settings

Probe schema reference

Used in: Probes

Table 39. Probe properties
Property	Type	Default	Description
`initialDelaySeconds`	int	5	Number of seconds after the container has started before probes are initiated.
`periodSeconds`	int	10	How often (in seconds) to perform the probe.
`timeoutSeconds`	int	10	Number of seconds after which the probe times out.
`failureThreshold`	int	3	Number of failures in a row before the overall check has failed.

Probes schema reference

Used in: ContainerTemplate

Table 40. Probes properties
Property	Type	Default	Description
`readiness`	`Probe`	No default value	Readiness probe configuration applied to the container.
`liveness`	`Probe`	No default value	Liveness probe configuration applied to the container.

RedisOffsetStore schema reference

Used in: Offset

Table 41. RedisOffsetStore properties
Property	Type	Default	Description
`address`	String	No default value	Redis host:port used to connect
`user`	String	No default value	Redis username
`password`	String	No default value	Redis password
`sslEnabled`	boolean	false	Redis username
`key`	String	No default value	Redis hash key
`wait`	`RedisStoreWaitConfig`	No default value	Configures verification of replica writes
`config`	Map	No default value	Additional store configuration properties.

RedisSchemaHistoryStore schema reference

Used in: SchemaHistory

Table 42. RedisSchemaHistoryStore properties
Property	Type	Default	Description
`address`	String	No default value	Redis host:port used to connect
`user`	String	No default value	Redis username
`password`	String	No default value	Redis password
`sslEnabled`	boolean	false	Redis username
`key`	String	No default value	Redis hash key
`wait`	`RedisStoreWaitConfig`	No default value	Configures verification of replica writes
`config`	Map	No default value	Additional store configuration properties.

RedisStoreWaitConfig schema reference

Used in: RedisOffsetStore, RedisSchemaHistoryStore

Table 43. RedisStoreWaitConfig properties
Property	Type	Default	Description
`enabled`	boolean	false	In case of Redis with replica, this allows to verify that the data has been written to replica
`timeoutMs`	long	1000	Timeout in ms when waiting for replica
`retry`	boolean	false	Enables retry on wait for replica
`retryDelayMs`	long	1000	Delay of retry on wait

SchemaHistory schema reference

Used in: Source

Table 44. SchemaHistory properties
Property	Type	Default	Description
`file`	`FileSchemaHistoryStore`	No default value	File backed schema history store configuration
`memory`	`InMemorySchemaHistoryStore`	No default value	Memory backed schema history store configuration
`redis`	`RedisSchemaHistoryStore`	No default value	Redis backed schema history store configuration
`kafka`	`KafkaSchemaHistoryStore`	No default value	Kafka backed schema history store configuration
`jdbc`	`JdbcSchemaHistoryStore`	No default value	JDBC backed schema history store configuration
`store`	`CustomStore`	No default value	Arbitrary schema history store configuration
`config`	Map	No default value	Additional common schema history store configuration properties.

Complete Debezium Server configuration example

The following example shows a complete Debezium Server custom resource that incorporates properties from the preceding tables.

Example 2. Complete Debezium server custom resource

apiVersion: debezium.io/v1alpha1
kind: DebeziumServer
metadata:
  name: my-debezium-server
spec:
  version: "3.2.0"
  source:
    class: io.debezium.connector.mysql.MySqlConnector
    config:
      database.hostname: mysql-server
      database.port: 3306
      database.user: debezium
      database.password: secret
      database.server.id: 184054
      topic.prefix: mysql
      database.include.list: inventory
    offset:
      type: kafka
      config:
        bootstrap.servers: kafka:9092
        topic: debezium-offsets
    schemaHistory:
      type: kafka
      config:
        bootstrap.servers: kafka:9092
        topic: debezium-schema-history
  sink:
    type: kafka
    config:
      bootstrap.servers: kafka:9092
  format:
    key:
      type: json
    value:
      type: json
      config:
        schemas.enable: false
  runtime:
    jmx:
      enabled: true
      port: 1099
    storage:
      data:
        type: persistent
        claimName: debezium-data-pvc
    metrics:
      jmxExporter:
        enabled: true
  transforms:
    - type: io.debezium.transforms.UnwrapFromEnvelope
      config:
        drop.tombstones: false
      predicate: inventory-filter
      negate: false
  predicates:
    inventory-filter:
      type: org.apache.kafka.connect.transforms.predicates.TopicNameMatches
      config:
        pattern: "mysql.inventory.*"
  quarkus:
    config:
      quarkus.log.level: INFO
      quarkus.http.port: 8080