Featured Posts

Tags
apache kafka apache kafka apicurio avro aws batch caassandra camel cassandra cdc channels community community stories connectors containers cqrs custom datalake db2 ddd debezium debezium server debezium ui deduplication discussion docker elasticsearch event sourcing exactly once semantics example examples features fedora flink hiring ibmi iceberg informix integration introduction jaeger jdbc json kafka kafka streams kafka streams kogito ksql kubernetes lakehouse machine learning mariadb microservices mongo mongodb mysql news newsletter notifications online learning operator oracle outbox performance postgres presentation production quarkus questdb rds releases schema scylla secrets sentry serialization signaling smt snapshots spanner spark sql sqlserver tensorflow testcontainers tests time series timescaledb topics tracing transactions vagrant vitess website

As the year starts to come to a close, I am happy to announce the first maintenance release for the Debezium 2.4 release series, Debezium 2.4.1.Final.

While our maintenance releases focus primarily on bug fixes and stability improvements, there are a few new features we back ported from our most recent Debezium 2.5 development series. All these new features focus on the Debezium Operator for Kubernetes, so lets take a quick deep dive into those details. As always, you can find the complete list of changes for this release in the release notes.

Debezium Operator - Specify image pull secrets

There may be situations where the Debezium Server image may not contain all the necessary components for your chosen deployment, such as proprietary drivers that aren’t included the image. In such cases, it’s common practice for users to construct their own image from the base image to include such files. It’s then common to place that newly created image onto an image container registry where you may require credentials to fetch that image for your deployment. Debezium 2.4.1.Final introduces the ability to specify those credentials directly as a part of your CRD. (DBZ-6962)

An example CRD that now includes the new imagePullSecrets configuration:
runtime:
  templates:
    pod:
      imagePullSecrets:
      - name: debezium-server-secret

Debezium Operator - Specify container resources

With Debezium Server on Kubernetes, you may want to limit the resources that your pod consumes, particularly if you are streaming large volumes of data regularly. Debezium 2.4.1.Final introduces the ability to specify these constraints as part of the Pod template as well. (DBZ-7052)

An example CRD specifying Pod resource constraints
runtime:
  templates:
    container:
      resources:
        limits:
          memory: "64Mi"
          cpu: "250m"
        requests:
          memory: "128Mi"
          cpu: "500m"

Please see the Kubernetes documentation for more details on the supported properties.

Debezium Operator metrics endpoint improvements

Debezium 2.4.1.Final introduces a new JMX metrics endpoint for better metrics gathering and exposure with the Debezium Operator and Debezium Server. (DBZ-7053) This feature can be enabled in the custom resource definition by simply adding the following lines:

spec:
  runtime:
    jmx:
      enabled: true
      port: 1099

The endpoint currently does not support authentication nor SSL but those are on the road-map and will be added as a supplemental change to this feature in a future release.

Other fixes

In addition, there were quite a number of stability and bug fixes that made it into this release. These include the following:

  • Oracle RAC throws ORA-00310: archive log sequence required DBZ-5350

  • oracle missing CDC data DBZ-5656

  • Missing oracle cdc records DBZ-5750

  • Multiple debezium:offsets Redis clients DBZ-6952

  • Wrong case-behavior for non-avro column name in sink connector DBZ-6958

  • Missing events from Oracle 19c DBZ-6963

  • Handle properly bytea field for jdbc sink to postgresql DBZ-6967

  • Debezium jdbc sink process truncate event failure DBZ-6970

  • Debezium jdbc sink should throw not supporting schema change topic exception DBZ-6990

  • OLM bundle version for GA releases is invalid DBZ-6994

  • Further refactoring to correct downstream rendering of incremental snapshots topics DBZ-6997

  • ALTER TABLE fails when adding multiple columns to JDBC sink target DBZ-6999

  • Invalid Link to zulip chat in CSV metadata DBZ-7000

  • Make sure to terminate the task once connectivity is lost to either the rebalance or sync topic DBZ-7001

  • Missing .metadata.annotations.repository field in CSV metadata DBZ-7003

  • Single quote replication and loss of data DBZ-7006

  • Oracle connector: Payload size over 76020 bytes are getting truncated DBZ-7018

  • Use optional schema for Timezone Converter tests DBZ-7020

  • DDL statement couldn’t be parsed DBZ-7030

  • Debezium Embedded Infinispan Performs Slowly DBZ-7047

  • Update operator dependencies and add qosdk platform bom DBZ-7048

  • Field exclusion does not work with events of removed fields DBZ-7058

  • Consolidate resource labels and annotations DBZ-7064

  • Allow DS JMX to use username-password authentication on k8 DBZ-7087

  • Debezium crashes on parsing MySQL DDL statement (specific INSERT) DBZ-7119

  • Debezium-ddl-parser crashes on parsing MySQL DDL statement (specific UNION) DBZ-7140

Altogether, 30 issues were fixed for this release. A big thank you to all the contributors from the community who worked on this release: Anisha Mohanty, Bertrand Paquet, Bob Roldan, Chris Cranford, David Remy, Don Seiler, Harvey Yue, Jakub Cechacek, Jiri Pechanec, Mario Fiore Vitale, Nancy Xu, Robert Roldan, Sergey Eizner, Thomas Thornton, Vojtech Juranek, and adityajain311995!

What’s next?

As mentioned in our last release announcement, the cadence for Debezium 2.5 is condensed due to the upcoming holiday season. The next preview release for Debezium 2.5 will be our first and most likely only Beta release, later this month. We plan to conclude the Debezium 2.5 release series with a release candidate most likely the first week of December and a final release mid-way through December, just before the holiday break.

We are also moving forward on our review and process for MariaDB support. There will likely be some news on this in the coming weeks as we begin to find a path forward around this particular advancement. The team is also continuing the work on the Debezium Engine improvements, and much more. You can find all the details for our continued plans for Debezium 2.5 on our roadmap.

As always, please be sure to get in touch with us on the mailing list or Zulip chat if you have questions or feedback. Until next time, stay warm out there!

Chris Cranford

Chris is a software engineer at Red Hat. He previously was a member of the Hibernate ORM team and now works on Debezium. He lives in North Carolina just a few hours from Red Hat towers.

   

About Debezium

Debezium is an open source distributed platform that turns your existing databases into event streams, so applications can see and respond almost instantly to each committed row-level change in the databases. Debezium is built on top of Kafka and provides Kafka Connect compatible connectors that monitor specific database management systems. Debezium records the history of data changes in Kafka logs, so your application can be stopped and restarted at any time and can easily consume all of the events it missed while it was not running, ensuring that all events are processed correctly and completely. Debezium is open source under the Apache License, Version 2.0.

Get involved

We hope you find Debezium interesting and useful, and want to give it a try. Follow us on Twitter @debezium, chat with us on Zulip, or join our mailing list to talk with the community. All of the code is open source on GitHub, so build the code locally and help us improve ours existing connectors and add even more connectors. If you find problems or have ideas how we can improve Debezium, please let us know or log an issue.