Debezium Blog
Since introducing the Debezium Management Platform (Debezium Platform), our goal has been to make it easy to build CDC data pipelines so you can focus on how your data flows from sources to destinations. Many users already run Debezium connector on Kafka Connect or with Debezium Server. So to further ease the onboarding and jump start the process, we introduced the ability to reuse the configuration of any existing Debezium connector on Kafka Connect or...
SQL Server offers change data capture (CDC) through change tables - a special set of system tables that record modifications in selected “ordinary” tables. If you want to monitor changes in real time, you query these change tables periodically. That’s exactly how Debezium works today: it polls SQL Server’s change tables at configured intervals and turns the results into a continuous stream of CDC records. This approach works fine, but could we do better?
Captured tables are filled by the SQL Server Agent, which reads the transaction log, extracts changes, and stores them in change tables. In theory, we could skip the middleman and parse the transaction log directly. That’s how tools like OpenLogReplicator handle CDC for Oracle databases. Let’s peek inside the SQL Server internals and explore little bit how it works and stores the records.
In this post, we’ll:
-
Prepare a local SQL Server instance for experimentation
-
Explore the internal structure of the SQL Server transaction log
-
Understand how the records are stored on the disk
Although the end of summer is near, the Debezium team has a hot off the pressed preview release available with a fresh batch of improvements and enhancements. With Debezium 3.3.0.Beta1, this release brings a variety of stability fixes, performance optimizations, and user experience improvements across the connector ecosystem. Let’s take a look at what those are.
Debezium 3.2.2.Final delivers critical stability improvements, including a fix for potential data loss during failed ad-hoc blocking snapshots, resolution of confusing connector startup errors, and enhanced JMX throughput metrics for Oracle LogMiner.
Debezium 3.3.0.Alpha2 is out, bringing key fixes and powerful enhancements!
Highlights include heartbeat handling fixes, the ability to start MongoDB streaming from a precise oplog position, faster PostgreSQL TOAST performance, extended TSVECTOR support in the JDBC sink, and improved publication DDL handling in PostgreSQL. The Debezium Platform also gets major usability boosts with clearer error messages, fine-grained UI logging, and better source/destination definitions.
