We typically try to avoid any breaking changes, even during minor releases such as this; however, sometimes breaking changes are inevitable given the circumstances. Debezium 2.2.0.Alpha3 includes one breaking change:

Debezium’s relational database initial snapshot process has always been single-threaded. This limitation primarily stems from the complexities of ensuring data consistency across multiple transactions.

Starting in Debezium 2.2, we’re adding a new and initially optional way to utilize multiple threads to perform consistent database snapshot for a connector. This implementation uses these multiple threads to execute table-level snapshots in parallel.

In order to take advantage of this new feature, specify snapshot.max.threads in your connector’s configuration and when this property has a value greater than 1, parallel snapshots will be used.

In the example above, if the connector needs to snapshot more than 4 tables, there will be at most 4 tables being snapshot in parallel. When one thread finishes processing a table, it will get a new table to snapshot from the queue and the process continues until all tables have been snapshot.

Debezium presently subscribes to the MongoDB change stream and evaluates whether an event is of relevance or not on the connector side. On the surface, there is nothing technically wrong with this approach, it has worked well; however, a recent contributor explained how this decision impacts them.

Overall, the current process effectively serializes across the network all changes from MongoDB to the connector. If you have a lower volume of changes, you likely don’t see any issue with this approach; however, in a high volume scenario, especially when you’re only interested in a subset of the data generated by change streams, you quickly begin to see how this approach is inefficient. Furthermore, if you’re running the connector in a cloud environment like AWS, you’ll likely see in a high volume scenario where utilization costs could be impacted.

By moving where the include/exclude list filters are evaluated from the connector to the MongoDB server’s change stream subscription, this adds a number of advantages for all MongoDB connector users.

By reducing the number of events seen by connector, this impacts both network and CPU utilization. When events are sent that the connector simply discards due to include/exclude filters, this leads to network usage that could be avoided. When the connector is configured with full document or pre-image settings, this adds even more utilization to the network that is entirely unnecessary. Furthermore, by receiving more events than the connector configuration is interested in, this leads to the connector doing more processing, raising CPU utilization.

While network and CPU utilization are critical regardless of one’s environment, these are often more scrutinized when operating a cloud-based environments as these two metrics directly impact the operating budget. Users should see an overall lower network and CPU utilization with Debezium MongoDB 2.2 connectors.

We hope to share more details the benefits of this change in a future blog post, so stay tuned!

Debezium’s incremental snapshot feature has been a tremendous success. It provides an efficient way to perform a consist snapshot of data that can be resumed, which is critical when the snapshot consists of large volumes of data.

However, incremental snapshots do have specific requirements that must be met before the feature can be used. One of those requirements is all tables being snapshot must use a primary key. You may ask, why does a table have no primary key, and we aren’t going to debate that here today; however, suffice to say this occurs more often than you may think.

With Debezium 2.2, incremental snapshots can be performed on key-less tables as long as there is one column that is unique and can be considered a "surrogate key" for incremental snapshot purposes.

To provide the surrogate key column data in an incremental snapshot signal, the signal’s payload must include the new surrogate key attribute, surrogate-key.

In the above example, an incremental snapshot will be started for table public.mytab and the incremental snapshot will use the customer_ref column as the primary key for generating the snapshot windows.

However, the surrogate key feature isn’t just applicable for tables with no primary keys. There are a series of advantages when using this feature with tables that have primary keys:

There were quite a number of other improvements, bug fixes, and stability changes in this release, some noteworthy are:

Altogether, 33 issues were fixed for this release. A big thank you to all the contributors from the community who worked on this release: Gabor Andras, Anisha Mohanty, Bob Roldan, Bobby Tiernay, Chris Cranford, Eugene Abramchuk, Gabor Andras, Gunnar Morling, Harvey Yue, Jakub Cechacek, Jeremy Ford, Jiri Pechanec, Mehmet Firat Komurcu, Plugaru Tudor, Stefan Miklosovic, Subodh Kant Chaturvedi, Vojtech Juranek, and Xinbin Huang!

In addition, we are nearing the end of the Debezium 2.2 development cycle. Assuming no unexpected problems, we do intend to release Beta1 next week, followed by a release candidate two weeks thereafter. Our goal is to finalize the Debezium 2.2 release in late March or early April at the latest.

We would love to hear your feedback or suggestions about our roadmap, changes in this release, or any that are outstanding or that we may haven’t mentioned. Be sure to get in touch with us on the mailing list or our chat if there is.

Also, the DevNexus 2023 conference is coming up in early April in Atlanta, and I have the privilege to be a guest speaker discussing Debezium and CDC patterns. Be sure to check out that talk in person if you have an opportunity!

And finally, be on the lookout for our first installment of our 2023 Newsletter later this month. I also will be wrapping up the blog series, "Debezium for Oracle" where I cover performance, debugging, and frequently asked questions about the Oracle connector.

Until next time…​