For Cassandra connector users who may have been using Debezium Server or who may have wanted to use Debezium Server, we previously only shipped Cassandra 4 with the Debezium Server distribution. With Debezium 2.4, we now include all three Cassandra connector variants with the distribution, meaning that Cassandra 3 and DSE can now be used directly.

However, for this to work, a new environment variable EXTRA_CONNECTOR was introduced to specify specifically which Cassandra connector variant should be used by Debezium Server. This means that if you were using Cassandra 4 with Debezium Server, you must include this environment variable when upgrading to have the same configuration continue to work as it did in prior versions.

This new environment variable should be set to dse, cassandra-3, or cassandra-4 depending on the Cassandra version you intend to use for your Debezium Server source connector.

The Debezium for MySQL connector was not properly setting the precision for BIGINT data types when configuring the connector with bigint.unsigned.handling.mode as precise. Unfortunately, this led to a situation where the schema for such fields did not include the correct precision value.

Debezium 2.4 includes DBZ-6714, which provides a fix to address the incorrect precision for such fields. This can lead to schema incompatibilities when using schema registry, so you may need to adjust your compatibility settings or take other actions if you need to use strict compatibility rules.

Debezium 2.4 introduces a change in the default values for the snapshot.fetch.size and the query.fetch.size Oracle connector configuration properties. Previously, these properties used a default of 2000; however, thanks to a community contributor, it was identified that these values may likely be too low for production use.

With this release, the Oracle connector will now use a default of 10000 for both properties, which should have a positive improvement on throughput for most users who were not explicitly setting these values. If you were previously using custom values for these settings in your connector configurations, then you will not see a change in your existing behavior. Only users who previously were not explicitly setting these values will notice that the new defaults will be used.

For collations that end with the _bin designator, Vitess maps these to a data type of VARBINARY. As a result, the Vitess connector was inferring that these columns should be emitted as binary data; however, for character-based columns that used such collations, this was incorrect.

Debezium 2.4 will now properly emit character-based columns that are collated with a _bin designator as string-based data rather than binary data. This means that if you are using schema registry, you may observe somee schema incompatibilities and you may need to adjust your compatibility settings or take other actions to mitigate this change.

Incremental snapshots were first introduced nearly two years ago in Debezium 1.6 and has remained quite popular in the community to deal with a variety of re-snapshot use cases. However, there are some use cases where the intertwined nature of read events with create, updates, and deletes may be less than ideal or even not supported by some consumer application. For those use cases, Debezium 2.4 introduces ad-hoc blocking snapshots.

An ad-hoc blocking snapshot works in a similar way that ad-hoc incremental snapshots work; however, with one major difference. The snapshot is still triggered by sending a signal to Debezium; however when the signal is processed by the connector, the key difference is that streaming is put on hold while the snapshot process runs. This means you won’t be receiving a series of read events interwoven with create, update, or delete events. This also means that we’ll be processing the snapshot in a similar way to traditional snapshots, so the throughput should generally be higher than incremental snapshots.

The signal to initiate an ad-hoc blocking snapshot is very similar to its ad-hoc incremental snapshot counterpart. The following signal below shows the payload to snapshot a specific table with a condition, but this uses the new blocking snapshot rather than the incremental snapshot:

Normally column names map directly to field names and vice versa when consumed by sink connectors such as a JDBC connector. However, there are situations where the serialization technology, such as Avro, has very specific rules about field naming conventions. When a column’s name in a database table conflicts with the serialization rule’s naming conventions, Debezium will rename the field in the event so that it adheres to the serialization’s rules. This often means that a field will be prepended with underscores or invalid characters replaced with underscores.

This can create problems for certain types of sinks, such as a JDBC sink connector, because the sink cannot easily deduce the original column name for the destination table nor can it adequately map between the event’s field name and a column name if they differ. This typically means users must use transformation chains on the sink side in order to reconstruct the event’s fields with namings that represent the source.

Debezium 2.4 introduces a way to minimize and potentially avoid that entirely by propagating the original column name as a field schema parameter, much in the same way that we do for data types, precision, scale, and length. The schema parameter __debezium.source.column.name now includes the original column name when column or data type propagation is enabled.

In order to use IAM authentication on AWS, a special MySQL driver is required to provide that type of functionality. With Debezium 2.4, you can now provide a reference to this specific driver and the connector will use that driver instead of the default driver shipped with the connector.

As an example, to connect using IAM authentication on AWS, the following configuration is needed:

The database.jdbc.driver specifies the driver that should be loaded by the connector and used to communicate with the MySQL database. The database.jdbc.protocol is a supplemental configuration property that may not be required in all contexts. It defaults to jdbc:mysql but since AWS requires jdbc:mysql:aws, this allows you to specify this derivative within the configuration.

We’ve love to hear feedback and whether something like this might be useful for other scenarios.