Databases have had support for storing binary data since the beginning, and there are a number of reasons that applications favor using a database for this over other alternatives. Binary data doesn’t always have to be information like images, thumbnails, or binary documents like PDFs, it can also include serialized objects too.

When using the outbox pattern with MongoDB and Avro serialization, the outbox’s payload field is the only value that is emitted in the message’s value. When the payload consists of binary data, as shown below, it is emitted as-is and can lead to some serialization issues with consumers of the topic.

In this release, we’ve deprecated the ByteBufferConverter and we’ve introduced two new converters in its place:

For this use case, we’re going to make use of the new ByteArrayConverter as we want to emit this payload’s value as the raw byte[]. In the connector configuration, the value.converter configuration option must be set to handle this use case.

Now the data will be emitted to the broker as-is, as a byte-array that can be safely consumed by consumers. Thanks to Nathan Bradshaw for this excellent contribution!

The heartbeat action query is a feature that allows a Debezium connector to write records to the source database and to capture those records during the event processing loop. This was first introduced for PostgreSQL to deal with situations when captured tables change less frequently than other non-captured tables, causing unintended WAL growth.

For Oracle using the LogMiner implementation, a similar problem occurs that impacts the connector’s ability to restart due to the offset SCN not being advanced on a regular interval due to changes being made to other tables that are not captured, whether they are part of the same pluggable database or another.

To enable heartbeat action queries, the connector must be configured with:

The heartbeat functionality must first be enabled by specifying the heartbeat.interval.ms. This controls how often the connector generates heartbeat events. If this value is not greater-than 0, then heartbeats are disabled.

Next, to specifically use the action query feature, the heartbeat.action.query option must be given. This specifies a SQL statement that will be executed on each heartbeat interval. This statement can be either an INSERT or an UPDATE, as long as the resulting SQL operation generates a row change.

Finally, the action query must operate on a table that is included in the connector’s filter configuration. Like any other captured table, the table must also be configured with the correct supplemental logging so that the event is captured.

With this configuration in place, and assuming no long-running transaction, the offset SCN will advance on each heartbeat.

The Debezium Oracle connector’s LogMiner session duration has always been based on how often the redo log switches. Generally, this behavior has worked well for most environments; however, in low traffic environments or during off-peak hours, this has the potential to re-use the same LogMiner session for a longer period of time that can lead to ORA-04030 exceptions when the PGA memory is exhausted.

A new configuration option has been added, log.mining.session.max.ms, allowing full control over the maximum duration of an Oracle LogMiner session. When set to a value greater-than 0, the connector will automatically close and restart the mining session if the maximum duration is reached or a log switch occurs, whichever comes first. Oracle environments with low volume, particularly during off-peak hours, should no longer notice any PGA memory concerns when enabling this new option. Coordinate with your database administrator team to determine the best value based on your environment’s configuration and activity.

But that’s not all, there were also a number of bugfixes that are noteworthy, including but not limited to:

We strongly recommend upgrading to 1.9.3.Final to get the latest improvements both in performance and stability.

Overall, 47 issues were fixed in this release. Please refer to the release notes to learn more about all fixed bugs, update procedures, etc.

Many thanks to the following individuals from the community which contributed to Debezium 1.9.3.Final: Alexey Miroshnikov, Anisha Mohanty, Bob Roldan, Chris Cranford, Connor Szczepaniak, César Martínez, Harvey Yue, Himanshu Mishra, Jakub Cechacek, Jiri Pechanec, Mark Allanson, Mark Bereznitsky, Nathan Bradshaw, Sergei Morozov, Stefan Miklosovic, Vadzim Ramanenka, and Vojtech Juranek!

The Debezium 1.9 release stream will remain the current long-running version for the next three months. During this time, we will continue to evaluate user reports and do micro-releases to address bugs and regressions depending on severity.

Also, quite a lot of work has gone into Debezium 2.0. We intend to release Debezium 2.0.0.Alpha2 next week, will releases to follow about every three weeks thereafter.

Stay tuned for more in the coming weeks and stay cool out there!