Debezium 2.6 introduces a brand-new connector for IBM fans to stream changes from Db2 iSeries/AS400 using the IBM iJournal system. This collaboration is a multi-year development effort from the community, and we’re pleased that the community has allowed this to be distributed under the Debezium umbrella.

The new connector can be obtained from Maven Central using the following coordinates or a direct download.

The PostgreSQL driver supports a SQL syntax called a row-value constructor using the ROW() function. This allows a query to express predicate conditions in a more efficient way when working with multi-columned primary keys that have a suitable index. The incremental snapshot process is an ideal candidate for the use of the ROW() function, the process involves issuing a series of select SQL statements to fetch data in chunks. Each statement, aka chunk query, should ideally be as efficient as possible to minimize the cost overhead of these queries to maximize throughput of your WAL changes to your topics.

There are no specific changes needed, but the query issued for PostgreSQL incremental snapshots has been adjusted to take advantage of this new syntax, and therefore users who utilize incremental snapshots should see performance improvements.

An example of the old query used might look like this for a simple table:

The new implementation constructs this query using the ROW() function as follows:

We’d be interested in any feedback on this change, and what performance improvements are observed.

An incremental snapshot process requires a signal table to write open/close markers to coordinate the change boundaries with the data recorded in the transaction logs, unless you’re using MySQL’s read-only flavor. In some cases, users would like to be able to track the window time slot, knowing when the window was opened and closed.

Starting with Debezium 2.6, the data column in the signal table will be populated with the time window details, allowing users to obtain when the window was opened and closed. The following shows the details of the data column for each of the two signal markers:

We have improved the Oracle connector’s event structure for inserts, updates, and deletes to optionally contain the SQL that was reconstructed by LogMiner in the source information block. This feature is an opt-in only feature that you must enable as this can easily more than double the size of your existing event payload.

To enable the inclusion of the REDO SQL as part of the change event, add the following connector configuration:

With this option enabled, the source information block contains a new field redo_sql, as shown below:

A new delay-strategy for transaction registration has been added when using LogMiner. This strategy effectively delays the creation of the transaction record in the buffer until we observe the first captured change for that transaction.

Delaying transaction registration has a number of benefits, which include:

We are looking into how we can explore this change for Infinispan-based users in a future build; however, due to the nature of how lob.enabled works with LogMiner, this feature won’t be possible for that use case.

Debezium 2.6 introduces a new community requested feature to improve the precision of timestamps in change events. Users will now notice the addition of 4 new fields, two at the envelope level and two in the source information block as shown below:

The envelope values will always provide both microsecond (ts_us) and nanosecond (ts_ns) values while the source information block may have both micro and nano -second precision values truncated to a lower precision if the source database does not provide that level of precision.

Informix databases only increases the transaction identifier when there are concurrent transactions, otherwise the value remains identical for sequential transactions. This can prove difficult for users who may want to utilize the transaction metadata to order change events in a post processing step.

Debezium 2.6 for Informix will now append the log sequence number (LSN) to the transaction identifier so that users can easily sort change events based on the transaction metadata. The transaction identifier field will now use the format <id>:<lsn>. This change affects transaction metadata events and the source information block for change events, as shown below:

While it’s common for users to utilize serialization based on Json, Avro, Protobufs, or CloudEvents, there may be reasons to use a more simplistic format. Thanks to a community contribution as part of DBZ-7512, Debezium can be configured to use two new formats called simplestring and binary.

The simplestring and binary formats are configured in Debezium server using the debezium.format configurations. For simplestring, the payload will be serialized as a single STRING data type into the topic. For binary, the payload will be serialized as a BYTES using a byte[] (byte array).

Debezium 2.6 also introduces a new Oracle LogMiner mining strategy called hyrid, which can be enabled by setting the configuration property log.mining.strategy with the value of hybrid. This new strategy is designed to support all schema evolution features of the default mining strategy while taking advantage of all the performance optimizations from the online catalog strategy.

The main problem with the online_catalog strategy is that if a mining step observes a schema change and a data change in the same mining step, LogMiner is incapable of reconstructing the SQL correctly, which will result in the table name being OBJ# xxxxxx or the columns represented as COL1, COL2, and so on. To avoid this while using the online catalog strategy, users are recommended to perform schema changes in a lock-step pattern to avoid a mining step that observes both a schema change and a data change together; however, this is not always feasible.

The new hybrid strategy works by tracking a table’s object id at the database level and then using this identifier to look up the schema associated with the table from Debezium’s relational table model. In short, this allows Debezium to do what Oracle LogMiner is unable to do in these specific corner cases. The table name will be taken from the relational model’s table name and columns will be mapped by column position.

Unfortunately, Oracle does not provide a way to reconstruct failed SQL operations for CLOB, BLOB, and XML data types. This means that the new hybrid strategy cannot be configured with configurations that use lob.enabled set to true. If a connector is started using the hybrid strategy and has lob.enabled set to true, the connector will fail to start and report a configuration failure.