Many database platforms support multi-tenancy out of the box, meaning you can have one installation of the database engine and have many unique databases. In cases like SQL Server, this traditionally required a separate connector deployment for each unique database. Over the last year, a large effort has been made to break down that barrier and to introduce a common way that any single connector deployment could connect and stream changes from multiple databases.

The first notable change is with the SQL Server connector’s configuration option, database.dbname. This option has been replaced with a new option called database.names. As multi-partition mode is now default, this new database.names option can be specified using a comma-separated list of database names, as shown below:

In this example, the connector is being configured to capture changes from two unique databases on the same host installation. The connector will start two unique tasks in Kafka Connect and each task will be responsible for streaming changes from its respective database concurrently.

The second notable change is with connector metrics naming. A connector exposes JMX metrics via beans that are identified with a unique name. With multi-partition mode the default with multiple tasks, each task requires its own metrics bean and so a change in the naming strategy was necessary.

In older versions of Debezium using SQL Server as an example, metrics were available using the following naming strategy:

In this release, the naming strategy now includes a new task component in the JMX MBean name:

Please review your metrics configurations as the naming changes could have an impact when collecting Debezium metrics.

In this release, we have introduced a new debezium-storage set of artifacts for file- and kafka- based database history and offset storage. This change is the first of several future implementations set to support platforms such as Amazon S3, Redis, and possibly JDBC.

For users who install connectors via plugin artifacts, this should be a seamless change as all dependencies are bundled in those plugin downloadable archives. For users who may embed Debezium in their applications or who may be building their own connector, be aware you may need to add a new storage dependency depending on which storage implementations used.

Debezium’s default topic naming strategy emits change events to topics named database.schema.table. If you require that topics be named differently, an SMT would normally be added to the connector configuration to adjust this behavior. But, this presents a challenge in situations where one of the components of this topic name, perhaps the database or table name, contains a dot (.) and perhaps an SMT doesn’t have adequate context.

In this release, a new TopicNamingStrategy was introduced to allow fully customizing this behavior directly inside Debezium. The default naming strategy implementation should suffice in most cases, but if you find that it doesn’t you can provide a custom implementation of the TopicNamingStrategy contract to fully control various namings used by the connector. To provide your own custom strategy, you would specify the topic.naming.strategy connector option with the fully-qualified class name of the strategy, as shown below:

This custom strategy is not just limited to controlling the names of topics for table mappings, but also for schema changes, transaction metadata, and heartbeats. You can refer to the DefaultTopicNamingStrategy found here as an example. This feature is still incubating and we’ll continue to improve and develop it as feedback is received.

The source information block of change events carry a variety of context about where the change event originated. In this release, the Oracle connector now includes the user who made the database change in the captured change event. A new field, user_name, can now be found in the source info block with this new information. This field is optional, and is only available when changes are emitted using the LogMiner-based implementation. This field may also contain the value of UNKNOWN if the user associated with a change is dropped prior to the change being captured by the connector.

A table does not have to have a primary key to be captured by a Debezium connector. In cases where a primary key is not defined, Debezium will inspect a table’s unique indices to see whether a reasonable key substitution can be made. In some situations, the index may refer to columns such as CTID for PostgreSQL or ROWID in Oracle. These columns are not visible nor user-defined, but instead are hidden synthetic columns generated automatically by the database. In addition, the index may also use database functions to transform the column value that is stored, such as UPPER or LOWER for example.

In this release, indices that rely on hidden, auto-generated columns, or columns wrapped in database functions are no longer eligible as primary key alternatives. This guarantees that when relying on an index as a primary key rather than a defined primary key itself, the generated message’s primary key value tuple directly maps to the same values used by the database to represent uniqueness.

There are several bugfixes and stability changes in this release, some noteworthy are:

Altogether, a total of 59 issues were fixed for this release.

A big thank you to all the contributors from the community who worked on this release: Andrew Walker, Anisha Mohanty, Bob Roldan, Chris Cranford, Giljae Joo, Harvey Yue, Henry Cai, Hossein Torabi, Ismail Simsek, Jan Doms, Jiri Pechanec, Martin Medek, Nathan Smit, Paweł Malon, Pengwei Dou, Sergei Morozov, Vojtech Juranek, and Zhongqiang Gong!

In these last few months, the team has made some incredible progress on Debezium 2.0, and we can begin to see the finish line in the distance. A large of this is in part to the grew work the community has done to contribute changes, provide feedback, and to test and help make new features stable. But we’re not done, so you can continue to expect another 2.0.0.Beta2 release in approximately 3 weeks, sticking with our usual cadence.

In addition, we do continue to backport changes to the 1.9 branch and will likely look at a 1.9.6.Final release sometime in August to round out that release stream just before we wrap up Debezium 2.0.0.Final.

So stay cool and safe and happy capturing!