Before we dive directly into the test results I want to illuminate some relevant facts about our environment:

In my tests I primarily focused on the performance related properties. These are

on Debezium side:
snapshot.max.threads
snapshot.fetch.size
max.batch.size
max.queue.size
poll.interval.ms

on Kafka Connect side:
batch.size
linger.ms
compression.type

I experimented with these properties during my tests and got interesting insights. At this point I already reveal the settings that proved to be most effective in our case:

producer.override.batch.size: 1000000,
producer.override.linger.ms: 500,
producer.override.compression.type: lz4,

By using these settings we were able to achieve an optimization of 25%: from initially 8 hours we reduced the time for a complete snapshot to 6 hours (see figure 1). The CPU consumption as well as the JVM memory used never exceeded 80% throughout the whole snapshotting process.

One metric I observed particularly was the source record poll rate. During my test this metric served as a helpful first indicator whether the performance was good or bad. As figure 2 illustrates a rate of up to 90k ops/s turned out to be the maximum. In any case I haven’t been able to reach a higher rate. Just as important was to have a look at its neighbor metric source record write rate which should show virtually the same diagram:

If polling is ok but pushing to Kafka wouldn’t be fast enough then the metric source record active count could be an identifier for that. Figure 3 shows that in our case we didn’t have to worry about any blocking:

As a matter of course we tried to be even faster and tested some more settings and combinations of them. Here are the results:

As you see none of these attempts really brought any improvement, most of them were slower. Setting the value of snapshot.max.threads to the total number of tables we extract data from didn’t accelerate the process either and furthermore was delicate because of the massive load on the shared database resources. With too many parallel threads we even encountered that the connector crashed owing to “ORA-12801: error signaled in parallel query server”.

Although we didn’t manage to reach our goal completely (we are still 2 hours above the targeted snapshot time) the performance optimization we accomplished is good enough for us at the moment. It is clear to us that performance optimization is an iterative process and we need to continuously monitor, analyze, and tune our whole system for optimal performance.

Performance optimization is definitely not an easy thing and requires perseverance. To obtain an optimal performance it is necessary to find the right parameters respectively properties and to get the ideal balance between them. If you consider changing the compression type don’t forget to think about the impact this change can have to all the downstream processes. For better end to end performance we decided to define the same compression type in ksqlDB’s default value configuration.

Finally I want to emphasize an interesting fact: in our tests we didn’t have to change any default value of Debezium’s performance related properties, which is kind of soothing. All the improvements we achieved was by modifying Kafka Connect producer properties.

If you have any questions or advice please don’t hesitate to contact me on zulipchat.