One possible way to slow down your tests, especially when you have a multi-threaded application, is to execute the tests on a single thread or limited number of threads. On the Linux operating system, it’s pretty easy with the taskset command. taskset tell the Linux scheduler to attach given process to specified CPU core. To run e.g. Debezium MySQL TransactionMetadataIT on a single CPU core, you just need to run

which would execute the tests on CPU #0.

On the Debezium project we use containers for tests heavily. We run the databases against the tests run in the containers. What we often need is not to slow down the tests itself, but the database. Docker provides quite a lot of options how to limit container resources. The most useful is usually limiting the CPU using --cpus parameter. This allows us to limit the amount of CPU Docker can use for running the container. The nice thing here is that it can be a float number, so you can e.g. limit containers to use only half of CPU time by setting --cpus=0.5. In a similar way you can also limit other resource, like e.g. RAM.

The common Debezium workflow is to run the containers from Maven, using Docker Maven plugin. The plugin provides long list of properties which you can configure, including properties for limiting container resources. However, there is one caveat with this option. With current release, docker.cpus expect long number instead of float and have a meaning, roughly saying, how many CPU nano seconds from one second cycle the container can take. E.g. equivalent of --cpus=0.5 would be:

This issue was fixed recently and should be in the next Docker Maven plugin release, so once Debezium upgrade to the next version, you should be able use docker.cpus in the same way as you would use when running the container from the command line. Other Docker Maven plugin properties, e.g. docker.memory should work as expected.

Another common source of random test failures is network latency. There’s probably not any easy way to simulate it on a local machine and one has to use some kind of proxy. Fortunately, there is a proxy exactly for this purpose - Toxiproxy. It’s a dedicated proxy to simulate various network failures and latencies. It has a rich feature set and moreover it’s pretty easy to set it up, so it’s a pleasure to work with it. Let’s see how to set it up with Debezium tests on a local machine.

You can install Toxiproxy locally (on Fedora by running sudo dnf install toxiproxy) or download it in a container:

We are going to run Toxiproxy in a container, but it’s also convenient to install it locally as it contains a CLI utility to send commands to the Toxyproxy. Otherwise we would have to run the commands from the container. For simplicity, we will use a CLI tool installed locally. Toxyproxi allows us to send commands over HTTP and listens on port 8474. Therefore, when we start Toxyproxy, we need to expose this port. Another port we need to expose is the one for the database for which the Toxiproxy will serve as a proxy. In our example we will use MySQL, therefore we need to expose port 3306. We can of course use any other port, but in such a case we would need to pass additional parameter to the Debezium test, namely database.port pointing to the port exposed by Toxiproxy. Again, for simplicity, let’s stick with the default port 3306. Also, as we are going to run the Debezium tests from the local machine (not from a container), we need to attach Toxiproxy to the localhost network, which is by default named host. Putting everything together, we can run Toxiproxy container as follows:

Now we also have to start our database. As the port 3306 is already occupied by Toxiproxy, we have to choose another one, let’s say 3307:

The last missing piece is to tell Toxiproxy for which ports it should create the proxy. In our case it’s from port 3306 (listen port -l) to 3307 (upstream port -u):

This command creates a new proxy within Toxiproxy, called mysql. There can be multiple proxies. We can list all the proxies by running

which gives you output like this:

Now let’s try if everything works and run some test:

Everything should run as normal as we haven’t created any toxics (latencies or failure) yet. It’s just a check that the proxy works correctly. If everything works, let’s create a toxic now:

This will add a network latency of 500 ms on the mysql proxy. The toxic is named "mysql_latency".

You can get more details about specified proxy by running inspect command:

with output like this:

Now, run the test again. Did you observe that the test ran substantially longer? If yes, everything works as expected, as we added latency to every call to the database.

This is a simple example of adding toxic to the Toxiproxy. Toxiproxy provides many more options and ways to configure the toxics. See Toxiproxy for more details.

Once we are done, we can remove toxic

as well as proxy itself:

or simply stop and delete the container.

In this blog post I tried to show a couple of techniques which may help you to simulate flaky test failures locally. All of them try to make the test environment less responsive, namely by limiting CPU or imposing network latencies using Toxiproxy. There are many other reasons why the tests can be flaky, in many parts of your application stack, and also there are many other tools which can inject various kinds of failures (e.g. disk failures). So this post is not by far exhaustive. But I hope it will help you to debug at least some of the flaky tests, if not in the Debezium project, then at least in your own project.

All these things, especially Toxiproxy, can be also used on a regular basis, even in the CI, to spot various hidden issues in the project which appears only when the environment where it runs doesn’t behave nicely.

Feel free to share in the discussion any other tips on how to debug flaky tests and what kind of tools you find handy.