No More Flaky

Do you remember that one time, when you executed the build and it was yellow. Then, you re-executed the build and—without a change—it was green? Yep, that was a flaky (or non-deterministic) test. Flaky tests are toxic to your test automation endeavor.

Left uncontrolled, non-deterministic tests can completely destroy the value of an automated regression suite.

Martin Fowler

Essentially, a test can be flaky due to the following reasons:

  1. Lack of isolation.
  2. Asynchronous behavior.
  3. Remote services.
  4. Unreliable component recognition.
  5. Flakiness inside your application (e.g. resource leaks or race conditions).

We address all of this—well, except the one that is your duty …

Lack of Isolation

By its very nature, ReTest enforces you to take control over your test environment and be able to "reset" it. And to do this repeatedly, after every test suite. With traditional GUI test tools, it is easy to miss the signs that come together with a lack of isolation, such as data in tables that randomly occurs. But ReTest captures the complete GUI state and makes a diff. This will very clearly show you any change that manifests in the GUI, making it easy to spot a lack of isolation early on.

Asynchronous Behavior

Asynchronous behavior means that your test tries to execute the next test step while your application is not ready yet (e.g. loading external data), causing it to fail. One study found that this reason alone accounts for about 45% of all flaky tests. With most tools, there is usually no reasonable global solution available, so each tests tries to fix this individually. This usually looks as follows:

//selenium-style
new WebDriverWait(driver, 10).until(ExpectedConditions.elementToBeClickable(By.id("someId")));

//or even worse
Thread.sleep(someTime);

This is bad for several reasons:

  1. It requires manual effort to come up with this, putting the burden on the test creator.
  2. It is boilerplate code that basically needs to be put on every step of the test.
  3. If you run into the timeout, you don't know whether your application just took longer this once or something else happened.
  4. The timeout is optimized locally, for this one step only.

ReTest solves this globaly by waiting until the JVM is done (even remotely). This greatly reduces flakiness. It even allows you to measure performance (although ReTest is not a performance test tool).

Remote Services

Remote services amplify the problem stated above and add additional non-determinism. ReTest is so extensible, that it is easy to adjust the mechanism stated above for the purpose of knowing in your test when the remote system is done. This is way better than waiting for a timeout and again, solves the problem globaly.

If you have no control of the remote services to reset them or even do not have test equivalences of the remote services, we strongly advise to use test doubles. For most situations, there are easy to use mechanisms to quickly set up test doubles in a reliable way. Contact us, if we can help you set up your test environment.

Unreliable Component Recognition

In many other tools, component recognition is a major pain point. Not only maintaining it in such a way that it doesn't break over time (making tests brittle), but already getting it right such that it doesn't result in failing tests. Because we nailed that problem, this is almost never a problem with ReTest.

Flakiness Inside Your Application

The remaining source of flakiness stems from inside your application. Because of resource leaks, caches, race conditions, or any other kind of problem that resides in your software, tests may also fail randomly. But we think these flaky tests then should be analyzed and the root cause should be fixed. These problems are the reason for all the testing effort in the first place: they are bugs.