A crucial step in the software creation process is effective testing. It ensures the programs’ general quality, security, and usefulness. Testing teams do, however, have a number of difficulties, one of which is finding faulty tests. Sometimes these tests fail for no obvious reason at all, which confuses people and impedes the progress of development. These tests are called flaky tests. This article covers various testing tools that can help you spot, control, and minimize the effect of flaky tests.
Challenges in Identifying Flaky TestsIn software testing, random test failures may be very frustrating. Flaky tests are unexpected since they can pass on one try but fail on another. These inconsistencies might be caused by difficult-to-replicate external dependencies, complex links between many code parts, or timing problems. Finding incorrect tests becomes more difficult when several tests are run simultaneously, application behavior changes, and unexpected test execution conditions occur.
The asynchronous and dynamic loading of web application components makes the job more difficult. As a result, it is difficult to create a reliable and consistent testing environment and find the root cause of random test failures. This is when certain testing tools become important. By resolving these problems, you may improve the dependability of your test results using these tools. They allow testing teams to identify, control, and mitigate the effect of irregular tests on the overall testing strategy. You can read more about the best practices to help mitigate it here
Tools for Detecting and Managing Flaky TestsSeveral testing tools can simplify the process of finding and handling flaky tests. Let’s explore some famous options:
TestNGTestNG, a famous Java testing framework, includes built-in features for handling flaky tests. It uses listeners and analyzers to closely watch test runs, discover problems, and properly group tests.
Adding TestNG to your project
Here is a step-by-step guide on installing TestNG in IntelliJ IDEA (it comes bundled with TestNG 7.1.0)
Alt Insert and choose Dependency.org.testng:testing dependency, choose its version from the search results, and then click Add.Ctrl Shift + O or clicking the refresh icon to import the changes.Alt Insert in build.gradle and choose Add Maven artifact dependent.org.testng:testing, choose its version from the search results, and click Add.Ctrl Shift O or refresh icon to import changesCtrl Alt Shift Sorg.testng:testng:6.14.3.Creating a new TestNG class
The simplest way to make a new test class in IntelliJ IDEA is to use a special intention action that can be called from the source code. In this case, the IDE makes a new test class and produces test code for it, the package, or function. Here are steps to take:
Place the caret at the class for which you wish to build a test in your production code in the editor, hit Alt Enter, and choose build Test.
Choose the library you wish to use from the Create Test dialog box. It will ask you to download the required library if you don’t already have it. Click Fix to accomplish it.
The IDE will add the missing dependencies to your pom.xml if you’re using Maven. Manually add the required dependencies for projects using Gradle.
Running TestNG tests Executing TestNG tests within IntelliJ IDEA is straightforward:
JUnit Flaky Test PluginThe JUnit Flaky Test Plugin is an extremely useful tool for developers working with JUnit test packages. It handles a prevalent problem which is flaky tests. These are tests that give inconsistent results, with some passing and others failing for no obvious reason. This difference can hinder development progress by wasting time studying mistakes that do not suggest true issues. JUnit 5 is the latest version of the JUnit testing system, offering a contemporary base for developer-side testing on the Java Virtual Machine (JVM). This includes emphasis on Java 8 and higher, as well as allowing for a range of testing methods. Here is how to plan and perform tests.
Creating Project
Add dependency
We must add JUnit as a dependency in order for our project to leverage JUnit functionalities.
Alt Insert in pom.xml and choose Dependency.org.junit.jupiter:junit-jupiter in the search box of the dialog that appears.Ctrl Shift O or selecting refresh to import changes.Ctrl Shift 0 may be used to swiftly go to a file by entering its name.Alt Insert in build.gradle and choose Add Maven artifact dependency.org.junit.jupiter:junit-jupiter in the search field of the tool window that appears.Ctrl Shift O or clicking refresh to import changesCtrl Alt Shift S) from the main menu.org.junit.jupiter:junit-jupiter:5.9.1.Creating Sample Code and Tests
In the Project tool window go to src/main/java and create a Java file called Calculator.java.
import java.util.stream.DoubleStream;public class Calculator { static double add(double... operands) { return DoubleStream.of(operands) .sum(); } static double multiply(double... operands) { return DoubleStream.of(operands) .reduce(1, (a, b) -> a * b); }}
Right-click on “Calculator” in the Project tool window and select “Create Test.” Choose the two methods you want to test (add and multiply).
The editor takes you to the newly created test class. Modify the add() test as follows:
@Test@DisplayName("Add two numbers")void add() { assertEquals(4, Calculator.add(2, 2));}
This short test will determine whether our algorithm adds two and two correctly. The test has a more practical and useful name thanks to the @DisplayName annotation. What would happen if you wanted to include more than one statement in a single test and run it all even if some of the assertions didn’t pass? The assertAll() function accepts a list of lambda expression assertions and checks that each one has been verified. Seeing a specific result instead of the test’s overall result is always more handy than having several single assertions.
Run tests
After we have set up the code for the testing, we can run the tests and find out if the tested methods are working correctly. Running individual test, click run in the gutter. To execute all tests in a test class, click execute against the test class declaration and then choose Run. The final result can be seen below:
RSpec’s –bisect OptionA potent method called –bisect is given by the well-known Ruby testing framework RSpec to overcome inconsistent test failures. When tests only fail after finishing particular tests before them, a situation that might be difficult to fix using standard methods, this flag becomes extremely important.
A repeating method is invoked by the --bisect option to find the source of test flakiness. It starts by running your whole test suite, dividing it in half carefully and running each half separately. Subdivision continues until the smallest set of tests successfully reproduces the mistake in the half that still shows the failure.
Practical Example Consider a Ruby test suite with ten files (spec/calculator_1_spec.rb to spec/calculator_10_spec.rb), and one of them occasionally fails. You assume the failure is caused by the execution order, but you can’t determine the specific combination.
Using --bisect:
rspec --seed 1234 --bisect
--seed 1234: This guarantees that tests are executed in a consistent sequence during the bisect process. RSpec will provide findings that include rounds of bisection, eventually narrowing down the failed samples. Here is a sample output:
Round 1: bisecting over non-failing examples 1-9 .. ignoring examples 6-9Round 2: bisecting over non-failing examples 1-5 .. ignoring examples 4-5Round 3: bisecting over non-failing examples 1-3 .. ignoring example 3Round 4: bisecting over non-failing examples 1-2 .. ignoring example 1Bisect complete! Reduced necessary non-failing examples from 9 to 1.
Success! RSpec has identified the minimal set causing the failure. The final output provides a minimal reproduction command that includes only the necessary failing specs. This command below allows you to re-run the failing tests in isolation, making it easier to diagnose the root cause.
rspec ./spec/calculator_10_spec.rb [1:1] ./spec/calculator_1_spec.rb [1:1] --seed 1234
You can use Ctrl-C to end the bisect operation at any time. You can test it with projects here
Visual Regression Testing ToolsVisual Regression Testing (VRT) tools are crucial in detecting even the slightest visual differences that could be overlooked during conventional testing. In this chapter, we will review Percy and Applitools, two prominent players in the VRT industry. Both platforms provide unique benefits that are worth exploring.
Overview of Percy’s Visual Testing Capabilities
The purpose of Percy, an automated visual testing tool, is to identify visual inconsistencies in online applications. It helps teams ensure the visual excellence of their user interface by capturing and comparing screenshots of the application’s visual state during testing. Jest, Cypress, and Selenium are just a few of the well-known testing frameworks that Percy easily interfaces with. Additionally, it works with CI/CD systems like GitHub Actions and Jenkins. It creates baseline images of the user interface for your programme and notifies you of any differences by comparing subsequent test runs with these baselines.
Integrating Percy with Jest:
npm install --save-dev @percy/puppeteer
* Set up Percy in your Jest configuration.
// jest.config.jsmodule.exports = { // ... other Jest configurations setupFilesAfterEnv: ['<rootDir>/percy.setup.js'],};
* Create a percy.setup.js file with the following content:
// percy.setup.jsconst { percySnapshot } = require('@percy/puppeteer');// Ensure Percy is runningbeforeAll(async () => { await page.goto('http://localhost:5338');});// Take snapshots using Percytest('Snapshot', async () => { await percySnapshot(page, 'Snapshot');});
* Run Jest with Percy: Execute your Jest tests with Percy.
npx percy exec -- jest
The integration with Percy enhances your visual testing strategy, providing a comprehensive view of your application’s UI changes.
Integration with Selenium:
If you’re working with Java and Selenium, you can use the percy-java-selenium library in your project. Add it to your Maven project dependencies
<dependency> <groupId>io.percy</groupId> <artifactId>percy-java-selenium</artifactId> <version>1.3.0</version></dependency>
A sample of test code in Java for taking snapshot:
import org.openqa.selenium.WebDriver;import org.openqa.selenium.chrome.ChromeDriver;import io.percy.selenium.Percy;public class Example { public static void main(String[] args) { // Create a new WebDriver instance using ChromeDriver WebDriver driver = new ChromeDriver(); // Navigate to the specified URL (in this case, "https://example.com") driver.get("https://example.com"); // Initialize Percy with the WebDriver instance Percy percy = new Percy(driver); // Take a snapshot of the current page with a descriptive name percy.snapshot("Java example"); }}
In short, this code sets up Percy, starts “https://example.com“, sets up a Chrome WebDriver, and takes a picture for testing and comparison. Percy will keep this sample for future use as a guide and for checking for bugs. Keep in mind that you should use the real URL of the site you want to test instead of “https://example.com.” You can test it with various Java projects here
Applitools Eyes: Visual AI Testing
One of the most complete Visual AI testing tools is Applitools Eyes, which has a lot of options for finding UI issues. Users pay based on how many tests they run and the services they need, using a commercial price plan. Test scripts for Applitools Eyes can be written in Python, Java, and JavaScript, among other languages. The Applitools SDK is used to compare snapshot that were taken while tests were running. In particular, Applitools Eyes cuts down on false results by using AI and machine learning. It ensures accurate test results by instantly finding and disregarding changing visual aspects.
To start using Applitools Eyes for Visual AI Testing, follow these steps:
Example (Java with Selenium):
import com.applitools.eyes.selenium.Eyes;import org.openqa.selenium.WebDriver;import org.openqa.selenium.chrome.ChromeDriver;public class YourVisualTest { public static void main(String[] args) { // Initialize the WebDriver for Chrome browser (you can modify this to use a different browser) WebDriver driver = new ChromeDriver(); // Create an Eyes object to perform visual testing Eyes eyes = new Eyes(); // Set your Applitools API key to enable communication with the platform (replace with your actual key) eyes.setApiKey("Your_Applitools_API_Key"); try { // Start the visual test by providing the application name and test name of your choice eyes.open(driver, "Your App Name", "Your Test Name"); // Navigate to the URL of your web application under test driver.get("Your App URL"); // Capture a visual checkpoint of the entire page. // Consider using more specific selectors for targeted testing in the future eyes.checkWindow("Full Page"); } finally { // Always close the WebDriver to avoid resource leaks driver.quit(); // If the test was aborted due to an exception, abort the Eyes test as well eyes.abortIfNotClosed(); } }}
* Run Your Visual Test: When you run your test script, Applitools will take screenshots of your application and compare them to a baseline.
* Review Results in the Dashboard: Return to the Applitools dashboard to see the visual tests’ outcomes. It will draw attention to any variations between your application’s baseline and current states. Update the baseline in the Applitools dashboard to reflect the new intended visual design as your application develops.
Adding visual testing to your workflow guarantees a visually consistent and enjoyable user experience, regardless of whether you want the powerful AI-driven analysis offered by Applitools or the seamless integration, collaboration, and ease of use offered by Percy.
ConclusionThis article discusses the challenges faced by flaky tests in software development, with a particular emphasis on the role of testing tools in resolving these issues. To study more of this tools and practical experiences in large-scale commercial settings and investigating technologies such as TestNG, JUnit, and Percy, check out this in-depth study. It becomes clear that the testing tools used have a substantial impact on test suite dependability. The study emphasizes the ongoing need for research and innovation in testing technologies to improve their ability to detect and manage flaky tests. As the software development community grows, a concentrated effort to refine and advance testing tools is critical for establishing confidence, assuring reliability, and successfully negotiating the complexities of flaky testing scenarios.
The post Testing Tools to Identify Flaky Tests appeared first on Semaphore.