Almost every company has seen the benefits of test automation and has bought into the idea of test automation throughout their entire stack. While automation has significantly improved delivery times and allowed us to deliver software at a quicker rate – a common problem I’ve seen with many automation frameworks is that they do not do a very good job of handling execution in parallel.
Benefits of Parallel Test Execution
Running test automation in parallel offers several significant benefits, particularly in terms of efficiency, speed, and resource utilization. Here are the key reasons why parallel execution of test automation is advantageous:
Reduced Test Execution Time
Faster Feedback Loop: Parallel execution significantly reduces the total time required to run your entire test suite. This is especially important in continuous integration and continuous deployment (CI/CD) pipelines where quick feedback is crucial.
Efficiency: By running tests concurrently, you make better use of available resources, thereby reducing idle time and improving the overall efficiency of the testing process.
Increased Test Coverage
Scalability: Parallel testing allows you to run a large number of tests across multiple environments and configurations simultaneously, increasing test coverage and ensuring that your application works correctly under various conditions.
Optimized Resource Utilization
Better Use of Hardware: Utilizing multiple cores or machines to run tests in parallel ensures that the available hardware is used more effectively, leading to improved performance and throughput.
Cost-Effectiveness: For cloud-based testing environments, running tests in parallel can be more cost-effective as it reduces the total time and resources required, potentially lowering the cost of cloud computing services.
Enhanced Reliability and Stability
Isolation of Tests: Running tests in parallel can help identify flaky tests (tests that sometimes pass and sometimes fail) and resource contention issues, as tests are run independently in isolated environments.
Error Detection: By testing in parallel, you can detect concurrency issues, race conditions, and other problems that might not be apparent in a sequential execution.
Improved Developer Productivity
Quick Turnaround: Developers receive faster feedback on their changes, enabling them to address issues promptly and maintain a high development velocity.
Concurrent Development and Testing: Parallel testing allows for concurrent development and testing activities, which can lead to a more streamlined and efficient development process.
Example Use Cases
Continuous Integration/Continuous Deployment (CI/CD): In CI/CD pipelines, it’s essential to get quick feedback on code changes. Running tests in parallel helps ensure that builds and deployments are not delayed by lengthy test executions.
Cross-Browser Testing: To ensure that a web application works correctly across different browsers and browser versions, parallel testing can be employed to run tests on multiple browser instances simultaneously.
API Testing: For microservices architectures, running API tests in parallel can validate multiple endpoints and services concurrently, speeding up the overall testing process.
Practical Implementation
So, we can easily see the benefits of executing automation tests in parallel, but he real challenge for many might be in how best to implement this into their respective frameworks. We are now going to have a look at different approaches with different tools to showcase how this can be done.
There are many others ways of doing it, but the below approaches should be able to suit the needs of most teams looking to achieve parallel test execution with their different tools. I have also limited the tools here to just a few of the most popular ones with a popular programming language too. These framework approaches can be easily adapted to different programming languages.
Selenium with TestNG
TestNG is a popular testing framework for Java, which supports parallel test execution out-of-the-box. Here’s how you can configure it, with Java used asa tge core programming language in these examples:
Add dependencies: Ensure you have the necessary dependencies in your pom.xml (if you’re using Maven).
<dependencies>
<dependency>
<groupId>org.testng</groupId>
<artifactId>testng</artifactId>
<version>7.4.0</version>
<scope>test</scope>
</dependency>
<dependency>
<groupId>org.seleniumhq.selenium</groupId>
<artifactId>selenium-java</artifactId>
<version>4.1.0</version>
</dependency>
</dependencies>Create Test Classes: Create test classes annotated with @Test from TestNG.
import org.testng.annotations.Test;
public class TestClass1 {
@Test
public void testMethod1() {
System.out.println("Test Method 1 - " + Thread.currentThread().getId());
}
}
public class TestClass2 {
@Test
public void testMethod2() {
System.out.println("Test Method 2 - " + Thread.currentThread().getId());
}
}Configure TestNG XML for Parallel Execution: Create a testng.xml file to configure parallel execution.
<!DOCTYPE suite SYSTEM "http://testng.org/testng-1.0.dtd">
<suite name="ParallelTestsSuite" parallel="tests" thread-count="2">
<test name="Test1">
<classes>
<class name="TestClass1"/>
</classes>
</test>
<test name="Test2">
<classes>
<class name="TestClass2"/>
</classes>
</test>
</suite>Run TestNG Suite: Execute the testng.xml suite in shell script.
mvn test -DsuiteXmlFile=testng.xmlSelenium with JUnit
JUnit, particularly JUnit 5, can also run tests in parallel using a few configurations.
Add dependencies: Ensure you have the necessary dependencies in your build.gradle (if you’re using Gradle).
dependencies {
testImplementation 'org.junit.jupiter:junit-jupiter:5.7.0'
testImplementation 'org.seleniumhq.selenium:selenium-java:4.1.0'
}Create Test Classes: Create test classes annotated with @Test from JUnit.
import org.junit.jupiter.api.Test;
public class TestClass1 {
@Test
void testMethod1() {
System.out.println("Test Method 1 - " + Thread.currentThread().getId());
}
}
public class TestClass2 {
@Test
void testMethod2() {
System.out.println("Test Method 2 - " + Thread.currentThread().getId());
}
}Configure JUnit for Parallel Execution: Create a junit-platform.properties file in src/test/resources.
junit.jupiter.execution.parallel.enabled = true
junit.jupiter.execution.parallel.mode.default = concurrent
junit.jupiter.execution.parallel.config.strategy = dynamicRun Tests: Execute the tests using Gradle.
./gradlew testUsing Python with pytest
Pytest is a popular testing framework in Python that supports parallel execution via the pytest-xdist plugin.
Install dependencies: Install pytest and pytest-xdist.
pip install pytest pytest-xdist seleniumCreate Test Files: Create test files with test functions.
import pytest
def test_method1():
print(f"Test Method 1 - {pytest.current.thread().id}")
def test_method2():
print(f"Test Method 2 - {pytest.current.thread().id}")Run Tests in Parallel: Use the -n flag to specify the number of parallel workers.
pytest -n 2Using C# with NUnit
NUnit is a widely used testing framework for C# and supports parallel test execution.
Add dependencies: Ensure you have the necessary NuGet packages.
dotnet add package NUnit
dotnet add package NUnit3TestAdapter
dotnet add package Microsoft.NET.Test.Sdk
dotnet add package Selenium.WebDriverCreate Test Classes: Create test classes annotated with [Test].
using NUnit.Framework;
[TestFixture]
public class TestClass1 {
[Test]
public void TestMethod1() {
Console.WriteLine($"Test Method 1 - {Thread.CurrentThread.ManagedThreadId}");
}
}
[TestFixture]
public class TestClass2 {
[Test]
public void TestMethod2() {
Console.WriteLine($"Test Method 2 - {Thread.CurrentThread.ManagedThreadId}");
}
}Configure Parallel Execution: You can configure parallel execution in the .runsettings file or programmatically.
<?xml version="1.0" encoding="utf-8"?>
<RunSettings>
<RunConfiguration>
<MaxCpuCount>2</MaxCpuCount>
</RunConfiguration>
</RunSettings>Run Tests: Use the dotnet test command.
dotnet test --settings .runsettingsCypress
To run tests in parallel with Cypress, you can use the Cypress Dashboard Service, which provides built-in support for parallelization. Here’s a step-by-step guide:
Step 1: Install Cypress
First, ensure you have Cypress installed in your project:
npm install cypress --save-devStep 2: Configure Cypress for Parallel Testing
You need to set up your project to use the Cypress Dashboard. If you don’t have a Cypress Dashboard project set up, you can create one by following the instructions on the Cypress Dashboard.
Step 3: Add Environment Variables
Add your CYPRESS_RECORD_KEY (which you get from the Cypress Dashboard) to your CI environment. For example, in a GitHub Actions workflow, you might add it like this:
env:
CYPRESS_RECORD_KEY: ${{ secrets.CYPRESS_RECORD_KEY }}Step 4: Configure CI for Parallel Execution
Modify your CI configuration to run Cypress tests in parallel. Here’s an example using GitHub Actions:
name: Cypress Tests
on: [push, pull_request]
jobs:
cypress-run:
runs-on: ubuntu-latest
strategy:
matrix:
containers: [1, 2, 3, 4]
steps:
- uses: actions/checkout@v2
- name: Use Node.js
uses: actions/setup-node@v2
with:
node-version: '14'
- name: Install Dependencies
run: npm install
- name: Run Cypress Tests
run: npx cypress run --record --parallel --group ${{ matrix.containers }}
env:
CYPRESS_RECORD_KEY: ${{ secrets.CYPRESS_RECORD_KEY }}This configuration will split your Cypress tests across four parallel containers.
Playwright
Playwright also supports parallel test execution out of the box. We are using Typescript as a language in these examples. Here’s how you can set it up:
Step 1: Install Playwright
First, ensure you have Playwright installed in your project:
npm install @playwright/test --save-devStep 2: Configure Playwright for Parallel Testing
Playwright runs tests in parallel by default. You can configure the level of parallelism in your playwright.config.ts file.
import { defineConfig } from '@playwright/test';
export default defineConfig({
// Set the maximum number of parallel workers
workers: 4,
use: {
// Browser settings
browserName: 'chromium',
headless: true,
},
projects: [
{
name: 'chromium',
use: { ...devices['Desktop Chrome'] },
},
{
name: 'firefox',
use: { ...devices['Desktop Firefox'] },
},
{
name: 'webkit',
use: { ...devices['Desktop Safari'] },
},
],
});Step 3: Run Playwright Tests
Run the tests using the Playwright test runner:
npx playwright testThis command will automatically use the parallelism level set in the configuration file.
Step 4: Configure CI for Playwright
Here’s an example of a GitHub Actions workflow for running Playwright tests in parallel:
name: Playwright Tests
on: [push, pull_request]
jobs:
playwright-run:
runs-on: ubuntu-latest
strategy:
matrix:
node-version: [14.x]
steps:
- uses: actions/checkout@v2
- name: Use Node.js
uses: actions/setup-node@v2
with:
node-version: ${{ matrix.node-version }}
- name: Install Dependencies
run: npm install
- name: Run Playwright Tests
run: npx playwright testSummary
Parallel test execution is one of the fundamental tenets of test automation effectiveness and yet one that is rarely executed well. Hopefully, with some of the tips gained in this blog, you will be able to work with your existing automation tools and adjust them to better capitalize on the benefits of parallel test execution.
Comments