The pros and cons of different testing tools – Gatling

Before I move on to other forms of testing tools, like Test Management Tools, I wanted to talk about one more performance testing tool, Gatling. Gatling is an open-source load and performance testing tool designed for testing the performance of web applications. Interestingly it is written in Scala, which is not one of the most performant programming languages as it has a high level of abstraction and a reliance on the JVM which slows it down and makes it not suitable for most pipelines.

However, the tool makes use of a Domain-specific language (DSL) for writing test scripts, making it easy to create and maintain complex test scenarios - though it does require some Scala knowledge to use optimally. The DSL is designed to be both expressive and concise. It also can simulate user behavior in the form of think times, loops, and specific request sequences allowing developers and testers to make their tests mimic human behavior.

Gatling scripts are organized around scenarios. A scenario represents a sequence of user actions. You can create a new scenario by extending the Simulation class and defining the scenario within it.

import io.gatling.core.Predef._
import io.gatling.http.Predef._

class MyScenario extends Simulation {

val httpProtocol = http.baseUrl("https://example.com"
val myScenario = scenario("My Scenario")
   .exec(.("Request 1")
        .get("/page1"))
   .pause(2) // Wait for 2 seconds
        .exec(http("Request 2")
    .post("/submitForm"))

setUp( myScenario.inject(atOnceUsers(10)) // Simulate 10 users in parallel ).protocols(httpProtocol) }

To then run this Gatling script, use the Gatling command-line tool. Navigate to the project directory and execute the following command:

gatling.sh -s MyScenario

In the example above, you define a scenario named "My Scenario" and specify a sequence of user actions using the .exec method. You can include HTTP requests, pauses, and other actions in your scenario.

The setUp method specifies how the scenario is executed. In this case, it's set to simulate 10 users executing the scenario simultaneously.

The httpProtocol variable defines the base URL for your HTTP requests.

The scripting is quite simple and easy to follow. Still, it’s also possible to customize by adding loops, conditional statements, and variable assignments to create more complex and realistic user interactions. You can also include assertions to validate the responses from the application and ensure that the behavior is as expected.

As I’ve mentioned before though in my previous analysis of performance testing tools, the scripting is actually not the most important feature of a tool and it’s the ability to monitor an application and make sense of what is going on while scripts are executing to find the bottleneck that makes a tool effective.

And this is the area that is perhaps Gatling’s strength. While a few other open source tools require another tool or form of reporting to visualize this Gatling has in-built reporting that can visualize a lot of the important metrics like minimum response time, maximum response time, mean response time, and response time percentiles (e.g., 95th percentile), throughput (number of requests made per second), error rates (error rates as a percentage of failed requests), Latency (time taken between the start of a request and the receipt of the first byte of the response and also showcasing a distribution of these metrics over time.

These reports can be generated in a variety of formats like HTML, JSON, and Junit and if you’re looking for a tool that provides combined scripting and good analysis of a specific website, then Gating should provide you with what you need. Its logging is also quite extensive providing the detail you need should the visualization not give you a detailed enough view of the information.

Below is a basic history of Gatling and how it grew over time:

2009: Gatling was created by Stéphane Landelle in 2009. Stéphane is a software engineer with a background in both performance testing and Scala, the programming language in which Gatling is primarily written.
Open-Source Release: Gatling was released as an open-source project from the beginning, allowing developers and organizations to use it freely and contribute to its development.
Early Development: In its early stages, Gatling focused on providing a modern and expressive DSL for creating performance tests. Its design aimed to make it easier for testers and developers to script realistic user scenarios.
Continuous Development: Gatling has seen continuous development and improvements over the years, with regular releases that have introduced new features and enhancements. This ongoing development has made it a competitive choice in the performance testing tool landscape.
Community Growth: Gatling's active community of users and contributors has played a significant role in its growth and success. The community has provided support, tutorials, and plugins, further enriching the tool's ecosystem.
Protocol Support: Gatling has expanded its protocol support to cover a wide range of web and application protocols, making it suitable for testing diverse applications and services.
Integration with CI/CD: The tool's integration with Continuous Integration (CI) and Continuous Delivery (CD) pipelines has made it a valuable tool for DevOps teams looking to automate performance testing as part of their deployment processes.
Commercial Offerings: While the core of Gatling remains open-source, there are commercial offerings, such as Gatling FrontLine, which provide additional features and support for enterprises that require advanced performance testing capabilities.
Growing Popularity: Gatling has gained popularity in various industries, including finance, e-commerce, and technology. Its ability to simulate realistic user behavior and generate detailed reports has made it a preferred choice for performance testing.

Below are the pros and cons of using Gatling:

Pros:

Highly Scalable: Gatling can simulate a large number of virtual users, allowing you to test your application's performance under heavy loads.
Expressive DSL: Gatling's domain-specific language (DSL) for scripting tests is expressive and easy to use, making it simple to create complex test scenarios.
Realistic Scenarios: You can model realistic user behavior by defining scenarios with user actions, wait times, and loops, which helps in more accurate testing.
Detailed Reporting: Gatling generates comprehensive reports with a wealth of performance metrics, making it easy to identify bottlenecks and performance issues.
Open Source: Gatling is open-source and freely available, which makes it cost-effective for small and large organizations. Many of Gatling’s best reporting features though are paid for – so important to just be aware of that.
Active Community: Gatling has an active and growing community, providing access to a wealth of resources, including documentation, tutorials, and community support.
Protocol Support: It supports a wide range of protocols, including HTTP, HTTPS, WebSocket, JMS, and more, making it suitable for testing various types of applications and services.
CI Integration: Gatling integrates well with popular Continuous Integration (CI) tools like Jenkins, TeamCity, and Bamboo, allowing you to automate performance testing in your CI/CD pipeline. However as mentioned, the JVM dependency could make it a little heavy for regular pipeline use.
Modularization: It encourages script modularization, making it easier to reuse and maintain test components.
Extensible: Gatling is highly extensible, allowing you to write custom plugins and extensions to adapt them to your specific testing needs.

Cons:

Learning Curve: Gatling's DSL, while expressive, may have a steeper learning curve for beginners, especially those who are not familiar with Scala.
Scala Dependency: Gatling is written in Scala, and if your team is not already familiar with Scala, it may require additional training or expertise to use effectively.
Resource Intensive: Running tests with a large number of virtual users can be resource-intensive, requiring a powerful machine or distributed test setups.
Limited Protocol Support: While it supports a variety of protocols, it may not cover all the specific protocols or technologies used in every application.
Complex Scenarios: For very complex scenarios, script maintenance can become challenging over time, as scenarios grow in size and complexity.
Limited User Community: While the community is active, it may not be as extensive as some other performance testing tools, which could impact the availability of third-party plugins and support.
Steep Performance Overhead: Simulating a high load can introduce a performance overhead on the machine running Gatling tests, affecting the results if not managed properly.

When to use Gatling:

You have strong Scala skills within your organization.
You are looking for a tool that allows you to easily scale up your load and performance demands while providing detailed reporting.
You want a comprehensive tool but don’t want to pay for a more extensive tool

In summary, Gatling allows you to conduct various types of performance tests, including load testing, stress testing, and scalability testing. It helps you assess how the application performs under different levels of load, ensuring that it can handle expected traffic and scaling demands. It features fantastic reporting skills but can also be tricky to learn and implement if you want to write complex scripts in its Scala-based framework.

CRAIG RISI