How To: Stress Test Web Applications

J.D. Meier, Carlos Farre, Prashant Bansode, Scott Barber

Applies To

Summary

This How To represents a high-level introduction to stress-testing a Web application. Stress testing allows you to identify potential application issues that surface only under extreme conditions. Such conditions range from exhaustion of system resources such as memory, processor cycles, network bandwidth, and disk capacity to excessive load due to unpredictable usage patterns, common in Web applications.

Stress testing centers around objectives and key user scenarios with an emphasis on the robustness, reliability, and stability of the application. The effectiveness of stress testing relies on applying the correct methodology and being able to effectively analyze testing results. Applying the correct methodology is dependent on the capacity for reproducing workload conditions for both user load and volume of data, reproducing key scenarios, and interpreting the key performance metrics.

Contents

Objectives

Overview

Stress testing is a type of performance testing focused on determining an application’s robustness, availability, and reliability under extreme conditions. The goal of stress testing is to identify application issues that arise or become apparent only under extreme conditions. These conditions can include heavy loads, high concurrency, or limited computational resources. Proper stress testing is useful in finding synchronization and timing bugs, interlock problems, priority problems, and resource loss bugs. The idea is to stress a system to the breaking point in order to find bugs that will make that break potentially harmful. The system is not expected to process the overload without adequate resources, but to behave (e.g., fail) in an acceptable manner (e.g., not corrupting or losing data).

Stress tests typically involve simulating one or more key production scenarios under a variety of stressful conditions. For example, you might deploy your application on a server that is already running a processor-intensive application; in this way, your application is immediately “starved” of processor resources and must compete with the other application for processor cycles. You can also stress-test a single Web page or even a single item such as a stored procedure or class.

Examples of Stress Conditions

Examples of stress conditions include:

Examples of Stress-Related Symptoms

Examples of stress-related symptoms include:

Input

To perform stress testing, you are likely to use as reference one or more of the following items:

Output

Output from a stress test may include:

Summary of Steps

The following steps are involved in stress-testing a Web application:


These steps are graphically represented below; the following sections discuss each step in detail.

StressTesting1.GIF

Figure 1 Stress Testing Steps

Step 1 - Identify Test Objectives

Identify the desired outcomes of your stress testing by asking yourself the following questions:
  1. Is the purpose of the test to identify the ways the system can possibly fail catastrophically in production?
  2. Is it to provide information to the team in order to build defenses against catastrophic failures?
  3. Is it to identify how the application behaves when system resources such as memory, disk space, network bandwidth, or processor cycles are depleted?
  4. Is it to ensure that functionality does not break under stress? For example, there may be cases where operational performance metrics meet the objectives, but the functionality of the application is failing to meet them — orders are not inserted in the database, the application is not returning the complete product information in searches, form controls are not being populated properly, redirects to custom error pages are occurring during the stress testing, and so on.

Step 2 - Identify Key Scenario(s)

To get the most value out of a stress test, the test needs to focus on the behavior of the usage scenario or scenarios that matter most to the overall success of the application. To identify these scenarios, you generally start by defining a single scenario that you want to stress-test in order to identify a potential performance issue. Consider these guidelines when choosing appropriate scenarios:


Scenarios that may need to be stress tested separately for a typical e-commerce application include the following:

Step 3 - Identify the Workload

The load you apply to a particular scenario should stress the system sufficiently beyond threshold limits to enable you to observe the consequences of the stress condition. One method to determine the load at which an application begins to exhibit signs of stress is to incrementally increase the load and observe the application behavior under various load conditions. The key is to systematically test with various workloads until you create a significant failure. These variations may be accomplished by adding more users, reducing delay times, adding or reducing the number and type of user activities represented, or adjusting test data.

For example, a stress test could be designed to simulate every registered user of the application attempting to log on during one 30-second period. This would simulate a situation where the application suddenly became available again after a period of downtime and all users were anxiously refreshing their browsers, waiting for the application to come back online. Although this situation does not occur frequently in the real world, it does happen often enough for there to be real value in learning how the application will respond if it does.

Remember to represent the workload with accurate and realistic test data — type and volume, different user logins, product IDs, product categories, and so on — allowing you to simulate important failures such as deadlocks or resource consumption.

The following activities are generally useful in identifying appropriate workloads for stress testing:

The table below provides an example of a peak workload profile identified for a sample application.

Table Sample Workload Profile for a Sample Application

User profile Percentage Simultaneous users
Browse 50% 500
Search 30% 300
Order 20% 200
Total 100% 1,000


The table below provides an example of an anti-profile workload identified for a sample application.

Table: Order Use Case Anti-Profile

User profile Percentage Simultaneous users
Browse 10% 100
Search 10% 100
Order 80% 800
Total 100% 1,000


Continue to increase the load for this scenario, and observe the application response at various load levels. The time duration of the tests depends upon the scenario. For example, it might take 10 to 15 days to simulate deadlocks on a four-processor machine, but the same deadlocks may be simulated in half that time on an eight-processor machine.

Step 4 - Identify Metrics

When identified and captured correctly, metrics provide information about how well or poorly your application is performing as compared to your performance objectives. In addition, metrics can help you identify problem areas and bottlenecks within your application.

Using the desired performance characteristics identified during the “Identify objectives” step, identify metrics to be captured that focus on potential pitfalls for each scenario. The metrics can be related to both performance and throughput goals as well as providing information about potential problems; for example, custom performance counters that have been embedded in the application.

When identifying metrics, you will use either direct objectives or indicators that are directly or indirectly related to those objectives. The following table describes performance metrics in terms of related performance objectives.

Performance metrics Category
Base set of metrics
Processor Processor utilization
Process Memory consumption
Processor utilization
Process recycles
Memory Memory available
Memory utilization
Disk Disk utilization
Network Network utilization
Transactions/business metrics Transactions/sec
Transactions succeeded
Transactions failed
Orders succeeded
Orders failed
Threading Contentions per second
Deadlocks
Thread allocation
Response times Transactions times

Step 5 - Create Test Cases

Identifying workload profiles and key scenarios generally does not provide all of the information necessary to implement and execute test cases. Additional inputs for completely designing a stress test include performance objectives, workload characteristics, test data, test environments, and identified metrics. Each test design should mention the expected results and/or the key data of interest to be collected, in such a way that each test case can be marked as a “pass,” “fail,” or “inconclusive” after execution.

The following is an example of a test case based on the order-placement scenario.

Test 1 – Place Order Scenario

Workload: 1,000 simultaneous users.
Think time: Use a random think time between 1 and 10 seconds in the test script after each operation.
Test Duration: Run the test for two days.

Expected results:

Step 6 - Simulate Load

After you have completed the previous steps to an appropriate degree, you should be ready to simulate the load executing the stress test. Typically, test execution follows these steps:
  1. Validate that the test environment matches the configuration that you were expecting and/or designed your test for.
  2. Ensure that both the test and the test environment are correctly configured for metrics collection.
  3. Before running the test, execute a quick “smoke test” to make sure that the test script and remote performance counters are working correctly.
  4. Reset the system (unless your scenario is to do otherwise) and start a formal test execution.

Note: Make sure that the client (a.k.a. load generator) computers that you use to generate load are not overly stressed. Utilization of resources such as processor and memory should remain low enough to ensure that the load-generation environment is not itself a bottleneck.

Step 7 - Analyze Results

Analyze the captured data and compare the results against the metric’s accepted level. If the results indicate that your required performance levels have not been attained, analyze and fix the cause of the bottleneck. To address observed issues, you might need to do one or more of the following:


In situations where performance issues are observed, but only under conditions that are deemed to be unlikely enough to warrant tuning at the current time, you may want to consider conducting additional tests to identify an early indicator for the issue in order to avoid unwanted surprises.

Usage Scenarios for Stress Testing

The following are examples of how stress testing is applied in practice:

Exploratory Stress Testing

Exploratory stress testing is an approach to subjecting a system, application, or component to a set of unusual parameters or conditions that are unlikely to occur in the real world but are nevertheless possible. In general, exploratory testing can be viewed as an interactive process of simultaneous learning, test design, and test execution. Most often, exploratory stress tests are designed by modifying existing tests and/or working with application/system administrators to create unlikely but possible conditions in the system. This type of stress testing is seldom conducted in isolation because it is typically conducted to determine if more systematic stress testing is called for related to a particular failure mode. The following are some examples of exploratory stress tests to determine the answer to “How will the system respond if…?”

Resources