How To: Coordinate Performance Testing with an Iteration-Based Process

J.D. Meier, Carlos Farre, Prashant Bansode, Scott Barber

Applies To

Summary

This How To provides guidance for coordinating performance testing with iteration-based processes that are found in Agile Software Development, Extreme Programming (XP), the Rational Unified Process (RUP), and other sources. The How To describes the concepts underlying the Steps necessary to make performance testing successful within an iterative process, as well as specific, actionable items that you can immediately apply to your project in order to gain a significant return on this investment.

Performance testing with iteration-based processes is a common practice in development cycles such as Agile, XP, RUP, and other sources. To be effective, performance testing should be managed correctly in the context of iteration planning and processes.

Contents

Objectives

Overview

Performance testing is a critical aspect of many software projects because it tests the architectural aspects of the customer experience and provides an indication of overall software quality. Because it is frequently expensive to set up and integrate performance testing, project teams often wait until the end of the project development/test life cycle to do so. The potential side effect to this approach is that when major issues are found near the end of the development life cycle, it becomes much more expensive to resolve them.

The key to working within an iteration-based work cycle is team coordination. For this reason, the performance tester must be able to adapt what he or she measures and analyzes per iteration cycle as circumstances change.

Introduction to the Approach

When viewed from a linear perspective, the approach starts by examining the software development project as a whole, the reasons why stakeholders have chosen to include performance testing in the project, and the value that performance testing is expected to bring to the project. The results of this examination include the team’s view of the success criteria for the performance-testing effort.

Once the success criteria are understood at a high level, an overall strategy is envisioned to guide the general approach to achieving those criteria by summarizing what performance testing Steps are anticipated to add the most value at various points during the development life cycle. Those points may include key project deliveries, checkpoints, sprints, iterations, or weekly builds. For the purposes of this How To, these events are collectively referred to as “performance builds.” Frequently, while the strategy is evolving, the performance specialist and/or the team will begin setting up a performance-test environment and a load-generation environment.

With a strategy in mind and the necessary environments in place, the test team draws up plans for major tests or tasks identified for imminent performance builds. When a performance build is delivered, the plan’s tasks should be executed in priority sequence (based on all currently available information), appropriately reporting, recording, revising, reprioritizing, adding, and removing tasks and improving the application and the overall plan as the work progresses.

Summary of Steps

This approach can be represented by using the following nine Steps:

Step 1. Understand the Project Vision and Context

The project vision and context are the foundation for determining what performance testing Steps are necessary and valuable. Because the performance tester is not driving these items, the coordination aspect refers more to team education about the performance implications of the project vision and context, and to identifying areas where future coordination will likely be needed for success.

A critical part of working with an iteration-based process is asking the correct questions, providing the correct value, and performing the correct task related to each step. Although situations can shift or add more questions, values, or tasks, a sample checklist is provided as a starting point for each step.

Checklist

Questions to ask:
Value provided:
Tasks accomplished:
Coordinate with:

Step 2. Identify Reasons for Testing Performance

The underlying reasons for testing performance on a particular project are not always obvious based on the vision and context alone. Project teams generally do not include performance testing as part of the project unless there is some performance-related risk or concern they feel needs to be mitigated. Explicitly identifying these risks and areas of concern is the next fundamental step in determining what specific performance testing Steps will add the most value to the project.

Having a full-time performance tester on the team from the start of the project may frequently be a good idea, but it does not happen frequently. Generally, when a performance tester is present at project inception, it means there is a specific, significant risk that the tester is there to address.

Regardless of when a performance tester joins the team, once the project vision and context are understood, it is worth taking the time to verbalize and/or document the overall objectives of the performance-testing effort based on the risks or concerns that the team has. The following checklist should help you to accomplish this step.

Checklist

Questions to ask:
Value provided:
Tasks accomplished:
Coordinate with:

Step 3. Identify the Value Performance Testing Adds to the Project

Using information gained from Steps 1 and 2, you can now clarify the value added through performance testing, and convert that value into a conceptual performance-testing strategy. The point is to translate the project- and business-level objectives into specific, identifiable, and manageable performance-testing Steps. The coordination aspect of this step involves team-wide discussion and agreement on which performance-testing Steps are likely to add value or provide valuable information, and if these Steps are worth planning for at this time.

Checklist

Questions to ask:
Value provided:
Tasks accomplished:
Coordinate with:

Step 4. Configure the Test Environment

With a conceptual strategy in place, prepare the tools and resources in order to execute the strategy as features and components become available for test. Take this step as soon as possible, so that the team has this resource from the beginning.

This step is fairly straightforward. Set up the load-generation tools and the system under test — collectively known as the performance test environment — and ensure that this environment will meet engineering needs. The coordination component of this step typically involves asking managers and administrators to obtain and/or configure equipment and other resources that are not under the direct control of the team or performance tester.

Checklist

Questions to ask:
Value provided:
Tasks accomplished:
Coordinate with:

Step 5. Identify and Coordinate Tasks

Performance testing tasks do not happen in isolation. The performance specialist needs to work with the team to prioritize and coordinate support, resources, and schedules to make the tasks efficient and successful.

During the pre-iteration planning meeting, look at where the project is now and where you want to be to determine what should and can be done next. When planning for the iteration cycle, the performance tester is driven by the goals that have been determined for this cycle. This step also includes signing up for the Steps that will be accomplished during this cycle.

Checklist

Questions to ask:
Value provided:
Tasks accomplished:
Coordinate with:

Step 6. Execute Task(s)

Conduct tasks in one- to two-day segments. See them through to completion, but be willing to take important detours along the way if an opportunity to add additional value presents itself. Step 5 defines what work the team members will sign up for in this iteration. Now it is time to execute the Steps for this iteration.

Checklist

Questions to ask:
Value provided:
Tasks accomplished:
Coordinate with:

Step 7. Analyze Results and Report

To keep up with an iterative process, results need to be analyzed and shared quickly. If the analysis is inconclusive, retest at the earliest possible opportunity to give the team maximum time to react to performance issues. As the project is wrapped for final shipping, it is usually worth having a meeting afterward to collect and pass along lessons learned. In most cases it is valuable to have a daily or every-other-day update to share information and coordinate next tasks.

Checklist

Questions to ask:
Value provided:
Tasks accomplished:
Coordinate with:

Step 8. Revisit Steps 1-3 and Consider Performance Acceptance Criteria

Between iterations, ensure that the foundational information has not changed. Integrate new information, such as customer feedback, and update the strategy as necessary.

Checklist

Questions to ask:
Value provided:
Tasks accomplished:
Coordinate with:

Step 9. Reprioritize Tasks

Based on the test results, new information, and the availability of features and components, reprioritize, add to, or delete tasks from the strategy, and then return to Step 5.

Checklist

Questions to ask:
Value provided:
Tasks accomplished:
Coordinate with:

Resources