How To: Consolidate Various Types of Performance Acceptance Criteria

J.D. Meier, Carlos Farre, Prashant Bansode, Scott Barber

Applies To

Summary

Performance requirements and testing objectives are typically derived from the perspectives of system users, business owners, and the project team, as well as compliance expectations and technological considerations.

To have confidence in the completeness of your system acceptance criteria, they should be based on the information collected from all these different perspectives, which should result in a single set of verifiable, complementary performance requirements and testing objectives.

It is important to always keep in mind that the performance characteristic that matters most is that application users must not be frustrated with poor performance.

Contents

Objectives

Overview

This How To demonstrates how to blend and consolidate the information collected from these sources into a single set of verifiable, complementary performance requirements and testing objectives.

Determining the desired performance characteristics for a system from particular perspectives is only the first part of determining the overall performance acceptance criteria for that system. After examining the desired performance characteristics from limited perspectives, you must resolve those characteristics against one another. For example, end users may desire sub-second response time for every transaction; business stakeholders may want the system to support millions of users; and compliance criteria may mandate strict security policies.

Individually, any of these characteristics may be achievable, but collectively they may not be possible due to time, technology, and/or budget constraints. Finding a way to achieve these desired characteristics presents team-wide challenges that should be addressed proactively, not reactively after conflicts become apparent through testing.

Summary of Steps

Consolidating acceptance criteria can be thought of in terms of the following activities:

These activities are discussed in detail in the following sections.

Step 1. Investigate End-User Requirements

Once your application reaches production, the most important performance characteristic that matters is that application users must not be frustrated with poor performance. If users are annoyed, they will find an alternative to your application. If they do become frustrated, it will not matter how many users your application supports, how much data it can process, or how efficient it is in its use of resources — in fact, even if you accomplished all of the above and were the first to market the application with the new features, it will still mean nothing to the user.

Users of your application will not know or care about the results of the performance tests, how many seconds it takes the screen to display past a user’s normal threshold for “too long,” or what the throughput is. Primarily, application users notice only whether the application seems slow or not. Also, users’ reactions (or not) can be based on anything, including how they feel at the time or their overall experience with applications.

Quantifying end-user satisfaction and/or frustration can be challenging but is far from impossible. To quantify end-user satisfaction, all you need is an application and some representative users. You do not need a completed application, since a prototype or demo will suffice for a first pass. For example, with only a few lines of code in the HTML of a demo or prototype, you can control the load time for each page, screen, graphic, control, or list. Using this method, you can create several versions of the application with different response characteristics. You can then ask users to try each version and tell you whether they find that version unacceptable, slow, reasonable, fast, or whatever terms correspond to the captured goals.

Because you know the actual response times, you can start equating those numbers to the users’ reported degrees of satisfaction. This is not an exact science, but it generally works well for goals — especially if you follow up by asking the same questions about the application’s performance each time you test it. This applies for functional testing, user acceptance testing, beta testing, or any other reason, as you are measuring response times in the background as testers interact with the system. This approach allows you to collect more data and enhance your end-user performance goals as the application evolves.

Step 2. Collect Business Requirements

The real engineering challenge is not only being able to meet your business requirements, but also achieving them on time and within budget. Before you can find this balance, you must first determine those business requirements. For example, you will need to determine the budget for new hardware, and what existing hardware is available. You will need to know how rigid the target delivery date is, whether an initial release to a limited number of users is acceptable, and what project aspects take priority. For instance, if a situation arises where you can achieve the desired performance on time and on budget by eliminating valuable features, you must ask yourself whether that tradeoff is worth considering.

Creating a performance plan — not a performance test plan, but an overall plan for achieving desired performance characteristics — is the process most teams use to demonstrate how to spend the technical and financial performance budget in order to create the desired experiences, and to determine the overall cost of that performance in terms of dollars, resources, and schedule. Some areas worthy of significant consideration in a performance plan include:
You may also be affected by external business requirements in addition to the internal requirements. An example of an external business requirement is achieving performance characteristics that are “as good as or better than” competing applications. Every application in every vertical industry will have different methods and sources for determining its competitive landscape. Even if you cannot run load tests against a competitor, you can still collect valuable response time information by, for example, periodically surfing the site and from researching sites such as www.keynote.com.
The bottom line: do not assume that your site will not be compared against others simply because there is no published standard; de-facto standards are more likely to set your users’ expectations than formal metrics do.

Step 3. Determine Technical Requirements

Usually, technical performance characteristics only indirectly relate to the other categories of requirements. These characteristics frequently concern the use of particular technologies, and usually consist of targets and thresholds that are specific metrics related to particular resources.

For example, it is generally agreed that once a server’s processor utilization reaches 80 percent, user requests will start to significantly queue, and therefore users will experience poor service even though the server might still be running fine. Based on this, many teams will set a processor utilization target of 70 percent and a threshold of 80 percent. By doing so, they are telling the performance tester to alert the team if he or she observes readings of more than 70-percent processor utilization sustained for more than a few seconds, and to register a defect if a processor utilization rate of more than 80 percent is observed for more than a few seconds under the target workloads.

Capturing and managing technical performance-testing requirements generally requires that you:

Step 4. Research Standards, Compliance, and Contracts

The performance characteristics in this category are frequently the least negotiable, whether or not they have a significant impact on end-user satisfaction, budget, schedule, or technology. Determining performance requirements in this category generally involves the following tasks:

Step 5. Establish Performance-Testing Objectives

Determining performance-testing objectives can be fairly easy. The challenge is that the performance tester does not always have easy access to either explicit or implied objectives, and therefore frequently must conduct a systematic search for these objectives. Determining performance-testing objectives generally involves the following tasks:

While it is most valuable to collect performance testing objectives early in the project life cycle, it is also important to periodically revisit these objectives and ask team members if they would like to see any new objectives added.

Step 6. Compare and Consolidate Performance Characteristics

After you have identified performance characteristics from each of these categories, it is important to compare and consolidate them. Often, the best way to do so is to employ a cross-functional team while developing or enhancing the performance plan. Some of the key discussions, activities, and tradeoffs to consider include:

The key here is to focus on the experience you are trying to create, and from that, determine the associated costs of achieving that experience. In many cases this can lead to reverse-engineering desired end-user performance characteristics into actionable technical requirements, and then extrapolating those technical requirements into such relevant costs as time and money.

Step 7. Review and Update the Performance Plan

As with many aspects of software development, the performance plan is a moving target. The previous six activities, if you accomplish them early in the development life cycle, will likely lead to a tentative application design that can be evaluated against technical and business requirements. As long as that design continues to make sense when evaluated against the requirements, stick with it and flesh it out incrementally as you go. When that design stops making sense when evaluated against the requirements, recalibrate until it does make sense.

You are likely to iterate through this process many times throughout the development cycle, each time potentially updating the application design, the performance plan, and the performance requirements. Do not try to flesh it out all at once, and do not expect perfection. The key is to keep the design, the plan, and the requirements in sync and moving together to achieve satisfied end users and happy stakeholders once the application goes live.

Resources