pnp.GIF

How To: Evaluate Systems to Improve Performance Testing Effectiveness

Carlos Farre, Scott Barber

Applies to

Summary

While System Evaluation is an ongoing process throughout the performance testing effort, it offers greater value if it is conducted early on. The intent is to collect information about the project as a whole, the functions of the system, the expected user activities, the system architecture, and any other details that are helpful in guiding performance testing to achieve the specific needs of the project. This information provides a foundation for collecting the performance goals and requirements, characterizing the workload, creating performance test strategies and plans, and assessing project and system risks.

Contents

Objectives

Overview

Evaluation of the system under test is critical to a successful performance testing effort. The measurements gathered during later aspects are only as accurate as the models that are developed and validated in this aspect. The evaluation provides a foundation for determining acceptable performance; specifying performance requirements of the software, system, or component(s); and identifying any risks to the effort before testing even begins.

Evaluating the system includes, but isn’t limited to, the following steps:

Steps

Step 1. Capture System Functions

This step identifies the system’s core functions that will help build the performance acceptance criteria. Subsequently, workload models can be assessed to validate both the acceptance criteria and the collection of system functions.
For performance testing, it is essential to know the core functions of the system under test, which makes it possible to initially determine performance acceptance criteria as well as the user community models used to assess the application’s success in meeting the acceptance criteria.
To ensure that all of the system functions are captured, start by meeting with stakeholders to determine the overall purpose of the system or application. Before you can determine how best to test a system, you must first completely understand the intent of the system. It is often the case that the project documents don’t explicitly express all of the functionality implied by the stakeholders’ vision. This is why it is a good idea to start with the stakeholders before moving on to evaluate documentation.
Valuable resources for determining system functionality include:

Considerations

Step 2. Capture User Activities

In this step, you will identify the key user activities, since it is not practical to simulate all user tasks. It is impractical and virtually impossible to simulate every possible user task or activity in a performance test. However, before you can decide which activities are most important to simulate during performance testing you need to determine what the possible user activities are.

One of the most effective places to start is to evaluate the competition’s Web site. Whether it is explicitly stated or not, at some point during the project it is likely to become very obvious that the goal is to allow users to perform all of the activities available from the competitor. Knowing up front what these activities are will keep you from being surprised when they show up in the application – whether or not they appear in any of the documentation.
Valuable resources for determining system functionality include: Once you collect a list of what you believe are all of the activities a user can perform, circulate the list to the team along with the question: “What else can a user of any type possibly do with this application that isn’t on this list?”

Considerations

Step 3. Capture Logical and Physical Architectures

This step identifies the relationship between the application and the structure of the hardware and software. This information is critical when you are designing performance tests to address specific areas of concern, and when you are trying to track down a performance bottleneck.

Of the three aspects of system evaluation, this is frequently the one performed least well. It can lead to adverse affects on performance testing later in the project and add time to the tuning process. To capture the logical and physical architectures, the performance tester generally meets with technical stakeholders, architects, and administrators to determine both the physical and programmatic architecture of the system or application for both the production and test environments. This is critical because designing an effective test strategy requires the performance tester to be aware of which components or tiers of the system communicate with one another and how they do so. It is also valuable to understand the basic structure of the code and contributing external software.

Because the term “architecture” is used in so many different ways by different teams, the following sections have been included for clarity

Logical Architecture

Logical architecture, as it is used in this How To, refers to the structure, interaction, and abstraction of software and/or code. That code may include everything from objects, functions, and classes, or even entire applications such as IIS. You will have to learn the code-level architecture from your team, as every team accomplishes this differently. However, there is a common aspect of logical architectures that is worth exploring further – the concept of logical architectural tiers.

The most basic architecture for Web-based applications is known as the three-tier architecture, where those tiers often correspond to physical machines with roles defined as follows: Evaluate System1.GIF
As architectures became more complex, more tiers were added and those tiers began to be comprised of clusters of machines that served the same role, or of single machines serving as the host for multiple logical tiers.
Evaluate System2.GIF

Specifically, this complexity implies the following:
To confuse the matter even further, virtually no one uses terms like “file storage tier.” The “file storage tier” has become colloquially named “the file server,” whether or not that tier resides on a dedicated server or not. The case is the same for the presentation tier (Web server), application or business logic tier (application server or app server in shorthand), data storage tier (database server), etc.

Put simply, the key to understanding a logical architecture is the following: In a logical architecture, each tier contains a unique set of functionality that is logically separated from the other tiers. However, even if a tier is commonly referred to as “server,” it is not safe to assume that every tier lives on its own dedicated machine

Physical Architecture

It should be clear that the physical architecture of the environment – i.e., the actual hardware that runs the software – is at least as important as the logical architecture.

Many teams refer to the actual hardware as the “environment” or the “network architecture,” but neither term actually encompasses everything a performance tester should be interested in. What concerns you is generally represented in diagrams where actual, physical computers are shown and labeled with the roles they play along with the other actual, physical computers that they talk to.
Evaluate System3.GIF

System Architecture


Putting these two pieces of the puzzle together adds most value for the performance testing effort. Having this information at your fingertips, along with the more detailed code architecture of what functions or activities are handled on which tiers is what allows you to design tests that can determine and isolate bottlenecks
Evaluate System4.GIF

Considerations

Additional Resources

<<TBD>>