How To: Model an Application Usage

J.D. Meier, Carlos Farre, Prashant Bansode, Scott Barber

Applies To

Summary

This How To explains to model workload characterization, which can be used for performance testing to simulate production characteristics of an application. When conducting performance testing with the intent of understanding, predicting, or tuning production performance, it is crucial that test conditions be similar or at least close to production usage or projected future business volume.

For accurate, predictive test results, user behavior must involve modeling the customer sessions based on page flow, frequency of hits, the length of time that users stop between pages, and any other factor specific to how users interact with your Web site.

Contents

Objectives

Overview

The process of identifying one or more composite application usage profiles for use in performance testing is known as workload modeling. The most common purpose of Web load tests is to simulate the user’s experience as realistically as possible. For performance testing to yield results that are directly applicable to understanding the performance characteristics of an application in production, the tested workloads must represent a real-world production scenario. To create a reasonably accurate representation of reality, you must understand the business context for the use of the application, expected transaction volumes in various situations, expected user path(s) by volume, and other usage factors. By focusing on groups of users and how they interact with the application, this How To demonstrates an approach to developing workload models that approximate production usage based on various data sources.

Testing a Web site in such a way that the test can reliably predict performance is often more art than science. As critical as it is to creating load and usage models that will predict performance accurately, the data necessary to create these models is typically not directly available to the individuals who conduct the testing. When it is, it is typically not complete or comprehensive.

While it is certainly true that simulating unrealistic workload models can provide a team with valuable information when conducting performance testing, you can only make accurate predictions about performance in a production environment, or prioritize performance optimizations, when realistic workload models are simulated.

Summary of Steps

Workload modeling can be accomplished in any number of ways, but to varying degrees the following activities are conducted, either explicitly or implicitly, during virtually all performance-testing projects that are successful in predicting or estimating performance characteristics in a production environment:

These activities are discussed in detail in the following sections.

Step 1 - Identify the Objectives

The objectives of creating a workload model typically center on ensuring the realism of a test, or on designing a test to address a specific requirement, goal, or performance-testing objective. When identifying the objectives, work with targets that will satisfy the stated business requirements. Consider the following key questions when formulating your objectives:

This information can be gathered from Web server logs, marketing documentation reflecting business requirements, or stakeholders. The following are some of the objectives identified during this process:
It is acceptable if these objectives only make sense in the context of the project at this point. The remaining activities will help you fill in the necessary details to achieve the objectives.

Considerations

Consider the following key points when identifying objectives:

Step 2 - Determine Key Usage Scenarios

To simulate every possible user task or activity in a performance test is impractical, if not a sheer impossibility. As a result, no matter what method you use to identify key scenarios, you will probably want to apply some limiting heuristic to the number of activities or key scenarios you identify for performance testing. You may find the following limiting heuristics useful:

The following information sources are frequently useful in identifying usage scenarios that fit into the categories above:

If you have access to Web server logs for a current implementation of the application ? whether it is a production implementation of a previous release, a representative prototype, or a beta release ? you can use data from those logs to validate and/or enhance the data collected using the resources above.

After you have collected a list of what you believe are the key usage scenarios, solicit commentary from the team members. Ask what they think is missing, what they think can be de-prioritized, and, most importantly, why. What does not seem to matter to one person may still be critical to include in the performance test. This is due to potential side effects that activity may have on the system as a whole, and the fact that the individual who suggests that the activity is unimportant may be unaware of the consequences.

Considerations

Consider the following key points when determining key usage scenarios:

Step 3 - Determine Navigation Paths for Key Scenarios

Now that you have a list of key scenarios, the next activity is to determine how individual users actually accomplish the tasks or activities related to those scenarios.

Human beings are unpredictable, and Web sites commonly offer redundant functionality. Even with a relatively small number of users, it is almost certain that real users will not only use every path you think they will to complete a task, but they also will inevitably invent some that you had not planned. Each path a user takes to complete an activity will put a different load on the system. That difference may be trivial, or it may be enormous ? there is no way to be certain until you test it. There are many methods to determine navigation paths, including:

After the application is released for unscripted user acceptance testing, beta testing, or production, you will be able to determine how the majority of users accomplish activities on the system under test by evaluating Web server logs. It is always a good idea to compare your models against reality and make an informed decision about whether to do additional testing based on the similarities and differences found.

Apply the same limiting heuristics to navigation paths as you did when determining which paths you wanted to include in your performance simulation, and share your findings with the team. Ask what they think is missing, what they think can be de-prioritized, and why.

Considerations

Consider the following key points when determining navigation paths for key scenarios:

Step 4 - Determine Individual User Data and Variances

No matter how accurate the model representing navigation paths and usage scenarios is, it is not complete without accounting for the data used by and the variances associated with individual users. While thinking of users as interchangeable entities leads to tests being simpler to design and analyze, and even makes some classes of performance issues easier to detect, it masks much of the real-world complexity that your Web site is likely to encounter in production. Accounting for and simulating this complexity is crucial to finding the performance issues most likely to be encountered by real users, as well as being an essential element to making any predictions or estimations about performance characteristics in production.

The sections that follow detail some of the sources of information from which to model individual user data and variances, and some of the data and variances that are important to consider when creating your model and designing your tests.

Web Site Metrics in Web Logs

Web site metrics are the variables that help you understand a site’s traffic and load patterns from the server’s perspective. Web site metrics are generally averages that may vary with the flow of users accessing the site, but they generally provide a high-level view of the site’s usage that is helpful in creating models for performance testing. These metrics ultimately reside in the Web server logs. (There are many software applications that parse these logs to present these metrics graphically or otherwise, but these are outside of the scope of this How To.) Some of the more useful metrics that can be read or interpreted from Web server logs (assuming that the Web server is configured to keep logs) include:

Step 5 - Determine the Relative Distribution of Scenarios

Having determined which scenarios to simulate and what the steps and associated data are for those scenarios, and having consolidated those scenarios into one or more workload models, you now need to determine how often users perform each activity represented in the model relative to the other activities needed to complete the workload model.

Sometimes one workload distribution is not enough. Research and experience have shown that user activities often vary greatly over time. To ensure test validity, you must validate that activities are evaluated according to time of day, day of week, day of month, and time of year. As an example, consider an online bill-payment site. If all bills go out on the 20th of the month, the activity on the site immediately before the 20th will be focused on updating accounts, importing billing information, and so on by system administrators, while immediately after the 20th, customers will be viewing and paying their bills until the payment due date of the 5th of the next month. The most common methods for determining the relative distribution of activities include:

Teams and individuals use a wide variety of methods to consolidate individual usage patterns into one or more collective models. Some of those include spreadsheets, pivot tables, narrative text, Unified Modeling Language (UML) collaboration diagrams, Markov Chain diagrams, and flow charts. In each case the intent is to make the model as a whole easy to understand, maintain, and communicate across the entire team.

One highly effective method is to create visual models of navigation paths and the percentage of users you anticipate will perform each activity that are intuitive to the entire team, including end users, developers, testers, analysts, and executive stakeholders. The key is to use language and visual representations that make sense to your team without extensive training. In fact, visual models are best when they convey their intended meaning without the need for any training at all. After you create such a model, it is valuable to circulate that model to both users and stakeholders for review/comment. Following the steps taken to collect key usage scenarios, ask the team members what they think is missing, what they think can be de-prioritized, and why. Often, team members will simply write new percentages on the visual model, making it very easy for everyone to see which activities have achieved a consensus, and which have not.
Once you are confident that the model is appropriate for performance testing, supplement that model with the individual usage data collected for each navigation path during the “Determine Individual User Data and Variances” activity, in such a way that the model contains all the data you need to create the actual test.


NavPaths.GIF

Figure 1 Visual Model of Navigation Paths

Considerations

Consider the following key points when determining the relative distribution of scenarios:

Step 6 - Identify Target Load Levels

A customer visit to a Web site comprises a series of related requests known as a user session. Users with different behaviors who navigate the same Web site are unlikely to cause overlapping requests to the Web server during their sessions. Therefore, instead of modeling the user experience on the basis of concurrent users, it is more useful to base your model on user sessions. User sessions can be defined as a sequence of actions in a navigational page flow, undertaken by a customer visiting a Web site.

Quantifying the Volume of Application Usage: Theory

It is frequently difficult to determine and express an application’s usage volume because Web-based multi-user applications communicate via stateless protocols. Although terms such as “concurrent users” and “simultaneous users” are frequently used, they can be misleading when applied to modeling user visits to a Web site. In Figures 2 and 3 below, each line segment represents a user activity, and different activities are represented by different colors. The solid black line segment represents the activity “load the Home page.” User sessions are represented horizontally across the graph. In this hypothetical representation, the same activity takes the same amount of time for each user. The time elapsed between the Start of Model and End of Model lines is one hour.


ServerPerspective.GIF

Figure 2 Server Perspective of User Activities

Figure 2 above represents usage volume from the perspective of the server (in this case, a Web server). Reading the graph from top to bottom and from left to right, you can see that user 1 navigates first to page “solid black” and then to pages “white,” “polka dot,” “solid black,” “white,” and “polka dot.” User 2 also starts with page “solid black,” but then goes to pages “zebra stripe,” “grey,” etc. You will also notice that virtually any vertical slice of the graph between the start and end times will reveal 10 users accessing the system, showing that this distribution is representative of 10 concurrent, or simultaneous, users. What should be clear is that the server knows that 10 activities are occurring at any moment in time, but not how many actual users are interacting with the system to generate those 10 activities.

Figure 3 below depicts another distribution of activities by individual users that would generate the server perspective graph above.

ServerDistribution.GIF

Figure 3 Actual Distribution of User Activities Over Time

In this graph, the activities of 23 individual users have been captured. Each of these users conducted some activity during the time span being modeled, and their respective activities can be thought of as 23 user sessions. Each of the 23 users began interacting with the site at a different time. There is no particular pattern to the order of activities, with the exception of all users who started with the “solid black” activity. These 23 users actually represent the exact same activities in the same sequence shown in Figure 2. However, as depicted in Figure 3, at any given time there are 9 to 10 concurrent users. The modeling of usage for the above case in terms of volume can be thought of in terms of total hourly users, or user sessions counted between “Start of Model” and “End of Model.”

Without some degree of empirical data (for example, Web server logs from a previous release of the application), target load levels are exactly that — targets. These targets are most frequently set by the business, based on its goals related to the application and whether those goals are market penetration, revenue generation, or something else. These represent the numbers you want to work with at the outset.

Quantifying the Volume of Application Usage

If you have access to Web server logs for a current implementation of the application — whether it is a production implementation of a previous release, a representative prototype, or a beta release — you can use data from these logs to validate and/or enhance the data collected by using the resources above. By performing a quantitative analysis on Web server logs, you can determine:

The following are the inputs and outputs used for determining target load levels.
Inputs
Output
By combining the volume information with objectives, key scenarios, user delays, navigation paths, and scenario distributions from the previous steps, you can determine the remaining details necessary to implement the workload model under a particular target load.

Integrating Model Variance

Because the usage models are “best guesses” until production data becomes available, it is a good idea to create no fewer than three usage models for each target load. This has the effect of adding a rough confidence interval to the performance measurements. Stakeholders can focus on the results from one test based on many fallible assumptions, as well as on how many inaccuracies in those assumptions are likely to impact the performance characteristics of the application.

The three usage models that teams generally find most valuable are:

The following chart is an example of the information that testing for all three of these models can provide. As you can see, in this particular case the Anticipated Usage and Best Case Usage resulted in similar performance characteristics. However, the Worst Case Usage showed that there is nearly a 50-percent drop-off in the total load that can be supported between it and the Anticipated Usage. Such information could lead to a reevaluation of the usage model, or possibly to a decision to test with the Worst Case Usage model moving forward as a kind of safety factor until empirical data becomes available.


UsageModels.GIF

Figure 4 Usage Models

Considerations

Consider the following key points when identifying target load levels:

Step 7 - Prepare to Implement the Model

Implementation of the workload model as an executable test is tightly tied to the implementation method — typically, creating scripts in a load-generation tool.

Considerations

Consider the following key points when preparing to implement the model:

Resources