pnp.GIF

How To: Model Application Usage without Empirical Data

Scott Barber,

Applies To

Summary

This How-To demonstrates an approach to developing user community models that represent a realistic approximation of the usage of the application when there is no empirical data available to model from by focusing on groups of users and how they interact from the perspective of the application. Applications typically allow for many ways to accomplish a task or many methods to navigate the application. To effectively predict performance in production, these variants of individual user patterns must be accounted for. To do this effectively, the model needs to available to and understandable by the entire team with little or no explanation allowing for ease of conversation and resulting in more accurate user models.

Contents

Objectives

Overview

Testing a website such that the test can reliably predict performance is often more of an art than a science. More than a few brilliant minds have published detailed and mathematically sound methods to plan for, predict and model performance characteristics very accurately… as long as there is sufficient empirical data to work from.

As critical as it is to creating load and usages models that will predict performance accurately, the data necessary to create these models is typically not directly available to the individuals actually conducting the testing, if it exists at all. Since all of the generally accepted methods and techniques for testing and predicting application performance depends on usage and load models, an alternative to empirical data is required. This How To explores an approach to creating load and usage models that is a quick and effective substitute for empirical data.

Steps

Step 1. Determine Key Scenarios

It is typically somewhere between impractical and impossible to simulate every possible user task or activity in a performance test. As a result, no matter what method you use to identify key scenarios, you will probably want to apply some limiting heuristic to the number of activities, or key scenarios you identify for performance testing. You may find the following limiting heuristics useful:
Determining these items without empirical data is an indirect process based on evaluating information collected from places like the following:
Once you collect a list of what you believe are the key tasks or usage scenarios, solicit commentary from the team. Ask what they think is missing, what they think can be de-prioritized and why. The “why” part is most important. What doesn’t seem to matter to one person may still be critical to include in the performance test due to side effects that activity has on the system as a whole that the individual suggesting that the activity isn’t important may know be aware of.

Considerations

Step 2. Determine Individual User Patterns

With a list of key scenarios, the next step is to determine how individual users actually accomplish the tasks or activities related to those scenarios and the user specific data associated with a user accomplishing that task or activity.

Navigation

Human beings are unpredictable and web sites commonly offer multiple paths to accomplish the same task or activity. Even with a relatively small number of users, it is almost certain that real users will not only use every path you think they will to complete a task, they will inevitably invent some that you hadn’t thought of. Each path they take to complete an activity will put a different load on the system. That difference may be trivial, it may be enormous. There is no way to be certain until we test it. There are many methods to determine navigation paths to complete a task or activity. Some include:

Once the application is released for unscripted user acceptance testing, beta testing or to production, you will be able to determine how the majority of users accomplish activities on the system under test. It is always a good idea to compare your models against reality and make an informed decision about whether to do additional testing based on the similarities and differences found.

Apply the same limiting heuristics to navigation paths as you did when determining activities to decide which paths you want to include in your performance simulation.

Considerations

Data

Unfortunately, navigation paths alone don’t provide all of the information required to implement a workload simulation. To fully implement the workload model, several more pieces of information are needed. This information includes items such as:
Below is an example of unique data identified for an eCommerce application:
Implementation Data
ScenarioPage/ StepData InputsData OutputsThink Time
Login
Login pageUsername (unique), Password (matched to username) 6 – 9 Sec, Random
Browse
Login PageUsername (unique), Password (matched to username) 6 – 9 Sec, Random
BrowseCatalog Tree/Structure (static), User Type (weighted)Product Description, Title, Category4 – 60 Sec, Random

Considerations

Step 3. Consolidate Individual Patterns Into One or More Collective Models

There are a wide variety of methods that teams and individuals use to consolidate individual usage patterns into one or more collective models. Some of those include spreadsheets, pivot tables, narrative text, UML collaboration diagrams, Markov Chain diagrams and flow charts. In each case the intent is to make the model as a whole easy to understand, maintain and communicate across the entire team.
One highly effective method is to create visual models that are intuitive to the entire team, including end-users, developers, testers, analysts and executive stakeholders. The key to this technique is to use language and visual representations that make sense to your team without extensive training. In fact, visual models that convey their intended meaning with no training what-so-ever are best. Consider the following visualization of a consolidated usage model for an eCommerce application.
ImpiricalData1.GIF
As you can see, the combination of labeled user activities, flow lines and color make it easy for the entire team to understand. Once such a model is created, it is valuable to circulate that model to both users and stakeholders for review/comment. Just like when you collected key usage scenarios, ask the team what they think is missing, what they think can be de-prioritized and why.
Once you are confident that the model is appropriate for performance testing, supplement that model with the individual usage data collected for each navigation path during step 2 such that the model contains all of the data that will be needed to create the actual test. The table below represents one way that supplementary data can be organized in a spreadsheet.
ImpiricalData2.GIF

Considerations

Step 4. Determine Distribution of Activities

Now that you’ve determined what scenarios you want to simulate, the steps and associated data are for those scenarios, and consolidated those scenarios into visual models, you need to determine how often each activity represented in the model is performed relative to the other activities to complete the workload model. The most common methods to determine the relative distribution of activities with no empirical data to draw from are:
Possibly the most effective method capturing and communicating the distribution of activities is to return to the visual model and make notes about what percentage of users you anticipate will perform each activity, then once again, circulate that model to both users and stakeholders for review/comment. Just like before, ask them to tell you if they believe the percentages are reasonable and why. Often, members of the team will simply write new percentages on the visual model, making it very easy for everyone to see which activities have achieved a consensus and which have not. The following visualization is for the same eCommerce application referenced in Step 3, only now it contains distribution data.

ImpiricalData3.GIF
Sometimes you will find that one workload model is not enough. Research and experience tell us that, user activities often vary greatly over time. To ensure test validity, you must validate that activities are evaluated by time of day, day of week, day of month and time of year.
As an example, consider an on-line bill payment site. If all bills go out on the 20th of the month, the activity on the site immediately before the 20th will be focused on updating accounts and importing billing information, etc. by system administrators, while immediately after the 20th, customers will be viewing and paying their bills until the payment due date of the 5th of the next month.

Considerations

Step 5. Identify Target Load Levels

Unless or until some degree of empirical data is available (for example, previous related applications, predetermined user base, etc.) target load levels are exactly that; targets. These targets are most frequently set by the business based on their goals related to the application, whether those goals are market penetration, revenue generation or something else. These are the numbers that you want to work with to get started. These numbers may or may end up correlating to the loads the application will actually encounter, but the business will want to know if and how well the application as developed or deployed will support those loads if their targets are met.
Since the models you have constructed represent the frequency of each activity as a percentage of the total load, your models should not need to be updated as a result of determining target load levels.

Step 6. Integrate Model Variance

Since the usage models are “best guesses” until empirical data becomes available, it is a god idea to create not fewer than three usage models for each target load. This has the effect of adding a rough confidence interval to the performance measurements so that stakeholders can focus on not just the results from one test based on many fallible assumptions, but also on how much inaccuracies in those assumptions are likely to impact the performance characteristics of the application.

The three usage models that teams generally find most valuable are:
The chart below is an example of the information that testing for all three of these models can provide. As you can see, in this particular case, the Anticipated Usage and Best Case Usage resulted in similar performance characteristics. However, the Worst Case Usage showed that there is nearly a 50% drop-off in the total load that can be supported between it and the Anticipated Usage. Information such as this could lead to a re-evaluation of the usage model or possibly a decision to test with the Worst Case Usage moving forward as a sort of a safety factor until empirical data becomes available
ImpiricalData4.GIF

Step 7. Prepare to Implement the Model

Preparing to implement the model is tightly tied to the method of implementation, typically a load generation tool. For more information about implementing a workload model using VSTS see {HowTo:SomeName}.
Considerations:

Resources

<<TBD>>