pnp.GIF

How To: Account for User Abandonment

Scott Barber,

Applies To

Summary

When you adopt concurrent, simultaneous, or overlapping users as a performance testing input parameter but fail to account for user abandonment, you run the risk of creating loads that are highly unrealistic and improbable. It is important to simulate user abandonment as realistically as possible. If you don’t, you may be creating a type of load that will never occur in a real-world business environment, while creating bottlenecks that might never happen with actual users. At the same time, you would be ignoring one of the most important load testing results: the number of users that might abandon your Web site due to poor performance. In other words, your test might be useless. This How To addresses modeling and implementing user abandonment in your performance tests.

Contents

Objectives

Overview

It is well documented that people who use the Internet often abandon the Web sites they visit, and that they do so for a variety of reasons. One major reason is that users get tired of waiting for a Web page to load and exit the site before completing the task they had in mind. When designing your performance tests, it is important to ask whether the tests account for this type of behavior. If you are using the default settings on your load-generation tool, the tool probably won’t allow for user abandonment. In fact, if you don’t manually write a custom abandonment routine for your load-generation tool, the tool either won’t account for abandonment at all or will account for it in a way that isn’t useful for testing purposes.

One of the advantages of most Web sites is that if the load gets too big for the system/application to handle, the site slows down, causing visitors to abandon it, thus decreasing the load until the system speeds back up to acceptable rates. Conversely, imagine the side effects to your business if a site slowed down and stayed slow until someone fixed the server. Fortunately, abandonment can offset any negative impacts in most cases. Assuming that the site performs well enough under a “reasonable” load, performance is generally self-regulating, but at the cost of losing some customers/users. One reason to correctly account for user abandonment when designing your test is to determine how many users will likely abandon a slow-running site. Another reason is to determine the actual volume of transactions your application can maintain before you start losing users. Yet another reason to account for user abandonment is to avoid simulating, and subsequently resolving, bottlenecks that might not even be possible in a real-world scenario.

Steps

Step 1. Determine if You Need to Account for User Abandonment

In the vast majority of cases, accounting for user abandonment is critical to conducting a performance test that provides accurate results as the response time starts to increase. However, there are some cases where not accounting for abandonment is an accurate representation of a real-world situation. One such case would be a Web-based application that has been exclusively created for an audience that has no choice but to wait for it to load; for example, a company that has a Web-based time-management system and a policy that all employees must enter the hours they worked in the past week between noon and 5 P.M. every Friday (with a few trivial exceptions). Because roughly 3,500 people accessed the time-management system during a five-hour period in addition to the normal traffic of users performing other tasks on the Web site, the system got very slow. Under other circumstances, users might abandon the site and try later or go somewhere else, but in this case, the users didn’t have a choice, so accounting for abandonment as part of the performance testing would make the tests less representative of a real-life situation. In this case it was more valuable to determine how long the employees would have to wait under different usage scenarios.

To determine whether accounting for user abandonment is appropriate for your application, consider the following questions:

If you can answer yes to any one of these questions, you should account for user abandonment when conducting performance testing.

Step 2. Determine Abandonment Thresholds

Presuming that you have determined that your site is subject to user abandonment, you now need to determine, for each page or class of page, how long users will wait for a page before abandoning it. To do this, you will need to establish the following three parameters for each type of page on the site:

These parameters are determined by either parsing Web server logs, researching sites with functionality similar to the one you are testing, or conducting your own informal research by observing representative users. It is important to note that abandonment parameters that are off by a few seconds one way or another are not going to have a significant impact on your results. These parameters need to be reasonable estimations, not precise measurements. It is also important to note that you can obtain valuable information by executing the same test with different parameters to determine ranges for abandonment, with the understanding that user expectations and loyalty are bound to change over the life of your application.

Minimum Abandonment Time

The minimum abandonment time is your estimate of the amount of time that you anticipate all users will wait for a page to load. If you do not have Web server logs from which to extract this information, you can estimate the time as falling somewhere between the performance goal for a given page and 1.5 times the goal for that page, depending on how much motivation a user has to complete his or her task, right now and specifically on your Web site. Obviously, the less motivated a user is, the shorter the minimum abandonment time should be.

Maximum Abandonment Time

No matter how patient a user may be, Web sites sometimes simply fail to respond due to circumstances such as the following:

While some of these situations don’t technically count as user abandonment (because the Web site abandoned the user and not the other way around), they do provide the upper boundary for how long users could potentially wait before ending their Web session.

The maximum abandonment time parameter is the most scientific of the parameters. The maximum abandonment time is simply the time after which either your browser stops waiting for a response (frequently, 120 seconds); the secure session expires and the user has to reestablish the session, typically by logging in again (most session managers have a default setting of 20 minutes); or some other number determined by parsing Web server logs. If you don’t already know these numbers, you can simply ask the architect/developer responsible for the presentation tier (or Web server) to provide you with the information.

Abandonment Distribution

The reasons that a user may abandon a Web site are extremely variable, not only between different users but also between visits by the same user. A user’s tolerance for waiting might change dramatically from session to session. While this makes it difficult to predict when an individual might abandon a session, it also means that abandonment is likely to follow standard distribution models such as normal or linear distributions when taken over a large enough sample. Your next task is to determine, for each page type, what distribution model best represents the likely abandonment behavior of users.

In the vast majority of cases, either a normal (bell curve) or a uniform (linear) distribution will be sufficiently accurate. If you have ever taken a statistics or psychology course, you know that almost everything that real human beings do (over a large enough sample) can be represented by a bell curve. You may also recall that the key to an accurate bell curve is the standard deviation. Virtually all relevant research indicates two things about standard deviations when it comes to Web usage: (1) they are exceptionally large (statistically) in comparison to the range of values, and (2) they are almost impossible for non-mathematicians to calculate accurately. What this means is that in most cases, if you were to graph abandonment rates for a production site, you would very likely end up with a very flat bell curve that, in effect, approaches a linear distribution. Statistics aside, if you do not have a strong reason to do otherwise, choose either a normal or a uniform distribution based on your best judgment. If you really don’t know which distribution is best in your specific situation, use a linear distribution because it is almost always close enough and can always be modified later as you obtain more information.
Abandonment Linear Dist.GIF
Abandonment Normal Dist.GIF
Programming or configuring your load generation tool to handle user abandonment with these three parameters is unique to every tool and out of scope for this How To

Step 3. Interpreting Abandonment Statistics

As with any metric or statistic, abandonment statistics can be grossly misinterpreted. Remember the following when reviewing your abandonment data:

Consequences of Not Accounting for User Abandonment

If you do not account for abandonment at all when you should, the performance script will wait indefinitely to receive the page or object it requested. When it eventually receives that object, it will move on to the next object as though nothing ever happened. If the object is never received, the script never ends. This adds no value to the performance-testing effort, unless there is a need to demonstrate to some stakeholder that, for example, “Under the specified conditions, the average page-load time was roughly 2.5 hours.” Unfortunately, points like this sometimes can only be made convincingly by generating meaningless numbers, which does not constitute a performance test and does not bring you any closer to delivering a quality, well-performing application.

Consequences of Improperly Accounting for User Abandonment

Improperly accounting for user abandonment is what most load-generation tools do by default. This is because most tools assume that all users abandon at a predetermined time (frequently 240 seconds) yet still continue on, requesting the following page as though nothing happened. Of course, with most tools, you can change settings to improve the situation by changing the time limit or having the virtual user actually exit, but that still is not context-specific by page. Regardless of your tool’s default settings, or how difficult or easy it may be to change those defaults, improper accounting for abandonment can bring results that are even more misleading than if abandonment were not accounted for at all. Consider the following examples and their side-effects.

While this may be useful during early testing, it is a very inaccurate representation of the actual abandonment rate for a variety of reasons, including:

Please note that the above examples represent cases where actual abandonment rates were grossly misrepresented. Inaccurately modeling the abandonment range by a few seconds is not going to cause problems on this scale. Your abandonment model needs to be reasonable, not perfect.

Resources