pnp.gif

How To: Capacity Plan For Web Applications Using Transaction Cost Analysis (TCA)

J. D. Meier, Prashant Bansode, Scott Barber, Mark Tomlinson

Applies To

Summary

This How To shows you how to perform capacity planning for Web applications by using Transaction Cost Analysis (TCA). Transaction Cost Analysis measures the cost of a user operation on the available server resource.

Contents

Objectives

Overview

When testing projects are limited by time, budget, or resource constraints, it is very common to perform mathematical estimates in an attempt to predict application performance. Transaction Cost Analysis (TCA) is a process for conducting a controlled estimate of application performance, based on key measurements of a single transaction through the application. Although TCA does not serve as a replacement for the complete and thorough performance testing of the entire system, it can be very helpful in determining any obvious risks or serious underestimates related to the hardware required to support the system. This How To will help you understand the basic TCA process and how to perform calculations for extrapolating an estimate of future system capacity.
As you might imagine, with any mathematical calculation, it is critical to ensure very exact inputs to the calculations if you want to have reliable outputs and results from the TCA model.

Summary of Steps

Step 1. Compile a User Profile

An individual user profile is also commonly referred to as a workload characterization (make link reference here), in which information about transaction volumes and application usage patterns is arranged in a matrix or table format. You can also compile individual user profiles based on analysis of existing production traffic data, or a transactional workload taken from your load test plan. One possible resource for identifying production usage is the Microsoft® Internet Information Services (IIS) log files.

The Usage Profile table should contain the following key information:
The following is an example of how to compile a user profile from an analysis of IIS log files:

User Requests per Page
IDURI# of RequestsPercentage
1/MyApp/login.aspx18,23435%
2/MyApp/home.aspx10,75620%
3/MyApp/logout.aspx9,99319%
4/MyApp/SellStock.aspx4,2008%
5/MyApp/BuyStock.aspx9,42318%
TotalNA52,606100%


Number of operations = # of requests / # of requests per operation

The Requests per operation column in the following table shows how many times the page was requested for a single operation.

User Requests per Operation
IDURI# of RequestsRequests / Operations*# of Operations
1/MyApp/login.aspx18,23429,117
2/MyApp/logout.aspx9,99319,993
3/MyApp/SellStock.aspx4,20022,100
4/MyApp/BuyStock.aspx9,42333,141
TotalNA41,850824,351

Average User Profile
Operations# of Operations Executed during an average session
Login1
SellStock3
BuyStock2
Logout1


Of course, if the application has never run in a production environment, you would not have any IIS logs to process. In this case, another way to create the user profile is to use the documents created from performance test planning. For more information about identifying user profiles, see "Workload Modeling" <<Add Url>>

Step 2. Execute Discrete Tests

In this step, you will create a very simple test script that will execute discrete tests for each user operation identified in Step 1. You will configure a load test to place enough stress on your system to reach maximum throughput. For example, you might need to run separate tests for the Login, BuyStock, and SellStock operations, increasing the load on the system to the point where the CPU resource is at 100-percent utilization. The test script only fires the requests for a dedicated user operation, such as a single .aspx page.

The procedure for executing the tests consists of the following tasks:

Performance Counters Used to Identify Cost
ObjectCounterInstance
Processor% Processor Time_Total
ASP.NET ApplicationsRequests/SecYour virtual directory


Note - It is important to have a throughput measure that is reliable and relates specifically to the requests being emulated by the load testing tool. For ASP.NET, you can use Requests/Sec to calculate the processor cost per request.

Output:

The output from executing this series of steps for each scenario would be a report like the following:

Following table shows a sample report for the results of the load tests.
Load Test Results comparing the Processor % utilization to the Requests/Sec

User operationProcess\% Processor TimeASP.NET Applications\Requests/Sec
Login90%441
SellStock78%241
BuyStock83%329
Logout87%510

Step 3. Calculate the Cost of Each Operation

Measure the cost of each operation in terms of the limiting resource identified in Step 2. Measuring the operation cost involves calculating the cost per request and then calculating the cost per operation. Use the following formulas for these tasks:
Cost (Mcycles/request) = ((number of processors x processor speed) x processor use) / number of requests per second 


For example, using the values identified for the performance counters in Step 2, where processor speed is 1.3 GHz or 1300 Mcycles/sec, processor usage is 90 percent, and Requests/Sec is 441, you can calculate the page cost as:
((2 x 1,300 Mcycles/sec) x 0.90) / (441 Requests/Sec) = 5.30 Mcycles/request 
Cost per operation = (number of Mcycles/request) x number of pages for an operation 

The cost of the Login operation is:
5.30 x 3 = 15.9 Mcycles 

If you cannot separate out independent functions in your application and need one independent function as a prerequisite to another, you should try to run the common function individually and then subtract the cost from all of the dependent functions. For example, to perform the BuyStock operation, you need to perform the login operation, calculate the cost of login separately, and then subtract the cost of login from the cost of the BuyStock operation.

Therefore the cost of a single BuyStock operation can be calculated as follows:
Single cost of BuyStock operation = Total cost of BuyStock – Cost of Login operation 

The cost of a single BuyStock operation is:
39.36 – 15.92 = 23.44 Mcycles 

Following table shows the cost of each user operation in a sample application using the following scenario.

Cost per Operation for Login, SellStock, BuyStock, and Logout Operations
User OperationsCPU % utilizationTotal net CPU McyclesASP.NET Requests / Sec# of RequestsOperation Costs (Mcycles)# pages without loginSingle Operation cost
Login90%2,340.00441315.92315.92
SellStock78%2,028.00241542.07226.16
BuyStock83%2,158.00329639.36323.44
Logout87%2,262.00510522.1826.26


The operation cost needs to be measured separately for each tier of an application.

Step 4. Calculate the Cost of an Average User Profile

The behavior of actual users can cause random crests and troughs in resource utilization. However, over time these variations even out statistically to average behavior. The user profile you compiled in Step 1 reflects average user behavior. To estimate capacity, you need to assume an average user and then calculate the cost in terms of the limiting resource identified in Step 2.

As shown in following table, during a ten-minute session, an average user needs 147.52 Mcycles of CPU on the server. The cost per second can be calculated as follows:
Average cost of profile in Mcycles/sec = Total cost for a profile / session length in seconds

Thus, the average cost for the profile shown in Table 7 is:
147.52/600 = 0.245 Mcycles/sec

This value can help you calculate the maximum number of simultaneous users your site can support.

Cost of an Average User Profile
Average User ProfileNumber of operations executed during an average sessionCost per operation (Mcycles)Total cost per operation (Mcycles)
Login115.9215.92
SellStock326.1678.47
BuyStock223.4446.87
Logout16.266.26
Total 147.52

Step 5. Calculate Site Capacity

Calculating site capacity involves knowing how many users your application can support on specific hardware and what your site's future resource requirements are. To calculate these values, use the following formulas:
Maximum number of simultaneous users with a given profile = (number of CPUs) x
 (CPU speed in Mcycles/sec) x (maximum CPU utilization) / (cost of user profile in Mcycles/sec) 

Therefore, the maximum number of simultaneous users with a given profile that the sample application can support is:
(2 x 1300 x 0.75)/0.245 = 7,959 users 
Based on the formulas used earlier, you can calculate the number of CPUs required for a given number of users as follows:
Number of CPUs = (Number of users) x (Total cost of user profile in Mcycles/sec) /
 (CPU speed in MHz) x (Maximum CPU utilization) 

If you want to plan for 10,000 users for the sample application and have a threshold limit of 75 percent defined for the processor, the number of CPUs required is:
10000 x 0.245 / (1.3 x 1000) x 0.75 = 2.51 processors 

Your resource estimates should also factor in the impact of possible code changes or functionality additions in future versions of the application. These versions may require more resources than estimated for the current version.

Step 6. Verify Site Capacity

It is very important to verify that TCA accurately predicts your application capacity and future requirements. Because TCA analysis estimates are based on extrapolations of very specific performance measurements, there is a good possibility that some discrepancies might result. Most test projects use TCA as only an initial estimate of performance, but then continue the load testing effort to execute load tests to verify that the TCA model accurately predicts your application capacity and future requirements. In short, TCA is not recommended as a replacement for load testing, and you should still execute a thorough and robust performance test if at all possible.

Verify the calculated application capacity by running load tests with the same characteristics you used to calculate TCA. The verification script is simply a collection of all TCA measurement scripts, aggregated and run as a single script.

The actual values and the estimated values should vary by an acceptable margin of error (+/- 15%). The acceptable margin of error may vary depending on the size of the setup and the budget constraints. You do not need to run load tests each time you perform TCA.

Additional Resources