pnp.gif

How To: Control the Number of Test Iterations During a Load Test in Microsoft Visual Studio Team System

J. D. Meier, Prashant Bansode, Carlos Farre, Scott Barber

Applies to

Summary

This How To explains how to create and use a Load Test plug-in that will control the number of test iterations for a particular Web test or for all Web tests in a test mix of a load test. It also demonstrates how to terminate the test after the required number of test iterations is reached.

Contents


Objectives


Overview

Frequently while designing or conducting load tests, it becomes clear that the only way to achieve the desired test duration is to increase the number of test iterations (e.g., the number of times that a specific scenario represented by a user session) rather than increasing the duration of individual user sessions. For example, this is desirable in the following scenarios:
This How To explains how to control the number of test iterations when using VSTS.

Scenario

This How To considers a typical e-commerce application with Web tests and their workload distribution shown in the following Load Distribution table.

Browse: Open Page->Login->Search->Browse->Logout.
Search: Open Page->Login->Search->Logout
Place Order: Open Page->Login->Search->Browse->AddToCart->Order->Logout

Load Distribution
User Scenarios% of usersUsers
Browse50250
Search30150
Place Order20100
Total100500


Typically, each execution of the above case can be understood as the user session, or the navigation path through the Web site. In the Visual Studio Team System (VSTS), the user session in Web tests can be also be viewed as test iteration. When the Web test is executed from a load test, after going through all pages, it executes the session path again during the test, iterating in a loop cycle.

Summary of Steps

Step 1. Create a Load Test Containing One or More Web Tests in a Test Mix

First, create a test project that will contain your load test and Web tests. Then, create individual Web tests that reflect key business user scenarios. Finally, create a load test to which you will add your Web tests. While creating the load test, you can set many run-time properties to generate desired load simulation; for example, you can specify the load pattern, browser, and network types and add the performance counters to be monitored.
For more information on creating load tests, refer to “How To: Create a Load Test Using VS.NET 2005” at << to link to how to >>

Step 2. Create a Load Test Plug-In Class to Monitor the Number of Tests to Be Executed.

In this step, you create a plug-in class for your load test project. The Load Test plug-in is an extensibility point that allows code to be executed when test iteration is complete. This Load Test plug-in will allow controlling of load test duration based on the number of test iterations of a particular Web test, or based on the test iterations of all Web tests.
This plug-in can help you gauge the number of users needed to simulate realistic load patterns for workload characterization of a Web site. For example, a load test must simulate placing 20,000 orders over a two-hour period when the toad test is running a mix of Web tests (Browse, Search, and Place Order)—a particularly challenging scenario to simulate for load tests containing many Web tests. Another case scenario is the running of a load test based on transaction count, after which another load test must be executed characterizing a business scenario. The process of writing a Load Test plug-in can be summarized as follows:

Create a plug-in Class

Perform the following steps to create a Load Test plug-in class:
using Microsoft.VisualStudio.TestTools.LoadTesting;

public class MaxTestIterationsLoadTestPlugin: ILoadTestPlugin

    private LoadTest mLoadTest;
    private int mMaxTestIterations;
    private int mTestsFinished;
    private bool mAbortTest;
    private string mTestName;


Your class should now look like this:

 using System;
 using System.Collections.Generic;
 using System.IO;
 using System.Text;
 using Microsoft.VisualStudio.TestTools.LoadTesting;
 
 namespace LoadIterationPlugin
 {
 
    public class MaxTestIterationsLoadTestPlugin : ILoadTestPlugin
    {
        #region ILoadTestPlugin Members
        private LoadTest mLoadTest;
        private int mMaxTestIterations;
        private int mTestsFinished;
        private bool mAbortTest;
        private string mTestName;
 
       }
 }


More Information
The purpose of each member variable is as follows:
The context variables used from the LoadTest object and the event handler used to control the number of test iterations are described below:

Implement the Initialize method.

In this example, you implement the Initialize method of the ILoadTestPlugin interface, which takes the LoadTest object as a parameter in the Initialize method; assigns the local variables mMaxTestIterations, mAbortTest, and mTestName; and subscribes the event handler mLoadTestTestFinished to the TestFinished event. To do this, add the following code to your class:

 public void Initialize(LoadTest mloadTest)
 {
   //load Test object
   this.mLoadTest = myloadTest;            
   if (this.mLoadTest.Context.ContainsKey("MaxTestIterations"))
   {
      try
      {
         //number of iterations to be executed
	   mMaxTestIterations = 
                int.Parse((string)mLoadTest.Context["MaxTestIterations"]);
      }
      catch (FormatException)
      {
	   throw new ApplicationException(
                "MaxTestIterations not in integer format");
      }
   }
                
   //The function to be executed after every test iteration
   this.mLoadTest.TestFinished += new 
                EventHandler<TestFinishedEventArgs>(mLoadTestTestFinished);
 
   if (mLoadTest.Context.ContainsKey("AbortWhenMaxTestsCompleted"))
   {
      try
      {
         // Bool value to specify if test is to be 
         // aborted when condition is met
         mAbortTest =
                bool.Parse((string)mLoadTest.Context[
"AbortWhenMaxTestsCompleted"]);
     }
     catch (FormatException)
     {
        throw new ApplicationException(
                "AbortWhenMaxTestsCompleted not in bool format");
     }
   }
  
   if (this.mLoadTest.Context.ContainsKey("TestName"))
   {
      try
      {
         //The name of the test where the test iterations are to be counted
         mTestName = (string)this.mLoadTest.Context["TestName"];
      }
      catch (FormatException)
      {
         throw new ApplicationException(
                        "AbortWhenMaxTestsCompleted not in string format");
      }
   }
 }


More Information
Before the test starts, the Initialize method is called once; the mLoadTest variable is assigned with LoadTest passed as a parameter; and the local variables mMaxTestIterations, mAbortTest, mTestName are assigned from the context parameters read from the LoadTest object. In addition, the callback function is subscribed to the TestFinished event handler. This will execute when iteration is completed.

Implement the callback function

You will implement the callback function as follows:

 void mLoadTestTestFinished(object sender, TestFinishedEventArgs e)
 {
   if (mMaxTestIterations > 0)
   {
      if (String.IsNullOrEmpty(mTestName) ||
               (String.Equals(mTestName,e.TestName)))
      {
         if (++mTestsFinished >= mMaxTestIterations)
         {
            SetLoadForAllScenariosToZero();
            if (mAbortTest)
            {
               mLoadTest.Abort();
            }
         }
      }
   }
 }


More Information
Every time a test is completed, the TestFinished event is raised, which will be handled by the callback function. Because the mloadTestTestFinished function checks whether mMaxTestIterations is greater than zero, it has been defined as a context parameter as an integer. If this context parameter is not defined, the test will continue as specified in the run-time settings.

If mMaxTestIterations is greater then zero, it checks whether mTestName is null or the same as the test being finished; in this case the condition will be tested for every test, including all of the tests executed for the load test. If mTestName is null or is the same as the test being finished, the counter controlling the tests being executed is incremented using the mTestsFinished local counter. If mTestName is null, the load test will count all iterations for all Web tests in the test mix; if not, it will do so only for that specific Web test.
If the counter mTestsFinished crosses the threshold set by mMaxTestIterations, it calls SetLoadForAllScenariosToZero, which sets the current load to zero for every scenario retrieved from the LoadTestScenarios collection of LoadTest. After setting the load to zero, if mAbortTest was defined, it will abort the test; otherwise the test continues with the load set to zero until the test completes as defined in the run-time settings.
Create the code to set the load for the scenarios as follows:

 void SetLoadForAllScenariosToZero()
 {
   foreach (LoadTestScenario scenario in mLoadTest.Scenarios)
   {
      scenario.CurrentLoad = 0;
   }
 }


More Information
LoadTestScenario is the individual scenario designed in the load test. A load test can have one or more scenarios, each of which can have one or more tests to be executed. The LoadTestScenario includes the CurrentLoad that can be read or set for the current load running on a scenario for each individual agent that is executing the test. If the test is executing locally without agents, CurrentLoad expresses the current load running on the scenario on the local machine.

It is important to note that the Iteration Control plug-in will hold variables pertaining to the local agent executing the load test. If mMaxTestIterations is set to 30, and if there are three agents executing the test, a total of 90 iterations will be executed. Also, when a scenario is set to zero current load as in scenario.CurrentLoad =0 , the load is set for the particular agent executing the load test plug in.

Example Load Testing Plug-In Class

Here is the complete code sample for the plug-in class, which you can simply copy paste.

 using System;
 using System.Collections.Generic;
 using System.IO;
 using System.Text;
 using Microsoft.VisualStudio.TestTools.LoadTesting;
 
 namespace LoadIterationPlugin
 {
    public class MaxTestIterationsLoadTestPlugin : ILoadTestPlugin
    {
        #region ILoadTestPlugin Members
        private LoadTest mLoadTest;
        private int mMaxTestIterations;
        private int mTestsFinished;
        private bool mAbortTest;
        private string mTestName;
 
        public void Initialize(LoadTest loadTest)
        {
            mLoadTest = loadTest;
            if (mLoadTest.Context.ContainsKey("MaxTestIterations"))
            {
                try
                {
                   mMaxTestIterations =
                   int.Parse((string)mLoadTest.Context["MaxTestIterations"]);
                }
                catch (FormatException)
                {
                    throw new ApplicationException("MaxTestIterations 
                                                    not in integer format");
                }
 
                mLoadTest.TestFinished += new 
                EventHandler<TestFinishedEventArgs>(mLoadTestTestFinished);

                if(mLoadTest.Context.ContainsKey
                  ("AbortWhenMaxTestsCompleted"))
                {
                  try
                  {
                  mAbortTest =                                               
                             bool.Parse((string)mLoadTest.Context
                             ["AbortWhenMaxTestsCompleted"]);
                    }
                    catch (FormatException)
                    {
                        throw new ApplicationException
                        ("AbortWhenMaxTestsCompleted not in bool format");
                    }
                }
                if (mLoadTest.Context.ContainsKey("TestName"))
                {
                    try
                    {
                        mTestName = (string)mLoadTest.Context["TestName"];
                    }
                    catch (FormatException)
                    {
                        throw new ApplicationException
                                  ("AbortWhenMaxTestsCompleted not in bool
                                    format");
                    }
                }
            }
        }
 
        void SetLoadForAllScenariosToZero()
        {
            foreach (LoadTestScenario scenario in mLoadTest.Scenarios)
            {
                scenario.CurrentLoad = 0;
            }
        }
 
        void mLoadTestTestFinished(object sender, TestFinishedEventArgs e)
        {
            if (mMaxTestIterations > 0)
            {
              if (String.IsNullOrEmpty(mTestName) ||
                 (String.Equals(mTestName,e.TestName)))
                {
                    if (++mTestsFinished >= mMaxTestIterations)
                    {
                        SetLoadForAllScenariosToZero();
                        if (mAbortTest)
                        {
                            mLoadTest.Abort();
                        }
                    }
                }
            }
        }
        #endregion
    }
  }

Step 3. Add the New Load Test Plug-In Class to the Test Project

After the new Load Test plug-in class has been developed, add the class project containing the plug-in into the test project. It is required for configuring the load test properties for the Load Test plug-in.
  1. Right-click the test solution.
  2. Select Add, and then click Existing Project….
  3. Browse to the project containing the Load Test Plug in.
  4. Select the Load Test Plug in. It will be added to the test project.

Step 4. Add the Load Test Plug-In to the Load Test.

In this step, you add the Load Test plug-in to the load test so it can be executed. For a load test to execute the Load Test plug-in, the load test needs to be configured after the Load Test plug-in has been added to the test project. The Load Test plug-in needs to be added in the properties of the root node of the load test, as detailed in the following steps:
LoadPlug-In 1.GIF LoadPlug-In 2.GIF

Step 5 Add Context Parameters.

Context parameters are data object variables that you pass to the Load Test plug-in. These parameters can be assigned to local variables either during the initialize phase before the load test executes, or during the execution of the callback function. Context parameters are used to allow the passing of variables during execution of a test, to make it possible to change load test settings such as current load, based on conditions such as the Web test name and the number of test executions.
You add the context parameters as follows: LoadPlug-In 3.GIF LoadPlug-In 4.GIF LoadPlug-In 5.GIF LoadPlug-In 6.GIF

Step 6 – Run the Load Test

In this step, you run the actual load test. The load test is created with one or more Web tests in the mix setting, with the Load Test plug-in hooked up as shown in previous steps.
The following is an example of load test run with three Web tests (Browse, Orders, and Search Products). The load test will run until 2,000 orders are placed. The load test was configured to run with 20 users and for two hours.
LoadPlug-In 7.GIF

Running Your Load Test

  1. In the Load Test Editor, right-click the load test and then select the Run Test option.
  2. The test runs until the condition is met and an abort message is displayed at the end. Clicking tables and selecting tests from the drop-down list will give the number of tests for each Web test in the test mix, and the total number of tests will be displayed at the bottom of the left pane. The following illustration depicts a result set totaling 2,000 orders executed:
Total tests: 10134
Browse tests: 5128
Orders tests: 2000
SearchProducts tests: 3006

LoadPlug-In 8.GIF

Resources