Explained: Load Agent Deployment Scenarios in Visual Studio Team System

- J.D. Meier, Carlos Farre, Prashant Bansode

Applies To

Contents

Overview

You can do performance and load testing with Visual Studio Team System solution, comprising of different products. The test and load agent solution complements the Visual Studio Team System and includes controller and agent, allowing for scalable test configuration, generating extra test load distributed across several machines.

The agent is used to simulate multiple clients from a single workstation. You can use multiple agents to increase the number of clients you simulate. The controller controls the test execution of the agents and collects performance counter metrics. You can share the controller among different Visual Studio workstations in two different scenarios:
For doing performance and load testing following two main options

Using Single Workstation (Scaled Down Load Test Framework)

In the simplest scenario, you can run load tests on your local workstation, without a separate controller and agent. In this case, VSTestHost.exe will act as the agent and controller. This scenario is a scaled down load test framework:

Using Separate Load Test Controller and Agent

If you need to simulate more users beyond what a single workstation can handle or if you need to create a shared environment for performance testing, then it will be required to implement scalable test configuration with controller and agent(s). The Visual Studio workstation acts both as a test development platform and local load generator; the Visual studio load agent acts as a distributed load testing platform, with controller hosted by QTController.exe and agent hosted QTAgentService.exe acting as a dispatcher and process QTAgent.exe acting as the load generator engine.

Scenario 1 – Single Workstation Scenario

Scenario1.GIF

Single Workstation Scenario

Key Characteristics

When to use

Key Points

More Info

Scenario 2 – Local Controller, Local Results Database, and Remote Agent


Scenario2.GIF

Local Controller, Local Results Database, and Remote Agent Scenario

Key Characteristics

When to use

Key Points

More Info

Scenario 3 – Remote Controller, Remote Results Database and Remote Agent(s)

Scenario3.GIF

Remote Controller, Remote Results Database and Remote Agent(s)

Key Characteristics

When to use

Key Points

More Info

Scenario 4 – Remote Controller, Remote Results Database and Multiple Remote Agents

Scenario4.GIF

Remote Controller, Remote Results Database and Multiple Remote Agents Scenario

Key Characteristics

When to use

Key Points

More Info

V-Lan Infrastructure Examples

Visual Studio Team System, controller and agents can be deployed in virtual lans in two ways:
  1. The lab infrastructure is not shared. Each performance testing project will have its own agents and controller assigned to only one Visual Studio workstation.
  2. The lab infrastructure is shared. Several performance testing projects will share the agents and controller and they will be assigned to different Visual Studio workstations, or each Visual Studio workstation will share the controller, but will be assigned unique agents.

Lab Infrastructure Not Shared

Following figure is representation of the lab performance infrastructure that is not shared.

Lab1.GIF

Lab Infrastructure Not Shared

Lab Infrastructure Shared

Following figure is representation of the lab infrastructure that is shared.
Lab2.GIF

Lab Infrastructure Shared

How Many Machines and Agents Do You Need

To determine the number of workstations and agents that you need, consider the following:

Processor and Memory Limits

The amount of load that a specific load agent can generate varies widely from test to test. Most tests are bound by the processor. Processor use is directly proportional to requests per second (RPS). For other load tests, memory is the limiting factor. The RPS you can expect to drive from a load agent depends on the following factors:

Resources Used By the Controller, Agents and SQL Server

When a load test is created, it defaults to critical performance counters and thresholds of all the agents in your test configuration. Monitor those performance counters on the agents during load test execution. Increase the number of agents if threshold violations are reported. The table below reports the summary of resources used by controller, agents and SQL Server for collecting data results.

Agent Controller Application Tier Controller Data Tier Controller Application tier/Data tier
CPU Depending on the test, the CPU is frequently the limiting factor Not heavily used Not heavily used Not heavily used
Disk Not heavily used Not heavily used 10 GB space required for 24 hours of test data 10 GB space required for 24 hours of test data
Memory Depending on the test, memory will be the limiting factor Not heavily used Heavily used by SQL Server Heavily used by SQL Server


Connection Pool Mode versus Connection Per User

Connection mode of execution is a load test setting, configurable during load test design. Agents can be bound by memory on tests that use the Connection Per User connection mode. Two connection modes can be configured in the load test run settings:

ConnectionPerUser

In Connection Per User mode, each user has a connection that consists of two actual connections open to the server. The ConnectionPerUser model most closely simulates the behavior of a real browser: each virtual user running a web test uses one or two connections to the web server that are dedicated to that virtual user. The first connection is established when the first request in the web test is issued. A second connection may be used when a page contains more than one dependent request; these requests may be issued in parallel using the two connections. These connections are re-used for subsequent requests within the web test, and are closed when the web test completes. The drawback of the ConnectionPerUser model is that the number of connections held open on the agent machine may be high (as high as 2 times the user load), and the resources required to support this high connection count may limit the user load that can be driven from a single load test agent.

ConnectionPool

In Connection Pool mode the connections are pooled but each user still uses two connections when active. In this mode all virtual users are multiplexed over the connection pool. The ConnectionPool model conserves the resources on the load test agent by sharing connections to the web server among multiple virtual web test users. When the Connection Model is ConnectionPool, the Connection Pool Size specifies the maximum number of connections to make between the load test agent and the web server. If the user load is larger than the connection pool size, then web tests executing on behalf of different virtual users will share a connection. This may mean that one web test may need to wait before issuing a request when another web test is using the connection, the average time that a web test waits before submitting a request is tracked by the load test performance counter “Avg. Connection Wait Time”. This number should ideally be less than the average response time for a page. If it is not, then the connection pool size is probably too small.

SQL Sizing Considerations

When deploying SQL database to act as a load test repository, there are considerations regarding the sizing, directly related with the number of samples for the performance testing metrics.

Multi-processor Considerations

If you are generating load from a multi-processor machine, consider enabling server GC. By default the process will use workstation GC.

To enable your application to use server GC, you need to modify either the VSTestHost.exe.config or the QTAgent.exe.config. If you are not using a Controller and Agent setup, then you need to modify the VSTesthost.exe.config. If you are using a controller and agent, then modify the QTAgent.exe.config for each agent machine.

Open the correct file. The locations are
To enable gcserver you need to add the following line in the runtime section:

<gcServer enabled="true" />

Below is the sample config file:
<?xml version ="1.0"?>
<configuration>
    <runtime>
        <gcServer enabled="true" />
        <assemblyBinding xmlns="urn:schemas-microsoft-com:asm.v1">
            <probing privatePath="PrivateAssemblies;PublicAssemblies"/>
        </assemblyBinding>
    </runtime>
</configuration>

Security Considerations

If the computers in a rig are running in Work Group mode, that is, they are not in a domain, or they are running in different non-trusted domains, create local computer accounts on the controller. The accounts should have a matching password for each user who will access the controller, including the Agent service users.

Below is an example of machines: Controller1, Agent1 and Agent2:
Computer Name User Account Password
Controller1 ControllerService ControllerServicePassword
Agent1 ControllerService ControllerServicePassword
Agent2 ControllerService ControllerServicePassword
Controller1 AgentService AgentServicePassword
Agent1 AgentService AgentServicePassword
Agent2 AgentService AgentServicePassword

Additional Resources