Creating a new filter

Understanding the AX UI

The most important tool in understanding parts of the AX UI is the inspect.exe tool. This is a tool that comes with the windows SDK. It is able to traverse the UI hierarchy of any accessible application as UI Automation sees it or as MSAA sees it. While traversing it's also possible to look at most properties of the UI controls.

When making a new filter it's very important to know the general hierarchy surrounding your targeted control. This allows you access to much more information to make runtime decisions than the individual UI control could.

Understanding actions

Your only interface into the CUIT environment is through reading, updating, and creating actions. The full set of action classes can be looked through here. The ones you're most likely to deal with are the subclasses of InputAction. While it is not a compile error to subclass UITestAction, it will break XML serialization. This means you will be limited by existing action classes. The most notable limitation is the lack of any conditional action.

The best way to understand what actions are produced by the CUIT recorder in practice is to turn on the TracerFilter and try clicking and typing around in the interface. You can then look in the logs, which will show the basic information of the actions that were recorded.

Creating the filter

To start off, simply create a new class implementing Filter and add it to the FilterGroup in AXActionFilter. The first thing to do is to add requirements to the new Filter. This is done in the constructor with Require calls. The Require pattern was created to avoid the nesting and code duplication involved with the start of every filter's Run method. You can see what this used to look like in the UnconditionalRun method of SegmentedEntryFilter. Filter's constructor requires there to be an action on a non-null UIElement to be on top of the action stack, as that applies to every filter, so you don't have to note that in the subclass.

Once you've added requirements that ensure that the UI control interaction you want to regulate is on the stack, you'll have to implement Run. This varies a great deal based on what you're trying to do. The simplest filters simply intend to change a property of an action's target. Examples of these are WindowTitleFilter and DropdownUncachingFilter. Things get more complicated when you want to create and delete actions.

Sample: Designing the navigation pane filter

The navigation pane filter is one of the most broad filters, along with the segmented entry filter. This makes it a good example for demonstrating all the techniques and steps involved in making a filter.

The purpose of the navigation pane filter is to optimize the speed of navigation by translating clicks on the navigation pane to direct navigation bar keyboard actions. This makes the requirements fairly simple, simply requiring that the user is clicking on a TreeItem. Because these only exist in the navigation pane on the left, you can be sure that you won't interfere with other aspects of recording. This means that we can have the constructor simply be this:

public NavigationPaneFilter()
{
    Require(stack => stack.Peek().UIElement.ControlTypeName == "TreeItem");
}

Next up, we want to make sure to remove any actions that are expanding or retracting the tree items. The easiest way to check this with the tools we have is by checking whether the ui element has any children. If it does, then it's not a leaf and the action is irrelevant, so we can just get rid of it and return. This creates the start of our Run method.

protected override void Run(IUITestActionStack stack)
{
    // If it has any children, this is just an organizational
    // TreeItem, so we can ignore that action.
    if (Playback.GetCoreTechnologyManager("MSAA").GetChildren(stack.Peek().UIElement, null).MoveNext())
    {
        stack.Pop();
        return;
    }

Before we continue, we're going to need to get the navigation bar so that we can create an action that types into it later. Before we can do any in detail navigation through the UI, we're going to want to make sure we have the right technology manager loaded. The AXUtils extension methods rely on the AXUtils.LoadManager method being called to set up a specific technology manager. In this case, the UIA manager is the best technology manager for the job. It gives a much cleaner hierarchy when it can give one at all, though it doesn't support getting an element's parent. Because we're going to simply move down the hierarchy from the main window down, we don't need to get an element's parent.

Unfortunately, it seems impossible to get the window of dynamics easily through just the UITechnologyElement interface. Because of this we use the GetDesktopWindow function from user32.dll in conjunction with the technology manager's GetElementFromWindowHandle method. Afterward, we just walk down the tree until we reach the toolbar:

private UITechnologyElement GetToolbar(UITestAction action)
{
    AXUtils.LoadManager("UIA");
    IUITechnologyElement parent = AXUtils.Manager.GetElementFromWindowHandle(GetDesktopWindow())
        .Child("Dynamics", "Window")
        .Child("WindowHeaderFrame", "Pane")
        .Child("Pane")
        .Child("TopRow", "Pane")
        .Child("AddressBarContainer", "Pane");

    IUITechnologyElement toolbar = parent.Child("ToolBar");

Through testing we notice that this toolbar IUITechnologyElement doesn't allow playback to find the toolbar, so we need to modify its search properties to properly specify the toolbar. This required poking around the UI hierarchy around the toolbar with inspect.exe. After seeing what properties the toolbar and its parent have, we redefine their search properties:

    toolbar.QueryId.Ancestor = new UITechnologyElementRedirect(parent,
        action.UIElement.TopLevelElement,
        Playback.GetCoreTechnologyManager("MSAA"), "MSAA");
    toolbar.QueryId.Condition = new AndCondition(
        new PropertyCondition("ControlType", "ToolBar"));
    parent.QueryId.Condition = new AndCondition(
        new PropertyCondition("ClassName", "WindowsForms10.Window", PropertyConditionOperator.Contains),
        new PropertyCondition("Instance", "35"),
        new PropertyCondition("ControlType", "Window"));
    return new UITechnologyElementRedirect(toolbar,
        action.UIElement.TopLevelElement,
        Playback.GetCoreTechnologyManager("MSAA"), "MSAA");
}

Now that we can get the toolbar, we can keep going with Run. Once we have the toolbar, we need to create a mouse action and a typing action to enter the path of the tree item's destination.

    UITestAction action = stack.Peek();
    UITechnologyElement toolbar = GetToolbar(action);
    
    var mouse = new MouseAction(toolbar,
        System.Windows.Forms.MouseButtons.Left,
        MouseActionType.Click);
    // The farthest right location on the toolbar.
    mouse.Location = new System.Drawing.Point(835, 5);
    var keys = new SendKeysAction();
    AXUtils.LoadManager("MSAA");
    keys.Text = "USMF" + PathRepresented(action.UIElement) + "{Enter}";

The PathRepresented method is a simple recursive method to get the path that the tree item clicked on leads to. It needs to get the parent of the element recursively, so we have to load the MSAA manager before calling it.

After all this is done, we have to actually deal with the actionstack. We've already popped all other navigation pane actions, but the user may have used the toolbar to get to an area page so that the navigation pane had what they wanted. We have to pop that as well, and after all that push the new click and type actions.

    stack.Pop();
    // If the user set the area page the newly pushed actions will do that anyway.
    if (stack.Count >= 2 &&
        stack.Peek(1).UIElement.ControlTypeName == "SplitButton")
    {
        stack.Pop();
        stack.Pop();
    }
    stack.Push(mouse);
    stack.Push(keys);
}

And that's it! The extension now changes any navigation through the pane to navigation through the toolbar. This represents a pretty large improvement in speed. However, though this seemed simple to create, there are many invisible steps involved in creating this filter:

Important considerations during development

While this seemed a fairly straightforward process, it actually took many iterations of each piece before it worked. The problem is that a large number of functionality that UITestAction, UITechnologyElement, and TechnologyManager either don't actually support or implement in unexpected ways.

As an example, the fact that the UIA technology manager doesn't always support GetParent was discovered experimentally after some confusion. Even more interestingly, it does work when used in the SegmentedEntryFilter, possibly because of the UI framework used by that part of AX UI.

Another interesting issue came up in the creation of the NavigationPaneFilter we just created. There were about 5 different ways of trying to get the main AX window. Most didn't work at all, simply returning null or returning an element in a strange hierarchy that didn't map to any existing window. Eventually I came upon the solution (using a user32.dll function).

To avoid most of this pain, it's important to change your mentality a little when developing on this extension framework. Instead of trying to find the single correct solution to a problem and make that work, it's better to maintain the mentality of trying out as many solutions as possible and then refining the one that works.