Creating a new filter
Understanding the AX UI
The most important tool in understanding parts of the AX UI is the inspect.exe tool. This is a tool that comes with
the windows SDK. It is able to traverse the UI hierarchy of any accessible application as UI Automation sees it or as MSAA sees it. While traversing it's also possible to look at most properties of the UI controls.
When making a new filter it's very important to know the general hierarchy surrounding your targeted control. This allows you access to much more information to make runtime decisions than the individual UI control could.
Understanding actions
Your only interface into the CUIT environment is through reading, updating, and creating actions. The full set of action classes can be looked through
here. The ones you're most likely to deal with are the subclasses of
InputAction. While it is not a compile error to subclass
UITestAction, it will break XML serialization. This means you will be limited by existing action classes. The most notable limitation is the lack of any conditional action.
The best way to understand what actions are produced by the CUIT recorder in practice is to turn on the
TracerFilter and try clicking and typing around in the interface. You can then
look in the logs, which will show the basic information of the actions that were recorded.
Creating the filter
To start off, simply create a new class implementing
Filter and add it to the
FilterGroup in
AXActionFilter. The first thing to do is to add requirements to the new
Filter. This is done in the constructor with
Require calls. The
Require pattern was created to avoid the nesting and code duplication involved with the start of every filter's
Run method. You can see what this used to look like in the
UnconditionalRun method of
SegmentedEntryFilter.
Filter's constructor requires there to be an action on a non-null UIElement to be on top of the action stack, as that applies to every filter, so you don't have to note that in the subclass.
Once you've added requirements that ensure that the UI control interaction you want to regulate is on the stack, you'll have to implement
Run. This varies a great deal based on what you're trying to do. The simplest filters simply intend to change a property of an action's target. Examples of these are
WindowTitleFilter and
DropdownUncachingFilter. Things get more complicated when you want to create and delete actions.
Sample: Designing the navigation pane filter
The navigation pane filter is one of the most broad filters, along with the segmented entry filter. This makes it a good example for demonstrating all the techniques and steps involved in making a filter.
The purpose of the navigation pane filter is to optimize the speed of navigation by translating clicks on the navigation pane to direct navigation bar keyboard actions. This makes the requirements fairly simple, simply requiring that the user is clicking on a
TreeItem. Because these only exist in the navigation pane on the left, you can be sure that you won't interfere with other aspects of recording. This means that we can have the constructor simply be this:
public NavigationPaneFilter()
{
Require(stack => stack.Peek().UIElement.ControlTypeName == "TreeItem");
}
Next up, we want to make sure to remove any actions that are expanding or retracting the tree items. The easiest way to check this with the tools we have is by checking whether the ui element has any children. If it does, then it's not a leaf and the action is irrelevant, so we can just get rid of it and return. This creates the start of our
Run method.
protected override void Run(IUITestActionStack stack)
{
// If it has any children, this is just an organizational
// TreeItem, so we can ignore that action.
if (Playback.GetCoreTechnologyManager("MSAA").GetChildren(stack.Peek().UIElement, null).MoveNext())
{
stack.Pop();
return;
}
Before we continue, we're going to need to get the navigation bar so that we can create an action that types into it later. Before we can do any in detail navigation through the UI, we're going to want to make sure we have the right technology manager loaded. The
AXUtils extension methods rely on the
AXUtils.LoadManager method being called to set up a specific technology manager. In this case, the UIA manager is the best technology manager for the job. It gives a much cleaner hierarchy when it can give one at all, though it doesn't support getting an element's parent. Because we're going to simply move down the hierarchy from the main window down, we don't need to get an element's parent.
Unfortunately, it seems impossible to get the window of dynamics easily through just the
UITechnologyElement interface. Because of this we use the
GetDesktopWindow function from user32.dll in conjunction with the technology manager's
GetElementFromWindowHandle method. Afterward, we just walk down the tree until we reach the toolbar:
private UITechnologyElement GetToolbar(UITestAction action)
{
AXUtils.LoadManager("UIA");
IUITechnologyElement parent = AXUtils.Manager.GetElementFromWindowHandle(GetDesktopWindow())
.Child("Dynamics", "Window")
.Child("WindowHeaderFrame", "Pane")
.Child("Pane")
.Child("TopRow", "Pane")
.Child("AddressBarContainer", "Pane");
IUITechnologyElement toolbar = parent.Child("ToolBar");
Through testing we notice that this toolbar
IUITechnologyElement doesn't allow playback to find the toolbar, so we need to modify its search properties to properly specify the toolbar. This required poking around the UI hierarchy around the toolbar with inspect.exe. After seeing what properties the toolbar and its parent have, we redefine their search properties:
toolbar.QueryId.Ancestor = new UITechnologyElementRedirect(parent,
action.UIElement.TopLevelElement,
Playback.GetCoreTechnologyManager("MSAA"), "MSAA");
toolbar.QueryId.Condition = new AndCondition(
new PropertyCondition("ControlType", "ToolBar"));
parent.QueryId.Condition = new AndCondition(
new PropertyCondition("ClassName", "WindowsForms10.Window", PropertyConditionOperator.Contains),
new PropertyCondition("Instance", "35"),
new PropertyCondition("ControlType", "Window"));
return new UITechnologyElementRedirect(toolbar,
action.UIElement.TopLevelElement,
Playback.GetCoreTechnologyManager("MSAA"), "MSAA");
}
Now that we can get the toolbar, we can keep going with
Run. Once we have the toolbar, we need to create a mouse action and a typing action to enter the path of the tree item's destination.
UITestAction action = stack.Peek();
UITechnologyElement toolbar = GetToolbar(action);
var mouse = new MouseAction(toolbar,
System.Windows.Forms.MouseButtons.Left,
MouseActionType.Click);
// The farthest right location on the toolbar.
mouse.Location = new System.Drawing.Point(835, 5);
var keys = new SendKeysAction();
AXUtils.LoadManager("MSAA");
keys.Text = "USMF" + PathRepresented(action.UIElement) + "{Enter}";
The
PathRepresented method is a simple recursive method to get the path that the tree item clicked on leads to. It needs to get the parent of the element recursively, so we have to load the MSAA manager before calling it.
After all this is done, we have to actually deal with the actionstack. We've already popped all other navigation pane actions, but the user may have used the toolbar to get to an area page so that the navigation pane had what they wanted. We have to pop that as well, and after all that push the new click and type actions.
stack.Pop();
// If the user set the area page the newly pushed actions will do that anyway.
if (stack.Count >= 2 &&
stack.Peek(1).UIElement.ControlTypeName == "SplitButton")
{
stack.Pop();
stack.Pop();
}
stack.Push(mouse);
stack.Push(keys);
}
And that's it! The extension now changes any navigation through the pane to navigation through the toolbar. This represents a pretty large improvement in speed. However, though this seemed simple to create, there are many invisible steps involved in creating this filter:
Important considerations during development
While this seemed a fairly straightforward process, it actually took many iterations of each piece before it worked. The problem is that a large number of functionality that
UITestAction,
UITechnologyElement, and
TechnologyManager either don't actually support or implement in unexpected ways.
As an example, the fact that the UIA technology manager doesn't always support
GetParent was discovered experimentally after some confusion. Even more interestingly, it does work when used in the
SegmentedEntryFilter, possibly because of the UI framework used by that part of AX UI.
Another interesting issue came up in the creation of the
NavigationPaneFilter we just created. There were about 5 different ways of trying to get the main AX window. Most didn't work at all, simply returning null or returning an element in a strange hierarchy that didn't map to any existing window. Eventually I came upon the solution (using a user32.dll function).
To avoid most of this pain, it's important to change your mentality a little when developing on this extension framework. Instead of trying to find the single correct solution to a problem and make that work, it's better to maintain the mentality of trying out as many solutions as possible and then refining the one that works.