CodeDom Test Suite Readme

Copyright (C) Microsoft Corp. 2004. Apache 2.0 License.

Introduction

The CodeDom test suite is a collection of tests that enables custom code providers a way to verify that they are generating and compiling code from their provider properly. Each test takes a pre-built CodeDom tree and generates and compiles the tree using the provider supplied to it on the command line. The tests then verify the assembly generated from the code provider through reflection.

The pre-built CodeDom trees are listed in the BuildTree methods of each test located in the tests\ directory. The generated C# code is listed above the CodeDom code that generates it. Even if you don't have a CodeDom provider to test, you can browse through these tests and get a feel for the constructs that are required to generate certain kinds of code.

You must have one of the SDK versions of the runtime installed on your machine to use this test suite. These are freely available on MSDN.

To get started, refer to common usage scenarios.

Test case writers may want to refer to the Writing Test Cases for the Test Suite document also located in this collection.

Usage

The test suite takes the following command line arguments two of which are required. Below are common usage scenarios, and below that are the full usage descriptions.

Compiling

A Visual Studio .NET 2005 Beta1 solution file has been included. Please upgrade to Whidbey Beta1 of the runtime if you haven't already. A batch file named build.bat is included for those that wish to build the test suite against v1.0 or v1.1 of the runtime, or for those preferring to build from the commandline.

If you would like to use this batch file to build for v1.0 or v1.1 of the runtime, simply run it as is. Those users wanting to use the batch file for Whidbey must set the environment variable located in the script called IS_WHIDBEY to 1.

It is recommended that you compile using the "Debug" release version in Visual Studio .NET or compile with the /debug+ switch on the command line. This will enhance your ability to troubleshoot problems with your provider while running the test cases.

Common Usage Scenarios

Test Microsoft's CSharpCodeProvider against all test cases in the library "tests.dll." Also be verbose.

   > cdtsuite /p:Microsoft.CSharp.CSharpCodeProvider /tl:tests.dll /v

Test MyCodeProvider located in "MyDll.dll" against all test cases in the library "tests.dll." Also, be verbose, and save all assemblies, logs and source files during the test.

   > cdtsuite /p:MyCodeProvider /pl:MyDll.dll /tl:tests.dll /v /sl /sa /ss

Test MyCodeProvider located in "MyDll.dll" against the test cases named "ArrayTest" and "ParamsTest" in the library "tests.dll."

   > cdtsuite /p:MyCodeProvider /pl:MyDll.dll /tl:tests.dll /t:ArrayTest /t:ParamsTest

Test MyCodeProvider located in "MyDll.dll" against all test cases that are of the class "Subset" in the library "tests.dll."

   > cdtsuite /p:MyCodeProvider /pl:MyDll.dll /tl:tests.dll /set:Subset

Test Microsoft's VBCodeProvider against the test case library "tests.dll." Also, be verbose, and save all assemblies, logs and source files during the test.

   > cdtsuite /p:Microsoft.VisualBasic.VBCodeProvider /tl:c:\tests.dll

Test Microsoft's JScriptCodeProvider against the test case library "tests.dll." The /lang option makes the test suite use the the language name to get the provider. Also, dump logs and source files for failing test cases.

   > cdtsuite /lang:js /tl:c:\tests.dll /dump

Full Usage Descriptions

Options are case insensitive. You may also specify options with a '-'.

/help or /? or /h

Shows the usage description.

/codedomprovider:<value> or /p:<value>

Either this or languagename required. CodeDomProvider you wish to use to run the test cases. This should be specified using the full name. You should also specify the codedomproviderlib option if this provider is not in an assembly in the GAC. If both languagename and codedomprovider options are specified, languagename is taken in favor of the two.

/languagename:<value> or /lang:<value>

Either this or languagename required. The language to run the test cases against. This is the name given to CreateProvider(). This only works in CLR versions >= 2.0. If both languagename and codedomprovider options are specified, languagename is taken in favor of the two.

/testcaselib:<value> or /tl:<value>

Required. Test case assembly that stores the test cases you wish to test against. You may specify more than one.

/codedomproviderlib:<value> or /pl:<value>

The assembly where the CodeDomProvider lives. This should be specified either by file name or the strong name. If this option is not specified, the current assembly is used.

/list or /l

Lists available tests. Use -listdesc to get descriptions. Note you must pass in valid test assemblies in order for this to print anything.

/listdesc or /ld

Lists available tests and their descriptions. Note you must pass in valid test assemblies for this to print anything.

/set:<value> or /s:<value>

The set of tests to run. Only those matching the given option will be run. See -setlist for a list of valid set names.

/setlist or /sls

The list of test sets you may run.

/optionfile:<value> or /of:<value>

Path to a option file(s) that lists these options one per line. Options specified in these file will 'add to' those also specified on the command-line. This is similar to the CSharp compiler's response file except options should not have '-' or '/' prepended before them.

/runtestcase:<value> or /t:<value>

Select specific test case(s) to run. You may specify more than one.

/dontruntestcase:<value> or /n:<value>

Select specific test case(s) *NOT* to run. You may specify more than one.

/savedirectory:<value> or /dir:<value> or /savedir:<value>

The directory to save all logs, assemblies and source files.

/savesources or /ss

Save the sources even if all tests pass.

/saveassemblies or /sa

Save the generated assemblies even if all tests pass.

/savelogs or /sl

Save the generated logs even if all tests pass.

/verbose or /v

Be verbose. All test case output is written to the console. If this is not given, no output is given unless the test case fails.

/dump or /d

If during the run a test case fails, dump the log and source file to the console for that test case.

Layout

The suite consists of a single command-line program called cdtsuite, a CodeDomTest.dll assembly that contains some utility classes and base types, and an assembly that contains a base line test suite. The tests in this assembly all derive from the class CodeDomTestTree as provided in CodeDomTest.dll.

Every test case derives from the base type CodeDomTest, and cdtsuite uses virtual methods on this type to run each test. The tests provided in the accompanying base line test case assembly (tests.dll), each exercise some part of a CodeDomProvider's ability to generate and compile CodeDom trees. These tests and a short description are given in the Test Case Description section.

Test Classification (Buckets)

These base line tests are classified into three buckets: Subset, Everett and Whidbey. Subset tests generate and test CodeDom trees that generate the absolute minimally required parts of CodeDom. Everett and Whidbey tests generate trees that fall outside of this subset but within the feature areas of each classification (i.e. Everett tests don't test generics while Whidbey tests do).

When you specify the classification on the command-line, every test in or below that classification will be tested. So if you specify Everett on the command-line, every test with the classification of Subset and Everett will be tested, but no test from Whidbey will be run.

The CodeDom subset is a moving target. Currently, the official document is being reviewed and finalized. Look for its release on MSDN "in the near future."

Output

cdtsuite generates .log files for every test it runs. This log file contains messages that each test case prints while it is running and is saved to (output_dir)\(test_name).log. If no output directory is specified with /savedir:, this directory defaults to (current_dir)\testoutput. Unless /savelogs is given, log files for passing test cases will be deleted.

Source File Generation

If a test case derives from CodeDomTestTree, a source file may be saved in the same directory as (test_name).src. As with .log files, .src files are deleted if the test case passes (unless /savesources is given).

Assembly File Generation

Assemblies generated from these tests may also be saved by specifying the /saveassemblies option on the command-line. Only test cases deriving from CodeDomTestTree will save these files. Since assemblies are only generated if specified on the command line, these will not be deleted even if test cases pass.

Console Output

cdtsuite's console output can be controlled with two arguments: /verbose and /dump. There are four combinations of these arguments; these are listed below along with how they change cdtsuite's behavior.

no arguments

Nothing is printed, and only a summary is given after all test cases are run. The summary lists the test cases that failed and where to find the log and source files for these failed cases are located.

/verbose only

As test cases are run, the output that would normally only go to the log file is printed on the console. This includes test cases that have succeeded. The summary is still given at the end of the run.

/dump only

Nothing is printed for successful test cases, but on failed cases, both the log and source files are printed to the console. A summary is given at the end of the test case run.

/dump and /verbose

Output is printed to the console as test cases are run as in /verbose mode, but also source files are printed for failed cases. As always, a summary is given at the end of all test case runs.

Test Cases and their Descriptions

Test NameBucketTest Description
CallMethodTestSubsetTests calling methods.
CastingCodeDomSubsetTests casting
CreateObjectTestSubsetTests object creation.
DelegateTestSubsetTests delegates.
EventTestSubsetTests events.
IterationsTestSubsetTests iterations.
NamespaceTestSubsetTests namespaces.
OverloadTestSubsetTests method overloading.
SubsetArrayTestSubsetTests arrays while conforming to the subset.
SubsetAttributesTestSubsetTests metadata attributes while staying in the subset.
SubsetBinaryOperatorsTestSubsetTests binary operators while staying within the subset spec.
SubsetFieldsTestSubsetSubset compatible test of calling fields.
PropertiesTestSubsetTests properties
TypeOfTestSubsetTests typeof statements.
ArrayTestEverettTests arrays.
AttributesTestEverettTests metadata attributes.
BinaryOperatorsTestEverettTests binary operators.
CallingFieldTestEverettTests calling fields.
CallMethodWDirectEverettCall a method with a direction
ClassTestEverettTests classes.
CodePrimitiveExpressionTestEverettTests coding primitive expressions.
CodeSnippetTestEverettTests code snippets.
CommentTestEverettTests comment statements.
ConditionalStatementTestEverettTests conditional statements.
ConstructorTestEverettTests constructors.
DeclareFieldEverettTests declarations of fields.
DeclareMethodEverettTests declaration of methods.
EnumTestEverettTest enumerations
GeneratorSupportsTestEverettTests GeneratorSupport enumeration.
GoToTestEverettTests goto statements
ImplementingStructsTestEverettTests structs that implement other things.
IndexersTestEverettTests indexers.
LinePragmaTestEverettTests CodeLinePragma.
StructTestEverettTests structs
TryCatchTestEverettTests try/catch/finally statements.
TypeTestEverettTests generating and using objects of different types.
UnicodeCharEscapeTestSubsetTest All the Unicode characters for escaping
CheckSumTestWhidbeyChecksum testing.
GenericsTestWhidbeyTests generating generics from CodeDom.
GlobalKeywordTestWhidbeyTest the global keyword used to differentiate global namespaces from local ones.
ParamsTestEverettTests variable method parameters.
PartialClassTestWhidbeyTests partial classes.
RegionDirectiveTestWhidbeyTests region directives on various code constructs.
VerbatimOrderingTestWhidbeyTests if types and members are generated in the order they are specified in the tree.