The CodeDom test suite is a collection of tests that enables custom code providers a way to verify that they are generating and compiling code from their provider properly. Each test takes a pre-built CodeDom tree and generates and compiles the tree using the provider supplied to it on the command line. The tests then verify the assembly generated from the code provider through reflection.
The pre-built CodeDom trees are listed in the BuildTree
methods
of each test located in the tests\
directory. The generated
C# code is listed above the CodeDom code that generates it. Even if you
don't have a CodeDom provider to test, you can browse through these tests
and get a feel for the constructs that are required to generate certain
kinds of code.
You must have one of the SDK versions of the runtime installed on your machine to use this test suite. These are freely available on MSDN.
To get started, refer to common usage scenarios.
Test case writers may want to refer to the Writing Test Cases for the Test Suite document also located in this collection.
The test suite takes the following command line arguments two of which are required. Below are common usage scenarios, and below that are the full usage descriptions.
A Visual Studio .NET 2005 Beta1 solution file has been included. Please upgrade to
Whidbey Beta1 of the runtime if you haven't already. A batch file
named build.bat
is included for those that wish to build the
test suite against v1.0 or v1.1 of the runtime, or for those preferring to
build from the commandline.
If you would like to use this batch file to build for v1.0 or v1.1 of the
runtime, simply run it as is. Those users wanting to use the batch file for
Whidbey must set the environment variable located in the script called
IS_WHIDBEY
to 1
.
It is recommended that you compile using the "Debug" release version in Visual Studio .NET or compile with the /debug+ switch on the command line. This will enhance your ability to troubleshoot problems with your provider while running the test cases.
Test Microsoft's CSharpCodeProvider against all test cases in the library "tests.dll." Also be verbose.
> cdtsuite/p
:Microsoft.CSharp.CSharpCodeProvider/tl
:tests.dll/v
Test MyCodeProvider located in "MyDll.dll" against all test cases in the library "tests.dll." Also, be verbose, and save all assemblies, logs and source files during the test.
> cdtsuite/p
:MyCodeProvider/pl
:MyDll.dll/tl
:tests.dll/v
/sl
/sa
/ss
Test MyCodeProvider located in "MyDll.dll" against the test cases named "ArrayTest" and "ParamsTest" in the library "tests.dll."
> cdtsuite/p
:MyCodeProvider/pl
:MyDll.dll/tl
:tests.dll/t
:ArrayTest/t
:ParamsTest
Test MyCodeProvider located in "MyDll.dll" against all test cases that are of the class "Subset" in the library "tests.dll."
> cdtsuite/p
:MyCodeProvider/pl
:MyDll.dll/tl
:tests.dll/set
:Subset
Test Microsoft's VBCodeProvider against the test case library "tests.dll." Also, be verbose, and save all assemblies, logs and source files during the test.
> cdtsuite/p
:Microsoft.VisualBasic.VBCodeProvider/tl
:c:\tests.dll
Test Microsoft's JScriptCodeProvider against the test case library "tests.dll."
The
option makes the test suite use the the language name to
get the provider. Also, dump logs and source files for failing test cases.
/lang
> cdtsuite/lang
:js/tl
:c:\tests.dll/dump
Options are case insensitive. You may also specify options with a '-'.
/help
or /?
or /h
Shows the usage description.
/codedomprovider:<value>
or /p:<value>
Either this or languagename required. CodeDomProvider you wish to use to run the test cases. This should be specified using the full name. You should also specify the codedomproviderlib option if this provider is not in an assembly in the GAC. If both languagename and codedomprovider options are specified, languagename is taken in favor of the two.
/languagename:<value>
or /lang:<value>
Either this or languagename required. The language to run the test cases against. This is the name given to CreateProvider(). This only works in CLR versions >= 2.0. If both languagename and codedomprovider options are specified, languagename is taken in favor of the two.
/testcaselib:<value>
or /tl:<value>
Required. Test case assembly that stores the test cases you wish to test against. You may specify more than one.
/codedomproviderlib:<value>
or /pl:<value>
The assembly where the CodeDomProvider lives. This should be specified either by file name or the strong name. If this option is not specified, the current assembly is used.
/list
or /l
Lists available tests. Use -listdesc to get descriptions. Note you must pass in valid test assemblies in order for this to print anything.
/listdesc
or /ld
Lists available tests and their descriptions. Note you must pass in valid test assemblies for this to print anything.
/set:<value>
or /s:<value>
The set of tests to run. Only those matching the given option will be run. See -setlist for a list of valid set names.
/setlist
or /sls
The list of test sets you may run.
/optionfile:<value>
or /of:<value>
Path to a option file(s) that lists these options one per line. Options specified in these file will 'add to' those also specified on the command-line. This is similar to the CSharp compiler's response file except options should not have '-' or '/' prepended before them.
/runtestcase:<value>
or /t:<value>
Select specific test case(s) to run. You may specify more than one.
/dontruntestcase:<value>
or /n:<value>
Select specific test case(s) *NOT* to run. You may specify more than one.
/savedirectory:<value>
or /dir:<value>
or /savedir:<value>
The directory to save all logs, assemblies and source files.
/savesources
or /ss
Save the sources even if all tests pass.
/saveassemblies
or /sa
Save the generated assemblies even if all tests pass.
/savelogs
or /sl
Save the generated logs even if all tests pass.
/verbose
or /v
Be verbose. All test case output is written to the console. If this is not given, no output is given unless the test case fails.
/dump
or /d
If during the run a test case fails, dump the log and source file to the console for that test case.
The suite consists of a single command-line program called cdtsuite
,
a CodeDomTest.dll
assembly that contains some utility classes and
base types, and an assembly that contains a base line test suite.
The tests in this assembly all derive from the class CodeDomTestTree
as provided in CodeDomTest.dll
.
Every test case derives from the base type CodeDomTest, and cdtsuite
uses virtual methods on this type to run each test. The tests
provided in the accompanying base line test case assembly (tests.dll
),
each exercise some part of a CodeDomProvider's ability to generate
and compile CodeDom trees. These tests and a short description are
given in the Test Case Description
section.
These base line tests are classified into three buckets: Subset, Everett and Whidbey. Subset tests generate and test CodeDom trees that generate the absolute minimally required parts of CodeDom. Everett and Whidbey tests generate trees that fall outside of this subset but within the feature areas of each classification (i.e. Everett tests don't test generics while Whidbey tests do).
When
you specify the classification on the command-line, every test in
or below that classification will be tested. So if you specify
Everett
on the command-line, every test with the
classification of
Subset
and Everett
will be tested, but
no test from Whidbey
will be run.
The CodeDom subset is a moving target. Currently, the official document is being reviewed and finalized. Look for its release on MSDN "in the near future."
cdtsuite
generates .log files for every test it runs. This log
file contains messages that each test case prints while it is
running and is saved to (output_dir)\(test_name).log
. If no
output directory is specified with /savedir
:, this directory
defaults to (current_dir)\testoutput
. Unless /savelogs
is
given, log files for passing test cases will be deleted.
If a test case derives from CodeDomTestTree, a source file may
be saved in the same directory as (test_name).src
. As with
.log files, .src files are deleted if the test case passes
(unless /savesources
is given).
Assemblies generated from these tests may also be saved by
specifying the /saveassemblies
option on the command-line. Only
test cases deriving from CodeDomTestTree will save these files.
Since assemblies are only generated if specified on the command
line, these will not be deleted even if test cases pass.
cdtsuite
's console output can be controlled with two arguments: /verbose
and /dump
. There are four combinations of these arguments; these
are listed below along with how they change cdtsuite
's behavior.
Nothing is printed, and only a summary is given after all test cases are run. The summary lists the test cases that failed and where to find the log and source files for these failed cases are located.
/verbose
only
As test cases are run, the output that would normally only go to the log file is printed on the console. This includes test cases that have succeeded. The summary is still given at the end of the run.
/dump
only
Nothing is printed for successful test cases, but on failed cases, both the log and source files are printed to the console. A summary is given at the end of the test case run.
/dump
and /verbose
Output is printed to the console as test cases are run as in
/verbose
mode, but also source files are printed for failed
cases. As always, a summary is given at the end of all test
case runs.
Test Name | Bucket | Test Description |
---|---|---|
CallMethodTest | Subset | Tests calling methods. |
CastingCodeDom | Subset | Tests casting |
CreateObjectTest | Subset | Tests object creation. |
DelegateTest | Subset | Tests delegates. |
EventTest | Subset | Tests events. |
IterationsTest | Subset | Tests iterations. |
NamespaceTest | Subset | Tests namespaces. |
OverloadTest | Subset | Tests method overloading. |
SubsetArrayTest | Subset | Tests arrays while conforming to the subset. |
SubsetAttributesTest | Subset | Tests metadata attributes while staying in the subset. |
SubsetBinaryOperatorsTest | Subset | Tests binary operators while staying within the subset spec. |
SubsetFieldsTest | Subset | Subset compatible test of calling fields. |
PropertiesTest | Subset | Tests properties |
TypeOfTest | Subset | Tests typeof statements. |
ArrayTest | Everett | Tests arrays. |
AttributesTest | Everett | Tests metadata attributes. |
BinaryOperatorsTest | Everett | Tests binary operators. |
CallingFieldTest | Everett | Tests calling fields. |
CallMethodWDirect | Everett | Call a method with a direction |
ClassTest | Everett | Tests classes. |
CodePrimitiveExpressionTest | Everett | Tests coding primitive expressions. |
CodeSnippetTest | Everett | Tests code snippets. |
CommentTest | Everett | Tests comment statements. |
ConditionalStatementTest | Everett | Tests conditional statements. |
ConstructorTest | Everett | Tests constructors. |
DeclareField | Everett | Tests declarations of fields. |
DeclareMethod | Everett | Tests declaration of methods. |
EnumTest | Everett | Test enumerations |
GeneratorSupportsTest | Everett | Tests GeneratorSupport enumeration. |
GoToTest | Everett | Tests goto statements |
ImplementingStructsTest | Everett | Tests structs that implement other things. |
IndexersTest | Everett | Tests indexers. |
LinePragmaTest | Everett | Tests CodeLinePragma. |
StructTest | Everett | Tests structs |
TryCatchTest | Everett | Tests try/catch/finally statements. |
TypeTest | Everett | Tests generating and using objects of different types. |
UnicodeCharEscapeTest | Subset | Test All the Unicode characters for escaping |
CheckSumTest | Whidbey | Checksum testing. |
GenericsTest | Whidbey | Tests generating generics from CodeDom. |
GlobalKeywordTest | Whidbey | Test the global keyword used to differentiate global namespaces from local ones. |
ParamsTest | Everett | Tests variable method parameters. |
PartialClassTest | Whidbey | Tests partial classes. |
RegionDirectiveTest | Whidbey | Tests region directives on various code constructs. |
VerbatimOrderingTest | Whidbey | Tests if types and members are generated in the order they are specified in the tree. |