Friday, 14 September 2012

Testing Framework Review:

In a previous post I reviewed NUnit. For my last post in this series I will focus on is a newer open source framework that is gaining some traction. From the website on CodePlex: is a unit testing tool for the .NET Framework. Written by the original inventor of NUnit, is the latest technology for unit testing C#, F#, VB.NET and other .NET languages. Works with ReSharper, CodeRush, and TestDriven.NET. is a developer testing framework, built to support Test Driven Development, with a design goal of extreme simplicity and alignment with framework features. It is compatible with .NET Framework 2.0 and later, and offers several runners: console, GUI, MSBuild, and Visual Studio integration via, CodeRush Test Runner and Resharper. It also offers test project integration for ASP.NET MVC. is even used internally by some high profile Microsoft projects such as:

Integration is a separate project meaning that direct Visual Studio integration support is not provided. However Visual Studio 2012 will allow different frameworks apart from MSTest to be used as the primary unit testing framework – this includes TFS builds too.

In the meantime, the following steps are required:

Download from NuGet

NuGet provides three packages for

Adding these packages to a Visual Studio project is very simple as NuGet will automatically download the latest versions and insert the correct project references required.

Project Items and Snippets

Unlike MSTest which provides project items and snippets with the IDE, does not provide any by default. However these items are not difficult to create yourself if required and the CodePlex project page even explains how to create snippets for


Although some initial setup is required one possible benefit is that is a standalone framework – it can be run anywhere without requiring installation, simply by copying the correct files.

Team Build

TFS 2012 will be able to use the same Unit Test plugin model that Visual Studio 2012 uses meaning in the future it will be a lot easier to integrate into the Team Build process.

Until then though it is possible to use within Team Build but only via a custom build activity and translating the XML output into MSTest results. This webpage explains how it is possible to do it, though the process looks quite longwinded to me.

Writing Tests

Tests are written like this in

   1: using System;
   2: using System.Collections.Generic;
   3: using System.Linq;
   4: using System.Text;
   5: using Xunit;
   6: using Xunit.Extensions;
   8: namespace SampleCode.xUnit
   9: {
  10:     // Classes do not require attributes, does not care
  11:     public class CalculatorTests
  12:     {
  13:         // A "fact" is a test without any parameters    
  14:         [Fact]
  15:         public void Add_AddOneAndTwo_ReturnsThree()
  16:         {
  17:             var result = Calculator.Add(1, 2);
  19:             // Many asserts are provided by default, API style is simple and concise
  20:             Assert.Equal(3, result);
  21:         }
  22:     }
  23: }

There are a much wider variety of assertions provided by by default compared to MSTest. A full list can be found here.

Data Driven Tests

Data driven tests in are known as theories. They are test methods that have parameters and can accept input from a number of sources. A theory looks like this:

   1: [Theory]
   2: [InlineData(1, 2, 3)]
   3: [InlineData(3, 4, 7)]
   4: [InlineData(30, 10, 40)]
   5: public void Add_AddDataValues_ReturnsExpectedResult(int first, int second, int expected)
   6: {
   7:     var actualResult = Calculator.Add(first, second);
   9:     Assert.Equal(expected, actualResult);
  10: }

Out of the box can accept input from the following sources:

  • Inline data
  • Property data
  • Excel spreadsheet
  • OleDb connection
  • SQL Server database

Running Tests

Until Visual Studio 2012 comes out tests cannot be run directly via the IDE but there are a number of other options available.

The console runner is the most basic test runner available and works from the command line.

The GUI runner is a simple standalone application with it’s own user interface and is not as full featured as the NUnit GUI runner but is capable enough. Instead of a tree view like the NUnit GUI runner this test runner presents a flat list of tests but they can be filtered down by search terms, assembly and/or trait values.

One aspect that is different from the NUnit GUI runner is that although this runner will detect and reload test assemblies when rebuilt it will not automatically run the selected tests again, unlike NUnit.


Unlike MSTest and NUnit, provides it’s own custom MSBuild task which allows direct integration into the build process. A project file can then use it similar to this:

   1: <UsingTask 
   2:     AssemblyFile="..\packages\xunit.1.9.1\lib\net20\xunit.runner.msbuild.dll" 
   3:     TaskName="Xunit.Runner.MSBuild.xunit" />
   4: <Target Name="AfterBuild">
   5:     <xunit Assembly="$(TargetPath)" />
   6: </Target>

Build output then looks similar to the following:

------ Build started: Project: SampleCode, Configuration: Debug Any CPU ------
SampleCode -> C:\TalentQ\Experiments\UnitTestAnalysis\SampleCode\bin\Debug\SampleCode.dll
------ Build started: Project: SampleCode.xUnit, Configuration: Debug Any CPU ------
SampleCode.xUnit -> C:\TalentQ\Experiments\UnitTestAnalysis\SampleCode.xUnit\bin\Debug\SampleCode.xUnit.dll MSBuild runner (32-bit .NET 4.0.30319.269)
xunit.dll: Version
Test assembly: C:\TalentQ\Experiments\UnitTestAnalysis\SampleCode.xUnit\bin\Debug\SampleCode.xUnit.dll
Tests: 4, Failures: 0, Skipped: 0, Time: 0.041 seconds
========== Build: 2 succeeded or up-to-date, 0 failed, 0 skipped ==========

The MSBuild task can be configured like the console runner meaning that XML/HTML results can also be saved too. See the documentation for more details.

Another useful thing is that, because it integrates into MSBuild, any failed tests will appear as errors in the IDE error list so by definition this would make it a failed build. The only slight oddity though is that, in its current form (version 1.9.1), double-clicking the errors in the error list does not take you to the correct source code as line numbers given are referring to the project file not source files.

Additional Runners also provides these test runners as standard:


Performance of running tests seems to be faster than MSTest, even with a significant number of tests to execute.


Apart from the output represented by various test runners, an XML report can be produced by either the console or MSBuild runner. Once in an XML format, this can then be transformed into another format, e.g. a HTML file to make it human readable or a *.trx (MSTest) output file so that Visual Studio can understand it.

Fortunately is able to do this transformation for you as long as you provide the XSLT stylesheet to use. Out of the box the following stylesheets are provided:

  • HTML – transforms the XML report into a HTML, human-readable report
  • NUnit – transforms the XML report into the same format that NUnit uses


This in my opinion is where falters. Because this is a newer framework documentation is thin on the ground, especially when compared to NUnit. Usually though the features are simple enough to figure out and there is sample code provided in the CodePlex repository, but you may also have to do some searching around on the internet for an explanation of some things.


One of the big selling points of is its extensibility which is far greater than either MSTest or NUnit. Some examples are:

Report Transformations

By default the console runner can provide XML, HTML or NUnit report output, but this is actually configurable by defining further command line switches mapped to a suitable XSLT stylesheet to transform in into another format (e.g. *.trx (MSTest) format). Extensions

The entire extensions assembly is a perfect example of its extensibility. For instance [Theory] methods are actually specialised [Fact] methods that do some additional work.

More Assertions

If there are not enough assertion functions for your liking you can implement more by extending the Assertions class rather than having to write your own wrappers for it. For example:

   1: public static class MyAssertions 
   2: {
   3:     // By using extension methods you can add more assertions
   4:     public static void Test(this Assertions assert)
   5:     { 
   6:         Assert.True(true);
   7:     }
   8: }
  10: // By deriving from TestClass a modifiable Assert class becomes available 
  11: public class CalculatorTests : TestClass 
  12: {
  13:     [Fact]
  14:     public void CustomAssert()
  15:     {
  16:         // This is our own assertion method
  17:         Assert.Test();
  18:     }
  19: }

My Opinion

This is a tricky one. Whereas I felt that NUnit was miles ahead of MSTest, the difference between NUnit and is a lot smaller. To be fair you could pick either one and be extremely productive so it all comes down to nit-picking.

Both NUnit and have their benefits and each have a few disadvantages but in the end, after much careful thought, I’ve decided to use as my primary test framework for the following reasons:

  1. I like the fact that XSLT stylesheets are provided with the framework so I don’t have to define my own HTML report format based on the XML output. And if I wanted to change the layout of the report I would at least have something to modify as a base.
  2. Overall the MSBuild task is a great way of integrating into the build process. Whereas MSTest and NUnit could be run as a task the fact was that they were just starting a new process; the only way you would know your tests had failed was by checking the exit code of the test runner, which wouldn’t tell you anything useful.
  3. The extensibility of framework is a real plus point. I haven’t needed to extend any features yet – consider that a testament to the basics it got right – but it’s nice to know that it is there if needed.
  4. Finally I just feel that it has a lot of potential. It may have some niggles to iron out but I feel confident that they will be.

All of the above are very minor points; like I said I could have just as easily went with NUnit, but just edged ahead in my opinion.