In the Coding Zone: Testing Framework Review: xUnit.net

In a previous post I reviewed NUnit. For my last post in this series I will focus on xUnit.net. xUnit.net is a newer open source framework that is gaining some traction. From the xUnit.net website on CodePlex:

xUnit.net is a unit testing tool for the .NET Framework. Written by the original inventor of NUnit, xUnit.net is the latest technology for unit testing C#, F#, VB.NET and other .NET languages. Works with ReSharper, CodeRush, and TestDriven.NET.
xUnit.net is a developer testing framework, built to support Test Driven Development, with a design goal of extreme simplicity and alignment with framework features. It is compatible with .NET Framework 2.0 and later, and offers several runners: console, GUI, MSBuild, and Visual Studio integration via TestDriven.net, CodeRush Test Runner and Resharper. It also offers test project integration for ASP.NET MVC.

xUnit.net is even used internally by some high profile Microsoft projects such as:

Integration

xUnit.net is a separate project meaning that direct Visual Studio integration support is not provided. However Visual Studio 2012 will allow different frameworks apart from MSTest to be used as the primary unit testing framework – this includes TFS builds too.

In the meantime, the following steps are required:

Download from NuGet

NuGet provides three packages for xUnit.net:

xUnit.net – the core framework
xUnit.net: Extensions – contains commonly used extensions for the framework (such as Theories)
xUnit.net: Runners – contains the various test runners needed to run test assemblies

Adding these packages to a Visual Studio project is very simple as NuGet will automatically download the latest versions and insert the correct project references required.

Project Items and Snippets

Unlike MSTest which provides project items and snippets with the IDE, xUnit.net does not provide any by default. However these items are not difficult to create yourself if required and the CodePlex project page even explains how to create snippets for xUnit.net.

Standalone

Although some initial setup is required one possible benefit is that xUnit.net is a standalone framework – it can be run anywhere without requiring installation, simply by copying the correct files.

Team Build

TFS 2012 will be able to use the same Unit Test plugin model that Visual Studio 2012 uses meaning in the future it will be a lot easier to integrate xUnit.net into the Team Build process.

Until then though it is possible to use xUnit.net within Team Build but only via a custom build activity and translating the xUnit.net XML output into MSTest results. This webpage explains how it is possible to do it, though the process looks quite longwinded to me.

Writing Tests

Tests are written like this in xUnit.net:

   1: using System;

   2: using System.Collections.Generic;

   3: using System.Linq;

   4: using System.Text;

   5: using Xunit;

   6: using Xunit.Extensions;

7:

   8: namespace SampleCode.xUnit

   9: {

  10:     // Classes do not require attributes, xUnit.net does not care

  11:     public class CalculatorTests

  12:     {

  13:         // A "fact" is a test without any parameters

  14:         [Fact]

  15:         public void Add_AddOneAndTwo_ReturnsThree()

  16:         {

  17:             var result = Calculator.Add(1, 2);

18:

  19:             // Many asserts are provided by default, API style is simple and concise

  20:             Assert.Equal(3, result);

  21:         }

  22:     }

  23: }

There are a much wider variety of assertions provided by xUnit.net by default compared to MSTest. A full list can be found here.

Data Driven Tests

Data driven tests in xUnit.net are known as theories. They are test methods that have parameters and can accept input from a number of sources. A theory looks like this:

   1: [Theory]

   2: [InlineData(1, 2, 3)]

   3: [InlineData(3, 4, 7)]

   4: [InlineData(30, 10, 40)]

   5: public void Add_AddDataValues_ReturnsExpectedResult(int first, int second, int expected)

   6: {

   7:     var actualResult = Calculator.Add(first, second);

8:

   9:     Assert.Equal(expected, actualResult);

  10: }

Out of the box xUnit.net can accept input from the following sources:

Inline data
Property data
Excel spreadsheet
OleDb connection
SQL Server database

Running Tests

Until Visual Studio 2012 comes out xUnit.net tests cannot be run directly via the IDE but there are a number of other options available.

The console runner is the most basic test runner available and works from the command line.

The GUI runner is a simple standalone application with it’s own user interface and is not as full featured as the NUnit GUI runner but is capable enough. Instead of a tree view like the NUnit GUI runner this test runner presents a flat list of tests but they can be filtered down by search terms, assembly and/or trait values.

One aspect that is different from the NUnit GUI runner is that although this runner will detect and reload test assemblies when rebuilt it will not automatically run the selected tests again, unlike NUnit.

MSBuild

Unlike MSTest and NUnit, xUnit.net provides it’s own custom MSBuild task which allows direct integration into the build process. A project file can then use it similar to this:

   1: <UsingTask

   2:     AssemblyFile="..\packages\xunit.1.9.1\lib\net20\xunit.runner.msbuild.dll"

   3:     TaskName="Xunit.Runner.MSBuild.xunit" />

   4: <Target Name="AfterBuild">

   5:     <xunit Assembly="$(TargetPath)" />

   6: </Target>

Build output then looks similar to the following:

------ Build started: Project: SampleCode, Configuration: Debug Any CPU ------
  SampleCode -> C:\TalentQ\Experiments\UnitTestAnalysis\SampleCode\bin\Debug\SampleCode.dll
------ Build started: Project: SampleCode.xUnit, Configuration: Debug Any CPU ------
  SampleCode.xUnit -> C:\TalentQ\Experiments\UnitTestAnalysis\SampleCode.xUnit\bin\Debug\SampleCode.xUnit.dll
  xUnit.net MSBuild runner (32-bit .NET 4.0.30319.269)
  xunit.dll:     Version 1.9.1.1600
  Test assembly: C:\TalentQ\Experiments\UnitTestAnalysis\SampleCode.xUnit\bin\Debug\SampleCode.xUnit.dll
    Tests: 4, Failures: 0, Skipped: 0, Time: 0.041 seconds
========== Build: 2 succeeded or up-to-date, 0 failed, 0 skipped ==========

The MSBuild task can be configured like the console runner meaning that XML/HTML results can also be saved too. See the documentation for more details.

Another useful thing is that, because it integrates into MSBuild, any failed tests will appear as errors in the IDE error list so by definition this would make it a failed build. The only slight oddity though is that, in its current form (version 1.9.1), double-clicking the errors in the error list does not take you to the correct source code as line numbers given are referring to the project file not source files.

Additional Runners

xUnit.net also provides these test runners as standard:

Performance

Performance of running tests seems to be faster than MSTest, even with a significant number of tests to execute.

Reports

Apart from the output represented by various test runners, an XML report can be produced by either the console or MSBuild runner. Once in an XML format, this can then be transformed into another format, e.g. a HTML file to make it human readable or a *.trx (MSTest) output file so that Visual Studio can understand it.

Fortunately xUnit.net is able to do this transformation for you as long as you provide the XSLT stylesheet to use. Out of the box the following stylesheets are provided:

HTML – transforms the XML report into a HTML, human-readable report
NUnit – transforms the XML report into the same format that NUnit uses

Documentation

This in my opinion is where xUnit.net falters. Because this is a newer framework documentation is thin on the ground, especially when compared to NUnit. Usually though the features are simple enough to figure out and there is sample code provided in the CodePlex repository, but you may also have to do some searching around on the internet for an explanation of some things.

Extensibility

One of the big selling points of xUnit.net is its extensibility which is far greater than either MSTest or NUnit. Some examples are:

Report Transformations

By default the console runner can provide XML, HTML or NUnit report output, but this is actually configurable by defining further command line switches mapped to a suitable XSLT stylesheet to transform in into another format (e.g. *.trx (MSTest) format).

xUnit.net Extensions

The entire xUnit.net extensions assembly is a perfect example of its extensibility. For instance [Theory] methods are actually specialised [Fact] methods that do some additional work.

More Assertions

If there are not enough assertion functions for your liking you can implement more by extending the Assertions class rather than having to write your own wrappers for it. For example:

   1: public static class MyAssertions

   2: {

   3:     // By using extension methods you can add more assertions

   4:     public static void Test(this Assertions assert)

   5:     {

   6:         Assert.True(true);

   7:     }

   8: }

9:

  10: // By deriving from TestClass a modifiable Assert class becomes available

  11: public class CalculatorTests : TestClass

  12: {

  13:     [Fact]

  14:     public void CustomAssert()

  15:     {

  16:         // This is our own assertion method

  17:         Assert.Test();

  18:     }

  19: }

My Opinion

This is a tricky one. Whereas I felt that NUnit was miles ahead of MSTest, the difference between NUnit and xUnit.net is a lot smaller. To be fair you could pick either one and be extremely productive so it all comes down to nit-picking.

Both NUnit and xUnit.net have their benefits and each have a few disadvantages but in the end, after much careful thought, I’ve decided to use xUnit.net as my primary test framework for the following reasons:

I like the fact that XSLT stylesheets are provided with the framework so I don’t have to define my own HTML report format based on the XML output. And if I wanted to change the layout of the report I would at least have something to modify as a base.
Overall the MSBuild task is a great way of integrating xUnit.net into the build process. Whereas MSTest and NUnit could be run as a task the fact was that they were just starting a new process; the only way you would know your tests had failed was by checking the exit code of the test runner, which wouldn’t tell you anything useful.
The extensibility of framework is a real plus point. I haven’t needed to extend any features yet – consider that a testament to the basics it got right – but it’s nice to know that it is there if needed.
Finally I just feel that it has a lot of potential. It may have some niggles to iron out but I feel confident that they will be.

All of the above are very minor points; like I said I could have just as easily went with NUnit, but xUnit.net just edged ahead in my opinion.

In the Coding Zone

Friday, 14 September 2012

Testing Framework Review: xUnit.net