Testing Framework Review: MSTest

Recently I wrote about what I learned from Test Driven Development (TDD) and for the most part glossed over the issue of which testing framework to use. In that post I said that I didn't like MSTest and ended up preferring xUnit.net, but was I too quick to judge? Was that a snap decision? Had I really investigated all the features of each framework?

As I am the sole TDD knowledge base at work currently I was also asked to write up an objective analysis of each the major frameworks that I was considering. I therefore thought this would be a good opportunity to share my findings to the world in case anyone else found them useful.

So I shall write three new posts giving my full review of the following .NET test frameworks I looked at and tried out: MSTest, NUnit and xUnit.net. I'm sure there are others out there but these were the ones I kept seeing through internet research on a regular basis. I shall try and present the facts as best I can but you will also see my opinions sprinkled throughout.

This first post will be focused on...

MSTest

MSTest is the testing framework from Microsoft and is built into some previous versions Visual Studio but is included as standard with Visual Studio 2010, which is the version I will be referring to from here on out. You can see the reference documentation here.

Integration

Because MSTest is built into Visual Studio itself, integration between Visual Studio and MSTest is very tight. Straight from the IDE you can:

• Create a test project
• Create a test class
• Use code snippets provided to define a test method
• Run tests using commands/shortcuts and view the results in an IDE tool window
• View report/result files.

Also, TFS 2010 has the option built-in to run MSTest assemblies as part of a Team Build process and it understands the MSTest result file format so it can be included in the build output.

A possible downside of deep integration though is that MSTest cannot run standalone without Visual Studio installed, unlike other frameworks. For instance, if a Continuous Integration (CI) build server were set up you would have to install the entire IDE just to use MSTest. It is possible to just find the necessary assemblies required for MSTest to force it to be standalone, as this blog post describes, but you would also have to fiddle around with the registry to include certain references which could be more painful than it is worth.

Writing Tests

So what do unit tests look like with MSTest? Here is a simple example:

using System;
using Microsoft.VisualStudio.TestTools.UnitTesting;

namespace SampleCode.MSTest
{
    // Denotes a Test Class for MSTest to look for
    [TestClass]
    public class CalculatorTests
    {        
        // Denotes a Test Method for MSTest to run
        [TestMethod]
        public void Add_AddOneAndTwo_ReturnsThree()
        {
            int first = 1;
            int second = 2;

            var result = Calculator.Add(first, second);

            // MSTest assertion
            Assert.AreEqual(3, result);
        }        
    }
}

MSTest provides several assertion methods via the Assert class, though it does not seem as feature complete as other frameworks.

Methods provided are:

• AreEqual
• AreNotEqual
• AreSame
• IsInstanceOfType
• IsNotInstanceOfType
• IsNull
• IsNotNull
• IsTrue
• IsFalse

There are also specialised StringAssert and CollectionAssert APIs.

Exception assertions must be done using the [ExpectedException] attribute. Other frameworks have moved on from this approach, making MSTest a bit antiquated in this area. Below is an example of how to test that something has thrown an exception:

[TestMethod]
[ExpectedException(typeof(ArgumentNullException))]
public void ExceptionTest()
{
    TestMethod(null);
}

It is possible to make your own assertions but you will have to define your own Assert class to provide additional assertions. For example, this blog post explains how to define an Assert.Throws() method to replace the clunky syntax for exception checking shown above for a more functional, fluid approach.

Data Driven Tests

Originally I thought that MSTest was not able to use data driven tests - that is tests which require input from an external source - but I was wrong. MSTest does support data driven tests but not in the same way as most other frameworks.

Below is an example of such a test:

[TestMethod]
[DataSource("Microsoft.VisualStudio.TestTools.DataSource.CSV", "|DataDirectory|\\Add_DataDrivenTestCases.csv", "Add_DataDrivenTestCases#csv", DataAccessMethod.Sequential)] 
[DeploymentItem("SampleCode.MSTest\\Add_DataDrivenTestCases.csv")]
public void Add_DataDriven_ReturnsCorrectResult()
{
    var first = Convert.ToInt32(TestContext.DataRow["First"]);
    var second = Convert.ToInt32(TestContext.DataRow["Second"]);
    var expectedResult = Convert.ToInt32(TestContext.DataRow["Result"]);

    var actualResult = Calculator.Add(first, second);

    Assert.AreEqual(expectedResult, actualResult);
}

Note the following:

1. MSTest does not support parameters in test methods like other frameworks do.
2. A [DataSource] attribute is used to define where the input data comes from. This example uses a CSV file but XML and databases are supported too.
3. Because parameters are not supported, all data input has to be extracted from a TestContext class (which you have to define yourself). Getting each column of the file actually returns an object, meaning you have to do yet more work to convert values into something meaningful.

In comparison to other frameworks (which I will highlight in future posts) this feature feels incredibly clunky to me. I used Visual Studio's IDE tools to auto-generate this test method for me as I doubt I would remember the syntax for it each time. It should also be noted that you cannot have inline data - that is data defined as constants such as what NUnit supports.

Running Tests

Running tests in MSTest is incredibly easy. Simply right-click in a test code file and click "Run Tests" for the Test Window to appear, such as below:

From this window you can view detailed results of each test.

For a data-driven test, only one row appears in this window yet it is run multiple times for each row in the data source, which can appear confusing. Viewing the detailed results of a data-driven test will show the individual results though.

It is also possible to run MSTest via an MSBuild task, e.g. as an after-build step. This is done by simply executing the MSTest console application using the <Exec> task like this:

<Target Name="AfterBuild">
    <Exec Command='"C:\Program Files (x86)\Microsoft Visual Studio 10.0\Common7\IDE\mstest.exe" /testcontainer:"$(TargetPath)"' />
</Target>

Build output then appears as follows:

------ Build started: Project: SampleCode.MSTest, Configuration: Debug Any CPU ------
  SampleCode.MSTest -> C:\Experiments\UnitTestAnalysis\SampleCode.MSTest\bin\Debug\SampleCode.MSTest.dll
  Microsoft (R) Test Execution Command Line Tool Version 10.0.30319.1
  Copyright (c) Microsoft Corporation. All rights reserved.

  Loading C:\Experiments\UnitTestAnalysis\SampleCode.MSTest\bin\Debug\SampleCode.MSTest.dll...
  Starting execution...

  Results               Top Level Tests
  -------               ---------------
  Passed                SampleCode.MSTest.CalculatorTests.Add_AddOneAndTwo_ReturnsThree
  Passed                SampleCode.MSTest.CalculatorTests.Add_DataDriven_ReturnsCorrectResult
  2/2 test(s) Passed

  Results               Add_DataDriven_ReturnsCorrectResult
  -------               -----------------------------------
  Passed                SampleCode.MSTest.CalculatorTests.Add_DataDriven_ReturnsCorrectResult
  Passed                SampleCode.MSTest.CalculatorTests.Add_DataDriven_ReturnsCorrectResult
  Passed                SampleCode.MSTest.CalculatorTests.Add_DataDriven_ReturnsCorrectResult
  3/3 test(s) Passed

  Summary
  -------
  Test Run Completed.
    Passed  5
    ---------
    Total   5
  Results file:  C:\Experiments\UnitTestAnalysis\SampleCode.MSTest\TestResults\petermonks_ACHILLES 2012-08-14 10_42_21.trx
  Test Settings: Default Test Settings
========== Build: 2 succeeded or up-to-date, 0 failed, 0 skipped ==========

Performance

Performance of running tests seems a little slower than other frameworks as MSTest saves all results to file meaning IO time is consumed a lot. This didn't cause any problems with my simple test suite but this might be significant if hundreds of tests are being run at once.

Extensibility

Since .NET 4 and Visual Studio 2010, MSTest has been able to be extended to some degree although doing so is hard work. For example, this blog post explains how to extend MSTest to use inline data-driven tests that have parameters, such as:

[TestMethod]
[Row(1, 2, 3)]
[Row(4, 5, 6)]
public void ParameterTest(int x, int y, int z)
{
    // Test code here...
}

However from this post it looks incredibly long-winded to implement and setup, whereas other frameworks have this ability as a built-in feature.

My Opinion

So these are the facts I researched, what do I think about MSTest now?

Overall I am still of the opinion that I was right to not use it. Having tried other frameworks, which I will give my review on in future posts, I can see that MSTest is clunky to use and not as fully featured as other frameworks. The only Unique Selling Point I can see with it is that it is built into Visual Studio so it is easy to get started with it and integrate it with other Microsoft tools, something we do require at work as we want to use TFS Build to get test results. But seeing as Visual Studio 2012 is going to include the new Unit Test Adapters to allow other test frameworks to be used instead, even this is starting to become a moot point. 

I've also noticed that some open-source Mircosoft projects don't even use MSTest and use something else. Maybe it is because they are trying to keep everything self-contained in these projects, but I read it as not even some developers in Mircosoft would consider using their own test framework.

So MSTest isn't for me. Next time I shall review NUnit.