Wednesday, 29 August 2012

Time for a Change

When I first started this blog I wasn’t sure how long it would last, so I made some initial decisions such as picking a free blogging platform as my host. As a result I may have gone a bit overboard with the design of it; it is hard to know what your blog looks like when there is no content on it to give you an idea. Because of this I felt that the look of this blog was too dark and too “busy”.

Taking a leaf out of Microsoft’s newfound focus on it’s Metro/Modern/Whatever-you-call-it UI style, I thought I would make the design simpler and focus on making it more readable and just focus on the content, now that I actually have some.

This is also the first time I’m writing a post in something other than Blogger’s online editing tools. They were fairly good but there were just too many annoyances I found, such as filling in hyperlinks would send me right back to the top of the page, and formatting (especially code snippets) was a real pain. This post was written in Windows Live Writer which I am very impressed with so far.

So just a quick update this time, normal service will resume shortly.

Sunday, 26 August 2012

Testing Framework Review: MSTest

Recently I wrote about what I learned from Test Driven Development (TDD) and for the most part glossed over the issue of which testing framework to use. In that post I said that I didn't like MSTest and ended up preferring, but was I too quick to judge? Was that a snap decision? Had I really investigated all the features of each framework?

As I am the sole TDD knowledge base at work currently I was also asked to write up an objective analysis of each the major frameworks that I was considering. I therefore thought this would be a good opportunity to share my findings to the world in case anyone else found them useful.

So I shall write three new posts giving my full review of the following .NET test frameworks I looked at and tried out: MSTest, NUnit and I'm sure there are others out there but these were the ones I kept seeing through internet research on a regular basis. I shall try and present the facts as best I can but you will also see my opinions sprinkled throughout.

This first post will be focused on...


MSTest is the testing framework from Microsoft and is built into some previous versions Visual Studio but is included as standard with Visual Studio 2010, which is the version I will be referring to from here on out. You can see the reference documentation here.


Because MSTest is built into Visual Studio itself, integration between Visual Studio and MSTest is very tight. Straight from the IDE you can:
  • Create a test project
  • Create a test class
  • Use code snippets provided to define a test method
  • Run tests using commands/shortcuts and view the results in an IDE tool window
  • View report/result files.
Also, TFS 2010 has the option built-in to run MSTest assemblies as part of a Team Build process and it understands the MSTest result file format so it can be included in the build output.

A possible downside of deep integration though is that MSTest cannot run standalone without Visual Studio installed, unlike other frameworks. For instance, if a Continuous Integration (CI) build server were set up you would have to install the entire IDE just to use MSTest. It is possible to just find the necessary assemblies required for MSTest to force it to be standalone, as this blog post describes, but you would also have to fiddle around with the registry to include certain references which could be more painful than it is worth.

Writing Tests

So what do unit tests look like with MSTest? Here is a simple example:

using System;
using Microsoft.VisualStudio.TestTools.UnitTesting;

namespace SampleCode.MSTest
    // Denotes a Test Class for MSTest to look for
    public class CalculatorTests
        // Denotes a Test Method for MSTest to run
        public void Add_AddOneAndTwo_ReturnsThree()
            int first = 1;
            int second = 2;

            var result = Calculator.Add(first, second);

            // MSTest assertion
            Assert.AreEqual(3, result);

MSTest provides several assertion methods via the Assert class, though it does not seem as feature complete as other frameworks.

Methods provided are:
  • AreEqual
  • AreNotEqual
  • AreSame
  • IsInstanceOfType
  • IsNotInstanceOfType
  • IsNull
  • IsNotNull
  • IsTrue
  • IsFalse
There are also specialised StringAssert and CollectionAssert APIs.

Exception assertions must be done using the [ExpectedException] attribute. Other frameworks have moved on from this approach, making MSTest a bit antiquated in this area. Below is an example of how to test that something has thrown an exception:

public void ExceptionTest()

It is possible to make your own assertions but you will have to define your own Assert class to provide additional assertions. For example, this blog post explains how to define an Assert.Throws() method to replace the clunky syntax for exception checking shown above for a more functional, fluid approach.

Data Driven Tests

Originally I thought that MSTest was not able to use data driven tests - that is tests which require input from an external source - but I was wrong. MSTest does support data driven tests but not in the same way as most other frameworks.

Below is an example of such a test:

[DataSource("Microsoft.VisualStudio.TestTools.DataSource.CSV", "|DataDirectory|\\Add_DataDrivenTestCases.csv", "Add_DataDrivenTestCases#csv", DataAccessMethod.Sequential)] 
public void Add_DataDriven_ReturnsCorrectResult()
    var first = Convert.ToInt32(TestContext.DataRow["First"]);
    var second = Convert.ToInt32(TestContext.DataRow["Second"]);
    var expectedResult = Convert.ToInt32(TestContext.DataRow["Result"]);

    var actualResult = Calculator.Add(first, second);

    Assert.AreEqual(expectedResult, actualResult);

Note the following:

  1. MSTest does not support parameters in test methods like other frameworks do.
  2. A [DataSource] attribute is used to define where the input data comes from. This example uses a CSV file but XML and databases are supported too.
  3. Because parameters are not supported, all data input has to be extracted from a TestContext class (which you have to define yourself). Getting each column of the file actually returns an object, meaning you have to do yet more work to convert values into something meaningful.
In comparison to other frameworks (which I will highlight in future posts) this feature feels incredibly clunky to me. I used Visual Studio's IDE tools to auto-generate this test method for me as I doubt I would remember the syntax for it each time. It should also be noted that you cannot have inline data - that is data defined as constants such as what NUnit supports.

Running Tests

Running tests in MSTest is incredibly easy. Simply right-click in a test code file and click "Run Tests" for the Test Window to appear, such as below:

From this window you can view detailed results of each test.

For a data-driven test, only one row appears in this window yet it is run multiple times for each row in the data source, which can appear confusing. Viewing the detailed results of a data-driven test will show the individual results though.

It is also possible to run MSTest via an MSBuild task, e.g. as an after-build step. This is done by simply executing the MSTest console application using the <Exec> task like this:

<Target Name="AfterBuild">
    <Exec Command='"C:\Program Files (x86)\Microsoft Visual Studio 10.0\Common7\IDE\mstest.exe" /testcontainer:"$(TargetPath)"' />

Build output then appears as follows:

------ Build started: Project: SampleCode.MSTest, Configuration: Debug Any CPU ------
  SampleCode.MSTest -> C:\Experiments\UnitTestAnalysis\SampleCode.MSTest\bin\Debug\SampleCode.MSTest.dll
  Microsoft (R) Test Execution Command Line Tool Version 10.0.30319.1
  Copyright (c) Microsoft Corporation. All rights reserved.

  Loading C:\Experiments\UnitTestAnalysis\SampleCode.MSTest\bin\Debug\SampleCode.MSTest.dll...
  Starting execution...

  Results               Top Level Tests
  -------               ---------------
  Passed                SampleCode.MSTest.CalculatorTests.Add_AddOneAndTwo_ReturnsThree
  Passed                SampleCode.MSTest.CalculatorTests.Add_DataDriven_ReturnsCorrectResult
  2/2 test(s) Passed

  Results               Add_DataDriven_ReturnsCorrectResult
  -------               -----------------------------------
  Passed                SampleCode.MSTest.CalculatorTests.Add_DataDriven_ReturnsCorrectResult
  Passed                SampleCode.MSTest.CalculatorTests.Add_DataDriven_ReturnsCorrectResult
  Passed                SampleCode.MSTest.CalculatorTests.Add_DataDriven_ReturnsCorrectResult
  3/3 test(s) Passed

  Test Run Completed.
    Passed  5
    Total   5
  Results file:  C:\Experiments\UnitTestAnalysis\SampleCode.MSTest\TestResults\petermonks_ACHILLES 2012-08-14 10_42_21.trx
  Test Settings: Default Test Settings
========== Build: 2 succeeded or up-to-date, 0 failed, 0 skipped ==========


Performance of running tests seems a little slower than other frameworks as MSTest saves all results to file meaning IO time is consumed a lot. This didn't cause any problems with my simple test suite but this might be significant if hundreds of tests are being run at once.


Since .NET 4 and Visual Studio 2010, MSTest has been able to be extended to some degree although doing so is hard work. For example, this blog post explains how to extend MSTest to use inline data-driven tests that have parameters, such as:

[Row(1, 2, 3)]
[Row(4, 5, 6)]
public void ParameterTest(int x, int y, int z)
    // Test code here...

However from this post it looks incredibly long-winded to implement and setup, whereas other frameworks have this ability as a built-in feature.

My Opinion

So these are the facts I researched, what do I think about MSTest now?

Overall I am still of the opinion that I was right to not use it. Having tried other frameworks, which I will give my review on in future posts, I can see that MSTest is clunky to use and not as fully featured as other frameworks. The only Unique Selling Point I can see with it is that it is built into Visual Studio so it is easy to get started with it and integrate it with other Microsoft tools, something we do require at work as we want to use TFS Build to get test results. But seeing as Visual Studio 2012 is going to include the new Unit Test Adapters to allow other test frameworks to be used instead, even this is starting to become a moot point. 

I've also noticed that some open-source Mircosoft projects don't even use MSTest and use something else. Maybe it is because they are trying to keep everything self-contained in these projects, but I read it as not even some developers in Mircosoft would consider using their own test framework.

So MSTest isn't for me. Next time I shall review NUnit.

Thursday, 16 August 2012

Visual Studio 2012: What's Your Problem?

Today I read that Visual Studio 2012 has finally been released (to MSDN subscribers at least). There are a lot of new features I'm looking forward to using in my work, some of which are highlighted in these videos. And yet instead of people being excited about it, all I see in blog comments are people still complaining about this:

Too little colour, TOO MUCH SCREAMING!!
Image courtesy of Scott Hanselman (

Seriously, are we still complaining about this?

I have to admit that when I first saw screenshots of the new interface I also thought "what have they done?" This was back when everything was grey and dull. I initially didn't like the UI but rather than just complain endlessly about it I thought I would wait until I could at least try it myself and not judge it on a few screenshots of a beta product. I even gave what I hoped was constructive criticism about it , like many others did. And eventually Microsoft got the hint and made changes and toned down the extreme parts of the new UI and now I feel more comfortable with it.

But this UI change has ended up masking any number of new features and improvements that actually make you productive to the point that people can only focus on one thing: they hate how it looks.

Now I'm not saying that it is perfect or anything, or that it ever could be, but I personally am happy enough with the final result and want to focus on what it can do rather than how it looks. Yet when I see comments like this still I wonder what people are really focusing on:

Dear Microsoft,
I've been a fan and supporter for 20+ years.  This time you guys outdid yourselves (yes, I am being sarcastic).
It's not even constructive, this is just a person ranting to the world because they can. Unfortunately a lot of people follow this tack so it just feels like I'm drowning in negativity.

I try very hard to not be critical of things and if I have to complain I try very hard to at least give suggestions to improve something, but it seems the age of the Internet and social media simply makes us into people who want to argue for arguments sake and that disappoints me.

And I want Microsoft to know that I at least am looking forward to using VS2012.

Friday, 10 August 2012

What I've Learned About Test Driven Development

At the end of April this year me and my colleagues prepared and deployed a big website/system update. A lot of things would change and get better and despite our rather lax testing procedures - testing being not very formal and done by us developers - we felt we had tested it as much as we possibly could and were very happy with what we did. We deployed it over a weekend with scheduled downtime, did our updates along with some server enhancements, and brought the site back up...

...Starting on the following Monday and continuing for the next two weeks we did nothing but patch holes with endless hotfixes and grovel to our customers promising that we will fix all the bugs they were reporting non-stop as soon as possible. One of our web services did not start up again for an entire weekend and our biggest client wanted to know why they couldn't access our systems. Users were complaining that the website kept kicking them out at random times which we eventually found out was being caused by a StackOverflow exception occurring every 30 minutes and crashing the IIS web process. It got to the point where the CEO got involved and at the eventual end of this nightmare my manager had to go and visit some key clients to personally assure them that all was well and this would never happen again.

We felt gutted. By our standards this was the most tested release we had ever done and yet it ended up being the worst one in our company's history.

This is all background to the real purpose of this post. When the dust settled we all sat down and discussed what went wrong and why and one of the key points we learned (though not the only one) was that our testing strategy was dire. In the absence of a real software tester (though we have one now) it was down to the people writing the code to do testing, and it tended to be a quick couple of run-throughs in certain areas to make sure things worked as we expected. This was not good enough and I simply felt that I couldn't trust our own system anymore.

A friend of mine talked to me about the benefits of Test Driven Development (TDD) a couple of weeks earlier and at this point I put it towards my team. I felt that, as well as having a proper software tester, we as developers needed to think about testing from the very start. In essence, we had to build quality into our code.

And so began my long journey into a development practice that I had never tried before and yet now I feel makes so much sense to me. Last week marked the very first unit and functional tests ever entered into our codebase and although they only cover a tiny fraction of our entire code I feel much more comfortable that that code works as it should, even before it goes to our tester to evaluate.

I will not go into the entire TDD process - there are enough resources on this already - but I thought as this was a learning process for me I would share with you what I've learned in the past few months.

Roy Osherove to the Rescue!

I did a lot of research into TDD, trawling the internet and reading through a book a colleague lent me but I still had many questions and things I was not sure about. What is the difference between a unit test and an integration test? What if your code has to read from a database or a file? What is a stub and what is a mock?

Then I listened to this Hanselminutes podcast of Scott Hanselman talking to Roy Osherove, author of The Art of Unit Testing, and all my questions were answered in 30 minutes. If anyone else is having trouble understanding unit testing and TDD I would very much recommend listening to this as it helped me a lot.

No-One Likes MSTest

Before you can write your tests though you need a unit testing framework to help you write your them, but which one do you choose?

Initially I started using MSTest as that came integrated with Visual Studio and I could experiment quickly with learning the TDD process, but in the end I saw some flaws with it - as do many others

  1. The benefit of having MSTest integral to Visual Studio can also be it's downfall as you simply cannot run it standalone.
  2. MSTest simply provided the bare minimum of features to make it worthy of calling it a unit testing framework. I learned that other unit testing frameworks were more capable and had far better features than MSTest, including assertion of exceptions and data-driven tests.
  3. The last problem I had was more of a personal one in that the test runner window didn't seem to make it that clear to me that all the tests had passed or failed. Compare the screenshots below of the MSTest and the NUnit test runner:
An example of the MSTest runner. Notice that tiny little tick at the top of the window? That tells me that all the tests passed.

Courtesy of Robust Haven (
Now compare that to the NUnit test runner. See that great big green line? Guess what that means?

Courtesy of litemedia (

So I tried out NUnit which I liked very much and seemed very dependable, and most recently which I finally settled on.

What I learned from this though is that, once you take MSTest out of the equation, any test framework is usually good enough and it simply comes down to personal preference. I went with because it provided several things I found useful straight out-of-the-box such as an MSBuild task to add to your project builds - if a test fails then the build fails - and it provides a XML stylesheet by default to create HTML reports rather than having to make one myself, a la NUnit.

Doing it Backwards

The very first stumbling block was that I had to mentally switch around my working style to effectively be back to front. Write tests before code? How was that meant to work, surely you need something there to try out? But as long as you follow the Red, Green, Refactor process it does actually work.

Even so, for a beginner like me it required a very large mental shift. For my very first unit test I must have sat there for a good half-hour thinking "How do I write this test? Where do I start?" Once you get past the first test - usually making sure something works - then the next test seems a bit more obvious: "What if I pass in the wrong input?" This could produce quite a few more unit tests so now the ball starts rolling. And once the first code function is tested you start to see a pattern so the next batch of tests seem a bit easier to write, and so on.

As long as you can start writing some unit tests to begin with, the process becomes a lot more intuitive up to the point where it seems like second nature and the way you were working before seems so arcane now.

Designing Your API

The other thing that struck me is that, despite the word "test" in Test Driven Development, initially it feels more like I'm writing a very detailed specification for all my objects. As I am building up each test it feels like I am defining what the code should be doing which to my mind is almost like a list of requirements - function A will allow a string to be passed in, but not an empty string or one that has more than 100 characters etc.

TDD has also made me think a lot more about object dependencies and code design. When I sat down and looked at my codebase as it stands now, many objects in the business logic layer call straight into the data-access layer and database to do something and these objects simply assume that that will just work - there has to be a database present to do anything in my system. For the real world that makes perfect sense as a system without a database is pretty much useless, but in a testing scenario it cannot be assumed that a database is there and for unit testing nothing should be touching external state.

This led me to wonder how I could actually test anything if all of my code is so inter-dependent on each other (or tightly coupled). The answer for TDD is to make your code loosely coupled instead which impacts your code design and makes you think a lot more about patterns and practices like Dependency Injection. As a result I'm starting to feel a lot happier with the overall design of my code as it has started to resemble smaller, Lego-sized blocks which can be combined together into bigger systems.

Start From Scratch

Following on from the previous section if you already have a large codebase and start to introduce TDD into it, like me you might find it a bit daunting. Where do you start? Do you retroactively add unit tests into existing code? And what about dependency problems you have?

So I learned that the best way to learn about TDD and introduce it into your working practices is to start on something brand new, a clean slate. That way you can build up your tests and knowledge as you go. And if you can't start from the very beginning? Then pick a new feature that you're due to develop and start thinking about how you would implement it in a TDD fashion. If only one part of your codebase has been written with testability in mind then that is better than nothing.

Test As Much As You Can

My final thought is that initially I thought that you would have to aim for 100% test coverage to make you feel that you had done a good job. While it is a target to aim for, you don't have to achieve it, or maybe it is just impossible to reach that goal for many reasons.

The very first feature I implemented into our product that was written with TDD in mind only has about 30 unit tests and that doesn't even cover the entire feature end-to-end, the reason being that existing code has dependencies that I cannot extricate just yet. But the new code I have written, that was unit tested. In all, the total test coverage is a tiny percentage but at least I can build upon it over time.

All in all I now think in a TDD style and find it difficult when I have to revert back to the old-fashioned cod-then-test mentality. It's only been a few months since I started investigating this process as a possibility but I'm glad that I did as I feel one step closer to being a better programmer.

Wednesday, 1 August 2012

Becoming a Father: One Year On - Part 3

This is the third part of my retrospective on my first year as a new father. To view all the parts, please click on one of the links below:

Part 1 - The Beginning
Part 2 - The First Two Weeks
Part 4 - The Accident

This post focuses on...

Getting the Hang of Things

Once I became more accustomed to waking up several times in the night followed by waking up early in the morning to go to work each day, I came to realise that things do tend to settle into a routine (almost). My wife and I learned many things about our son in the first 6 months:

  1. He liked to be on the go all the time, he simply did not like sitting still much. This resulted in me usually having to administer some kind of silly walk whilst carrying him so he felt the exaggerated movement and which kept him happy. Although after 30 seconds of this around the living room my leg muscles would be screaming at me to stop.
  2. He also didn't think much of napping during the day. Whilst other parents might boast that their little angel would have a 1 - 2 hour nap in the afternoon, we were forced to endure almost a whole day of feeling exhausted and looking after him since he would sleep for at most 30 minutes at a time (with usually a 4 - 5 hour gap inbetween).
  3. The flip-side of not napping during the day though was that he at least seemed to understand the day-night cycle and would sleep longer and better during the night. My wife and I started off going to bed at around 8pm for at least 3 months because we simply couldn't stay awake any longer!
  4. He was heavy and I was weak. When he was born he weighed 8lb 11oz - which for those not in the know is fairly large - and he pretty much worked his way up to the 91st percentile of his baby contemporaries and was generally the largest of all his baby friends. I am not a physically strong person and having to carry him around started to get very draining, though I eventually got more used to it - perhaps it could have been considered as weight training for me!
  5. My wife tried feeding him expressed milk as breastfeeding did not go well but eventually he started to drink more and more formula milk to the point where he could, after a few months, drink almost an entire bottle of it every 3 hours. That could explain the weight gain...
  6. He got to a point 3 months in where he slept pretty much through the night - the holy grail of parenting! But then, at 4 months, he switched things around a bit and did the dreaded sleep regression. Why babies apparently do this I do not know. I was having dinner with some friends and boasting about how he slept through the night now at such a young age when I got a text message from my wife telling me that he started waking up at night; that was the night when he started waking up several times a night for the next few months. I resolved from then on to never discuss his sleeping habits with anyone other than my wife lest I somehow jinx everything.
  7. Most importantly though was that we could start to see that he had a personality. It is hard to describe in writing but there were little things he did that you could just tell were a part of him.

The Daily Routine

I found it incredibly helpful that a daily routine seemed to materialise over time - I am a programmer after all! My wife and I both agreed that I would do the (very) early morning feed as once I'm awake I tend to stay awake; I'm a very light sleeper. 

Once that was done and everyone was still asleep I could get to my morning chores which started about 6am. These involved having a shower and breakfast for me of course but also cleaning up all the milk bottles, getting the bottle steriliser ready and boiling water for his formula milk for the day. If I had time I'd also prepare his reusable nappies after they had been washed and dried. 

At about 7:30am I would go wake up my wife so she could have a shower before I went to work and I would look after the boy in the meantime. We decided that my wife would get as much rest as possible and not the other way around since she would be the one spending all day with him on the go - remember, he didn't like napping. She basically needed as much rest as possible whereas, despite me having to do a full day's work too, at least I would be sitting down for most of it and downing cups of tea through the day.

Once I got home from work I would take over from my wife and give him his bath and get him ready for bed. In the first month that tended to be quite late as he was still getting used to a regular bedtime so that meant I had to learn how to hold him in one arm whilst I ate my dinner with the other. Eventually though I was responsible for putting him to bed by giving him his bedtime bottle, burping him and letting him fall asleep on my shoulder before putting him in his moses basket.

Here's a little secret I'll let you in on: whilst letting him fall asleep on me I spent many a night playing through Angry Birds on my phone. I actually got quite far through the game in this fashion!

"What dear? No, I'm putting our son to bed, what did you think I was doing?"
Image courtesy of Echo Enduring Blog (

It Does Get Better

My last post may have left you thinking that raising a child is a hellish experience, and initially it is incredibly demanding. It still is but as 6 months roll on you realise that you can deal with things better now, that he also grows up more and changes with you and somehow you both meet in the middle to end up having fun together.

He seems a lot happier now :)