Writing Working Code at Hyperspeed

Image courtesy of Cycling Cosmonaut

Today I thought I would share with you something I had to do recently at work. To give some background I work for Talent Q who are a provider of online psychometric tests; I am one of the developers who works on all the various systems we have such as the candidate tests, the administration side and all sorts of backend services to keep everything working as it should.

In our last sprint we were working on adding a new suite of tests to our product list. In fact we’ve already got them running now as part of some trials before global release and have already witnessed tens of thousands of completions – they clearly work so that’s always a good start! Me and my colleagues did want to go back over the code though before the final release as developers always have that feeling of wanting to perfect their code; we also had a view of making sure long-term maintainability could be achieved.

So we looked through what we had already and found some areas that we wanted to improve. The main part was that the code was trying to be too clever for its own good:

1. Most classes were trying to be very generic but ended up not incorporating the specifics of each individual test, resulting in some code smells such as lots of object casting which was unnecessary.
2. We also had several God objects that tried to contain every conceivable detail you could imagine yet at the end of day even missing some vital details.
3. Due to all the various classes being very generic it was difficult to follow through the logic of the code. There was clearly a design in there to not have too many strong dependencies interlinked with each other – something that I believe is definitely worthwhile – but you could also get easily lost down a codepath; bear in mind that these tests were built up in an organic manner as most things are so the end result does tend to have more than you actually need (which I will discuss later on).

So I was the one tasked with implementing the changes we wanted. We decided that we would not make all these generic types which somehow had to be tied together, instead each test would have a dedicated class with a clear focus even if it meant some duplication of logic – as long as we have working code we would be able to compare the differences and refactor later if required.

All seems reasonable so far. Oh, did I mention that due to a number of planning issues I had to do all of this in a matter of days?

Umm, OK…

But I managed it, all within working hours (no overtime) and it all worked first time! Here’s how I did it…

Plan Your Work First

You might think that with a tight deadline in front of you that you should not waste any time and start writing code immediately, right?

Wrong!

You could do this but without a clear focus on what you are trying to achieve you’ll struggle to keep up with what you are doing; you don’t want to be thinking of what you need to do and how to translate that into code at the same time.

So first of all I spent a few hours going over what I needed to do. I looked at each of the new tests we were creating and wrote down notes to help me understand the high-level logic that drove them, such as:

• How do I load the questions?
• How do I keep track of state and where the candidate is in the test?
• What if the candidate quits a session and restarts from the last question they answered? Will we be able to know where to pick up the test from?
• How do I get a final result from the test?

Fortunately out of the three new test types we were creating two of them were similar to previous assessments we had already released so it wasn’t too hard to work out the details for them. By the end of this I had a one-page document of notes for each test which explained to me everything they needed to do. With this now written down and not cluttering my mind I could finally get down to writing the code.

Test Driven Development is Your Friend

Now I knew what to do I had to figure out how I was going to get my code written down and make sure it all worked quickly. The trial code we currently had was not designed to be testable; it was all written under the assumption that there would always be a database and always have a user interface driving it, meaning that testing that code was manual, laborious and the feedback loop was painfully slow.

I’m a big fan of Test Driven Development though; the idea of writing lots of little tests to quickly verify that your code is working is something I’ve jumped onto with gusto. In the end, although I don’t particularly like the idea of rewriting code from scratch, I decided that I was going to have to write the code for these assessments from first principles again and this time focus on making them work in an isolated environment meaning no database or website required.

Start Typing

Image originally from Tumblr

With all this in place it was time for me to start typing, and fast! One of the benefits of Test Driven Development is that because unit tests are so small you can clearly focus on one tiny bit at a time. For example if I wrote the constructor for the assessment then under the scenario of a brand new assessment being run I knew that the outcome of creating the object would mean:

1. The internal state would be set to “Running Test” instead of “Completed”
2. That the first question number would be one
3. That the total number of questions available would be a specific number
4. That the first question was loaded into the test
5. etc

So I simply wrote one test at a time, used the test runner to make sure it passed, and moved to the next one. I managed to keep up this pace and produce quite a number of test cases quickly, all the while improving the code and verifying that it all worked as I expected it to.

Getting “In the Zone”

My blog is called “In the Coding Zone” as it refers to that mythical place where developers love to roam and simply be lost in their own thoughts without any kind of interruption. When “in the zone” developers are able to churn out vast amounts of code and be hugely productive.

Getting there is a challenge in itself, especially when working in an open office environment. Fortunately I got very few distractions during this time so that I really could just ignore everything around me and focus purely on hitting this deadline.

The End Result

After a lot of hard work I hit that deadline but what else did I have to show for it?

1. With the code rewritten with testing principles in mind we now had our first ever set of assessments which we could run using test automation. This means we can now verify that these assessments work even without a real database and just using dummy data, or build upon the unit tests to create integration tests using a real bank of questions.
2. For the first time our assessments finally make sense; figuring out how our assessments worked used to be a daunting task but I realised this is because it was down to the sheer amount of extraneous code that was written, not due to the complexity of the logic itself. Test Driven Development is really good at focusing you on what you need to achieve to get your tests running; once all tests pass then you don’t need to write any more code so your final implementation tends to be very lean and ultimately simpler to understand.
3. One drawback though is that I was mentally tired after this exercise. Keeping up a relentless pace of writing code/tests almost non-stop for days does take it’s toll on you and you can’t keep it up forever.

Developing software seems to have this aurora of always being delayed and not working until you test it over and over again but I wanted to prove that it is possible to have something work first time (and on time!) as long as you plan your work effectively, focus on the job in hand and basically test as you go, not leaving it as an afterthought.

Unit Testing ASP.NET MVC Views

I have successfully jumped onto the Test Driven Development (TDD) bandwagon now and loving it. Over the last 6 months I’ve been trying to incorporate it wherever I can; my code at work is now starting to gain a number of unit tests within a codebase that initially I thought would be difficult to do automated testing on.

Recently I’ve started a personal project using ASP.NET MVC – I haven’t had much experience of MVC, only Web Forms, and wanted to expand my knowledge on .NET web development. I’ve heard many stories how ASP.NET MVC was designed from the ground up to be very easy to test so I jumped at the chance of reading up and experimenting with it. One bonus as well is that this would be all-new code; the one thing I’ve learned about TDD is that it works brilliantly with a clean-slate, with legacy code it requires quite a bit more work and slower iterations in order to not break what you’ve already got.

For most of the MVC framework I could clearly see how you could apply unit testing to your code as it focuses a lot on plain objects. One of the things I couldn’t initially get my head around though is testing controllers and verifying that views returned were correct.

Take this example of a simple controller:

public class AccountController : Controller
{
    public ActionResult Index(int id)
    {
        Account account = new Account()
        {
            Id = id,
            Name = "My Account"
        };
        return View("Index", account);
    }
}

It is actually quite easy to write a unit test for this controller and action as, thankfully, controllers are not strongly tied to anything to do with HTTP requests or responses, so testing becomes as simple as:

[Fact]
public void IndexActionReturnsView()
{
    AccountController controller = new AccountController();
    ActionResult result = controller.Index(42);
    Assert.NotNull(result);
}

But at this point I got a little stuck. How do I know that the controller action returned the view that I wanted? Initially I was thinking only in terms of controller actions returning HTML output; how can we verify HTML text output in automated tests?

The answer is you don’t, and actually the answer is a lot simpler and cleaner; we don’t verify the overall output, we simply verify that the view has the necessary information (i.e. model data) it needs to generate the HTML output (or whatever output format is required).

This can be split up into several parts:

Check the View Name

One test can be defined to determine that the correct view was returned by checking the name of the view, such as below:

[Fact]
public void IndexActionReturnsCorrectViewName()
{
    AccountController controller = new AccountController();
    
    // Cast the result of the action to a ViewResult, this will then provide
    // view information for the test to confirm
    ViewResult result = controller.Index(42) as ViewResult;
    
    Assert.Equal("Index", result.ViewName);
}

One caveat I’ve found with this though is that in the controller you must specifically request which view you want; letting the MVC framework determine it by convention – based on the action name – doesn’t seem to work for some reason.

Check the View Model Type

The next test you can run is to ensure that the view returned was supplied with the correct model type, like so:

[Fact]
public void IndexActionReturnsCorrectViewModelType()
{
    AccountController controller = new AccountController();    
    ViewResult result = controller.Index(42) as ViewResult;
    
    // Test that the model object passed to the view was the correct 
    // type
    Assert.IsType<Account>(result.Model);
}

Check the View Model Data

Finally you can then test that the view returned was supplied with the correct model data, like so:

[Fact]
public void IndexActionReturnsCorrectViewModelData()
{
    AccountController controller = new AccountController();
    ViewResult result = controller.Index(42) as ViewResult;
    
    // Cast the model in the view result to the correct type and
    // now we can test against it
    Account model = result.Model as Account;
    
    Assert.Equal(42, model.Id);
}

Conclusion

This concept, once understood, feels incredibly clean to me. If you think about it you don’t want to test how a web page looks because that could change over time thanks to re-designs, plus parsing such information would be incredibly painful. All you need to do is verify that the view was given enough details for it to carry out it’s job; can it display a title for the page, does it have the correct list of customer orders to render, etc. It has actually made me re-think some of my code designs to better match this concept.

I can see why developers love ASP.NET MVC compared to Web Forms now!

What I've Learned About Test Driven Development

At the end of April this year me and my colleagues prepared and deployed a big website/system update. A lot of things would change and get better and despite our rather lax testing procedures - testing being not very formal and done by us developers - we felt we had tested it as much as we possibly could and were very happy with what we did. We deployed it over a weekend with scheduled downtime, did our updates along with some server enhancements, and brought the site back up...

...Starting on the following Monday and continuing for the next two weeks we did nothing but patch holes with endless hotfixes and grovel to our customers promising that we will fix all the bugs they were reporting non-stop as soon as possible. One of our web services did not start up again for an entire weekend and our biggest client wanted to know why they couldn't access our systems. Users were complaining that the website kept kicking them out at random times which we eventually found out was being caused by a StackOverflow exception occurring every 30 minutes and crashing the IIS web process. It got to the point where the CEO got involved and at the eventual end of this nightmare my manager had to go and visit some key clients to personally assure them that all was well and this would never happen again.

We felt gutted. By our standards this was the most tested release we had ever done and yet it ended up being the worst one in our company's history.

This is all background to the real purpose of this post. When the dust settled we all sat down and discussed what went wrong and why and one of the key points we learned (though not the only one) was that our testing strategy was dire. In the absence of a real software tester (though we have one now) it was down to the people writing the code to do testing, and it tended to be a quick couple of run-throughs in certain areas to make sure things worked as we expected. This was not good enough and I simply felt that I couldn't trust our own system anymore.

A friend of mine talked to me about the benefits of Test Driven Development (TDD) a couple of weeks earlier and at this point I put it towards my team. I felt that, as well as having a proper software tester, we as developers needed to think about testing from the very start. In essence, we had to build quality into our code.

And so began my long journey into a development practice that I had never tried before and yet now I feel makes so much sense to me. Last week marked the very first unit and functional tests ever entered into our codebase and although they only cover a tiny fraction of our entire code I feel much more comfortable that that code works as it should, even before it goes to our tester to evaluate.

I will not go into the entire TDD process - there are enough resources on this already - but I thought as this was a learning process for me I would share with you what I've learned in the past few months.

Roy Osherove to the Rescue!

I did a lot of research into TDD, trawling the internet and reading through a book a colleague lent me but I still had many questions and things I was not sure about. What is the difference between a unit test and an integration test? What if your code has to read from a database or a file? What is a stub and what is a mock?

Then I listened to this Hanselminutes podcast of Scott Hanselman talking to Roy Osherove, author of The Art of Unit Testing, and all my questions were answered in 30 minutes. If anyone else is having trouble understanding unit testing and TDD I would very much recommend listening to this as it helped me a lot.

No-One Likes MSTest

Before you can write your tests though you need a unit testing framework to help you write your them, but which one do you choose?

Initially I started using MSTest as that came integrated with Visual Studio and I could experiment quickly with learning the TDD process, but in the end I saw some flaws with it - as do many others: 

1. The benefit of having MSTest integral to Visual Studio can also be it's downfall as you simply cannot run it standalone.
2. MSTest simply provided the bare minimum of features to make it worthy of calling it a unit testing framework. I learned that other unit testing frameworks were more capable and had far better features than MSTest, including assertion of exceptions and data-driven tests.
3. The last problem I had was more of a personal one in that the test runner window didn't seem to make it that clear to me that all the tests had passed or failed. Compare the screenshots below of the MSTest and the NUnit test runner:

An example of the MSTest runner. Notice that tiny little tick at the top of the window? That tells me that all the tests passed.

Courtesy of Robust Haven ( http://www.robusthaven.com/blog/linux-development/Unit-test-runner-in-bash-for-C-and-CPP-libraries)

Now compare that to the NUnit test runner. See that great big green line? Guess what that means?

Courtesy of litemedia ( http://litemedia.info/data-driven-test-cases-in-nunit)

So I tried out NUnit which I liked very much and seemed very dependable, and most recently xUnit.net which I finally settled on.

What I learned from this though is that, once you take MSTest out of the equation, any test framework is usually good enough and it simply comes down to personal preference. I went with xUnit.net because it provided several things I found useful straight out-of-the-box such as an MSBuild task to add to your project builds - if a test fails then the build fails - and it provides a XML stylesheet by default to create HTML reports rather than having to make one myself, a la NUnit.

Doing it Backwards

The very first stumbling block was that I had to mentally switch around my working style to effectively be back to front. Write tests before code? How was that meant to work, surely you need something there to try out? But as long as you follow the Red, Green, Refactor process it does actually work.

Even so, for a beginner like me it required a very large mental shift. For my very first unit test I must have sat there for a good half-hour thinking "How do I write this test? Where do I start?" Once you get past the first test - usually making sure something works - then the next test seems a bit more obvious: "What if I pass in the wrong input?" This could produce quite a few more unit tests so now the ball starts rolling. And once the first code function is tested you start to see a pattern so the next batch of tests seem a bit easier to write, and so on.

As long as you can start writing some unit tests to begin with, the process becomes a lot more intuitive up to the point where it seems like second nature and the way you were working before seems so arcane now.

Designing Your API

The other thing that struck me is that, despite the word "test" in Test Driven Development, initially it feels more like I'm writing a very detailed specification for all my objects. As I am building up each test it feels like I am defining what the code should be doing which to my mind is almost like a list of requirements - function A will allow a string to be passed in, but not an empty string or one that has more than 100 characters etc.

TDD has also made me think a lot more about object dependencies and code design. When I sat down and looked at my codebase as it stands now, many objects in the business logic layer call straight into the data-access layer and database to do something and these objects simply assume that that will just work - there has to be a database present to do anything in my system. For the real world that makes perfect sense as a system without a database is pretty much useless, but in a testing scenario it cannot be assumed that a database is there and for unit testing nothing should be touching external state.

This led me to wonder how I could actually test anything if all of my code is so inter-dependent on each other (or tightly coupled). The answer for TDD is to make your code loosely coupled instead which impacts your code design and makes you think a lot more about patterns and practices like Dependency Injection. As a result I'm starting to feel a lot happier with the overall design of my code as it has started to resemble smaller, Lego-sized blocks which can be combined together into bigger systems.

Start From Scratch

Following on from the previous section if you already have a large codebase and start to introduce TDD into it, like me you might find it a bit daunting. Where do you start? Do you retroactively add unit tests into existing code? And what about dependency problems you have?

So I learned that the best way to learn about TDD and introduce it into your working practices is to start on something brand new, a clean slate. That way you can build up your tests and knowledge as you go. And if you can't start from the very beginning? Then pick a new feature that you're due to develop and start thinking about how you would implement it in a TDD fashion. If only one part of your codebase has been written with testability in mind then that is better than nothing.

Test As Much As You Can

My final thought is that initially I thought that you would have to aim for 100% test coverage to make you feel that you had done a good job. While it is a target to aim for, you don't have to achieve it, or maybe it is just impossible to reach that goal for many reasons.

The very first feature I implemented into our product that was written with TDD in mind only has about 30 unit tests and that doesn't even cover the entire feature end-to-end, the reason being that existing code has dependencies that I cannot extricate just yet. But the new code I have written, that was unit tested. In all, the total test coverage is a tiny percentage but at least I can build upon it over time.

All in all I now think in a TDD style and find it difficult when I have to revert back to the old-fashioned cod-then-test mentality. It's only been a few months since I started investigating this process as a possibility but I'm glad that I did as I feel one step closer to being a better programmer.