The Urge to Rewrite Code

I recently read a fantastic article on Ars Technica about the inner workings of the new WinRT technology powering Windows 8. It’s a lengthy article but it explains how we got all the way from Win16 to Win32 to OLE to COM to .NET and eventually to WinRT and is well worth a read if you’re interested in Windows development and its history.

What got me thinking though is that it gave me a snapshot into Microsoft’s process of keeping Windows and its technology up-to-date yet also be able to support a vast legacy of applications that have been running sometimes for the past 20 – 30 years. What you realise is that to support that legacy and keep pushing forwards it basically built everything on top of one another. If you were to start from the latest WinRT API’s and strip away all the layers of strata you would eventually hit the ancient Win32 API and kernel, the very same system that has been powering Windows since Windows NT. The same can be said for .NET and COM; it doesn’t matter how new or fancy the latest technology trend is, eventually they all become nicer wrappers over what came previously.

This is interesting to me because I’ll bet every single developer out there has had one thought cross their minds at some point in their career: “I need to rewrite this”. Maybe you have a codebase written from another era or you’ve inherited code that looks like a monkey tap danced on a keyboard, sometimes we all fall prey to thinking that we need to invest time in rewriting huge chunks of code (or entire systems) because, obviously, we can do a better job and that the end result will be “better”.

This is dangerous!

I can speak from experience plus cite any number of references saying that taking on a rewrite of a codebase, especially an enterprise/commercial one, is just a bad idea. What you are effectively suggesting is to take a (mostly) working system, spend 6 – 12 months (maybe more) ditching it and starting all over again, and end up with the exact same system you started with plus countless additional bugs you’ve introduced due to human error and/or lack of domain knowledge. Also, all that time you wasted rewriting code meant that you couldn’t do any current development work, meaning your competitors have now charged ahead of you with brand new features that you will never be able to keep up with.

All of this so that your code looks better, a matter that no-one else but you cares about.

Now I am being a bit extreme here. Of course there are times when you have to rewrite something in order to progress forward. Maybe your code is so stuck in the dark ages that adding new features becomes increasingly time consuming or complex, maybe impossible. Yet I’ve learned over the years that you simply cannot ditch what you already have; your customers don’t give two hoots how you managed to kludge together their feature, the fact remains that it was done and it works so breaking it now is not an option.

So what can be done to refactor code effectively? Below are some ideas that I thought of and some of which I even try and implement myself.

Do Nothing

By far the simplest strategy as this requires no work at all! Simply learn to live with your codebase, quality be damned.  If you can overcome your initial feelings of revulsion at the spaghetti code you see daily and just accept it for what it is you might overcome some hurdles.

Of course doing nothing also means making no improvements so it’s quite a trade-off, but as the old saying goes “if it ain’t broke, don’t fix it”.

The Big Bang Strategy

Image courtesy of http://memegenerator.net/

Or the polar opposite of doing nothing is doing everything in one go, but as I’ve already said this is very extreme and hardly ever needed as there are better ways of improving your codebase without greatly affecting anything else.

The Microsoft/Onion Strategy

What I’ve seen Microsoft tend to do is build layers upon all their existing technologies so that the next layer up has a better API than the one below and each new layer will handle the fiddly, lower level details so you don’t have to.

For example, consider when .NET first came into existence which introduced Windows Forms. This was meant to replicate the drag-and-drop style of development that Visual Basic programmers have long been used to. But do you think that all the framework classes designed to handle windows and controls were written from scratch for a brand new, untested technology? No, Windows Forms was simply an easier to use wrapper over the interop’ed Win32 code because it already worked; why re-invent the wheel?

Of course once you’ve introduced these layers and made sure they are working effectively you could start to clean up the lower layers or possibly even remove and replace them so that you don’t need so much API coverage; that is assuming of course you can remove all the dependencies on low level code.

The Side-by-Side Strategy

This is a refactoring strategy I tend to use myself. Let’s say you have a feature that, for whatever reason, you are going to rewrite. What I do is actually never touch the old code and instead create a separate layer of classes alongside the existing code to replicate the same functionality but written differently; usually I separate these classes with appropriate namespaces.

Now I can work on the new code whilst the old code can still be deployed if necessary and also not affect my other team member’s builds. Eventually I will have fleshed out the rewritten code enough for calling sites to start using the new code which will then phase out the old. Once all references to the old code have been removed, you can safely delete the old code from your codebase. This might take quite a while to achieve fully but it is certainly a lot safer than starting from a blank canvas with nothing to show for a long time.

I also use this strategy with the ObsoleteAttribute to make it clear that code is old and should no longer be used; it also helps find all the references to the old code by giving me compiler warnings that I can work my way through.

The Inheritance Strategy

As an alternative to having old and new code side-by-side, you could also implement it top-to-bottom within an inheritance hierarchy. The new code would be contained in a base class while the old code would derive from the new and still keep it’s existing API. This means that legacy code could in theory be passed into functions which require the new class and it would still work.

Conclusion

There are many alternatives to refactoring code in one large chunk – I’m sure there are also many other strategies thought up by people far cleverer and more experienced by me. Essentially I have learned from my career that the “big bang” approach never works out well and a more long-term, slower strategy usually gives the best results.

What I've Learned About Test Driven Development

At the end of April this year me and my colleagues prepared and deployed a big website/system update. A lot of things would change and get better and despite our rather lax testing procedures - testing being not very formal and done by us developers - we felt we had tested it as much as we possibly could and were very happy with what we did. We deployed it over a weekend with scheduled downtime, did our updates along with some server enhancements, and brought the site back up...

...Starting on the following Monday and continuing for the next two weeks we did nothing but patch holes with endless hotfixes and grovel to our customers promising that we will fix all the bugs they were reporting non-stop as soon as possible. One of our web services did not start up again for an entire weekend and our biggest client wanted to know why they couldn't access our systems. Users were complaining that the website kept kicking them out at random times which we eventually found out was being caused by a StackOverflow exception occurring every 30 minutes and crashing the IIS web process. It got to the point where the CEO got involved and at the eventual end of this nightmare my manager had to go and visit some key clients to personally assure them that all was well and this would never happen again.

We felt gutted. By our standards this was the most tested release we had ever done and yet it ended up being the worst one in our company's history.

This is all background to the real purpose of this post. When the dust settled we all sat down and discussed what went wrong and why and one of the key points we learned (though not the only one) was that our testing strategy was dire. In the absence of a real software tester (though we have one now) it was down to the people writing the code to do testing, and it tended to be a quick couple of run-throughs in certain areas to make sure things worked as we expected. This was not good enough and I simply felt that I couldn't trust our own system anymore.

A friend of mine talked to me about the benefits of Test Driven Development (TDD) a couple of weeks earlier and at this point I put it towards my team. I felt that, as well as having a proper software tester, we as developers needed to think about testing from the very start. In essence, we had to build quality into our code.

And so began my long journey into a development practice that I had never tried before and yet now I feel makes so much sense to me. Last week marked the very first unit and functional tests ever entered into our codebase and although they only cover a tiny fraction of our entire code I feel much more comfortable that that code works as it should, even before it goes to our tester to evaluate.

I will not go into the entire TDD process - there are enough resources on this already - but I thought as this was a learning process for me I would share with you what I've learned in the past few months.

Roy Osherove to the Rescue!

I did a lot of research into TDD, trawling the internet and reading through a book a colleague lent me but I still had many questions and things I was not sure about. What is the difference between a unit test and an integration test? What if your code has to read from a database or a file? What is a stub and what is a mock?

Then I listened to this Hanselminutes podcast of Scott Hanselman talking to Roy Osherove, author of The Art of Unit Testing, and all my questions were answered in 30 minutes. If anyone else is having trouble understanding unit testing and TDD I would very much recommend listening to this as it helped me a lot.

No-One Likes MSTest

Before you can write your tests though you need a unit testing framework to help you write your them, but which one do you choose?

Initially I started using MSTest as that came integrated with Visual Studio and I could experiment quickly with learning the TDD process, but in the end I saw some flaws with it - as do many others: 

1. The benefit of having MSTest integral to Visual Studio can also be it's downfall as you simply cannot run it standalone.
2. MSTest simply provided the bare minimum of features to make it worthy of calling it a unit testing framework. I learned that other unit testing frameworks were more capable and had far better features than MSTest, including assertion of exceptions and data-driven tests.
3. The last problem I had was more of a personal one in that the test runner window didn't seem to make it that clear to me that all the tests had passed or failed. Compare the screenshots below of the MSTest and the NUnit test runner:

An example of the MSTest runner. Notice that tiny little tick at the top of the window? That tells me that all the tests passed.

Courtesy of Robust Haven ( http://www.robusthaven.com/blog/linux-development/Unit-test-runner-in-bash-for-C-and-CPP-libraries)

Now compare that to the NUnit test runner. See that great big green line? Guess what that means?

Courtesy of litemedia ( http://litemedia.info/data-driven-test-cases-in-nunit)

So I tried out NUnit which I liked very much and seemed very dependable, and most recently xUnit.net which I finally settled on.

What I learned from this though is that, once you take MSTest out of the equation, any test framework is usually good enough and it simply comes down to personal preference. I went with xUnit.net because it provided several things I found useful straight out-of-the-box such as an MSBuild task to add to your project builds - if a test fails then the build fails - and it provides a XML stylesheet by default to create HTML reports rather than having to make one myself, a la NUnit.

Doing it Backwards

The very first stumbling block was that I had to mentally switch around my working style to effectively be back to front. Write tests before code? How was that meant to work, surely you need something there to try out? But as long as you follow the Red, Green, Refactor process it does actually work.

Even so, for a beginner like me it required a very large mental shift. For my very first unit test I must have sat there for a good half-hour thinking "How do I write this test? Where do I start?" Once you get past the first test - usually making sure something works - then the next test seems a bit more obvious: "What if I pass in the wrong input?" This could produce quite a few more unit tests so now the ball starts rolling. And once the first code function is tested you start to see a pattern so the next batch of tests seem a bit easier to write, and so on.

As long as you can start writing some unit tests to begin with, the process becomes a lot more intuitive up to the point where it seems like second nature and the way you were working before seems so arcane now.

Designing Your API

The other thing that struck me is that, despite the word "test" in Test Driven Development, initially it feels more like I'm writing a very detailed specification for all my objects. As I am building up each test it feels like I am defining what the code should be doing which to my mind is almost like a list of requirements - function A will allow a string to be passed in, but not an empty string or one that has more than 100 characters etc.

TDD has also made me think a lot more about object dependencies and code design. When I sat down and looked at my codebase as it stands now, many objects in the business logic layer call straight into the data-access layer and database to do something and these objects simply assume that that will just work - there has to be a database present to do anything in my system. For the real world that makes perfect sense as a system without a database is pretty much useless, but in a testing scenario it cannot be assumed that a database is there and for unit testing nothing should be touching external state.

This led me to wonder how I could actually test anything if all of my code is so inter-dependent on each other (or tightly coupled). The answer for TDD is to make your code loosely coupled instead which impacts your code design and makes you think a lot more about patterns and practices like Dependency Injection. As a result I'm starting to feel a lot happier with the overall design of my code as it has started to resemble smaller, Lego-sized blocks which can be combined together into bigger systems.

Start From Scratch

Following on from the previous section if you already have a large codebase and start to introduce TDD into it, like me you might find it a bit daunting. Where do you start? Do you retroactively add unit tests into existing code? And what about dependency problems you have?

So I learned that the best way to learn about TDD and introduce it into your working practices is to start on something brand new, a clean slate. That way you can build up your tests and knowledge as you go. And if you can't start from the very beginning? Then pick a new feature that you're due to develop and start thinking about how you would implement it in a TDD fashion. If only one part of your codebase has been written with testability in mind then that is better than nothing.

Test As Much As You Can

My final thought is that initially I thought that you would have to aim for 100% test coverage to make you feel that you had done a good job. While it is a target to aim for, you don't have to achieve it, or maybe it is just impossible to reach that goal for many reasons.

The very first feature I implemented into our product that was written with TDD in mind only has about 30 unit tests and that doesn't even cover the entire feature end-to-end, the reason being that existing code has dependencies that I cannot extricate just yet. But the new code I have written, that was unit tested. In all, the total test coverage is a tiny percentage but at least I can build upon it over time.

All in all I now think in a TDD style and find it difficult when I have to revert back to the old-fashioned cod-then-test mentality. It's only been a few months since I started investigating this process as a possibility but I'm glad that I did as I feel one step closer to being a better programmer.

Is It Really That Complicated?

I've recently been reading Fabien Sanglard's blog, particularly his fascinating code reviews on the Doom and Quake source engines. Apart from making me feel a bit stupid not understanding how BSP trees work, something else strikes me regarding my own work:

What I do isn't really that complicated.

At the heart of my job I write code to make a website work, and if I consider the bare essential tasks that it needs to do I think I'd have the following shortlist:

1. Get data from a database
2. Do something to the data
3. Put that data into a webpage and send back to the client
4. Profit

By now it's safe to assume we've figured out how to do all of these steps easily - except perhaps the last one! There are a ton of ways to read and write to a database from code, either using ADO.NET or a variety of ORM frameworks, just as many ways to visualise that data (WebForms, MVC, Silverlight etc), and the only real work you should do in theory is the logic that sits between those parts.

Yet if this is all I need to do, why does it still seem so complicated to get anything done? Why does it still take me ages to implement a new feature when "code-reuse" should be helping me? I mean, it's not like I'm writing a game engine with crazy 3D maths and physics to calculate in nanoseconds, attempting to cure cancer or even writing software for spacecraft. Apart from some technical challenges that do exist, I think it mainly boils down to two things.

Other People

Your customers usually don't tend to think in the same way as you do. They might say "hey, can you add in the ability to search our website like Google? In fact, can we not just use The Google on our database or something? And possibly email all the search results to us? It can't take that long, right?" I like to try and keep things as simple and generic as possible, yet find that when "other people" get involved it tends to spiral out of control.

Take, for example, the financial sub-system I inherited from my predecessor. At work we have a system which calculates how many units each of our clients uses based on what assessments have been completed and how many reports have been generated. These units are then used by account managers to translate into real money to calculate invoices, such as 1 unit = £10 for instance.

So this sounds simple, right? Surely it's just adding up numbers and computers love adding up numbers. Did I mention though that if component A was generated before component B then you only add units for B, but if C and B are combined into one then we halve the units of A. Oh, I forgot, if D is generated between B and A then we add C and E together but only cap to a maximum of 6 (which can be changed based on which country you are in) and then if you bring F into the mix... and this kind of logic goes on and on in an endless spiral of doom.

Why does the system work this way? Because that is how other people decided it should work, because apparently business rules should never be simple. It also didn't help that this evolved, as most products do, into a big mess which means that business rules end up circling around each other without you even being fully aware of it.

When I first started this job I had to spend 3 months understanding all the many intricacies of this system because only my predecessor knew how it truly worked - despite being told by the managers what they wanted. It has now gotten to the point where only I know how every item is charged for and even the management have to double check with me.

And to be fair, there is one other thing that tends to complicate matters just as much as other people...

Myself

I tend to get very attached to the code I write. In an ideal world I feel as though my source code should be beautiful to gaze upon, to be able to admire it's sheer simplicity whilst also marvelling at how it can handle so many complex tasks at once. Sometimes I look back on code I wrote maybe a month or two ago, which at the time I was perfectly happy with, yet now I feel it is ugly and horrible and what was I thinking? This simply has to go because now I know exactly how it should be written...

But this is the real world and experience has taught me that sometimes I have to just "get it done" no matter how it ends up. And if we're being honest who else apart from me really cares about what my code looks like? The other day I decided to bash-out three quick features for my website as I felt that customers really needed them; I created them as quickly as possible (since I do have other projects to complete) so the code wasn't pretty, sometimes even copied-and-pasted with some changes. I look back on it now and I don't like how it looks. Yet my customers won't care, because what they see are three new features that make their life easier and they work, which at the end of the day is the most important thing in your product.

Keep It Simple, Stupid

Simplicity is what I strive for in life. I genuinely feel that the simplest solution is always the best as there are fewer complications to consider and less chance of anything going wrong, and I really try to impress that upon people. Yet at the end of the day outside factors have a way of creeping in and adding complications to your best laid plans, and I've built up enough work experience to realise that although the simplest solution may be the best, it's not necessarily what people want.