Writing Working Code at Hyperspeed

Image courtesy of Cycling Cosmonaut

Today I thought I would share with you something I had to do recently at work. To give some background I work for Talent Q who are a provider of online psychometric tests; I am one of the developers who works on all the various systems we have such as the candidate tests, the administration side and all sorts of backend services to keep everything working as it should.

In our last sprint we were working on adding a new suite of tests to our product list. In fact we’ve already got them running now as part of some trials before global release and have already witnessed tens of thousands of completions – they clearly work so that’s always a good start! Me and my colleagues did want to go back over the code though before the final release as developers always have that feeling of wanting to perfect their code; we also had a view of making sure long-term maintainability could be achieved.

So we looked through what we had already and found some areas that we wanted to improve. The main part was that the code was trying to be too clever for its own good:

1. Most classes were trying to be very generic but ended up not incorporating the specifics of each individual test, resulting in some code smells such as lots of object casting which was unnecessary.
2. We also had several God objects that tried to contain every conceivable detail you could imagine yet at the end of day even missing some vital details.
3. Due to all the various classes being very generic it was difficult to follow through the logic of the code. There was clearly a design in there to not have too many strong dependencies interlinked with each other – something that I believe is definitely worthwhile – but you could also get easily lost down a codepath; bear in mind that these tests were built up in an organic manner as most things are so the end result does tend to have more than you actually need (which I will discuss later on).

So I was the one tasked with implementing the changes we wanted. We decided that we would not make all these generic types which somehow had to be tied together, instead each test would have a dedicated class with a clear focus even if it meant some duplication of logic – as long as we have working code we would be able to compare the differences and refactor later if required.

All seems reasonable so far. Oh, did I mention that due to a number of planning issues I had to do all of this in a matter of days?

Umm, OK…

But I managed it, all within working hours (no overtime) and it all worked first time! Here’s how I did it…

Plan Your Work First

You might think that with a tight deadline in front of you that you should not waste any time and start writing code immediately, right?

Wrong!

You could do this but without a clear focus on what you are trying to achieve you’ll struggle to keep up with what you are doing; you don’t want to be thinking of what you need to do and how to translate that into code at the same time.

So first of all I spent a few hours going over what I needed to do. I looked at each of the new tests we were creating and wrote down notes to help me understand the high-level logic that drove them, such as:

• How do I load the questions?
• How do I keep track of state and where the candidate is in the test?
• What if the candidate quits a session and restarts from the last question they answered? Will we be able to know where to pick up the test from?
• How do I get a final result from the test?

Fortunately out of the three new test types we were creating two of them were similar to previous assessments we had already released so it wasn’t too hard to work out the details for them. By the end of this I had a one-page document of notes for each test which explained to me everything they needed to do. With this now written down and not cluttering my mind I could finally get down to writing the code.

Test Driven Development is Your Friend

Now I knew what to do I had to figure out how I was going to get my code written down and make sure it all worked quickly. The trial code we currently had was not designed to be testable; it was all written under the assumption that there would always be a database and always have a user interface driving it, meaning that testing that code was manual, laborious and the feedback loop was painfully slow.

I’m a big fan of Test Driven Development though; the idea of writing lots of little tests to quickly verify that your code is working is something I’ve jumped onto with gusto. In the end, although I don’t particularly like the idea of rewriting code from scratch, I decided that I was going to have to write the code for these assessments from first principles again and this time focus on making them work in an isolated environment meaning no database or website required.

Start Typing

Image originally from Tumblr

With all this in place it was time for me to start typing, and fast! One of the benefits of Test Driven Development is that because unit tests are so small you can clearly focus on one tiny bit at a time. For example if I wrote the constructor for the assessment then under the scenario of a brand new assessment being run I knew that the outcome of creating the object would mean:

1. The internal state would be set to “Running Test” instead of “Completed”
2. That the first question number would be one
3. That the total number of questions available would be a specific number
4. That the first question was loaded into the test
5. etc

So I simply wrote one test at a time, used the test runner to make sure it passed, and moved to the next one. I managed to keep up this pace and produce quite a number of test cases quickly, all the while improving the code and verifying that it all worked as I expected it to.

Getting “In the Zone”

My blog is called “In the Coding Zone” as it refers to that mythical place where developers love to roam and simply be lost in their own thoughts without any kind of interruption. When “in the zone” developers are able to churn out vast amounts of code and be hugely productive.

Getting there is a challenge in itself, especially when working in an open office environment. Fortunately I got very few distractions during this time so that I really could just ignore everything around me and focus purely on hitting this deadline.

The End Result

After a lot of hard work I hit that deadline but what else did I have to show for it?

1. With the code rewritten with testing principles in mind we now had our first ever set of assessments which we could run using test automation. This means we can now verify that these assessments work even without a real database and just using dummy data, or build upon the unit tests to create integration tests using a real bank of questions.
2. For the first time our assessments finally make sense; figuring out how our assessments worked used to be a daunting task but I realised this is because it was down to the sheer amount of extraneous code that was written, not due to the complexity of the logic itself. Test Driven Development is really good at focusing you on what you need to achieve to get your tests running; once all tests pass then you don’t need to write any more code so your final implementation tends to be very lean and ultimately simpler to understand.
3. One drawback though is that I was mentally tired after this exercise. Keeping up a relentless pace of writing code/tests almost non-stop for days does take it’s toll on you and you can’t keep it up forever.

Developing software seems to have this aurora of always being delayed and not working until you test it over and over again but I wanted to prove that it is possible to have something work first time (and on time!) as long as you plan your work effectively, focus on the job in hand and basically test as you go, not leaving it as an afterthought.

The Urge to Rewrite Code

I recently read a fantastic article on Ars Technica about the inner workings of the new WinRT technology powering Windows 8. It’s a lengthy article but it explains how we got all the way from Win16 to Win32 to OLE to COM to .NET and eventually to WinRT and is well worth a read if you’re interested in Windows development and its history.

What got me thinking though is that it gave me a snapshot into Microsoft’s process of keeping Windows and its technology up-to-date yet also be able to support a vast legacy of applications that have been running sometimes for the past 20 – 30 years. What you realise is that to support that legacy and keep pushing forwards it basically built everything on top of one another. If you were to start from the latest WinRT API’s and strip away all the layers of strata you would eventually hit the ancient Win32 API and kernel, the very same system that has been powering Windows since Windows NT. The same can be said for .NET and COM; it doesn’t matter how new or fancy the latest technology trend is, eventually they all become nicer wrappers over what came previously.

This is interesting to me because I’ll bet every single developer out there has had one thought cross their minds at some point in their career: “I need to rewrite this”. Maybe you have a codebase written from another era or you’ve inherited code that looks like a monkey tap danced on a keyboard, sometimes we all fall prey to thinking that we need to invest time in rewriting huge chunks of code (or entire systems) because, obviously, we can do a better job and that the end result will be “better”.

This is dangerous!

I can speak from experience plus cite any number of references saying that taking on a rewrite of a codebase, especially an enterprise/commercial one, is just a bad idea. What you are effectively suggesting is to take a (mostly) working system, spend 6 – 12 months (maybe more) ditching it and starting all over again, and end up with the exact same system you started with plus countless additional bugs you’ve introduced due to human error and/or lack of domain knowledge. Also, all that time you wasted rewriting code meant that you couldn’t do any current development work, meaning your competitors have now charged ahead of you with brand new features that you will never be able to keep up with.

All of this so that your code looks better, a matter that no-one else but you cares about.

Now I am being a bit extreme here. Of course there are times when you have to rewrite something in order to progress forward. Maybe your code is so stuck in the dark ages that adding new features becomes increasingly time consuming or complex, maybe impossible. Yet I’ve learned over the years that you simply cannot ditch what you already have; your customers don’t give two hoots how you managed to kludge together their feature, the fact remains that it was done and it works so breaking it now is not an option.

So what can be done to refactor code effectively? Below are some ideas that I thought of and some of which I even try and implement myself.

Do Nothing

By far the simplest strategy as this requires no work at all! Simply learn to live with your codebase, quality be damned.  If you can overcome your initial feelings of revulsion at the spaghetti code you see daily and just accept it for what it is you might overcome some hurdles.

Of course doing nothing also means making no improvements so it’s quite a trade-off, but as the old saying goes “if it ain’t broke, don’t fix it”.

The Big Bang Strategy

Image courtesy of http://memegenerator.net/

Or the polar opposite of doing nothing is doing everything in one go, but as I’ve already said this is very extreme and hardly ever needed as there are better ways of improving your codebase without greatly affecting anything else.

The Microsoft/Onion Strategy

What I’ve seen Microsoft tend to do is build layers upon all their existing technologies so that the next layer up has a better API than the one below and each new layer will handle the fiddly, lower level details so you don’t have to.

For example, consider when .NET first came into existence which introduced Windows Forms. This was meant to replicate the drag-and-drop style of development that Visual Basic programmers have long been used to. But do you think that all the framework classes designed to handle windows and controls were written from scratch for a brand new, untested technology? No, Windows Forms was simply an easier to use wrapper over the interop’ed Win32 code because it already worked; why re-invent the wheel?

Of course once you’ve introduced these layers and made sure they are working effectively you could start to clean up the lower layers or possibly even remove and replace them so that you don’t need so much API coverage; that is assuming of course you can remove all the dependencies on low level code.

The Side-by-Side Strategy

This is a refactoring strategy I tend to use myself. Let’s say you have a feature that, for whatever reason, you are going to rewrite. What I do is actually never touch the old code and instead create a separate layer of classes alongside the existing code to replicate the same functionality but written differently; usually I separate these classes with appropriate namespaces.

Now I can work on the new code whilst the old code can still be deployed if necessary and also not affect my other team member’s builds. Eventually I will have fleshed out the rewritten code enough for calling sites to start using the new code which will then phase out the old. Once all references to the old code have been removed, you can safely delete the old code from your codebase. This might take quite a while to achieve fully but it is certainly a lot safer than starting from a blank canvas with nothing to show for a long time.

I also use this strategy with the ObsoleteAttribute to make it clear that code is old and should no longer be used; it also helps find all the references to the old code by giving me compiler warnings that I can work my way through.

The Inheritance Strategy

As an alternative to having old and new code side-by-side, you could also implement it top-to-bottom within an inheritance hierarchy. The new code would be contained in a base class while the old code would derive from the new and still keep it’s existing API. This means that legacy code could in theory be passed into functions which require the new class and it would still work.

Conclusion

There are many alternatives to refactoring code in one large chunk – I’m sure there are also many other strategies thought up by people far cleverer and more experienced by me. Essentially I have learned from my career that the “big bang” approach never works out well and a more long-term, slower strategy usually gives the best results.

Is It Really That Complicated?

I've recently been reading Fabien Sanglard's blog, particularly his fascinating code reviews on the Doom and Quake source engines. Apart from making me feel a bit stupid not understanding how BSP trees work, something else strikes me regarding my own work:

What I do isn't really that complicated.

At the heart of my job I write code to make a website work, and if I consider the bare essential tasks that it needs to do I think I'd have the following shortlist:

1. Get data from a database
2. Do something to the data
3. Put that data into a webpage and send back to the client
4. Profit

By now it's safe to assume we've figured out how to do all of these steps easily - except perhaps the last one! There are a ton of ways to read and write to a database from code, either using ADO.NET or a variety of ORM frameworks, just as many ways to visualise that data (WebForms, MVC, Silverlight etc), and the only real work you should do in theory is the logic that sits between those parts.

Yet if this is all I need to do, why does it still seem so complicated to get anything done? Why does it still take me ages to implement a new feature when "code-reuse" should be helping me? I mean, it's not like I'm writing a game engine with crazy 3D maths and physics to calculate in nanoseconds, attempting to cure cancer or even writing software for spacecraft. Apart from some technical challenges that do exist, I think it mainly boils down to two things.

Other People

Your customers usually don't tend to think in the same way as you do. They might say "hey, can you add in the ability to search our website like Google? In fact, can we not just use The Google on our database or something? And possibly email all the search results to us? It can't take that long, right?" I like to try and keep things as simple and generic as possible, yet find that when "other people" get involved it tends to spiral out of control.

Take, for example, the financial sub-system I inherited from my predecessor. At work we have a system which calculates how many units each of our clients uses based on what assessments have been completed and how many reports have been generated. These units are then used by account managers to translate into real money to calculate invoices, such as 1 unit = £10 for instance.

So this sounds simple, right? Surely it's just adding up numbers and computers love adding up numbers. Did I mention though that if component A was generated before component B then you only add units for B, but if C and B are combined into one then we halve the units of A. Oh, I forgot, if D is generated between B and A then we add C and E together but only cap to a maximum of 6 (which can be changed based on which country you are in) and then if you bring F into the mix... and this kind of logic goes on and on in an endless spiral of doom.

Why does the system work this way? Because that is how other people decided it should work, because apparently business rules should never be simple. It also didn't help that this evolved, as most products do, into a big mess which means that business rules end up circling around each other without you even being fully aware of it.

When I first started this job I had to spend 3 months understanding all the many intricacies of this system because only my predecessor knew how it truly worked - despite being told by the managers what they wanted. It has now gotten to the point where only I know how every item is charged for and even the management have to double check with me.

And to be fair, there is one other thing that tends to complicate matters just as much as other people...

Myself

I tend to get very attached to the code I write. In an ideal world I feel as though my source code should be beautiful to gaze upon, to be able to admire it's sheer simplicity whilst also marvelling at how it can handle so many complex tasks at once. Sometimes I look back on code I wrote maybe a month or two ago, which at the time I was perfectly happy with, yet now I feel it is ugly and horrible and what was I thinking? This simply has to go because now I know exactly how it should be written...

But this is the real world and experience has taught me that sometimes I have to just "get it done" no matter how it ends up. And if we're being honest who else apart from me really cares about what my code looks like? The other day I decided to bash-out three quick features for my website as I felt that customers really needed them; I created them as quickly as possible (since I do have other projects to complete) so the code wasn't pretty, sometimes even copied-and-pasted with some changes. I look back on it now and I don't like how it looks. Yet my customers won't care, because what they see are three new features that make their life easier and they work, which at the end of the day is the most important thing in your product.

Keep It Simple, Stupid

Simplicity is what I strive for in life. I genuinely feel that the simplest solution is always the best as there are fewer complications to consider and less chance of anything going wrong, and I really try to impress that upon people. Yet at the end of the day outside factors have a way of creeping in and adding complications to your best laid plans, and I've built up enough work experience to realise that although the simplest solution may be the best, it's not necessarily what people want.