Thoughts in the Wilderness

This is not the simplistic analogy you may be expecting.

Where I live in Minneapolis MN, we have a high profile bridge project going on right now, due to the tragic collapse of the previous structure. If you look at the project page on the state’s website, you can see the following “features of the new bridge”:

• 100-year life span
• 10 lanes of traffic, five in each direction—two lanes wider than the former bridge
• 189 feet wide—the previous bridge was 113 feet wide
• 13 foot wide right shoulders and 14 foot wide left shoulders, the previous bridge had no shoulders
• Light Rail Transport-ready which may help accommodate future transportation needs
• Design-build project complete in 437 days.
• Designed to be aesthetically pleasing and fit in with its environment

These are the most high-level (public) stakeholder values for the new bridge. From an engineer’s perspective, they are the constraints under which the bridge must be delivered. In addition, we see an architect’s rendition (picture) of what the new bridge will look like. This is also a constraint – an engineer cannot add things that will substantially change the appearance of the bridge.

Now, we can be sure that the design of the bridge was a collaborative effort of a team of people. Engineers, Marketing, Architects and the client. Is the design ongoing as they build the structure? Yes – they are using a growing technique known as Design-Build – they purposely start construction before the design is complete.

At first glance, design-build might sounds like a simple case of parallel work – one team is working on designing just-in-time, and another is working on the construction. In practice though, there is a collaborative environment that reportedly results in avoidance of disputes, faster project delivery, and less need for project management oversight.

The Analogy
So how is this similar to building software?

Firstly, the process of programming it is like the design of a bridge – it is the bringing together of people in different roles to creatively find ways to build the end result. Ideally, development involves a lot of Thinking, Talking, and Tweaking, just like a bridge design. In design, we often find that two heads are better than one. Pair-programming has been suggested as one way to do this in software development. Of course, we have many other collaborative techniques to communicate and discuss design.

Like a bridge design, the output of building software can be represented by piles of paper. The bridge has drawings, engineering specifications and requirements. A program has something better though – its code. (No, I’m not arguing that “the code is the design”. I’m just saying that the code “is a representation of the design”). The code accurately describes the parts of the design that it touches.

This “programming code = bridge design” point is key to what I’m trying to convey – the process of programming produces a design output, not a product. The final product is the result of implementing that design (just as the bridge itself is the result of implementing its design).

Specifically, the “building” is the deployment of software in its final environment. Deployments are where the “tires meet the road” – they are the intersection of the design with reality (just like construction). Mostly, the design holds up and does not need tweaking after deployment. Sometimes though, the harsh lights of reality expose the hidden flaws in the design. (In light of that, it is best to expose an application to its first deployment as soon as possible).

Some software groups have QA (quality) departments. Historically, these departments have taken the role of performing trial deployments – they will take the software, and expose it to a simulation of the real environment. Large construction projects also have this role – an independent group audits the designs, with the hope of spotting problems that would cause a problem when the construction occurs.

Finally, we find that the best way of constructing a large bridge project is to simultaneously design and build. The analogy for software is small frequent releases. Research and experience has shown this to be a good way deliver quality software that meets the requirements.

Conclusion
If we accept that building a bridge and building software are similar (they contain the same basic steps), then we can use that information to produce some interesting insights:

• That thing we need to do before developing is “architecture” - There is a fine distinction between architecture and design. The way I like to define it is that architecture describes the parts are visible from the outside, and design describes the inside. A bridge architect is able to construct a working model and rendition of the outside of a bridge without the full engineering specs. To do this, he needs to take into account all of the stakeholder values. Similarly, we need to be able to draw the edges of a software application before we start – we need to understand how the software will interact with the outside world, and how the outside world will interact with the software.
• QA is a misnomer – the primary purpose of a separate QA department should not be to assure quality. We can get quality in better ways than that. The purpose of the QA department should be to validate the design of the software, by simulating real environments. Many QA professionals already know this, of course.

This blog post is inspired by a set of three essays by Jack W. Reeves.

In any well designed software application, one of the fundamental principles we try to abide by is the open-closed principle. Basically, this means that we try and structure the code in such a way that adding new functionality does not require us to change existing code. That is, we strive to add code instead of changing code.

Following this principle dramatically improves quality. The reason is that changing code allows a risk of altering the meaning (semantics) of some part of that code. Because of that, whenever we change code, we run the risk of breaking code that depends on that code.

So how does that relate to fixing bugs? Fixing a bug almost always involves altering the semantics. The old meaning was wrong (buggy), so we need to fix it. This leads to some interesting paradoxes. First…

Fixing bugs is one of the most risky things you can do to a software application.

The better you have followed good design principles, the more this is true. Conversely, in less well designed systems, it is less true…

If you have a poorly designed system, then bug changes are not particularly risky.

This is because in a poorly designed system, all changes are equally risky.

You can mitigate the impact of bug-fixes through the following techniques:

• reduce dependencies – the less components that are linked to the code being changed, the less risky a change is.
• regression testing – A regression test will improve the likelihood of discovering breaking changes. Regression tests can take many forms – anything you run daily, (or at the same time as your builds) is a regression test. This includes NUnit-style unit tests, and FIT tests.
• Code reviews – part of a bug-fix code-review can be to review the impact of the change. Any tool that shows coupling (such as NDepends) can probably help with this.
• Impact analysis – Pre-identify parts of the system that others depend on heavily. Changes in these parts of the system are particularly risky and need careful attention.

Debugging. The process by which a programmer discovers how his program works.

I am a hyperproductive programmer. That means I can output 3-20 times the work that an average programmer can. (Arrogant? Maybe. Still true though).

One of the biggest reasons I am productive is that I make a habit of not “debugging”. This is my theory:

Debugging code is always wasted time

The only output of debugging is a greater understanding of the code. But there are better ways to understand code. I can read it. I can refactor it so as to make it easier to read. Leading to Corollary one:

Code reading and refactoring are more important skills than debugging.

Time spent debugging is not only wasted, it smells of poor code quality. If I lack understanding, that means that the code was too hard to read.

Programmers that do Test-Driven development understand this productivity boost. Once you write unit tests, you find that you no longer have to debug. Leading to Corollary two:

Improved up-front quality leads to less debugging.

Moving on. Maintainability. An ugly word. A nicer word is Soluble, (or grokkable). My productivity is far higher when I am dealing with code that I “grok”. Put me on a new project, and it will take me a while to come up to full speed. Much of that time will be debugging, refactoring, or reading code. Leading to Corollary three:

Solubility of code has a direct impact on time spent debugging.

(That is so obvious that it may be a truism).

In summary – debugging is a symptom caused by underlying causes of poor code quality, poor programmer skills, and code that is hard to read.

If you notice yourself, or other programmers debugging code, then ask yourself – which combination of the above is a problem? Then fix it.

Lean principles teach us to recognize a constraint in the system, and “elevate” (fix) it. I was reading what Amit Rathore was writing about this, and it got me to thinking…

My current work environment is a going-out-of-business software company. Our parent company is continuing in the same market, but they will be using a different piece of software than the one we created. (Due to acquisitions, there were 2 parts of the company creating the same kind of product).

The way our business unit worked is that we would get a sales order for a new deployment of our software. The client would specify their requirements in some sort of RFP. We would then spend time developing the missing pieces (required functionality that was not yet present), convert their existing data to our own format, train the users, and deploy.

In doing this, it appeared to me that Testing and Inventory (time between completing work and Deploying it) were large constraints. In other words, we would build a lot of software, but no-one would look closely at it until we deployed it months (up to a year) later.

It was a decision of the business to expect high quality work, but acceptance-testing that same work was so unimportant that not a single person was assigned full-time to it. Even traditional regression, or smoke testing was not prioritized. To the test-infected, this sounds a little crazy!

But it wasn’t. Because of the RFP (contract-driven) style of the client relationship, there was simply no justification for spending more on acceptance testing, (we did not have sufficient real client input to know for sure that we were building what they wanted). If it meet the letter of the RFP, then it was ok.

During big-bang style deployments, we would run around and fix the high-priority issues (mixture of bugs and changed requirements) until the product was working to the satisfaction of the client. Without built in development quality and Agile responsiveness, this is a nightmare. With those things in place, it is just highly stressful.

The extended deployment periods were our acceptance tests. They were also when we found the most about *actual* client requirements. Ultimately (with exceptions) the client was satisfied (but not necessarily happy). And the Product Manager made sure to build more realistic requirements into the next version.

Back to the failure of the business…

In this model, the satisfaction of the next client is directly proportional to how closely their RFP matched a previous one. This was our downfall, because we came into an expansionist period where each new client was a completely new RFP. We were breaking into new product areas and new geographic areas. And our unit was expensive (because we were developing large amounts of good quality, well designed, extensible software).

In this expansionist period we had great difficulty in creating happy clients, because they would each get the worst possible result – an initial, painful deployment where the best possible outcome was meeting the “letter” of the requirements. (Actually, we did a little better than that, but only through a lot of good, dedicated people doing heroic things).

We tried to slow down the expansion but conditions (new-sales-driven upper management) would not allow this. Although they did not think of it in those terms, management did try to elevate the testing/deployment constraint, by working more closely with new clients. This was done by increasing the size of the client-relationship staff.

In the end, we had some successes (and some failures), and the parent company eventually came to the decision to end our unit. This was not a direct result of our failures, but I am sure that had we been more successful, it would have gone down differently.

I have been struggling to try and convey to our team what the difference is between an Acceptance Test and a Component Test.

This morning in the shower, I had some insight into this. Why not define testing by its primary purpose? Seems obvious, but I have not seen it stated this way before. So, here goes:

• Acceptance Testing – ensure that the software meets the business requirement
• Component Testing – ensure that the component does not regress
• Unit Testing (in the Agile meaning) – drive the design of the code

Looking at it this way, it definitely matters whether it is a component test or an acceptance test, because they meet different needs.

As a side-note, I have found it difficult to perform acceptance testing unless unit testing has also been employed. That is because unit testing tends to create those testable-hooks that let us attach the acceptance test to the system with minimal fuss.