Thoughts in the Wilderness

When I was a junior-level programmer, I felt like I had one responsibility – be good at programming.

Later in my career, once I became a lead developer, I learned that I now had another responsibility – deliver a project on time.

Still later, as a software architect I picked up another responsibility – to make it easier for the team of developers to be successful.

Today, I recognize another responsibility – deliver value to the business.

In the above chart, blue represents focus on Self, red is focus on the Team, and yellow is focus on the Business.

What does your chart look like?

This is not the simplistic analogy you may be expecting.

Where I live in Minneapolis MN, we have a high profile bridge project going on right now, due to the tragic collapse of the previous structure. If you look at the project page on the state’s website, you can see the following “features of the new bridge”:

• 100-year life span
• 10 lanes of traffic, five in each direction—two lanes wider than the former bridge
• 189 feet wide—the previous bridge was 113 feet wide
• 13 foot wide right shoulders and 14 foot wide left shoulders, the previous bridge had no shoulders
• Light Rail Transport-ready which may help accommodate future transportation needs
• Design-build project complete in 437 days.
• Designed to be aesthetically pleasing and fit in with its environment

These are the most high-level (public) stakeholder values for the new bridge. From an engineer’s perspective, they are the constraints under which the bridge must be delivered. In addition, we see an architect’s rendition (picture) of what the new bridge will look like. This is also a constraint – an engineer cannot add things that will substantially change the appearance of the bridge.

Now, we can be sure that the design of the bridge was a collaborative effort of a team of people. Engineers, Marketing, Architects and the client. Is the design ongoing as they build the structure? Yes – they are using a growing technique known as Design-Build – they purposely start construction before the design is complete.

At first glance, design-build might sounds like a simple case of parallel work – one team is working on designing just-in-time, and another is working on the construction. In practice though, there is a collaborative environment that reportedly results in avoidance of disputes, faster project delivery, and less need for project management oversight.

The Analogy
So how is this similar to building software?

Firstly, the process of programming it is like the design of a bridge – it is the bringing together of people in different roles to creatively find ways to build the end result. Ideally, development involves a lot of Thinking, Talking, and Tweaking, just like a bridge design. In design, we often find that two heads are better than one. Pair-programming has been suggested as one way to do this in software development. Of course, we have many other collaborative techniques to communicate and discuss design.

Like a bridge design, the output of building software can be represented by piles of paper. The bridge has drawings, engineering specifications and requirements. A program has something better though – its code. (No, I’m not arguing that “the code is the design”. I’m just saying that the code “is a representation of the design”). The code accurately describes the parts of the design that it touches.

This “programming code = bridge design” point is key to what I’m trying to convey – the process of programming produces a design output, not a product. The final product is the result of implementing that design (just as the bridge itself is the result of implementing its design).

Specifically, the “building” is the deployment of software in its final environment. Deployments are where the “tires meet the road” – they are the intersection of the design with reality (just like construction). Mostly, the design holds up and does not need tweaking after deployment. Sometimes though, the harsh lights of reality expose the hidden flaws in the design. (In light of that, it is best to expose an application to its first deployment as soon as possible).

Some software groups have QA (quality) departments. Historically, these departments have taken the role of performing trial deployments – they will take the software, and expose it to a simulation of the real environment. Large construction projects also have this role – an independent group audits the designs, with the hope of spotting problems that would cause a problem when the construction occurs.

Finally, we find that the best way of constructing a large bridge project is to simultaneously design and build. The analogy for software is small frequent releases. Research and experience has shown this to be a good way deliver quality software that meets the requirements.

Conclusion
If we accept that building a bridge and building software are similar (they contain the same basic steps), then we can use that information to produce some interesting insights:

• That thing we need to do before developing is “architecture” - There is a fine distinction between architecture and design. The way I like to define it is that architecture describes the parts are visible from the outside, and design describes the inside. A bridge architect is able to construct a working model and rendition of the outside of a bridge without the full engineering specs. To do this, he needs to take into account all of the stakeholder values. Similarly, we need to be able to draw the edges of a software application before we start – we need to understand how the software will interact with the outside world, and how the outside world will interact with the software.
• QA is a misnomer – the primary purpose of a separate QA department should not be to assure quality. We can get quality in better ways than that. The purpose of the QA department should be to validate the design of the software, by simulating real environments. Many QA professionals already know this, of course.

This blog post is inspired by a set of three essays by Jack W. Reeves.

Overheard on the XP list, John A De Goes describing his attitude towards teams and processes.

There’s a growing realization that companies, organizations, and teams, they’re all comprised of humans, and being a human is radically different than being a machine. You can’t engineer something comprised of humans — you have to grow it, nourish it, cultivate it. You can fix a broken machine by following a recipe, but you can’t fix a group of people that way. Prescriptions for success amount to nothing, because they ignore the human component, which, when all is said and done, is the most significant factor determining success — development methodology be damned.

Several leaders in their fields have been giving a great deal of thought lately to the state of planning and estimation, and how it relates to agile and lean projects.

Alistair Cockburn said:

The purpose of planning to be able to answer the question “With what we NOW know, what will the most useful system we can create look like on THAT date?”

Glen Alleman said:

The primary function of project management is to ensure that the project is implemented to meet the established budget, schedule, safety, and performance requirements to satisfy its objectives.

Corey Ladas said:

When we agree to take on a work request, we intend to deliver it within n days

There seems to be a general idea that the concepts of Lean can be applied very well to the process of software development, but it leaves unanswered the question of how to plan a long term project.

Agile and Lean both take the view that we will apply some fixed team size to the software, and we will deliver as fast as possible, and allow as much flexibility as possible to the client’s ability to adjust the course (scope) of the project. This is undeniably the current most efficient way to produce software, but that does not mean that it is always the right choice.

The core of the problem is that detailed estimation is a wasteful effort – it does not add business value. So we can deliver more efficiently if we do not estimate. But we have to balance that against the requirements of the business, which has some objective that they are hoping to meet by some date. As the implementors of the solution, we are obligated to be able to tell the business the “when” of the delivery, and they are obligated to tell us the “what”. Unfortunately, the exact “what” always changes. So even if we estimate the “when”, we can very easily be wrong. Its a vicious circle.

In theory, Agile handles long term estimation better than Lean. This is because teams can estimate backlogs at a high level, months (but not years) in advance. Lean is very “just-in-time”, so it can only answer the question at the last instant. The best “lean” answer so far is that we can offer smaller commitments to minimal marketable features. That is, basically what the quote (above) is from Corey.

Neither Lean nor Agile answers the problem of the longer term, more complex project with multiple groups and critical paths and a well-defined scope. The scope is the primary problem – Lean and Agile will deliver faster, but they are not willing to tell the client what he wants to hear – that the particular scope will be completed 2 years from today. They can still give some very attractive answers though – Alistair Cockburn has a list on his wiki.

My conclusion? There is no question in my mind that traditional planned projects are suboptimal, but for now, they are still appropriate for many situations. First though, I would try to sell the client on one of the approaches from Alistair’s list – if they are willing to define what they want in a way that is compatible with Agile or Lean, then we can give them the result for much cheaper, or (more often) a better result for the same price.

Heard on the altdotnet group. (SVN = Subversion, TFS = Team Foundation Server, both of which are primarily source control systems):

Oh, and as someone who is doing a lot with both SVN and TFS recently. I would trust SVN much more than TFS.

SVN make me think: Wow, someone special has thought about doing this.
TFS make me think: Wow, someone… special has thought about doing this.

- Ayende Rahien (aka Oren)

I have been lurking in a recent discussion of using [Transaction]-like attributes in C# to indicate that certain methods can participate in, or require a transaction. Castle ActiveRecord has a another technique of allowing the user to specify a TransactionContext, like

using new TransactionContext
{
...do stuff that will automatically be in the transaction
}

The problem with all of these techniques is that they are essentially procedural. Specifically, anything that you want to participate in the transaction has to be manually called as part of the call-stack. Put another way, they fail to separate the concerns of ATOMic persistence and the identification of what needs to be persisted (the unit of work). The result is that it becomes difficult to implement some aspects of persistence, leading to an increase in artificial complexity.

For example, an aspect of saving a deposit into an account is that there should be a “dual” entry in another account, and balances must be updated. A single aspect like this is somewhat manageable using the proposed semantics, but if you have just a few more, then they quickly lead to very wordy, procedural and possibly complex “Save” methods. You will also end up adding additional state variables to classes that contain “Save” methods, in order to support the logic of the save.

Another way of looking at this is as the problem of the typical “business entity” class that simply does too much. There are cross-cutting concerns that do not belong in one “business entity” or another. That is the major weakness of the ActiveRecord-style of data access – when you take the world-view that every business entity is a table, then you encumber your ability to clearly work with the aspects that are orthogonal or cross-cutting to the entities.

My own solution (there may be better ones) is to explicitly expose the unit of work, and have a technique that allows class instances to intelligently enlist into it. Its worth describing in a little more detail. First, you need an interface that a class can implement to enlist in the work:

Interface IWorkEnlistee
Sub Participate(work as UnitOfWork)
Readonly Property UniqueKey() as String
End Interface

The Participate method is called just before the database Save, but after validation of user-data. The UniqueKey property is necessary to prevent two identical instances from participating (I usually just return the hash-code of some entity instance). You could add methods to the interface to get greater functionality, such as in-memory rollback.

For the dual account entry example, I would have an instance of the above interface that participates by adding the reverse entry and updating the balances. Any state data it needs will be passed in the constructor, which is called before the in-memory data is changed (so that it can get a clear before-picture). It will probably have several related state variables that otherwise would have found themselves complicating some other piece of code.

Using this technique, the entity, presentation and flow logic of the application remains clean, and the cross-cutting aspects that participate in transactions are nicely separated and encapsulated.

Currently I am giving a lot of thought to what the next part of my career should look like. I have always done this from time to time, and the time is right for me to move on from my current position. (Right now, my role can best be described as a mix between Application Architect, Lead Developer and Scrum-master).

Anyway, I got to wondering about how sustainable a career based on Lean and Agile will be. More specifically, I wondered:

Is Lean a one-time improvement that companies can make, and then once they “get it” have it be self-sustaining, or is it something that needs constant fresh input?

Articles like When the Low Hanging Fruit Disappears… Sustaining Lean for the Long Term imply that it remains hard work. You can successfully implement lean, but it remains a process that requires constant gardening to keep in shape.

It is touched on by the previous link, but this pdf article specifically points out that lean gains can be very short term (6 months) when employees do not completely “buy in” to the approach. This leads to the theory of “critical mass” for lean – you must have some critical mass of people that “think lean” in order to have a “lean culture” that can sustain itself.

Lean leadership (pdf) is an interesting participant. I’ve stayed away from management roles in my career, and in so doing, I’ve come to realize that I can lead without being a manager. Perhaps that sounds obvious, but many people think that that leadership requires authority. This is simply not the case. If someone has a compelling vision, the ability to communicate that vision, and the competency to back it up, then people will share that vision and help to achieve it.

Getting back to my original question…the consensus answer seems to be that lean is sustainable, but it requires appropriate leadership and a new culture. Involvement throughout the organization is required in order to apply the necessary continual adjustments and improvements.

In my own career, I’m leaning toward a more formal leadership role in software. Not as a manager, but in a way that allows me to exercise my skills in a way that can cross internal organizational boundaries. The best fit I can think of right now is Enterprise Architect.

If anyone has other ideas, please let me know.

Chris Wheeler on the Extreme Programming (XP) group:

I think values are important, but I’m almost certain I could derive XP’s practices from Peace,Love, and Understanding as easily as they have been derived from the values that currently undergird them.

Incidentally, the XP group is *very* high volume. Its hard to keep up, but its fun because there are so many names that I recognize from books and blogs – always some interesting discussion going on.

In my ramblings on Values, it came to me that it is relatively easy to redefine “traditional” and “agile” project management in terms of Values.

Traditional project management has the values of “fixed budget“, “fixed scope“, “fixed delivery date“. Sure, the project manager can vary these (iron triangle), but they are the only values that traditional project management recognizes. (I know, thats an oversimplification of traditional project management).

Contrast this with agile project management. It has values of “maintainability”, “time-to-market”, and “feature prioritization”.

No wonder the two are so hard to reconcile! But that is not the point of this post.

I am proposing a new style of project management, “Value-Driven Project Management“. This new style encompasses both Agile and traditional project management – they are two subsets that focus on specific sets of values.

At the core of Value-Driven Project Management is the notion that all of the practices of project management exist only to support the values. An example is in order…

Lets consider “risk”. Traditional project management devotes large amounts of effort to risk management. This is because the values are inherently risky – “FIXED budget”, “FIXED scope”, “FIXED delivery date”. Let us contrast that with Agile project management, which has far fewer risk management tools (spikes, tracer bullets). The simple Agile risk management techniques, support the value of “time-to-market”.

There are many more values than the ones I have mentioned. Too many to mention, but consider “high initial quality”, “low development cost”, “certain set of core features”, “performance measures”. Both traditional and agile project management fall short of addressing these values. And when you ignore them, they suffer. We need to help our clients to identify and prioritize their values. Only then can we truly design appropriate project management strategies.

Once we know the values, we need to be able to pick and choose our practices in a way that supports those values. We must pick practices that will combine to provide the client with a satisfying project experience, much as a chef must pick ingredients that combine to provide a satisfying meal.

Let me bash Agile project management for a moment. I have little doubt that Agile project management is one of the most efficient strategies for delivering software (because it prioritizes features). However, it does not directly address “quality” as a Value. Because of that, it is difficult for teams to know how much of the quality practices (TDD, code reviews, automated acceptance tests, formal QA) they should be applying. All it takes is a conversation to identify the value of quality to the client (relative to other values), and then a team can motivate the quality-effort.

That brings me to the final point of this post – it is all relative. The iron triangle is an ugly and blunt tool, but it is correct in the sense that you have to choose certain values above others. That is still the nature of project management.

I mentioned a few posts ago that I really liked the quote from Kent Beck

It is about values, out of which flow principles, out of which flow practices.

Some recent online discussions have made it clearer why this is. Values are things that we can all share. As a business analyst or a software architect, I can talk to a client about the value of long-term maintainability of the code, or the value of time-to-market, or the value of high initial quality, or the value of low development cost, or the value of a certain set of core features, or the value of performance measures.

As an experienced software professional, I can choose principles and practices that support the values that a client is interested in. I can explain to a client that there is a cost to applying certain practices, and he may have to weigh one value (e.g. maintainability) against another (e.g. time-to-market).

I use my own judgment in these tradeoffs, but I would much rather use hard-data. I think as an industry, software development can mature if we get serious about identifying the various practices, their relative costs and their relationship with the values that a client is interested in. But that cannot happen yet, and I’ll tell you why…

The iron triangle of project management grossly misrepresents the trade-offs, because it only understands the values of Time, Cost and Scope. You want to know why software quality is low? Because it has no choice – simplistic project management will steal from quality to pay scope or cost or time. Developers have very little real control over quality.

Some have suggested adding a fourth “quality” side to the triangle, or turning it into a pyramid. This misses the point. There are things that the client values. These are the things that we have to balance against each other to find the path to a successful project. Sorry, its just not as simple as a polygon of any fixed number of sides.

It has been left to the developers to try and motivate good practices that support other values, but only be able to apply them if they can do so without affecting the golden project management values. The end-result is that less good practices are used, and many developers fight against good practices as much as project managers.