Thoughts in the Wilderness

Several leaders in their fields have been giving a great deal of thought lately to the state of planning and estimation, and how it relates to agile and lean projects.

Alistair Cockburn said:

The purpose of planning to be able to answer the question “With what we NOW know, what will the most useful system we can create look like on THAT date?”

Glen Alleman said:

The primary function of project management is to ensure that the project is implemented to meet the established budget, schedule, safety, and performance requirements to satisfy its objectives.

Corey Ladas said:

When we agree to take on a work request, we intend to deliver it within n days

There seems to be a general idea that the concepts of Lean can be applied very well to the process of software development, but it leaves unanswered the question of how to plan a long term project.

Agile and Lean both take the view that we will apply some fixed team size to the software, and we will deliver as fast as possible, and allow as much flexibility as possible to the client’s ability to adjust the course (scope) of the project. This is undeniably the current most efficient way to produce software, but that does not mean that it is always the right choice.

The core of the problem is that detailed estimation is a wasteful effort – it does not add business value. So we can deliver more efficiently if we do not estimate. But we have to balance that against the requirements of the business, which has some objective that they are hoping to meet by some date. As the implementors of the solution, we are obligated to be able to tell the business the “when” of the delivery, and they are obligated to tell us the “what”. Unfortunately, the exact “what” always changes. So even if we estimate the “when”, we can very easily be wrong. Its a vicious circle.

In theory, Agile handles long term estimation better than Lean. This is because teams can estimate backlogs at a high level, months (but not years) in advance. Lean is very “just-in-time”, so it can only answer the question at the last instant. The best “lean” answer so far is that we can offer smaller commitments to minimal marketable features. That is, basically what the quote (above) is from Corey.

Neither Lean nor Agile answers the problem of the longer term, more complex project with multiple groups and critical paths and a well-defined scope. The scope is the primary problem – Lean and Agile will deliver faster, but they are not willing to tell the client what he wants to hear – that the particular scope will be completed 2 years from today. They can still give some very attractive answers though – Alistair Cockburn has a list on his wiki.

My conclusion? There is no question in my mind that traditional planned projects are suboptimal, but for now, they are still appropriate for many situations. First though, I would try to sell the client on one of the approaches from Alistair’s list – if they are willing to define what they want in a way that is compatible with Agile or Lean, then we can give them the result for much cheaper, or (more often) a better result for the same price.

Heard on the altdotnet group. (SVN = Subversion, TFS = Team Foundation Server, both of which are primarily source control systems):

Oh, and as someone who is doing a lot with both SVN and TFS recently. I would trust SVN much more than TFS.

SVN make me think: Wow, someone special has thought about doing this.
TFS make me think: Wow, someone… special has thought about doing this.

- Ayende Rahien (aka Oren)

I have been lurking in a recent discussion of using [Transaction]-like attributes in C# to indicate that certain methods can participate in, or require a transaction. Castle ActiveRecord has a another technique of allowing the user to specify a TransactionContext, like

using new TransactionContext
{
...do stuff that will automatically be in the transaction
}

The problem with all of these techniques is that they are essentially procedural. Specifically, anything that you want to participate in the transaction has to be manually called as part of the call-stack. Put another way, they fail to separate the concerns of ATOMic persistence and the identification of what needs to be persisted (the unit of work). The result is that it becomes difficult to implement some aspects of persistence, leading to an increase in artificial complexity.

For example, an aspect of saving a deposit into an account is that there should be a “dual” entry in another account, and balances must be updated. A single aspect like this is somewhat manageable using the proposed semantics, but if you have just a few more, then they quickly lead to very wordy, procedural and possibly complex “Save” methods. You will also end up adding additional state variables to classes that contain “Save” methods, in order to support the logic of the save.

Another way of looking at this is as the problem of the typical “business entity” class that simply does too much. There are cross-cutting concerns that do not belong in one “business entity” or another. That is the major weakness of the ActiveRecord-style of data access – when you take the world-view that every business entity is a table, then you encumber your ability to clearly work with the aspects that are orthogonal or cross-cutting to the entities.

My own solution (there may be better ones) is to explicitly expose the unit of work, and have a technique that allows class instances to intelligently enlist into it. Its worth describing in a little more detail. First, you need an interface that a class can implement to enlist in the work:

Interface IWorkEnlistee
Sub Participate(work as UnitOfWork)
Readonly Property UniqueKey() as String
End Interface

The Participate method is called just before the database Save, but after validation of user-data. The UniqueKey property is necessary to prevent two identical instances from participating (I usually just return the hash-code of some entity instance). You could add methods to the interface to get greater functionality, such as in-memory rollback.

For the dual account entry example, I would have an instance of the above interface that participates by adding the reverse entry and updating the balances. Any state data it needs will be passed in the constructor, which is called before the in-memory data is changed (so that it can get a clear before-picture). It will probably have several related state variables that otherwise would have found themselves complicating some other piece of code.

Using this technique, the entity, presentation and flow logic of the application remains clean, and the cross-cutting aspects that participate in transactions are nicely separated and encapsulated.

Currently I am giving a lot of thought to what the next part of my career should look like. I have always done this from time to time, and the time is right for me to move on from my current position. (Right now, my role can best be described as a mix between Application Architect, Lead Developer and Scrum-master).

Anyway, I got to wondering about how sustainable a career based on Lean and Agile will be. More specifically, I wondered:

Is Lean a one-time improvement that companies can make, and then once they “get it” have it be self-sustaining, or is it something that needs constant fresh input?

Articles like When the Low Hanging Fruit Disappears… Sustaining Lean for the Long Term imply that it remains hard work. You can successfully implement lean, but it remains a process that requires constant gardening to keep in shape.

It is touched on by the previous link, but this pdf article specifically points out that lean gains can be very short term (6 months) when employees do not completely “buy in” to the approach. This leads to the theory of “critical mass” for lean – you must have some critical mass of people that “think lean” in order to have a “lean culture” that can sustain itself.

Lean leadership (pdf) is an interesting participant. I’ve stayed away from management roles in my career, and in so doing, I’ve come to realize that I can lead without being a manager. Perhaps that sounds obvious, but many people think that that leadership requires authority. This is simply not the case. If someone has a compelling vision, the ability to communicate that vision, and the competency to back it up, then people will share that vision and help to achieve it.

Getting back to my original question…the consensus answer seems to be that lean is sustainable, but it requires appropriate leadership and a new culture. Involvement throughout the organization is required in order to apply the necessary continual adjustments and improvements.

In my own career, I’m leaning toward a more formal leadership role in software. Not as a manager, but in a way that allows me to exercise my skills in a way that can cross internal organizational boundaries. The best fit I can think of right now is Enterprise Architect.

If anyone has other ideas, please let me know.