Quote of the Day - John A. De Goes on Teams

Overheard on the XP list, John A De Goes describing his attitude towards teams and processes.

There's a growing realization that companies, organizations, and teams, they're all comprised of humans, and being a human is radically different than being a machine. You can't engineer something comprised of humans -- you have to grow it, nourish it, cultivate it. You can fix a broken machine by following a recipe, but you can't fix a group of people that way. Prescriptions for success amount to nothing, because they ignore the human component, which, when all is said and done, is the most significant factor determining success -- development methodology be damned.

Current thoughts on Estimation and Planning

Several leaders in their fields have been giving a great deal of thought lately to the state of planning and estimation, and how it relates to agile and lean projects.

Alistair Cockburn said:

The purpose of planning to be able to answer the question “With what we NOW know, what will the most useful system we can create look like on THAT date?”

Glen Alleman said:

The primary function of project management is to ensure that the project is implemented to meet the established budget, schedule, safety, and performance requirements to satisfy its objectives.

Corey Ladas said:

When we agree to take on a work request, we intend to deliver it within n days

There seems to be a general idea that the concepts of Lean can be applied very well to the process of software development, but it leaves unanswered the question of how to plan a long term project.

Agile and Lean both take the view that we will apply some fixed team size to the software, and we will deliver as fast as possible, and allow as much flexibility as possible to the client's ability to adjust the course (scope) of the project. This is undeniably the current most efficient way to produce software, but that does not mean that it is always the right choice.

The core of the problem is that detailed estimation is a wasteful effort - it does not add business value. So we can deliver more efficiently if we do not estimate. But we have to balance that against the requirements of the business, which has some objective that they are hoping to meet by some date. As the implementors of the solution, we are obligated to be able to tell the business the "when" of the delivery, and they are obligated to tell us the "what". Unfortunately, the exact "what" always changes. So even if we estimate the "when", we can very easily be wrong. Its a vicious circle.

In theory, Agile handles long term estimation better than Lean. This is because teams can estimate backlogs at a high level, months (but not years) in advance. Lean is very "just-in-time", so it can only answer the question at the last instant. The best "lean" answer so far is that we can offer smaller commitments to minimal marketable features. That is, basically what the quote (above) is from Corey.

Neither Lean nor Agile answers the problem of the longer term, more complex project with multiple groups and critical paths and a well-defined scope. The scope is the primary problem - Lean and Agile will deliver faster, but they are not willing to tell the client what he wants to hear - that the particular scope will be completed 2 years from today. They can still give some very attractive answers though - Alistair Cockburn has a list on his wiki.

My conclusion? There is no question in my mind that traditional planned projects are suboptimal, but for now, they are still appropriate for many situations. First though, I would try to sell the client on one of the approaches from Alistair's list - if they are willing to define what they want in a way that is compatible with Agile or Lean, then we can give them the result for much cheaper, or (more often) a better result for the same price.

Deciding on my next Career

Currently I am giving a lot of thought to what the next part of my career should look like. I have always done this from time to time, and the time is right for me to move on from my current position. (Right now, my role can best be described as a mix between Application Architect, Lead Developer and Scrum-master).

Anyway, I got to wondering about how sustainable a career based on Lean and Agile will be. More specifically, I wondered:

Is Lean a one-time improvement that companies can make, and then once they "get it" have it be self-sustaining, or is it something that needs constant fresh input?

Articles like When the Low Hanging Fruit Disappears... Sustaining Lean for the Long Term imply that it remains hard work. You can successfully implement lean, but it remains a process that requires constant gardening to keep in shape.

It is touched on by the previous link, but this pdf article specifically points out that lean gains can be very short term (6 months) when employees do not completely "buy in" to the approach. This leads to the theory of "critical mass" for lean - you must have some critical mass of people that "think lean" in order to have a "lean culture" that can sustain itself.

Lean leadership (pdf) is an interesting participant. I've stayed away from management roles in my career, and in so doing, I've come to realize that I can lead without being a manager. Perhaps that sounds obvious, but many people think that that leadership requires authority. This is simply not the case. If someone has a compelling vision, the ability to communicate that vision, and the competency to back it up, then people will share that vision and help to achieve it.

Getting back to my original question...the consensus answer seems to be that lean is sustainable, but it requires appropriate leadership and a new culture. Involvement throughout the organization is required in order to apply the necessary continual adjustments and improvements.

In my own career, I'm leaning toward a more formal leadership role in software. Not as a manager, but in a way that allows me to exercise my skills in a way that can cross internal organizational boundaries. The best fit I can think of right now is Enterprise Architect.

If anyone has other ideas, please let me know.

Quote of the day - Kent Beck on defining XP

From Kent Beck on the XP yahoo group (emphasis mine):

XP is not defined by its practices. It is about values, out of which flow principles, out of which flow practices. Enough of software development is similar enough that, even in different circumstances, the practices derived from the same values and principles end up looking similar. Different values, different principles and you end up with different practices. They might still produce valuable software, but they wouldn't be XP.

2007 Retrospective

For me, 2007 was mostly about learning the best way to run a development team in a way that adds the best value to the business.

I became convinced that it is not enough to optimize development locally - to achieve the best value, we must treat development as part of a manufacturing-like process. In doing so, we can understand that requirements are like inventory - they go stale and lose value if they sit around. We also can understand that we need to link the day-to-day tasks we do to business value. Finally, how we define "done" is one of the fundamental questions for any development team.

I already was using Scrum, but began experimenting with Kanban Boards. This showed me that sometimes planning meetings are wasted time. Estimating has value only as much as it assists in release-planning. My newly-won experience is invaluable in knowing how much planning and estimating is appropriate for a given situation. I feel like I could easily fulfill the role of ScrumMaster on any team, or else introduce existing teams to more optimal ways of doing things.

One of the lessons I have learned from being a Software Architect is that you cannot always dictate architecture, but you can guide it by providing simple and effective ways to do things. There are some very strong strategies that can be applied to achieve good long-term viability of software. To this end, I registered a new domain (perfectapi.com), and put up a website on that domain. I did not find the time I needed to do it justice though.

My idea was to try and apply my own hard-learned lessons of software development in a way that improved the architecture for others. This is not a pipe-dream - it has been done before, most recently with Ruby-on-Rails. If you can find the correct paradigms, you can quickly create a community that can be very effective at developing new applications.

In 2005, I had decided not to work on web applications anymore. ASP.NET was just too complex, too barebones to develop seriously on. Ruby on Rails is probably ok, but it has a Unix-like learning curve (I've grown used to Intellisense). In 2007, Silverlight was introduced. I have not worked with Silverlight yet, but the feeling I get is that it will change the paradigm enough away from the horrible POST-RESPONSE loop of web development.

Personally, it has been a difficult year. My lone-wolf tendencies have hurt family life, and I find myself with too few friends. I have put on weight. Work was all-consuming at times. It is not that I work long hours - it is just that the hours I work are very intense and stressful. Plus, there is just too much to know - it is a truly exciting period in software development.

I will find a new job in 2008. I hope that it will allow me the freedom to relax more with family and friends.

Shu Ha Ri, Summarized (or, the story of my life)

You may occasionally see the term Shu Ha Ri bandied about in people that think a lot about Agile and Lean. I think Alistair Cockburn was the first to use it in relation to software development.

The idea is that we should teach the beginner (Shu) some techniques, or best practices. Ha is the stage where the beginner has learned that there are some underlying principles, and begins to explore those. Ri is the master stage, where the master adapts and invents new techniques.

"Shu Ha Ri" is implicitly a craft-based way of thinking of software development. The individual words imply the different levels of craftsman, from apprentice through journeyman, to master.

The more interesting aspect of the term relates to communication. It can be hard for Shu-level and Ri-level to communicate. This is because Ri believes there is no single best solution to a problem (everything depends - he will implement a best solution by adapting as he creates it). But Shu needs firm guidance - he needs to be told a good-enough way to solve the problem.

One way for Ri to communicate is using the principle of strong opinions, weakly held. That is, he should communicate as if he is certain that what he says is correct. But it is all an act. He is not certain, and definitely not attached to the opinion.

In short, if I am Ri then I should feel free to present my opinions or theories as dogma.

If the other individual can show a better way, then I will accept that (it's easy, since I was never attached to the original opinion anyway).

Goal Driven Development

One of the foundations of all Agile techniques is that of frequent iterations developing working software. I think maybe this is a developer-centric view of a deeper concept.

When we talk about iterations, what we are really trying to do is to work with a client to break down their "big goals" into smaller ones. We ask them to call these mini-goals "stories" or "backlog items". Perhaps "goal" is a better word. Why...?

For one thing, it is less feature-attached. A backlog item or story usually translates to a feature. Maybe thats not the intention,but that is how it works out.

I'm digressing though. The main thought is that one or more iterations fulfill some client goal. This goal often corresponds with a release. That is a very concrete view of reality. Anything smaller is not meaningful to a client.

We ask them to shuffle backlog items around in order to decide which features make it into a release. What we should be asking them to do is to decide on the minimal set of functionality that meets their goal. Anything after that is meeting some other goal.

On another tangent...

When we ask developers to think in terms of backlog items and stories, we are making it too easy for them. They need to be made aware that the code they write has to work towards the client's goals. If a "feature" can be avoided or changed by meeting the goal in another way, then the developer needs to realize that.

For example? .... The client proposes a feature. Under certain circumstances, when the user clicks the Save button, we should pop up a dialog asking them if they are sure they want to do this. They must click "Yes" to continue.

Seems like a pretty normal feature. It will make a fine backlog item. But what is the goal of the client? In this example (based on real-life), the answer was that we needed users to manually assert their intentions.

The implementation of a dialog box is a poor solution (as it almost always is). A better one was to supply a checkbox that became enabled (and required) when the right conditions occurred. Thus, the user had to pro-actively assert that they were sure this is what they wanted.

The point is this - when we tell developers to think in terms of backlog items, we are really saying "think in terms of features". Experienced developers will intuitively sidestep this and get to the real goal. Less experienced developers will do exactly what the backlog item suggests.

If we had stated the requirement as a goal instead of a feature, then the less experienced developer would have had to suggest the details of the feature. This implies design, and thought. Exactly what we want in an Agile environment.

The end of a software company

Lean principles teach us to recognize a constraint in the system, and "elevate" (fix) it. I was reading what Amit Rathore was writing about this, and it got me to thinking...

My current work environment is a going-out-of-business software company. Our parent company is continuing in the same market, but they will be using a different piece of software than the one we created. (Due to acquisitions, there were 2 parts of the company creating the same kind of product).

The way our business unit worked is that we would get a sales order for a new deployment of our software. The client would specify their requirements in some sort of RFP. We would then spend time developing the missing pieces (required functionality that was not yet present), convert their existing data to our own format, train the users, and deploy.

In doing this, it appeared to me that Testing and Inventory (time between completing work and Deploying it) were large constraints. In other words, we would build a lot of software, but no-one would look closely at it until we deployed it months (up to a year) later.

It was a decision of the business to expect high quality work, but acceptance-testing that same work was so unimportant that not a single person was assigned full-time to it. Even traditional regression, or smoke testing was not prioritized. To the test-infected, this sounds a little crazy!

But it wasn't. Because of the RFP (contract-driven) style of the client relationship, there was simply no justification for spending more on acceptance testing, (we did not have sufficient real client input to know for sure that we were building what they wanted). If it meet the letter of the RFP, then it was ok.

During big-bang style deployments, we would run around and fix the high-priority issues (mixture of bugs and changed requirements) until the product was working to the satisfaction of the client. Without built in development quality and Agile responsiveness, this is a nightmare. With those things in place, it is just highly stressful.

The extended deployment periods were our acceptance tests. They were also when we found the most about *actual* client requirements. Ultimately (with exceptions) the client was satisfied (but not necessarily happy). And the Product Manager made sure to build more realistic requirements into the next version.

Back to the failure of the business...

In this model, the satisfaction of the next client is directly proportional to how closely their RFP matched a previous one. This was our downfall, because we came into an expansionist period where each new client was a completely new RFP. We were breaking into new product areas and new geographic areas. And our unit was expensive (because we were developing large amounts of good quality, well designed, extensible software).

In this expansionist period we had great difficulty in creating happy clients, because they would each get the worst possible result - an initial, painful deployment where the best possible outcome was meeting the "letter" of the requirements. (Actually, we did a little better than that, but only through a lot of good, dedicated people doing heroic things).

We tried to slow down the expansion but conditions (new-sales-driven upper management) would not allow this. Although they did not think of it in those terms, management did try to elevate the testing/deployment constraint, by working more closely with new clients. This was done by increasing the size of the client-relationship staff.

In the end, we had some successes (and some failures), and the parent company eventually came to the decision to end our unit. This was not a direct result of our failures, but I am sure that had we been more successful, it would have gone down differently.

"Done", or "Done-Done"

Maybe we're different from other Agile shops, but "what does 'done' look like?" is one of the primary questions in our process. It is almost a running joke - "is it done, or is it done-done?"

To prove you are "done", you have to agree (with the client) on a reasonable repeatable demonstration (i.e. test). Thus, "done" is also synonymous with "accepted", and to some extent, "tested".

It is obvious (to me) that you can never really move a piece of functionality out of "development" until you are able to agree on a definition of "done".

For large tasks (taking weeks or months), it is very helpful to break the task down into meaningful chunks. Being able to define "done" for each of those tasks is an invaluable technique. The "doneness" criteria allows clear borders to be drawn between each chunk, and helps assure that each is meaningful functionality.

Acceptance Test vs Component Test

I have been struggling to try and convey to our team what the difference is between an Acceptance Test and a Component Test.

This morning in the shower, I had some insight into this. Why not define testing by its primary purpose? Seems obvious, but I have not seen it stated this way before. So, here goes:

• Acceptance Testing - ensure that the software meets the business requirement
• Component Testing - ensure that the component does not regress
• Unit Testing (in the Agile meaning) - drive the design of the code
  

Looking at it this way, it definitely matters whether it is a component test or an acceptance test, because they meet different needs.

As a side-note, I have found it difficult to perform acceptance testing unless unit testing has also been employed. That is because unit testing tends to create those testable-hooks that let us attach the acceptance test to the system with minimal fuss.

Acceptance Testing with Fit

At my job (a software development company), we use a variant of Scrum to manage our development. We have short development iterations of 2 weeks, at the end of which we deliver something of value to the business.

Since we started though, it has been difficult for the business to accept our output as complete. We did not have any formal acceptance process, or acceptance tests. The result was that we were seldom able to promise a completed feature in a single 2 week sprint, because the acceptance criteria were not clear enough. Each feature evolved to where it needed to be over several sprints, severely impacting our long term planning.

Code quality is good, due to a focus on unit tests, but we struggle to quickly bring a completed user-valued feature to market because we are not able to home-in on what the completed feature should look like.

In recent retrospectives, the team decided that we want our user-stories (planned work items) to be more completely defined. We will require acceptance test documents before we start working on functionality!

A bold step, because it will increase upfront planning time. I was a little nervous of this, mainly because it sounds too much like waterfall - plan upfront, and then deliver per the spec.

I read something that calmed me though, and confirmed (for me) that we are on the right track. Mike Cohn writes that user stories should embody these 3 aspects:

• a written description of the story used for planning and as a reminder
• conversations about the story that serve to flesh out the details of the story
• tests that convey and document details and that can be used to determine when a story is complete

According to that definition, we have been missing a whole aspect of our requirements!

So, we are now using Fit to create automated acceptance tests during Sprint planning. This helps us better define what will constitute success up-front. We will spend more time planning, but we will gain in the longer term because we will be delivering completed client value at the end of the sprints.

Already, the automated acceptance tests have been improving product quality and team communication. We have gained a valuable set of regression tests for the system. But we have decreased productivity in the short term, because developers and testers have had to learn a new tool and re-invent the way we communicate.

Whether the promise of improved long-term productivity follows remains to be seen. I'm confident, but time will tell...

Blogger is finished screwing with me

My last post was almost a year ago...

The reason for that was that for a while now Google's blogger has been screwing with me - ever since they started the beta blogger service. The short version is that it was too much trouble to login to make posts. Now however, they have gone out of beta, and I can actually login again! Woohoo :)

The new interface is much the same, with some minor additions, such as tag support and integrated login with my Google (g-mail) account. There's more, but I couldn't find a link to a simple "new features" page.

(In unrelated news) I found a nice quote today:

"In preparing for battle I have always found that plans are useless, but planning is indispensable." - Dwight D Eisenhower

Sort of relevant to Agile software development. We like to plan, but we are not bound by the plan as we would be in the older waterfall model.