Thoughts in the Wilderness

I’m into my second month at my new job.  I remain amazed at the amount of documentation that is being generated.  Being a Microsoft shop, almost everything finds its way into Sharepoint.  I’m not against Sharepoint per-se, but I do find it very opaque, i.e. hard to find stuff.

Its like having a lot of interesting papers on my desk, but they are all inside sealed envelopes.  I can open the envelopes, but when I’m done with it, it goes back in the envelope.  Sometimes there are other envelopes inside the envelopes.  As I said, very opaque, and very non-agile.  Contrast with what I am used to, which is a Wiki.  Totally searchable, easy to find what people have been editing.  More like a nice folder with happy little index tabs.

Sharepoint rant over, moving on…

The Business Analysts (BAs) had a slow start, with the result that we started developing our product before they had really written any requirement documents about it.  The result is that they are playing catch-up, and documenting what we have provided, and asking all of the questions that we should have asked while we developed it.

We do not have a single customer, or product champion.  This means that the questions are difficult for anyone to answer effectively.  It also means that the team is pulled in several directions:

• software architect tries to get us to do multi-tier and service-based and ultra secure at the expense of usability
• managers care mostly that it is done on time
• business analysts care about “checking off” features
• there is very little optimization of features, i.e. everything is equal priority

The result is that we have a fairly well architected, feature-full product that is hastily put together.  Features are surface deep, i.e. we are not “done” to the level that an Agile team would normally consider “done”.  Usability is low, and the product lacks perceived integrity.

Even given all of that, I remain optimistic.  We have a solid team, and every day we work more closely together.  As I understand more about the environment, I am able to make small adjustments to improve it.  We have a lot of time left to deliver, and upper management expects that it will be very much a v1.0 product.

Well today we had our first retrospective, a day before the end of our first sprint.  I used the tried & true format of “What went well” followed by “what went not as well”.  We scheduled an hour for the meeting.

As usual, people had a lot of trouble coming up with positive things to say, and not much trouble coming up with negative   One person refused to write anything positive!

It came together pretty easily that the topic that concerned the most people was “requirements”, or rather the lack of them.  There was a fair bit of conversation, which could easily have got out of hand.  In fact, it did for a little bit when one person tried to hijack the meeting for a topic other than the one we agreed to talk about.  I redirected that to another 1-on-1 meeting between the 2 of us later.

I tried to keep it in front of everyone that we wanted to come up with concrete actions that could be used to address the problems.  This worked ok to keep the conversation on topic, but no-one really suggested any actions except for our manager.  Being that she is the manager, no-one was seemed ready to contradict her suggestions.  They were good enough action items though, so I went with the flow.

The organization is very document-centric, which bothers me.  Rather than talk about ways that the developers could *pull* requirements through 1-on-1 conversations, there were a lot of suggestions on how we could better use email and documents on SharePoint.  Ugh.  I’ll have to live with the document-centric viewpoint for a while I think.  The best I can probably hope for, short term, is to find ways to make the documents useful to the team.

In the end, we finished on time and came out with some actions that we are taking to address our requirements-concerns.  Nobody was insulted or put off, and the team is fairly positive.  I’m pretty happy with those results.

As mentioned a few weeks ago, I have started a new job. My role is still unclear, but that doesn’t bother me at all – this way I can define my own role.

I’m working in a small team of web developers, trying to start on the next generation of the web portion of our core business product. The team was supposed to be Agile, but was decidedly *not*. I’m working on that, i.e. so far I’m defining myself as the Agile-guru.

We had our first planning meeting that I was allowed to run, and I went with my favorite planning tool (FreeMind). The team is very tool and framework oriented, so I think introducing the tool as part of the planning was less intimidating than if I had just gone at it with story cards.

My purpose in using FreeMind is to enable open discussion, and also to focus that discussion. It allows me to acknowledge everyone’s input, by reflecting it on the mind-map. The tree-like structure naturally allows everyone to be focused on the current area of discussion without being too distracted by other concerns. Once we have everything noted, then we can more effectively make estimates.

We had a very short estimating session. This was difficult, because individuals on the team seemed information-starved, i.e. they wanted to gather all of the requirements then and there. I tried with limited success to focus on just getting rough estimates. To keep it ultra-simple, I allowed only 4 time estimates: 1 day, 2 days, 5 days, or more (needs to be broken down further to estimate). As expected for a new Agile team, estimates were *very* ambitious, and we did not complete.

My final note is that it is interesting to be able to observe the team forming. I was the last member of the team to be hired, but before my joining, the others have barely interacted as far as I can tell. This means that we started in the “forming” stage of team evolution. This is characterised by the our manager being very hands-on, and directive. Basically, we are not a team so much as a set of individuals working under the co-ordination of a manager.

The next team-stage after forming is “storming”. This is a little stressful, and I hope we don’t lose any team members, or the confidence of our manager. I can already see some people coming out strongly on opposite sides of issues, and coming into conflict because of it.

Anyway, that’s my update. Wish me luck.

Overheard on the XP list, John A De Goes describing his attitude towards teams and processes.

There’s a growing realization that companies, organizations, and teams, they’re all comprised of humans, and being a human is radically different than being a machine. You can’t engineer something comprised of humans — you have to grow it, nourish it, cultivate it. You can fix a broken machine by following a recipe, but you can’t fix a group of people that way. Prescriptions for success amount to nothing, because they ignore the human component, which, when all is said and done, is the most significant factor determining success — development methodology be damned.

Several leaders in their fields have been giving a great deal of thought lately to the state of planning and estimation, and how it relates to agile and lean projects.

Alistair Cockburn said:

The purpose of planning to be able to answer the question “With what we NOW know, what will the most useful system we can create look like on THAT date?”

Glen Alleman said:

The primary function of project management is to ensure that the project is implemented to meet the established budget, schedule, safety, and performance requirements to satisfy its objectives.

Corey Ladas said:

When we agree to take on a work request, we intend to deliver it within n days

There seems to be a general idea that the concepts of Lean can be applied very well to the process of software development, but it leaves unanswered the question of how to plan a long term project.

Agile and Lean both take the view that we will apply some fixed team size to the software, and we will deliver as fast as possible, and allow as much flexibility as possible to the client’s ability to adjust the course (scope) of the project. This is undeniably the current most efficient way to produce software, but that does not mean that it is always the right choice.

The core of the problem is that detailed estimation is a wasteful effort – it does not add business value. So we can deliver more efficiently if we do not estimate. But we have to balance that against the requirements of the business, which has some objective that they are hoping to meet by some date. As the implementors of the solution, we are obligated to be able to tell the business the “when” of the delivery, and they are obligated to tell us the “what”. Unfortunately, the exact “what” always changes. So even if we estimate the “when”, we can very easily be wrong. Its a vicious circle.

In theory, Agile handles long term estimation better than Lean. This is because teams can estimate backlogs at a high level, months (but not years) in advance. Lean is very “just-in-time”, so it can only answer the question at the last instant. The best “lean” answer so far is that we can offer smaller commitments to minimal marketable features. That is, basically what the quote (above) is from Corey.

Neither Lean nor Agile answers the problem of the longer term, more complex project with multiple groups and critical paths and a well-defined scope. The scope is the primary problem – Lean and Agile will deliver faster, but they are not willing to tell the client what he wants to hear – that the particular scope will be completed 2 years from today. They can still give some very attractive answers though – Alistair Cockburn has a list on his wiki.

My conclusion? There is no question in my mind that traditional planned projects are suboptimal, but for now, they are still appropriate for many situations. First though, I would try to sell the client on one of the approaches from Alistair’s list – if they are willing to define what they want in a way that is compatible with Agile or Lean, then we can give them the result for much cheaper, or (more often) a better result for the same price.

Currently I am giving a lot of thought to what the next part of my career should look like. I have always done this from time to time, and the time is right for me to move on from my current position. (Right now, my role can best be described as a mix between Application Architect, Lead Developer and Scrum-master).

Anyway, I got to wondering about how sustainable a career based on Lean and Agile will be. More specifically, I wondered:

Is Lean a one-time improvement that companies can make, and then once they “get it” have it be self-sustaining, or is it something that needs constant fresh input?

Articles like When the Low Hanging Fruit Disappears… Sustaining Lean for the Long Term imply that it remains hard work. You can successfully implement lean, but it remains a process that requires constant gardening to keep in shape.

It is touched on by the previous link, but this pdf article specifically points out that lean gains can be very short term (6 months) when employees do not completely “buy in” to the approach. This leads to the theory of “critical mass” for lean – you must have some critical mass of people that “think lean” in order to have a “lean culture” that can sustain itself.

Lean leadership (pdf) is an interesting participant. I’ve stayed away from management roles in my career, and in so doing, I’ve come to realize that I can lead without being a manager. Perhaps that sounds obvious, but many people think that that leadership requires authority. This is simply not the case. If someone has a compelling vision, the ability to communicate that vision, and the competency to back it up, then people will share that vision and help to achieve it.

Getting back to my original question…the consensus answer seems to be that lean is sustainable, but it requires appropriate leadership and a new culture. Involvement throughout the organization is required in order to apply the necessary continual adjustments and improvements.

In my own career, I’m leaning toward a more formal leadership role in software. Not as a manager, but in a way that allows me to exercise my skills in a way that can cross internal organizational boundaries. The best fit I can think of right now is Enterprise Architect.

If anyone has other ideas, please let me know.

From Kent Beck on the XP yahoo group (emphasis mine):

XP is not defined by its practices. It is about values, out of which flow principles, out of which flow practices. Enough of software development is similar enough that, even in different circumstances, the practices derived from the same values and principles end up looking similar. Different values, different principles and you end up with different practices. They might still produce valuable software, but they wouldn’t be XP.

For me, 2007 was mostly about learning the best way to run a development team in a way that adds the best value to the business.

I became convinced that it is not enough to optimize development locally – to achieve the best value, we must treat development as part of a manufacturing-like process. In doing so, we can understand that requirements are like inventory – they go stale and lose value if they sit around. We also can understand that we need to link the day-to-day tasks we do to business value. Finally, how we define “done” is one of the fundamental questions for any development team.

I already was using Scrum, but began experimenting with Kanban Boards. This showed me that sometimes planning meetings are wasted time. Estimating has value only as much as it assists in release-planning. My newly-won experience is invaluable in knowing how much planning and estimating is appropriate for a given situation. I feel like I could easily fulfill the role of ScrumMaster on any team, or else introduce existing teams to more optimal ways of doing things.

One of the lessons I have learned from being a Software Architect is that you cannot always dictate architecture, but you can guide it by providing simple and effective ways to do things. There are some very strong strategies that can be applied to achieve good long-term viability of software. To this end, I registered a new domain (perfectapi.com), and put up a website on that domain. I did not find the time I needed to do it justice though.

My idea was to try and apply my own hard-learned lessons of software development in a way that improved the architecture for others. This is not a pipe-dream – it has been done before, most recently with Ruby-on-Rails. If you can find the correct paradigms, you can quickly create a community that can be very effective at developing new applications.

In 2005, I had decided not to work on web applications anymore. ASP.NET was just too complex, too barebones to develop seriously on. Ruby on Rails is probably ok, but it has a Unix-like learning curve (I’ve grown used to Intellisense). In 2007, Silverlight was introduced. I have not worked with Silverlight yet, but the feeling I get is that it will change the paradigm enough away from the horrible POST-RESPONSE loop of web development.

Personally, it has been a difficult year. My lone-wolf tendencies have hurt family life, and I find myself with too few friends. I have put on weight. Work was all-consuming at times. It is not that I work long hours – it is just that the hours I work are very intense and stressful. Plus, there is just too much to know – it is a truly exciting period in software development.

I will find a new job in 2008. I hope that it will allow me the freedom to relax more with family and friends.

You may occasionally see the term Shu Ha Ri bandied about in people that think a lot about Agile and Lean. I think Alistair Cockburn was the first to use it in relation to software development.

The idea is that we should teach the beginner (Shu) some techniques, or best practices. Ha is the stage where the beginner has learned that there are some underlying principles, and begins to explore those. Ri is the master stage, where the master adapts and invents new techniques.

“Shu Ha Ri” is implicitly a craft-based way of thinking of software development. The individual words imply the different levels of craftsman, from apprentice through journeyman, to master.

The more interesting aspect of the term relates to communication. It can be hard for Shu-level and Ri-level to communicate. This is because Ri believes there is no single best solution to a problem (everything depends – he will implement a best solution by adapting as he creates it). But Shu needs firm guidance – he needs to be told a good-enough way to solve the problem.

One way for Ri to communicate is using the principle of strong opinions, weakly held. That is, he should communicate as if he is certain that what he says is correct. But it is all an act. He is not certain, and definitely not attached to the opinion.

In short, if I am Ri then I should feel free to present my opinions or theories as dogma.

If the other individual can show a better way, then I will accept that (it’s easy, since I was never attached to the original opinion anyway).

One of the foundations of all Agile techniques is that of frequent iterations developing working software. I think maybe this is a developer-centric view of a deeper concept.

When we talk about iterations, what we are really trying to do is to work with a client to break down their “big goals” into smaller ones. We ask them to call these mini-goals “stories” or “backlog items”. Perhaps “goal” is a better word. Why…?

For one thing, it is less feature-attached. A backlog item or story usually translates to a feature. Maybe thats not the intention,but that is how it works out.

I’m digressing though. The main thought is that one or more iterations fulfill some client goal. This goal often corresponds with a release. That is a very concrete view of reality. Anything smaller is not meaningful to a client.

We ask them to shuffle backlog items around in order to decide which features make it into a release. What we should be asking them to do is to decide on the minimal set of functionality that meets their goal. Anything after that is meeting some other goal.

On another tangent…

When we ask developers to think in terms of backlog items and stories, we are making it too easy for them. They need to be made aware that the code they write has to work towards the client’s goals. If a “feature” can be avoided or changed by meeting the goal in another way, then the developer needs to realize that.

For example? …. The client proposes a feature. Under certain circumstances, when the user clicks the Save button, we should pop up a dialog asking them if they are sure they want to do this. They must click “Yes” to continue.

Seems like a pretty normal feature. It will make a fine backlog item. But what is the goal of the client? In this example (based on real-life), the answer was that we needed users to manually assert their intentions.

The implementation of a dialog box is a poor solution (as it almost always is). A better one was to supply a checkbox that became enabled (and required) when the right conditions occurred. Thus, the user had to pro-actively assert that they were sure this is what they wanted.

The point is this – when we tell developers to think in terms of backlog items, we are really saying “think in terms of features”. Experienced developers will intuitively sidestep this and get to the real goal. Less experienced developers will do exactly what the backlog item suggests.

If we had stated the requirement as a goal instead of a feature, then the less experienced developer would have had to suggest the details of the feature. This implies design, and thought. Exactly what we want in an Agile environment.