Thoughts in the Wilderness

It was years ago when I first heard of the manufacturing concept of “stop the line whenever something goes wrong”.  At the time, it seemed crazy…why should everyone stop what they are doing just because one person has a problem?  After all, others can continue working, and they’ll just get a little ahead.

Once I started learning lean, I thought that it was about maintaining the flow, and not building up inventory.  More recently, I have understood something else:

“When something go wrong, the cause is almost always inherent in the system”.

In other words, when we “stop the line” it is because we want to understand the cause of the problem, and create a solution that eliminates the cause.

How is this applicable to software?  Developers become inured to day-to-day problems. Few developers would consider saying “stop work – the ASP.NET form life-cycle is too complex for our needs”, or “stop work, the deployment is too manual and error prone”.

There is even a culture wherein individuals get recognition for working around problems, rather than solving root causes.   Let Tom do the deployments, because Joe screws it up.  Anne is good at working with the complex ASP.NET lifecycle, what a great developer she must be!

Its not exactly wrong – Anne is more skilled, Tom is more careful, they should be commended for that.  But they did not go the extra mile – to automate and error-proof the process.  Why would they – they have not been trained to do it, and their manager would probably chastise them for wasting time if they tried.

We have not been taught scientific thinking practices, such as “Plan, Do, Check, Act“.   We’re woefully uneducated when it comes to the basic tools that would allow us to improve our own processes. We are not even aware that it should be our responsibility to improve the process.

Process improvement?  That’s someone else’s job – I’m here to code!

That’s an easy line to buy – when you think of process as a big, abstract thing.  However, if you take a look at the day-to-day things that we do, then there are many, many opportunities to improve.  The real challenge is to have the right thinking tools (PDCA or similar), and to just try.

I’m into my second month at my new job.  I remain amazed at the amount of documentation that is being generated.  Being a Microsoft shop, almost everything finds its way into Sharepoint.  I’m not against Sharepoint per-se, but I do find it very opaque, i.e. hard to find stuff.

Its like having a lot of interesting papers on my desk, but they are all inside sealed envelopes.  I can open the envelopes, but when I’m done with it, it goes back in the envelope.  Sometimes there are other envelopes inside the envelopes.  As I said, very opaque, and very non-agile.  Contrast with what I am used to, which is a Wiki.  Totally searchable, easy to find what people have been editing.  More like a nice folder with happy little index tabs.

Sharepoint rant over, moving on…

The Business Analysts (BAs) had a slow start, with the result that we started developing our product before they had really written any requirement documents about it.  The result is that they are playing catch-up, and documenting what we have provided, and asking all of the questions that we should have asked while we developed it.

We do not have a single customer, or product champion.  This means that the questions are difficult for anyone to answer effectively.  It also means that the team is pulled in several directions:

• software architect tries to get us to do multi-tier and service-based and ultra secure at the expense of usability
• managers care mostly that it is done on time
• business analysts care about “checking off” features
• there is very little optimization of features, i.e. everything is equal priority

The result is that we have a fairly well architected, feature-full product that is hastily put together.  Features are surface deep, i.e. we are not “done” to the level that an Agile team would normally consider “done”.  Usability is low, and the product lacks perceived integrity.

Even given all of that, I remain optimistic.  We have a solid team, and every day we work more closely together.  As I understand more about the environment, I am able to make small adjustments to improve it.  We have a lot of time left to deliver, and upper management expects that it will be very much a v1.0 product.

Several leaders in their fields have been giving a great deal of thought lately to the state of planning and estimation, and how it relates to agile and lean projects.

Alistair Cockburn said:

The purpose of planning to be able to answer the question “With what we NOW know, what will the most useful system we can create look like on THAT date?”

Glen Alleman said:

The primary function of project management is to ensure that the project is implemented to meet the established budget, schedule, safety, and performance requirements to satisfy its objectives.

Corey Ladas said:

When we agree to take on a work request, we intend to deliver it within n days

There seems to be a general idea that the concepts of Lean can be applied very well to the process of software development, but it leaves unanswered the question of how to plan a long term project.

Agile and Lean both take the view that we will apply some fixed team size to the software, and we will deliver as fast as possible, and allow as much flexibility as possible to the client’s ability to adjust the course (scope) of the project. This is undeniably the current most efficient way to produce software, but that does not mean that it is always the right choice.

The core of the problem is that detailed estimation is a wasteful effort – it does not add business value. So we can deliver more efficiently if we do not estimate. But we have to balance that against the requirements of the business, which has some objective that they are hoping to meet by some date. As the implementors of the solution, we are obligated to be able to tell the business the “when” of the delivery, and they are obligated to tell us the “what”. Unfortunately, the exact “what” always changes. So even if we estimate the “when”, we can very easily be wrong. Its a vicious circle.

In theory, Agile handles long term estimation better than Lean. This is because teams can estimate backlogs at a high level, months (but not years) in advance. Lean is very “just-in-time”, so it can only answer the question at the last instant. The best “lean” answer so far is that we can offer smaller commitments to minimal marketable features. That is, basically what the quote (above) is from Corey.

Neither Lean nor Agile answers the problem of the longer term, more complex project with multiple groups and critical paths and a well-defined scope. The scope is the primary problem – Lean and Agile will deliver faster, but they are not willing to tell the client what he wants to hear – that the particular scope will be completed 2 years from today. They can still give some very attractive answers though – Alistair Cockburn has a list on his wiki.

My conclusion? There is no question in my mind that traditional planned projects are suboptimal, but for now, they are still appropriate for many situations. First though, I would try to sell the client on one of the approaches from Alistair’s list – if they are willing to define what they want in a way that is compatible with Agile or Lean, then we can give them the result for much cheaper, or (more often) a better result for the same price.

Currently I am giving a lot of thought to what the next part of my career should look like. I have always done this from time to time, and the time is right for me to move on from my current position. (Right now, my role can best be described as a mix between Application Architect, Lead Developer and Scrum-master).

Anyway, I got to wondering about how sustainable a career based on Lean and Agile will be. More specifically, I wondered:

Is Lean a one-time improvement that companies can make, and then once they “get it” have it be self-sustaining, or is it something that needs constant fresh input?

Articles like When the Low Hanging Fruit Disappears… Sustaining Lean for the Long Term imply that it remains hard work. You can successfully implement lean, but it remains a process that requires constant gardening to keep in shape.

It is touched on by the previous link, but this pdf article specifically points out that lean gains can be very short term (6 months) when employees do not completely “buy in” to the approach. This leads to the theory of “critical mass” for lean – you must have some critical mass of people that “think lean” in order to have a “lean culture” that can sustain itself.

Lean leadership (pdf) is an interesting participant. I’ve stayed away from management roles in my career, and in so doing, I’ve come to realize that I can lead without being a manager. Perhaps that sounds obvious, but many people think that that leadership requires authority. This is simply not the case. If someone has a compelling vision, the ability to communicate that vision, and the competency to back it up, then people will share that vision and help to achieve it.

Getting back to my original question…the consensus answer seems to be that lean is sustainable, but it requires appropriate leadership and a new culture. Involvement throughout the organization is required in order to apply the necessary continual adjustments and improvements.

In my own career, I’m leaning toward a more formal leadership role in software. Not as a manager, but in a way that allows me to exercise my skills in a way that can cross internal organizational boundaries. The best fit I can think of right now is Enterprise Architect.

If anyone has other ideas, please let me know.

For me, 2007 was mostly about learning the best way to run a development team in a way that adds the best value to the business.

I became convinced that it is not enough to optimize development locally – to achieve the best value, we must treat development as part of a manufacturing-like process. In doing so, we can understand that requirements are like inventory – they go stale and lose value if they sit around. We also can understand that we need to link the day-to-day tasks we do to business value. Finally, how we define “done” is one of the fundamental questions for any development team.

I already was using Scrum, but began experimenting with Kanban Boards. This showed me that sometimes planning meetings are wasted time. Estimating has value only as much as it assists in release-planning. My newly-won experience is invaluable in knowing how much planning and estimating is appropriate for a given situation. I feel like I could easily fulfill the role of ScrumMaster on any team, or else introduce existing teams to more optimal ways of doing things.

One of the lessons I have learned from being a Software Architect is that you cannot always dictate architecture, but you can guide it by providing simple and effective ways to do things. There are some very strong strategies that can be applied to achieve good long-term viability of software. To this end, I registered a new domain (perfectapi.com), and put up a website on that domain. I did not find the time I needed to do it justice though.

My idea was to try and apply my own hard-learned lessons of software development in a way that improved the architecture for others. This is not a pipe-dream – it has been done before, most recently with Ruby-on-Rails. If you can find the correct paradigms, you can quickly create a community that can be very effective at developing new applications.

In 2005, I had decided not to work on web applications anymore. ASP.NET was just too complex, too barebones to develop seriously on. Ruby on Rails is probably ok, but it has a Unix-like learning curve (I’ve grown used to Intellisense). In 2007, Silverlight was introduced. I have not worked with Silverlight yet, but the feeling I get is that it will change the paradigm enough away from the horrible POST-RESPONSE loop of web development.

Personally, it has been a difficult year. My lone-wolf tendencies have hurt family life, and I find myself with too few friends. I have put on weight. Work was all-consuming at times. It is not that I work long hours – it is just that the hours I work are very intense and stressful. Plus, there is just too much to know – it is a truly exciting period in software development.

I will find a new job in 2008. I hope that it will allow me the freedom to relax more with family and friends.

You may occasionally see the term Shu Ha Ri bandied about in people that think a lot about Agile and Lean. I think Alistair Cockburn was the first to use it in relation to software development.

The idea is that we should teach the beginner (Shu) some techniques, or best practices. Ha is the stage where the beginner has learned that there are some underlying principles, and begins to explore those. Ri is the master stage, where the master adapts and invents new techniques.

“Shu Ha Ri” is implicitly a craft-based way of thinking of software development. The individual words imply the different levels of craftsman, from apprentice through journeyman, to master.

The more interesting aspect of the term relates to communication. It can be hard for Shu-level and Ri-level to communicate. This is because Ri believes there is no single best solution to a problem (everything depends – he will implement a best solution by adapting as he creates it). But Shu needs firm guidance – he needs to be told a good-enough way to solve the problem.

One way for Ri to communicate is using the principle of strong opinions, weakly held. That is, he should communicate as if he is certain that what he says is correct. But it is all an act. He is not certain, and definitely not attached to the opinion.

In short, if I am Ri then I should feel free to present my opinions or theories as dogma.

If the other individual can show a better way, then I will accept that (it’s easy, since I was never attached to the original opinion anyway).

Lean principles teach us to recognize a constraint in the system, and “elevate” (fix) it. I was reading what Amit Rathore was writing about this, and it got me to thinking…

My current work environment is a going-out-of-business software company. Our parent company is continuing in the same market, but they will be using a different piece of software than the one we created. (Due to acquisitions, there were 2 parts of the company creating the same kind of product).

The way our business unit worked is that we would get a sales order for a new deployment of our software. The client would specify their requirements in some sort of RFP. We would then spend time developing the missing pieces (required functionality that was not yet present), convert their existing data to our own format, train the users, and deploy.

In doing this, it appeared to me that Testing and Inventory (time between completing work and Deploying it) were large constraints. In other words, we would build a lot of software, but no-one would look closely at it until we deployed it months (up to a year) later.

It was a decision of the business to expect high quality work, but acceptance-testing that same work was so unimportant that not a single person was assigned full-time to it. Even traditional regression, or smoke testing was not prioritized. To the test-infected, this sounds a little crazy!

But it wasn’t. Because of the RFP (contract-driven) style of the client relationship, there was simply no justification for spending more on acceptance testing, (we did not have sufficient real client input to know for sure that we were building what they wanted). If it meet the letter of the RFP, then it was ok.

During big-bang style deployments, we would run around and fix the high-priority issues (mixture of bugs and changed requirements) until the product was working to the satisfaction of the client. Without built in development quality and Agile responsiveness, this is a nightmare. With those things in place, it is just highly stressful.

The extended deployment periods were our acceptance tests. They were also when we found the most about *actual* client requirements. Ultimately (with exceptions) the client was satisfied (but not necessarily happy). And the Product Manager made sure to build more realistic requirements into the next version.

Back to the failure of the business…

In this model, the satisfaction of the next client is directly proportional to how closely their RFP matched a previous one. This was our downfall, because we came into an expansionist period where each new client was a completely new RFP. We were breaking into new product areas and new geographic areas. And our unit was expensive (because we were developing large amounts of good quality, well designed, extensible software).

In this expansionist period we had great difficulty in creating happy clients, because they would each get the worst possible result – an initial, painful deployment where the best possible outcome was meeting the “letter” of the requirements. (Actually, we did a little better than that, but only through a lot of good, dedicated people doing heroic things).

We tried to slow down the expansion but conditions (new-sales-driven upper management) would not allow this. Although they did not think of it in those terms, management did try to elevate the testing/deployment constraint, by working more closely with new clients. This was done by increasing the size of the client-relationship staff.

In the end, we had some successes (and some failures), and the parent company eventually came to the decision to end our unit. This was not a direct result of our failures, but I am sure that had we been more successful, it would have gone down differently.