LINQ 201 – Changing the way I work with lists

LINQ brings the set-based power of SQL-like syntax to VB.NET  and C#.  I thought I’d highlight some of the ways this has changed and simplified my code.

Find an element in a list

Pre-LINQ:
function FindItem(key as string)
  For each item in myList
    if item.Key = key then return item
  Next
end function
 
dim foundItem = FindItem(key)
if foundItem isnot nothing then
  ...process item
else
  ...not found
end if
Post-LINQ:
dim foundItems = from item in MyList where item.Key = key
if foundItems.Any then
  ...process item (foundItems.First)
else
  ...not found
end if

Find matching elements in a list

Pre-LINQ:
dim newList = new List(Of SomeClass)
for each item in oldList
  if item.SomeField = "Foo" then newList.Add(item)
next
Post-LINQ:
dim newList = from item in oldList where item.SomeField = "Foo"

Convert a list of items of one type into another

Pre-LINQ:
dim newList = new List(Of NewType)
for each item in oldList
  dim newItem = new NewType
  newItem.Field1 = item.FieldA
  newItem.Field2 = item.FieldB
 
  newList.Add(newItem)
next
Post-LINQ:
dim newList = from item in oldList _
  select new NewType with _
  {.Field1 = item.FieldA, _
   .Field2 = item.FieldB}

Sorting a list

Pre-LINQ:
Way too hard. Suffice to say it involved creating a new IComparer class or
implementing IComparable in the existing class
Post-LINQ:
dim sortedList = from item in myList _
  order by item.Field1

(or you could use lambda expression, but I find the above more readable)

Toyota “Stop the Line” mentality

It was years ago when I first heard of the manufacturing concept of “stop the line whenever something goes wrong”.  At the time, it seemed crazy…why should everyone stop what they are doing just because one person has a problem?  After all, others can continue working, and they’ll just get a little ahead.

Once I started learning lean, I thought that it was about maintaining the flow, and not building up inventory.  More recently, I have understood something else:

“When something go wrong, the cause is almost always inherent in the system”.

In other words, when we “stop the line” it is because we want to understand the cause of the problem, and create a solution that eliminates the cause.

How is this applicable to software?  Developers become inured to day-to-day problems. Few developers would consider saying “stop work – the ASP.NET form life-cycle is too complex for our needs”, or “stop work, the deployment is too manual and error prone”.

There is even a culture wherein individuals get recognition for working around problems, rather than solving root causes.   Let Tom do the deployments, because Joe screws it up.  Anne is good at working with the complex ASP.NET lifecycle, what a great developer she must be!

Its not exactly wrong – Anne is more skilled, Tom is more careful, they should be commended for that.  But they did not go the extra mile – to automate and error-proof the process.  Why would they – they have not been trained to do it, and their manager would probably chastise them for wasting time if they tried.

We have not been taught scientific thinking practices, such as “Plan, Do, Check, Act“.   We’re woefully uneducated when it comes to the basic tools that would allow us to improve our own processes. We are not even aware that it should be our responsibility to improve the process.

Process improvement?  That’s someone else’s job – I’m here to code!

That’s an easy line to buy – when you think of process as a big, abstract thing.  However, if you take a look at the day-to-day things that we do, then there are many, many opportunities to improve.  The real challenge is to have the right thinking tools (PDCA or similar), and to just try.

Quote of the Day – Kent Beck on Responsible Development

Heard on the XP group:

Responsible Development is the style of development I aspire to now. It can be summarized by answering the question, “How would I develop if it was my money?” I’m amazed how many theoretical arguments evaporate when faced with this question.

Responsible Development shares many practices with XP but the roots are different. Responsible Development’s values are honesty, transparency, accountability, and responsibility. These lead me to pairing, test-first, incremental design, continuous integration, and so on because they support the values.

Agile in a non-Agile Environment

I’m into my second month at my new job.  I remain amazed at the amount of documentation that is being generated.  Being a Microsoft shop, almost everything finds its way into Sharepoint.  I’m not against Sharepoint per-se, but I do find it very opaque, i.e. hard to find stuff.

Its like having a lot of interesting papers on my desk, but they are all inside sealed envelopes.  I can open the envelopes, but when I’m done with it, it goes back in the envelope.  Sometimes there are other envelopes inside the envelopes.  As I said, very opaque, and very non-agile.  Contrast with what I am used to, which is a Wiki.  Totally searchable, easy to find what people have been editing.  More like a nice folder with happy little index tabs.

Sharepoint rant over, moving on…

The Business Analysts (BAs) had a slow start, with the result that we started developing our product before they had really written any requirement documents about it.  The result is that they are playing catch-up, and documenting what we have provided, and asking all of the questions that we should have asked while we developed it.

We do not have a single customer, or product champion.  This means that the questions are difficult for anyone to answer effectively.  It also means that the team is pulled in several directions:

  • software architect tries to get us to do multi-tier and service-based and ultra secure at the expense of usability
  • managers care mostly that it is done on time
  • business analysts care about “checking off” features
  • there is very little optimization of features, i.e. everything is equal priority

The result is that we have a fairly well architected, feature-full product that is hastily put together.  Features are surface deep, i.e. we are not “done” to the level that an Agile team would normally consider “done”.  Usability is low, and the product lacks perceived integrity.

Even given all of that, I remain optimistic.  We have a solid team, and every day we work more closely together.  As I understand more about the environment, I am able to make small adjustments to improve it.  We have a lot of time left to deliver, and upper management expects that it will be very much a v1.0 product.

First Retrospective with New Team

Well today we had our first retrospective, a day before the end of our first sprint.  I used the tried & true format of “What went well” followed by “what went not as well”.  We scheduled an hour for the meeting.

As usual, people had a lot of trouble coming up with positive things to say, and not much trouble coming up with negative :)   One person refused to write anything positive!

It came together pretty easily that the topic that concerned the most people was “requirements”, or rather the lack of them.  There was a fair bit of conversation, which could easily have got out of hand.  In fact, it did for a little bit when one person tried to hijack the meeting for a topic other than the one we agreed to talk about.  I redirected that to another 1-on-1 meeting between the 2 of us later.

I tried to keep it in front of everyone that we wanted to come up with concrete actions that could be used to address the problems.  This worked ok to keep the conversation on topic, but no-one really suggested any actions except for our manager.  Being that she is the manager, no-one was seemed ready to contradict her suggestions.  They were good enough action items though, so I went with the flow.

The organization is very document-centric, which bothers me.  Rather than talk about ways that the developers could *pull* requirements through 1-on-1 conversations, there were a lot of suggestions on how we could better use email and documents on SharePoint.  Ugh.  I’ll have to live with the document-centric viewpoint for a while I think.  The best I can probably hope for, short term, is to find ways to make the documents useful to the team.

In the end, we finished on time and came out with some actions that we are taking to address our requirements-concerns.  Nobody was insulted or put off, and the team is fairly positive.  I’m pretty happy with those results.

Bringing Agile to my new team

As mentioned a few weeks ago, I have started a new job. My role is still unclear, but that doesn’t bother me at all – this way I can define my own role.

I’m working in a small team of web developers, trying to start on the next generation of the web portion of our core business product. The team was supposed to be Agile, but was decidedly *not*. I’m working on that, i.e. so far I’m defining myself as the Agile-guru.

We had our first planning meeting that I was allowed to run, and I went with my favorite planning tool (FreeMind). The team is very tool and framework oriented, so I think introducing the tool as part of the planning was less intimidating than if I had just gone at it with story cards.

My purpose in using FreeMind is to enable open discussion, and also to focus that discussion. It allows me to acknowledge everyone’s input, by reflecting it on the mind-map. The tree-like structure naturally allows everyone to be focused on the current area of discussion without being too distracted by other concerns. Once we have everything noted, then we can more effectively make estimates.

We had a very short estimating session. This was difficult, because individuals on the team seemed information-starved, i.e. they wanted to gather all of the requirements then and there. I tried with limited success to focus on just getting rough estimates. To keep it ultra-simple, I allowed only 4 time estimates: 1 day, 2 days, 5 days, or more (needs to be broken down further to estimate). As expected for a new Agile team, estimates were *very* ambitious, and we did not complete.

My final note is that it is interesting to be able to observe the team forming. I was the last member of the team to be hired, but before my joining, the others have barely interacted as far as I can tell. This means that we started in the “forming” stage of team evolution. This is characterised by the our manager being very hands-on, and directive. Basically, we are not a team so much as a set of individuals working under the co-ordination of a manager.

The next team-stage after forming is “storming”. This is a little stressful, and I hope we don’t lose any team members, or the confidence of our manager. I can already see some people coming out strongly on opposite sides of issues, and coming into conflict because of it.

Anyway, that’s my update. Wish me luck.

Bye blogger.com

Sorry if the feeds are all messed up – I just moved my blog to my own hosting provider (it was hosted by blogger.com). I was fairly content with the blogger.com service, but I decided to scrap the website at www.perfectapi.com, and felt I needed something to put in its place. (So the blog will be the default website on my domain now).

New Job

Its been a while since my last post! Some of the reason is that I went on an intense 3 week job search, followed by a new job that I started on Monday. Its weird that I really enjoy interviews – but they did drain my energy levels substantially.

I’m still working as a senior software developer in a team environment (pretty much my goal for this job). I’ve been picking up a lot of new stuff, as the job involves getting back into the ASP.NET world that I had left behind a few years ago. My first task is to design and implement a new claims based (federated) security system that works across a multiple tiered application. Its fun, and I’m making good progress. Yay.

It is also interesting to be in a new environment that tries to be Agile, but is much more limited by environment than it was in my previous position. I hope I will be able to improve what they have started, but for now I have been just observing.

My biggest shock on the new job so far? Seeing VSS again, in active use. Ouch. Its a very Microsoft-centric shop, so at least they are looking at TFS.

Detecting idle state in WinForms apps

Its not my usual thing to post code on my blog, but so many people keep asking me for this, so here it is:

Public Class DetectActivity
   Implements IDisposable
   Implements System.Windows.Forms.IMessageFilter
 
   Private _lastActivity As Date
 
   Public Sub New()
       _lastActivity = Date.Now
       Windows.Forms.Application.AddMessageFilter(Me)
   End Sub
 
   Public ReadOnly Property LastActivity() As Date
       Get
           Return _lastActivity
       End Get
   End Property
 
   Public Function PreFilterMessage(ByRef m AsSystem.Windows.Forms.Message) As Boolean ImplementsSystem.Windows.Forms.IMessageFilter.PreFilterMessage
       Select Case m.Msg
           Case Win32Message.WM_KEYUP, Win32Message.WM_KEYDOWN
               _lastActivity = Date.Now
           Case Win32Message.WM_LBUTTONDOWN, Win32Message.WM_LBUTTONUP, _  Win32Message.WM_MBUTTONDOWN, Win32Message.WM_MBUTTONUP, _ Win32Message.WM_RBUTTONDOWN, Win32Message.WM_RBUTTONUP, _  Win32Message.WM_MOUSEMOVE, Win32Message.WM_MOUSEWHEEL
 
               _lastActivity = Date.Now
       End Select
 
       Return False
   End Function
 
   Public Sub Dispose() Implements System.IDisposable.Dispose
       Windows.Forms.Application.RemoveMessageFilter(Me)
   End Sub
 
   Enum Win32Message
       WM_KEYFIRST = &H100
       WM_KEYDOWN = &H100
       WM_KEYUP = &H101
       WM_CHAR = &H102
       WM_DEADCHAR = &H103
       WM_SYSKEYDOWN = &H104
       WM_SYSKEYUP = &H105
       WM_SYSCHAR = &H106
       WM_SYSDEADCHAR = &H107
 
       WM_MOUSEFIRST = &H200
       WM_MOUSEMOVE = &H200
       WM_LBUTTONDOWN = &H201
       WM_LBUTTONUP = &H202
       WM_LBUTTONDBLCLK = &H203
       WM_RBUTTONDOWN = &H204
       WM_RBUTTONUP = &H205
       WM_RBUTTONDBLCLK = &H206
       WM_MBUTTONDOWN = &H207
       WM_MBUTTONUP = &H208
       WM_MBUTTONDBLCLK = &H209
       WM_MOUSEWHEEL = &H20A
   End Enum
End Class

This simple class can be instantiated when a Windows Forms application starts up, and then you can ask it at any time when the last user-activity was.

This class works better than most other techniques, because:

  • it only detects activity in your own application
  • it is simple
  • it is 100% managed code
  • it works

Do programmers need to learn to program for multi-core?

Much has been written on the inability of the need for future applications to take advantage of dual-core and multi-core systems. I think most of it is hogwash.

Intel is a big proponent of the argument for more parallel-processing tools for programmers. This is no big surprise – Moore’s law has flattened out for the current generation of processors. They are expanding outwards to multiple cores now, no longer upwards to higher CPU speeds. Intel is understandably terrified that their PC CPU market demand is going to dry up. Its all marketing.

I prefer to look at reality.

Looking at whats running on my system, these are the some threading stats:

  • Firefox – 18 threads
  • Outlook – 31 threads
  • Perforce P4V – 8 threads
  • Custom .NET application – 13 threads

Seems like plenty of threading goodness is already going on.

Taking the example of the custom .NET application, there is a very noticeable performance hit when I run it within a virtual machine. I attribute this to the virtual environment only making use of a single processing core. Those 13 threads have to fight over the single processor. In addition, the .NET runtime (garbage collection etc) is highly concurrent – when running on a single-processor environment, it is crippled.

I am not arguing that all application threads are particularly busy. Truth be told, of the 13 threads, only one, sometimes 2-3 are ever particularly busy. I’m just saying that there are as many threads as are necessary to get the job done. As programmers, we need not care at all about multi-core, except where there is some advantage to be had in using it.

Applications are written do as many simultaneous things as the user expects. It has nothing to do with multi-core as to whether there are multiple threads – it has to do with what the user expects and what the application needs to do to meet those expectations.

Multiple CPUs are like RAM – you will never have enough for tomorrow’s needs. Heck, you can barely run Vista without a dual-core machine – the OS itself has too much going on to leave enough over for decent performance. Multi-core PCs are here to stay, but as programmers it is simply a case of “nothing to see here – move along”.