Thoughts in the Wilderness

This is not the simplistic analogy you may be expecting.

Where I live in Minneapolis MN, we have a high profile bridge project going on right now, due to the tragic collapse of the previous structure. If you look at the project page on the state’s website, you can see the following “features of the new bridge”:

• 100-year life span
• 10 lanes of traffic, five in each direction—two lanes wider than the former bridge
• 189 feet wide—the previous bridge was 113 feet wide
• 13 foot wide right shoulders and 14 foot wide left shoulders, the previous bridge had no shoulders
• Light Rail Transport-ready which may help accommodate future transportation needs
• Design-build project complete in 437 days.
• Designed to be aesthetically pleasing and fit in with its environment

These are the most high-level (public) stakeholder values for the new bridge. From an engineer’s perspective, they are the constraints under which the bridge must be delivered. In addition, we see an architect’s rendition (picture) of what the new bridge will look like. This is also a constraint – an engineer cannot add things that will substantially change the appearance of the bridge.

Now, we can be sure that the design of the bridge was a collaborative effort of a team of people. Engineers, Marketing, Architects and the client. Is the design ongoing as they build the structure? Yes – they are using a growing technique known as Design-Build – they purposely start construction before the design is complete.

At first glance, design-build might sounds like a simple case of parallel work – one team is working on designing just-in-time, and another is working on the construction. In practice though, there is a collaborative environment that reportedly results in avoidance of disputes, faster project delivery, and less need for project management oversight.

The Analogy
So how is this similar to building software?

Firstly, the process of programming it is like the design of a bridge – it is the bringing together of people in different roles to creatively find ways to build the end result. Ideally, development involves a lot of Thinking, Talking, and Tweaking, just like a bridge design. In design, we often find that two heads are better than one. Pair-programming has been suggested as one way to do this in software development. Of course, we have many other collaborative techniques to communicate and discuss design.

Like a bridge design, the output of building software can be represented by piles of paper. The bridge has drawings, engineering specifications and requirements. A program has something better though – its code. (No, I’m not arguing that “the code is the design”. I’m just saying that the code “is a representation of the design”). The code accurately describes the parts of the design that it touches.

This “programming code = bridge design” point is key to what I’m trying to convey – the process of programming produces a design output, not a product. The final product is the result of implementing that design (just as the bridge itself is the result of implementing its design).

Specifically, the “building” is the deployment of software in its final environment. Deployments are where the “tires meet the road” – they are the intersection of the design with reality (just like construction). Mostly, the design holds up and does not need tweaking after deployment. Sometimes though, the harsh lights of reality expose the hidden flaws in the design. (In light of that, it is best to expose an application to its first deployment as soon as possible).

Some software groups have QA (quality) departments. Historically, these departments have taken the role of performing trial deployments – they will take the software, and expose it to a simulation of the real environment. Large construction projects also have this role – an independent group audits the designs, with the hope of spotting problems that would cause a problem when the construction occurs.

Finally, we find that the best way of constructing a large bridge project is to simultaneously design and build. The analogy for software is small frequent releases. Research and experience has shown this to be a good way deliver quality software that meets the requirements.

Conclusion
If we accept that building a bridge and building software are similar (they contain the same basic steps), then we can use that information to produce some interesting insights:

• That thing we need to do before developing is “architecture” - There is a fine distinction between architecture and design. The way I like to define it is that architecture describes the parts are visible from the outside, and design describes the inside. A bridge architect is able to construct a working model and rendition of the outside of a bridge without the full engineering specs. To do this, he needs to take into account all of the stakeholder values. Similarly, we need to be able to draw the edges of a software application before we start – we need to understand how the software will interact with the outside world, and how the outside world will interact with the software.
• QA is a misnomer – the primary purpose of a separate QA department should not be to assure quality. We can get quality in better ways than that. The purpose of the QA department should be to validate the design of the software, by simulating real environments. Many QA professionals already know this, of course.

This blog post is inspired by a set of three essays by Jack W. Reeves.

I have been lurking in a recent discussion of using [Transaction]-like attributes in C# to indicate that certain methods can participate in, or require a transaction. Castle ActiveRecord has a another technique of allowing the user to specify a TransactionContext, like

using new TransactionContext
{
...do stuff that will automatically be in the transaction
}

The problem with all of these techniques is that they are essentially procedural. Specifically, anything that you want to participate in the transaction has to be manually called as part of the call-stack. Put another way, they fail to separate the concerns of ATOMic persistence and the identification of what needs to be persisted (the unit of work). The result is that it becomes difficult to implement some aspects of persistence, leading to an increase in artificial complexity.

For example, an aspect of saving a deposit into an account is that there should be a “dual” entry in another account, and balances must be updated. A single aspect like this is somewhat manageable using the proposed semantics, but if you have just a few more, then they quickly lead to very wordy, procedural and possibly complex “Save” methods. You will also end up adding additional state variables to classes that contain “Save” methods, in order to support the logic of the save.

Another way of looking at this is as the problem of the typical “business entity” class that simply does too much. There are cross-cutting concerns that do not belong in one “business entity” or another. That is the major weakness of the ActiveRecord-style of data access – when you take the world-view that every business entity is a table, then you encumber your ability to clearly work with the aspects that are orthogonal or cross-cutting to the entities.

My own solution (there may be better ones) is to explicitly expose the unit of work, and have a technique that allows class instances to intelligently enlist into it. Its worth describing in a little more detail. First, you need an interface that a class can implement to enlist in the work:

Interface IWorkEnlistee
Sub Participate(work as UnitOfWork)
Readonly Property UniqueKey() as String
End Interface

The Participate method is called just before the database Save, but after validation of user-data. The UniqueKey property is necessary to prevent two identical instances from participating (I usually just return the hash-code of some entity instance). You could add methods to the interface to get greater functionality, such as in-memory rollback.

For the dual account entry example, I would have an instance of the above interface that participates by adding the reverse entry and updating the balances. Any state data it needs will be passed in the constructor, which is called before the in-memory data is changed (so that it can get a clear before-picture). It will probably have several related state variables that otherwise would have found themselves complicating some other piece of code.

Using this technique, the entity, presentation and flow logic of the application remains clean, and the cross-cutting aspects that participate in transactions are nicely separated and encapsulated.

There are several universally accepted software architecture laws. These have the characteristic that if you heed their principles, your software will be of better quality and will last longer. These are the ones I am aware of:

Law of Demeter
aka principle of least knowledge, aka only talk to your immediate friends, aka low coupling and high cohesion.

Separation of Concerns
The notion that it is better to allow the code (and the developer) to concentrate on one concern at a time. This is the mother of many other principles, for example layering, or splitting software along logical lines.

Conway’s Law
Any piece of software reflects the organizational structure that produced it. The cause is more sociological than technical. The antidote is better communication, or smaller teams.

Shalloway’s Law
aka DRY. “When N things need to change and N>1, Shalloway will find at most N-1 of these things”. I like Shalloway’s version, because it manages to capture the essence of DRY, with the added subtlety that it is ok to duplicate stuff, as long as you don’t have to manually change it.

My first rule of good software architecture dealt with ways of making a particular code-base last a long time. The focus of the 2nd rule is different – it assumes that a problem domain will be solved multiple times by different software, or multiple versions of the same software. It suggests ways that we can make each new re-solving of the problem easier than the last.

To review, my 2nd law of good software architecture is:

Keep as much information as possible in an accessible, declarative form. This will eliminate duplication, and enable your software to be discarded and re-written without losing quite as much

This law is all about re-use of information, and describing how a particular problem domain can become better understood, even to the extreme where the “software” is just data.

We’ve all seen the tool-sets for generating entire applications – enter your requirements (mostly just your data structure) using vendor X’s WonderMaker(tm) and lo and behold, out springs an application with handy generated forms for doing wonderful things. As it turns out, those wonderful things are pretty much Create, Read, Update and Delete. Not so useful after all.

Those generic tools do solve particular problems well – but it is usually not the problem we want to solve. Understandably, users demand more than just create, read update and delete – they want to use the software to perform some task that meets their goals.

We can achieve the goal of a tool that generates most of an application, but only once we understand the problem domain well enough. We need to understand the domain, because we need to know what we can generate, and what we need to leave open to extension.

It is always a mistake to design a v1.0 system where logic is executed based on models of application logic. There are plenty of horror stories about the architect who thought he could model the business logic using XML. Don’t be the next one. This post is about evolving your understanding of a particular domain to the point where you can create models with confidence they will work.

That said, even in version 1.0, there are some things we can recognize. The first is that there are at least two easily identified models of the system. From the user’s perspective, there is the model that they understand and interact with. At the other end, there is the database. The important logic of the application sits between the user’s model and the database. This is the origin of the old 3-tiered concept – UI + Application + Database.

What we have to realize is that we can model each of these things in a way that is declarative. In version 1.0, we may not understand the way the user wants to use the UI well enough to do much work in this regard. However, we can certainly model the database in declarative form, and have that model persist after version 1.0.

Modeling the Application layer is the last evolutionary step. You will not reach that point until the core application requirements are stable and well-understood.

For existing products, the process of modeling requires a re-write of some portion of the system. This is unavoidable, because you have to extract information from where it is hidden in the code, and represent it outside of the code. The code will no longer work. The good news is that once you have correctly modeled a part of the system, the model can be extended to capture new types of information, and need not be re-written again.

A re-usable database Model
So what does a re-usable database model look like?

• It treats relationships as a first class concept – they have names, and they have attributes (one-to-many, cascade-delete behavior).
• It describes fields in a rich, descriptive manner. Strings have maximum lengths, phone numbers are represented by a phone-number data type, etc.
• It describes lookups (sets of values that are acceptable for a field)
• It describes roles – how field values come together to represent a particular flavor of record that has meaning to the user. A particular flavor may be extended with additional fields and properties.
• It should be directly and easily accessible to the rest of the code (re-usable).
• It should be able to be transformed into something that the data access layer (or ORM tool) can use directly.
• It should be able to be transformed into an empty database (it is complete).

The most obvious storage form of the model is as XML, because it is very accessible, and because it can represent hierarchical data. Other forms are ok, as long as they meet the above criteria.

Why all of the richness? We want to capture as much information as possible in a single place. This allows us to make use of that information at higher layers of the application, in ways that enhance the user and the developer experience. DRY (Don’t Repeat Yourself) is a powerful architectural technique.

Not all database structures represent the model we wish we had. We may have inherited a database, and it may be a horrible thing to behold. My first law of good software architecture applies – since we are exposing the model directly to the developer, we want it to be the one we wish we had. If the real database is too far from what we want, then we need to take steps to address that inconsistency. To do otherwise is to invite artificial complexity in the application code.

A re-usable UI Model
As mentioned previously, I do not expect that many version 1.0 products have a very good UI model. Still, if we can understand what a UI model looks like, then we can work towards it.

Firstly, a UI model has a relationship with the database model. The relationship is mapped – i.e. there is some automated transformation that can be used to relate a field on the UI back to one or more fields in the database. This is important, because the relationship is what allows us to re-use information defined at the database model (such as rich data types, lookups, maximum field lengths etc). If we’re re-using information, then we are not duplicating it.

A UI model can grow in pieces. First, you can model screens, then larger pieces that describe how various screens fit together. Screen models are the easiest. (Even today, many applications make use of screen models).

Again, XML is a good choice for representing the UI model.

Beware of including layout information in the UI model. That is a different aspect that belongs in a different model. The primary purpose of the UI model is to bring together fields and screens in a way that represents how the user sees them. This may include their likely order on the screen, but should not include their actual co-ordinates.

UI models are re-usable in several ways. Security, Form Design, and Ad-hoc user queries are a few.

Layout of Forms (Views)
Form layout can be defined declaratively, but it is seldom worth the trouble to do that manually. We cannot predict the next evolution of UI well enough to design a representation that is good enough. The best you can probably do is favor form-design tools that save themselves declaratively (for example, XAML).

Your form layout should make use of the UI Model directly (via data binding and control-binding). Otherwise, you are just duplicating yourself. (Control-binding is the technique of having the final appearance of a particular control determined based on metadata. See the screen shots in my Egg UI post for
an example).

Form layouts can be generated. This is a dangerous path, because it can limit your ability to satisfy the needs of the end-user.

Security
It is particularly useful to relate security to the UI model. One reason is that security is highly contextual – whether a user has rights to touch particular data elements can be driven by many factors, including the time of day. Another reason is that users need to understand security in order to effectively define it. The UI model’s shared understanding of the user’s perspective allows a good point of interaction for security.

Ad-hoc user queries
Often, we may want to expose the ability for users to query a database in some way that is fairly dynamic. A UI model that is mapped back to the database model provides a simple way to provide that feature.

First, I should touch on the intent of good architecture. The intent is to build something of quality that will last a long time. The “something” we build will not be static – it will be changed, and should be amenable to those changes without loss of quality. Small applications are easy to replace, rather than change – so good architecture is most relevant to medium to large applications.

To review, my first law of good architecture is:

Identify all core services to the application. Code against the interface of the service you wish you had, not to the implementation of the one you actually have.

Now when I talk of services here, I am specifically *not* talking about SOA. I am talking about all the pieces of your application that are not business logic. In this context, a “core service” is pretty much everything that is not the business logic itself. This includes the entire user interface, the database, the file system, and the application settings.

User Interface
Lets talk about the user interface as a service. This is a little-known technique, so I’ll take the time to motivate it as best I can.

Consider this statement: Logic naturally wants to be at the points of control. By default, the main point of control of an application is its user interface. This is why it is so hard for developers to keep it out of there! For non-visual applications, this is still true – the logic wants to be on the edges. From an architectural perspective, this tendency is very dangerous – the user interface is the most likely part of the application to be discarded, and the hardest (practically impossible) to re-use.

One well-known technique for limiting the damage is layering of the user-interface on top of the application logic. With discipline, this can work well. However, good architecture does not assume discipline. It assumes team members of average talent at best, and structures the application so that they are as effective as possible. Layering is not the best answer.

Given that logic wants to be at the point of control, we can make a conscious decision to put the application logic in control. This will make the other parts to the application subservient to the application logic. As it turns out, that is the definition of a service – a part of the application that is subservient to another.

Another important characteristic of a service is that it has a well-defined API. So well defined in fact, that we can define its interface, and code against that interface rather than the actual service. So score 1 for the user interface as a service – it makes automated testing of business logic easy.

Of course, the user will still interact with the application – clicking on menus, entering data, and generally driving the flow of the application. However, they will be doing that within the context that the application logic has defined and supplied to the user interface.

In practice, this is a lot easier than it sounds. You can evolve the interface as you develop the application. Modern inversion of control techniques make it easy to inject the actual user interface at the time of execution, and to supply an appropriate context that the user interface can operate within.

The File System
Some pre-built services, such as the file system are very broad. Do we create an interface over that entire surface? No. We code against the interface we wish we had. The file system may provide the implementation, but we would be introducing unnecessary complexity if we dealt with the file system directly.

Settings
My own view is that you can combine settings with the user interface service. This is because settings can often be user-choices in one implementation, and settings in another, and hard-coded in yet another. There is no perceivable downside to having the user interface implementation control the settings.

The Database
It turns out, the most difficult aspect of the application to make into a service is the database. A database is like a pool of data. Most times, the interface we wish for is to be able to scoop up the data with a bucket, play with it, then throw the data back into the pool. A simple data access layer can be good enough for this.

Sometimes we want more – for example, we may want to have data access run on a different application server, or be scaled across multiple servers. We may want to provide the ability to have the application run disconnected from a server. Or we may want to totally insulate the application from the data structure or vendor. These are all up-front choices we must make. All come with a cost. In the more expensive cases, the interface we wish for will be more service-like than a simple data access layer would.

If we do have to make data access into a service, we should still be sure to make it the service we wish we had. This implies that design of the service should be driven based on the needs of the application.

Conclusion
Good architecture puts the important logic at the center and treats the less important logic as subservient (services). We code against the services we wish we had, because to do otherwise introduces artificial complexity. (The implementation of the services can take care of translating back to the reality of the underlying provider).

“User Interface as-a-service” is a new concept. I have implemented it with success, although I didn’t understand it then as well as I do now. Others have too – Cockburn’s hexagonal architecture is a similar concept to what I have described. I think I will be writing more about it in later posts, because I have treated it too high-level here. People will want to know how to actually do it before they believe it is a good idea.

For the longest time, I have been trying to characterize the UI design I have been using for the last few years. I think I finally have a handle on explaining it. I hope you find it as interesting as I do.

To summarize up front…the Egg UI pattern is a technique for creating a rich user interface that is decoupled from specific application logic, but coupled to a large piece of infrastructure code. Some business value is invested in a common infrastructure, enabling the business to quickly add more modules that behave in similar ways. For a particular module, business value is invested in application logic, where it can be re-used independent of the infrastructure or UI. Almost zero business value is invested in the actual UI for a particular module.

If it sounds like I have frameworkitus, hold off on the judgment for a minute. A framework can be bad, because it risks coupling of your application logic to the framework. The Egg UI pattern does not do that. The UI is an egg, and the application logic is an egg. I’m calling them eggs, because they are self-contained (as opposed to layers, which have one-way dependencies).

In the diagram above, direct dependencies are shown as solid lines, and indirect dependencies are shown with dashed lines. The defining characteristic is that the UI is provided as a stateless service to the application. Everything else flows from that. The target platform is a rich-forms environment, for very large, modular applications that display and edit lots of data. (It may work for other environments too, but I have only used it in the one).

Some components:

• Presenter – responsible for applying form-level logic, providing field metadata, and validation.
• Menus - represent possible user actions. These may be rendered on the UI as buttons, or menus. They have captions, and metadata describing their required context.
• Commands - represent the details of the actions that menus execute.
• UI Service Interface – defines all of the activities that can be requested of the user interface. Also defines all settings that the application may need, and methods for sending messages to the user.
• UI Service Implementation – an implementation of the user interface. Uses data binding and infrastructure code to interact with the context (mostly the Presenter and the Menus)
• Views – Simple data forms, or pieces of more complex forms.
• UI Model - metadata, representing a shared understanding of the structure of the data in the user’s view of the system. Provides a means for the Views to be bound, and a mapping of UI fields to the database.
• Context - A holder for any context that the UI may need. Includes a minimum of the Menus, the Presenter, and other supporting methods. The Application Egg owns the context, but the UI Egg can see it and add to it.

So what is this pattern good for? I’m glad you asked.

Most importantly, User Interface and Application logic are decoupled from each other as much as is feasible. This pattern is almost at the extreme end of user interface decoupling. Any further and the forms would be drawing themselves (not a good thing, in my experience).

This decoupling provides an environment where it is very, very obvious to the developers where their code should go (hint – a presenter or a command). We can partially or completely re-work the UI infrastructure (e.g. Winforms => WPF) without concern for the application logic. We can extend the application logic (e.g. add additional user choices or change the types of fields) without touching the user interface code. We can test the Application without being concerned with the UI.

We can also repeat the pattern over and over in many modules that together comprise the application as a whole. In other words, it is amenable to vertical layering of the system, a factor which increases the workable size of the application by at least an order of magnitude.

There are many benefits, but there is also a big one-time cost – a significant amount of infrastructure (framework) code. This is necessary for any pattern where you want the user interface to be dumb. (And this user interface is particularly stupid). The UI needs to be able to act as a reflection of the application logic. This requires an investment in components that can read metadata and use that metadata to extend on the “drawn” user interface.

For example, this is a screen shot of a form in design mode:
Here is the same form at runtime:
And this is the user-code behind the form (the presenter contains all the meaningful code):
And here is the grid from which the form was accessed (no user-code):
And this is an intentionally blurred image of the context in which the grid was accessed (to demonstrate that this works at multiple levels, not just a simple master-detail example):
The infrastructure code has taken metadata, and used it to show labels, buttons, menus, images, treeviews, icons, dates, times, and dropdown controls. It also applies security, handles validation errors and generally gives a very rich user interaction experience. Unfortunately, this sort of power requires an investment. To me, that investment represents direct business advantage – in the ability to provide a unique, consistent experience with the richness and stability the users demand, while still leaving the door open to future possibilities.

Meditate on these, and you may achieve some enlightenment

Steve’s first law of good architecture:

Identify all core services to the application. Code against the interface of the service you wish you had, not to the implementation of the one you actually have.

Steve’s second law of good architecture:

Keep as much information as possible in an accessible, declarative form. This will eliminate duplication, and enable your software to be discarded and re-written without losing quite as much.

Corollary to Steve’s second law of good architecture:

A particular application domain is effectively solved (and no longer requires custom code) once all information about the application can be represented in declarative form.

Another Corollary Steve’s second law of good architecture:

When using a tool to generate some or all of an application, ensure that the declarative data of the tool is stored in an accessible form.

Steve’s third law of good architecture:

Usable components may evolve, but practical, re-usable components must be designed.

Corollary to Steve’s third law of good architecture:

Component re-use is only practical once you have designed an approachable, stable interface to the component.

Enlightenment Image by Sakka, licensed under Creative Commons ShareAlike version 2.5

See Matt Blodgett’s First Law of Software Development

A development process that involves any amount of tedium will eventually be done poorly or not at all.

I like that. To me, it is yet another argument for DRY (Don’t Repeat Yourself), which I consider to be the most important aspect of long term software quality.

If you are doing DRY, then you are not repeating yourself. Therefore, you are doing the least amount that you can in order to solve the problem. Any tedium is thus inherent in the problem, and could not be avoided.

(Of course, if you find or invent the right tool, you can also mitigate the remaining tedium. For example, using a diagramming tool to draw your database relationships rather than typing them in XML or SQL).

Is…

The separation of user interface (screens, forms, web pages) from application logic.

My current estimate is that 1% of programmers understand the abstraction and apply it successfully.

If 100% of programmers could make this leap, then I predict software quality would improve 1000x.

Enough said.

So you develop a framework for a project, and it is great. All is good. It does what it needed to, and it does it well. Maybe a few unexpected requests come in, and you manage to incorporate them into the framework. You are very satisfied. Other people are impressed too. Co-workers on other projects start to notice, and they want to use part of your framework to make their own project easier.

Except…that part that they want to re-use has some dependencies on other parts of the framework, that they do not want. “No thanks”, they say.

Sound familiar? Perhaps you have been the co-worker, asking after the framework? Was the framework wasted? Is re-use unachievable in your organization?

The fact is, the only way that re-use has been shown to work is to “componentize”. A framework is just *too big* to do this with.

The best you can do is ensure that your framework has many individual parts, each of which has potential for re-use. Even then, coupling is a big challenge. The parts become dependent on each other (for good reason).

The missing piece of the puzzle is packaging (componentizing). Only when you do this can you truly achieve re-usability across a broad range of projects. This is not as trivial as the word “packaging” implies. There is documentation, removal (or internalizing) of dependencies, retrofitting it into the existing system, versioning and deployment. There is also the challenge of letting others know it is available.

When your co-worker came and asked you for part of your framework, the sad truth is that it was already too late. Packaging takes time, and the other project needed its answer today.

Your organization needed to be more pro-active. It needed *someone* to notice the potential of the situation, and have the time and the resources to make something of it.