ThoughtShapes Shaping Ideas With Thought

I see a great many projects start off on the wrong foot by trying to decide all of their implementation technologies at the beginning of a project. Usually there’s a project kick-off meeting where everyone flies in from various locations to get introduced to all of the players and decide on things like what database engine are we going to use? What technology will we use to handle our data access? What user interface technologies should we use? Should we use web services?

While it seems like deciding these things up front would be worthwhile, it’s an endeavor that’s fraught with peril due to a number of uncertainties that are common at the start of most non-trivial projects:

• Technology will evolve over the course of the project. The cool new user interface framework will be in alpha test at the start of the project and the features will change by the time it’s released. The marketing hype around the new persistence framework turns out to be just that. We get bought by a company that wants us to integrate with a different technology stack.
• We often don’t know enough about the problem we’re trying to solve at the start of the project. The business may not have a clear business plan yet. The target market may be evolving rapidly. The system requirements will still be in flux because we won’t know everything we need the system to do and we don’t know how the users will want it to do the things it’s going to do. We haven’t learned anything about what it takes to implement the system because we haven’t done any coding yet.

Trying to make these decisions early is an exercise in making decisions with minimal understanding of the problem to be solved and the options available for solving it. “But”, I hear you ask, “Don’t I need to make these decisions now since they’ll drive how we build the application?” I’d like to convince you that you shouldn’t make these decisions now and to let that drive how you build the system.

That doesn’t mean that you don’t have to make those decisions, but it means that you can make them later with better information and still have confidence that the system can accommodate your decisions. And by later I don’t mean late. I mean at the latest responsible moment that allows you to reach your release goals.

So, how would I build the system if I didn’t make a decision about how I would ultimately store the data or interact with the user? I’d begin by focusing on building the classes that represent the domain of the system and look at how they interacted with the boundaries of the system. These boundaries would include those places where I was deferring a decision. Since I’m an Agile guy, I’d go through the exercise of determining which stories would have the highest business value and which would allow me to address the areas of highest risk (quite possibly those areas where I put off making a decision) and address them first. After a few short iterations (probably several weeks of calendar time), I’d have a much better understanding of the technology needs of my application and would be in a much better position to analyze what my technology needs really were.

To be a little more concrete here, let’s say the decision that I had deferred was about how I was going to manage persistence for my system. I’d try to implement several of the high business value user stories that would exercise the persistence requirements of my system. If the problem looked like it was moderately complex (more than just editing data that would be stored in a database), I’d probably use a Domain Driven Design approach to the problem. I’d follow the tenants of Domain Driven Design and use Repositories to manage the lifetime of created objects. I’d start with just an in-memory implementation of the repositories to allow me to do rapid development and testing of the domain model. The in-memory implementations would also ensure that my tests ran rapidly and would provide enough support for early demos to solicit feedback from users and customers.

From this early feedback, I’d continue to evolve the domain model and the interfaces to the repositories. At some point (often surprisingly quickly), the model would begin to settle down and I would be able to use the knowledge I’d gained to determine my real requirements for the persistence layer.

For example,

• If the Domain Model that was developed required heavy use of inheritance and the storing and retrieving a great many types of objects, I’d know that I needed a technology that would support mapping my inheritance model into and out of a relational database, which would drive me toward an ORM solution like NHibernate and away from technologies that didn’t support inheritance well (like the first version of LINQ which seems to only support single table inheritance mapping).
• If my Domain Model only required storage and retrieval of a few objects and had simple querying needs, I might go with LINQ or just roll my own data access layer. I might even use the file system.
• If I discovered that my Domain Model needed to differ greatly from a legacy database schema that was shared with multiple applications, I’d be moved to look deeply into the mapping capabilities of any technology I might use to implement the data access layer.
• If my Domain Model made heavy use of Value Objects, I’d need to look somewhere other than LINQ To Sql (see Jimmy Nilsson for more on this http://www.jnsk.se/weblog/).
• If my application needed significant undo/redo capabilities, I’d probably want to implement the Unit of Work pattern and need a persistence mechanism that supported Unit of Work (or at least one that would cooperate with one I developed).

So, what did I gain and what did I lose by waiting in this case?

I gained:

• The ability to make a more informed decision.
• A more modular design because I was forced to create an interface (the repositories) that allowed me to select from different implementations of that interface.
• The ability to iterate more quickly to arrive at a clearer understanding of the business logic in the system.
• A more testable system, both because my system is modular and because my in-memory tests allow me to run the tests faster and more often.
• The ability to build and optimize the domain model independently of how the persistence of the data is optimized.

What did I lose?

• The false sense of security I got by making the decision of how to persist the data too early.

“But”, I hear you ask, “Didn’t I just increase my risk of finishing the project on time? Aren’t I just going to end up in a situation where I finally make the decision to use a particular technology and then realize that the cost of using it (usually in schedule time) will be too great to recover from?”

No, I’d argue. What you’ve done is traded in invisible unmanageable risks for visible manageable ones.

The invisible unmanageable risk of making the decision too early has to do with how the software gets built if those decisions are made too early. The risk is of incurring Opportunity Costs and being faced with the Sunk Cost Dilemma. These risks will materialize on your project because of the Path of Least Resistance – Software Development Variation and the Kudzu Effect™.

The Path of Least Resistance – Software Variation states that under pressure, developers will take the path of least resistance to achieve their goal (which itself is subject to the Law of Diminishing Expectations). This pressure may take the form of schedule pressure or the pain involved in learning new ways to do things. The end result is often taking shortcuts that enable the developer (or team) to meet a short term goal. These shortcuts often involve bypassing architectural boundaries for the sake of expediency as in “Hey, we can just bind these datasets directly to the controls and we’re done.” Or “If we just generate the classes from the schema, we can get those edit screens up for that first iteration.”

A few instances of taking the path of least resistance quickly leads to the Kudzu Effect in which those underlying technologies begin to take over the application like Kudzu has taken over the American South (see http://en.wikipedia.org/wiki/Kudzu). Very quickly half the classes in the system seem to know about (i.e., have dependencies upon) what should have been an implementation technology that was only meant to be used to keep the access ramps (see Kudzu) of the system working properly.

The Opportunity Costs are the costs of doing what you did versus what you might have done with that money or resource. In this case, it’s the cost of having each developer spend the time and effort to understand the underlying technologies that have now crept throughout the system versus having them focus on learning the domain of the application and getting further along as team in understanding the domain (which in turn increases the odds of building a successful system and provides you with information to choose the most appropriate underlying technology).

The Sunk Cost Dilemma surfaces when the team finally understands enough of the system’s requirements and what needs to be done that it’s apparent that the initial technology choice isn’t going to work out. And the dilemma is that now you know what you should do, but so much of the initial technology choice has found its way into so many nooks and crannies of the system that there is no way to pull it out and replace it, and even if you could do that, so much of the schedule time and development budget has been spent getting to this point that you’ll never meet your release date.

The end result is probably familiar. Some dedicated people will find a way to slog through getting something working mostly the way it needs to work. And they’ll get there later (maybe a lot later) than they (and their customers) had hoped.

“Okay”, I hear, “If those are the invisible manageable risks, what are the visible ones that I can manage?”

The visible risk is of choosing the implementation technology too late to implement it. You can start to manage that risk by what you do at the kick-off meeting.

So what would I do at that kick-off meeting?

I’d create an initial risk management plan. These technology decisions that you’re deferring aren’t the only risks you’ll have in that plan (unless you’re really lucky), but for the technology risks I’d do the following:

• I’d identify the areas that I needed to make technology decisions about.
  • For example, how I was going to persist classes (assuming I was creating a domain model).
• I’d estimate the latest time that I could make that decision.
  • As an example, I worked on a project where the persistence layer implementation that actually wrote to the database (and the database design itself) was done in a few weeks for quite a large and complex model (for a trading application). The time at which the decision could be made might be later than you would first think.
• I’d identify someone technical to get up to speed on each of the various options.
  • Preferably someone who would be working on the development team as a developer that would be involved in developing the system.
• I’d make remaining neutral with respect to the piece of technology a design requirement and ensure that the development team knew to isolate that decision behind the appropriate interfaces.
  • Repositories in our persistence example.

And then what…?

Assuming I’m using a typical Agile development plan with multiple short iterations (e.g., two weeks), I’d allocate appropriate tasks to drive decreasing the risk. This could vary from ensuring that stories that would drive our understanding were implemented, to creating prototypes using the technology, to sending someone to a class or having them search the web for the information we needed. And I'd timebox those efforts so they didn't become resource sinks.

I’d review the risk plan at the end of every iteration. This would include updating estimates of when the decision needed to be made and ensuring that we were learning what we needed to in order to make a good decision. Then I’d repeat the process. And finally, I’d make the decision at the latest responsible time. And I’d be making it on much sounder footing.