Evolutionary Design...

Quite a remarkable thing occurred with the FPS project and I thought I would document it before I forgot the details.  The project started over a year ago and has since completed with universally acknowledged success.  We had an excellent team consisting of 1 project manager, 5 programmers, 1 x-programmer who knew the business logic through and through and most importantly we had engaged customers:  engineers to be exact.

There are several reasons why this project, in my opinion, was a success:

1. Engaged Customers.  I list this first because it is the most important.  They new exactly what they wanted, had "skin" in the game, and were willing to meet and verify every change we made.
2. Testing philosophy.  Our 80% + JUnit coverage helped us do radical refactoring (the main reason for this post) right up to the end of the project.  I was amazed at the extent of changes made each iteration.
3. Agile project management (Scrum).  Our project manager was very focused and kept us focused.  
4. Weekly code reviews with teeth.  We would go over every change (quickly) and record what needed fixing.  We would even follow up...
5. Evolutionary Design...

The last one I would like to elaborate on...

When someone buys an engine from us it has a stated horsepower.  This horsepower can be increased with some additional iron added to the engine, and a modification to the ECU.  The modification to the ECU is done using passwords and such and we charge for these upgrades.  FPS is a money maker at Caterpillar, but we don't just let anyone do it.  We have the EPA and foreign countries to satisfy and we certainly don't want to just give stuff away, so there is plenty of security behind all of this.

We designed our object model based on the ECU.  The input is a bunch of information that allows us to recreate the ECU as a model object.  This ECU is then validated and billed and passwords are generated.

The ECU model started out as a Java Map of values that was sent throughout the application and was acted upon.  It did have some responsibilities such as data hiding and such, but was pretty anemic.  Early on I knew I wanted it to have responsibilities, but we were not sure what they would be or even what the nature of the ECU itself was.  Our project had 2 main phases and in the first phase our ECUData didn't need much validation.

Step One:  I can tell you if the input I have been given is valid, and I can create a password string...

 Phase One went out with a bang and everyone loved it.  Phase two began and we slowly got requirements and started coding.  I have always been one to not worry about performance.  Dan Long told me long ago, "Make it work, Make it pretty, Make it fast".  In that order, please and stop when people stop complaining.  There are lots of verifications we had to do to decide if and when to charge someone for a password.  We have to find out if the request is valid (can we upgrade the engine from 50hp to 5000hp?), if they have already been billed for this upgrade once (folks don't like paying twice), and find out who and the amount to charge.  Some of these decisions are very complex and take many database calls to do.  As a matter of fact, one time through our app can trigger over 200 database calls. (wow!)  One of the team member was quite adamate that we need to make each database call only once.  I however, didn't care how many times we called for duplicate values because it was very fast already and (according to Dan) we shouldn't worry about that yet (defer decisions).  The discussion got pretty heated several times and was starting to influence the design.  We discussed creating a "reference book" object that would hold all this stuff that could be toated around our app and referenced when needed (kinda like a cache).  We (Tim and I had a discussion by my desk about it) finally decided on a MetaData map in our ECUData object that would hold this stuff.

Step Two:  I can tell you if the input I have been given is valid, I can create a password string, and I can tell you some stuff about the data I have been given (metadata).

I thought this was a great idea.  We started stuffing data in there and bam, our code became simpler.  Work progressed and we discovered that everywhere we asked for some metadata, we would have to find out if it had already been fetched (if (metadata.contains(productId)) else get it).  This code was everywhere and was making our code base a bunch of "ifs".  Tim (again.  I wish I could think of these things) came up with the idea of "providers".  So we code up a provider and then we can just ask the ECUData object, "Hey, do you have the product ID for this serial number?"  The ECUData object would look in the metadata cache, and if it found it, return it.  If it didn't find it, it would look in the database (using the provider) and get it, stick it in the cache, and give it to you.  GENIUS!  It was as if my eyes had been opened!

Step Three:  I can tell you if the input I have been given is valid, I can create a password string, and I can tell you some stuff about the data (metadata) and go get it if I don't know it already.

Finally, I don't have an anemic data model!  The amount of code this eliminated was astronomical.  We looked for example, for serial number productIds everywhere (some things 3 different places) and now all that code is in one place and we don't care if we have it already or we need to get it for the first time.  Since we had a great set of JUnits, this code change was trivial (I believe it took about 4 hours).  Me and another programmer did all the work in an afternoon and there was much rejoicing!  We also spread the philosophy of providers to the rest of the team and everyone jumped in.  We now have 12 providers and love them.

Since we now have a smart object, we looked for other ways to make it even smarter.  There were an abundance of comparisons made to the ECUData object and we refactored those into Wrappers that we inserted at create time.  This eliminated another bunch of code.

Step Four:  I can tell you if the input I have been given is valid, I can create a password string, I can tell you some stuff about the data (metadata) and go get it if I don't know it already, and I can compare myself to other things easily.

All of the significant design changes to our ECUData object came in the second half of the overall project, required significant code changes, did not take much time to implement because of extensive JUnits and customer functional testing, and because we had engaged programmers, they jumped in to help..  In his article Is Design Dead? Martin Fowler calls these items "Enabling Practices".  I believe this is the first (and hopefully not last) time in my career when all the stars aligned!

When the project began I was given a document prepared by an engineer that contained a flow chart and tens of pages of documentation on how to Decode the ECU parameters.  This document was changed several times by the customer and I implemented it pretty much as it was given to me.  The JUnits on this thing were exhaustive and we have had no trouble with it so far.  However, there are no Model objects involved;  it is entirely procedural coding. I have often thought of how I would create a Model  to implement it.  I think it could be very cool and flexible (currently it is not flexible.  It does one thing, does it well, and nothing else).  About 2/3rds the way through writing it I was getting brain cramps because of how procedural it was becoming, but there was no turning back.  I believe it is an illustration of the design being done up front.  When this decoder was done, we discovered that it could not create an alternate ECUData object (we have Standard and Legacy).  We had to code, from scratch, a different decoder to handle the other one.  I believe that had we created a Model for the decoder, we could have one decoder to create both types of data.

All things cannot be designed using Evolutionary Design.  I built a deck this year on my house.  I used Evolutionary design (really!).  I had many problems.  I was always fixing/correcting things I did in the previous phase.  It turned out OK, but could have been much better had I known what I was doing up front.  I believe software is different:

If you can easily change your decisions, this means it's less important to get them right.  (Fowler)

We have a unique opportunity with software.  If the stars align!