Monday, September 19, 2005

Swiming Up the Waterfall

Sometimes, you write something that you have to keep a lid on for a little while, because you don't want to offend someone who might be reading. This is one of those cases.

A fellow co-worker and I found ourselves in a client site who was a big believer in my most favorite of dysfunctional of methodologies, Waterfall. We were hired to help them design an advanced new system (that, of course, interacted with their still very active Mainframe development) with innovative technology. My coworker and I showed up with great enthusiasm (because we didn't know what was going on yet).

Here is what we were tasked to create: First, we need a Functional Design document (a technical-ish set of requirements) to pass off to the business unit to get approval for the project (which was already approved -- that's why we were there). It took man-weeks of effort to produce this utterly useless document, which was created by emailing Word documents around, because they don't put design documents in version control (What? Did I hear that right? Why not? "We just don't" “No, why?” “If it’s in version control, others can see it”). OK.

Next comes the Technical Design Document, another multi-man-week project, where we take the Functional Design and produce a Technical Design, meaning that we finally get to reveal all the technical decisions we made while writing the Functional Design but couldn't put in that document. And we can't produce "prototypes", because that's a bad word -- business people want to see prototypes run. "Can we call them spikes? Sure, no one knows that that means". A few times, I thought my coworker's head was actually going to explode. We set up our own secret version control system and starting writing our design documents (oh, sorry, Design Documents) using XML so that we could diff them, and converted them to Word documents when the big flurry of Consolidating the Documents happened.

Then we made a decision. Given the amount of time that we had to create the Technical Design Document, we could just write the code, using XP techniques (test-first coding, iterative design, etc.) and produce the design document based on the working, tested code. And that's what we did with part of the system. We kept this under wraps (after all, we're supposed to be designing, not coding) and made good progress. To show the quality of the code, I implemented code coverage using Cobertura so that we could demonstrate that the unit tests were actually exercising the code. With just a couple of days left in the Technical Design phase, we were asked to provide estimtes as to how long it would take to implement our design. That's when we let the cat out of the bag: "Do you want us to tell you how long it would take, or how long we have left? We can give you really good estimates because it's already done". We showed them the implementation and the unit tests, along with code coverage (96% code coverage, 100% branch coverage) . The Technical Design document that was already based on working code.

They weren't upset, but weren't particularly happy either -- just kind of stunned. This way of writing software is so foreign that they have no perspective on it. They've never seen code with unit tests, or code coverage, nor any code that was produced in an iterative way. You are supposed to spend months on design documents, then write the code, right? How is it that you can not do all that? Instead of the elation that we expected, we just got a weak "Good job, guys", and they took their working code and incorporated it where it was supposed to go. We were finished with our tasks, so we left.

But not without planting some seeds. You see, we involved some of their other programmers in what we were doing. We didn't code with them, but explained how we were going about it. And we showed them unit tests (which they had heard of) and code coverage (which was new). The other developers clearly saw the benefit of what we had done. Sometimes, you can't dig a deep hole all at once because you have to move too much dirt. But a river can dig a deep hole a little at a time, eroding away grime and stone bit by bit. Even if we didn't dig a deep hole, maybe we created a rivulet.


John said...

Like many other "Methodologies", Waterfall is really a reporting technique, not a tool that one can follow to do development. The higher up the management ladder you go, the more Waterfall fits the way people think.

I know of no software developement project that has actually followed Waterfall (but many, many that used it as the reporting tequnique). [OK, some of the really big Military software catastrophes may have actuualy come close to following Waterfall].

The art is to follow the way you know works, while you document progress according the the proscribed methodology. Or maybe I'm just getting cynical?

Anonymous said...

How are you generating the .doc formatted documents using XML? Apache FOP?

Anonymous said...

Did you consider not immediately showing the code, and offering instead to do a very rapid code development period, using the time instead to 'test' the code with the local programmers? ie: use the time as a mentoring program for their people? It's possible doing this would have made them more involved in the process and allowed them to really take pleasure in the amazingly fast development time - and possibly onsell future business for you with the firm?

Neal Ford said...

I'm using DocBook => WordML, which Word likes

We discussed several different options on when to best "spring" our subversive agility on them, and decided to do it when we did. I've left out lots of political stuff that surrounded this to protect the...innocent.

Anonymous said...


I'm always a little wary about introducing (process) change by stealth as you have described. Whilst it can be extremely successful in the short term, it can be viewed as abuse of trust by the customer, something which can have far longer reaching political ramifications.

Have you seen any other fallout (beyond the muted thanks at the end of your tenure) on that front?

Neal Ford said...

No, actually, our immediate supervisors eventually came around and understood the implications of what we had done. But, introducing proceess change above their head was even more daunting (we were working with the Java group, and they are peers with the Mainframe group).

It was a risk, no doubt, but it was really the only way we could provide reasonable value. The analogy I used: they had hired cario-vascular surgeons to hack off limbs. At the end of the day, we delicately made out point. In fact, other development groups are working on engaging us to apply Agility more directly, if they can get upper management buy-in.

tweedledeetweedledum said...

This blog is awesome! If you get a chance you may want to visit this pos software site, it's pretty awesome too!

Anonymous said...

One good resource for security software and many more free software alternatives is
Yes it is my site and I would love for you to drop by for a second. From there you will have free access to several of my products such as The Marketing Toolbar (which is goldmine of information on how to do things quickly and on the cheap, thus saving you time and money).
Also for webmasters I have created Webmaster Wizards, which will help you with almost every aspect of putting code on your site from things such as legal and privacy disclaimers to popunders. I have so much content and free software there I could fill up this whole blog, so check it out Thanks, Sincerely Rob Rudd

Anonymous said...


I have some really good websites below that can help you with credit repair and legal law forms.
You can also check out the news section at "Latest Legal Articles"

Preparing legal documents has never been easier
with our library of forms, agreements, and checklists. It is valuable to small and growing businesses and entrepreneurs.

Have you ever needed a legal form but didn't know where to get one? Have you ever wanted to cover yourself in a legal agreement or make sure that those you love are protected? is an easy to use online legal forms provider that can and will save you thousands of dollars of lawyer's fees every year.

Repair your own credit , Credit Repair is an ongoing matter for many consumers. Consumers have paid up to $500 or more to credit repair companies who follow this same process to remove their credit errors. Without credit you can't get a car, travel, loan money, or even get mortgage.

Have you been victimized by credit repair companies? Many companies promise or guarantee to clean up your credit report so you can get a car loan, a home mortgage, insurance, etc. In reality, they can't deliver what they promise. You pay them hundreds dollars in fees and these companies do nothing to improve your credit. Many of them just disappear.

Everything a credit repair companies can do for you, you can do for yourself at no cost. No one can do a better job in repairing your own credit as you can. You are the only one responsible to make sure your credit report is accurate. Our guide will help you every step of the way.

If your application for credit is denied due to information supplied by a credit bureau, the company you applied to must be liable to provide you with that credit bureau's contact information. Furthermore you are entitled to a free copy of your credit report if you've been denied credit within the last 60 days

Disputing credit mistakes or outdated items is free. Just ask the credit reporting agency for a dispute form or submit your dispute in writing, attach any supporting documentation. Always send them a copy instead of the original Team

Also checkout our sponsors at: - Get Paid to Drive in UK ---- Only website on the internet that offers paid to drive companies exclusively in United Kingdom

Doniette said...

How would you like to know about a Walking Marathon resource, that displays all the Walking Marathon information on one website? Check out
Walking Marathon today. Not only do they have a good fitness book section, but they also have great articles on fitness. I was really impressed with the resources their webmster put together.