Guerrilla SOA (SOA & The Tarpit of Irrelevancy)

This is the sixth in a series of blog posts where I discuss what I see wrong with SOA (Service Oriented Architecture) in the way that it's being sold by vendors. For those who are behind, you can read the previous installments here:

• Tactics vs. Strategy
• Standards-based vs. Standardized.
• Tools & Anti-behavior
• Rubick's Cubicle
• Triumph of Hope over Reason

In all the previous posts, I've been basically elucidating all the reasons why I think that most SOA projects are a quagmire: mis-understanding the goals of why you are involved in this approach, the way most vendors are poisoning the water by selling ridiculously complex tools and services, why it's so seductive for developers who fetish complexity to get knee deep in these projects, and hype. If you read all these posts back to back, you'll surely have the impression that I think all hope is lost for enterprise integration.

But it's not.

Just like any software project, it is possible to do SOA right. My colleague Jim Webber has done lots of outstanding work in this area, under a rubric called Guerilla SOA. The basic ideas are:

• treat SOA projects like any other project, and apply the same agile principles: continuous integration, testing, simplicity


• don't use tools just for the sake of using tools. Try to keep the tool stack as simple as possible, and as automated as possible


• use loosely typed endpoints and document/resource orientation to allow the different parts of your architecture evolve independently

It's best said from Jim's own mouth. Jim's British, so when he curses in technical presentations, people just think it's quaint (whereas when I do it, it's crass).

This is the way we generally approach SOA project: like other pursuits. SOA doesn't have to be a huge scary thing. It's just software, and we aren't going to throw our playbook out the window just because it sounds scary.

The term SOA has been so co-opted by vendors trying to sell stuff that I think it will die off as a term. We'll still be doing soa (note the lower case letters), but we'll have develop another three-letter acronym because the old one has slipped into the tarpit of irrelevancy.

Speaking at the Colgne JUG Monday, April 20th

I'll be speaking at the Cologne Java Users Group on the eve of the JAX Conference, on Monday, April 20th at 7 PM. I'm letting the JUG organizer pick the topic, so I'm not sure what I'll be talking about, but I'm looking forward to it: I've never spoken at a German JUG before. If you're in the neighborhood, stop by and we'll geek out, then have a real beer afterwards.

Update: Here is a link to the slides I presented at the JUG. Thanks for having me; I had a great time.

Real World Refactoring in NFJS the Magazine

Several people have asked me what ever happened to the NFJS Anthology book series (The NFJS Anthology, Volume 1 and The NFJS Anthology, Volume 2: What Every Software Developer Should Know. Both books contained essays built around some subject speakers were passionate about that year. Alas, the publishing business being what it is, there wasn't enough demand to justify continuing the series.

After much discussion, it was decided that the series would be more dynamic in magazine form rather than book form, which explains the formation of the NFJS the Magazine. This is a monthly publication written by NFJS speakers about something they are talking about this year, and of course something they are interested enough to speak and write about. Being in an magazine format makes it a bit easier to keep up to date, and the volume of material is higher because you get several articles a month.

I have a Real World Refactoring talk in the upcoming issue, based on my talk of the same name this year. If you go to a No Fluff, Just Stuff show, you get a free copy of the magazine, but anyone can subscribe

RailsConf Interview by Chad Fowler with Paul Gross and me

One of the marketing tools that RailsConf uses is a series of interviews with upcoming talks. Chad sent some interview questions to Paul and myself around our upcoming talk Rails in the Large: Building the Largest Rails Application in the World, and he's posted it on his site. Want some opinionated conversation about Rails and how to use it to build Enterprise applications, then here it is (and we didn't use an Scala!). Enjoy!

The Triumph of Hope over Reason (SOA & The Tarpit of Irrelevancy)

This is the fifth in a series of blog posts where I discuss what I see wrong with SOA (Service Oriented Architecture) in the way that it's being sold by vendors. For those who are behind, you can read the previous installments here:

• Tactics vs. Strategy
• Standards-based vs. Standardized
  
• Tools & Anti-behavior
  
• Rubick's Cubicle
  

A very funny site shows that this Internet thing might not be a fad. The Chuck Norris Facts web site has lots of great hyperbolic claims about Chuck Norris, American actor and legendary bad-ass. Some of the "facts":

• If you have five dollars and Chuck Norris has five dollars, Chuck Norris has more money than you.
• There is no 'ctrl' button on Chuck Norris's computer. Chuck Norris is always in control.
• Apple pays Chuck Norris 99 cents every time he listens to a song.
• Chuck Norris can kill two stones with one bird.
• When the Boogeyman goes to sleep every night, he checks his closet for Chuck Norris.

Some of this may be just a tad exaggerated. I'm pretty sure that when Chuck Norris does pushups, he is not in fact pushing the earth down instead of pushing himself up. The site is here, if you want to go read more of them. I'll wait.

OK, now that you understand more about Chuck Norris, here's another site of over-the-top exaggeration about an over-hyped subject: SOA Facts, modeled after Chuck Norris Facts:

• SOA is the only thing Chuck Norris can't kill.
• SOA invented the internet, and the internet was invented for SOA.
• SOA is not complex. You are just dumb.
• SOA can always win at TicTacToe. Even if you go first.
• One person successfully described SOA completely, and immediately died.
• In a battle between a ninja and a jedi, SOA would win.
• SOA knows what you did last summer, and is disappointed that it wasn't SOA.

I used a bunch of these in one of my SOA talks as bumper slides between the various topics, which provided a nice fun icebreaker. But I reserved two of them for the last part of the talk because I think they aren't exaggerations at all, merely deep truths:

• Implementing SOA for the first time is the triumph of imagination over intelligence.


• Implementing SOA for the second time is the triumph of hope over experience.

SOA has gotten so complex, with so many moving parts, that getting it right is extraordinarily difficult. Once you've lived through one of these projects (especially if you've fallen into the other tarpits I discuss in the previous installments), you understand the first quote at a deep level. That you would try it again truly is the triumph of hope over reason.

Rubick's Cubicle (SOA & the Tarpit of Irrelevancy)

This is the fourth in a series of blog posts where I discuss what I see wrong with SOA (Service Oriented Architecture) in the way that it's being sold by vendors. For those who are behind, you can read the previous installments here:

• Strategy vs. Tactics
• Standards-based vs. Standardized
• Tools & Anti-behavior

Developers love to solve puzzles. One project manager with which I used to work kept a jar full of little nail puzzle (like this) on his desk:

Any time he was having a conversation that he didn't want developers to listen in on, he'd grab one of the puzzles and toss it to them. Inevitably, the developer would grab the toy and immediately become totally absorbed in solving the puzzle. After about 10 minutes, the puzzle would yield up it's secret, and the developer would look up and ask "Did anything important just happen?"

Developers tend to be problem solvers -- it's one of the appealing things about fiddling with computers. But what happens when you take a natural problem solver and present them with dull work, with no interesting challenges? What happens frequently is what I've deemed the Rubick's Cubicle anti-pattern.

If the presented problem isn't complex enough, developers will figure out ways to make it complicated and therefore challenging.

Writing the same dull CRUD application over and over is boring. But what if you could figure out a way to get all the simple CRUD applications to talk to one another? That's a nice and juicy puzzle. This perhaps explains the complexity fetish I see in so many "Enterprise" architectures and applications. Some of it is accidental complexity, accrued from years of piecing together parts that were never meant to work with one another. But I don't think accidental complexity covers the entirety of why things are so convoluted.

I remember back in the mid-90s, I was the CTO of a small training and consulting company. We were absolutely delighted when we first saw EJB: here was a technology no one could understand without extensive training. The same thing happened with all the variations of COM, DCOM, and CORBA. Those were flush times for training companies because we knew that developers would be drawn like moths to a flame, frequently with the same result.

Building the simplest thing that can work is sometimes duller than crafting some devilishly complex Rube Goldberg machine, but keeping it simple is a worthy challenge in it's own right. If you find yourself in Rubick's Cubicle, stop and ask yourself: "is there a simpler way to do this? Perhaps dismantling something that no longer serves it's purpose? What is the simplest thing that could possibly work?"

Emergent Design & Evolutionary Architecture at DeveloperWorks

For the last few months, I've been toiling away on an article series for IBM DeveloperWorks, and it's rolling out today! From the abstract for the series opener:

This series aims to provide a fresh perspective on the often-discussed but elusive concepts of software architecture and design. Through concrete examples, Neal Ford gives you a solid grounding in the agile practices of evolutionary architecture and emergent design. By deferring important architectural and design decisions until the last responsible moment, you can prevent unnecessary complexity from undermining your software projects.

The first two articles in the series appeared today:

• Evolutionary architecture and emergent design: Investigating architecture and design


• Test-driven Design, Part 1

I plan to use this series to start a conversation about something that we all do everyday that we can't really describe well, even to other technical people (much less our grand parents). I don't supposed to know the answers (I'm not even sure I know all the questions), but at some point we have to talk about it. In the first installment, I provide some working definitions and some overarching concerns. Let me know what you think about it.

Speaking at the IT Architect Regional Conference in Atlanta

At the end of February (the 25th - 27th), I'll be making a rare Atlanta conference appearance at the IT Architect Regional Conference, hosted by the International Association of Software Architects (IASA). This is the first in a series of regional conferences focused on an important but generally neglected segment of the developer population, software architects. What that title actually means is some matter of debate (hey, maybe this conference will help define that term), but the topics covered certainly trod some important ground. I'm doing a short version of my Smithying in the 21st Century keynote (overviewed here). My ThoughtWorks colleague Steven "Doc" List will also be there, imported from the west coast, convening an open space called Beyond Fight or Flight: Meetings Don’t Have to be Gladiatorial Combat, which sounds quite interesting. It's still not too late to sign up for it; hope to see you there.

Tools & Anti-Behavior (SOA & the Tarpit of Irrelevancy)

This is the third in a series of blog posts where I discuss what I see wrong with SOA (Service Oriented Architecture) in the way that it's being sold by vendors. For those who are behind, you can read the first installment and second installments.

While rank and file developers go to conferences to soak in deep technical content, their peripherally technical managers (the ones who wrote some rockin' good Cobol code back in the day, but now they make decisions about modern enterprise architecture) go to different conferences in Palm Springs. At those conferences, they have a 2-hour morning session, run by a big tool vendor, then play golf for the balance of the afternoon. And what the vendors show them is poison.

Mostly what they see these days are tools that support SOA and ESBs. And in particular, the favorite demo-ware application is their BPEL (Business Process Execution Language) designer. This designer allows you to wire together services by drawing lines between boxes. The lines can include transformations and other sexiness. And it demos great. "Look, just draw a couple of lines here and here, click on the Run button and voila! Instant SOA".

Then, the manager brings it back home and notifies the developers that this is the new tool of choice. When developers start using it, they realize the awful truth: they've been sold a hairball generator. Tools like this work great on really small scales, when it's easy to see all the connections between things. But, as things get complicated, they start suffering from the hairball effect: all the lines start running together, and you can't create a diagram that makes sense to anyone anymore. Perhaps maybe you can fight through this, by creating workflows in chunks, and zooming in and out.

Then reality arrives. Because you create workflows using these tools, you are coding, in the worst possible language (a graphical representation). Thus, you are defining behavior, just like you do when you write source code. But the behavior you define lacks all the benefits you get from writing it in code.

• reuse: you can't really reuse portions of your workflow because their is no method or subroutine functionality (you might get lucky with a sub-workflow). Mostly, people achieve "reuse" by copy and pasting, which you never do in code.

• refactoring: no refactoring, making it harder to identify common workflow chunks for reuse. When you don't have refactoring, you don't watch for opportunities for refactoring as much.

• limited programmability: you don't get if statements and for loops, you get whatever this particular BPEL designer supports. You get flow-chartly looking stand-ins for real decision statements, but they are much more brittle than the facilities offered in modern languages.

• testing: you can't write unit, functional, or integration tests for these workflows. The only real testing option is user acceptance, meaning that the entire universe must be up and running. If you have no unit testing, you also don't have mock objects or other testing techniques common in code.

• hard to diff: lets say you fought the beast and get a non-trivial workflow up and running, and everything is great. In six months, you change it in non-trivial ways, and all is good. Then it comes time to see what's different. BPEL tools don't have diff facilities, so you can either visually diff the diagrams (yuck) or diff 2 10,000 line XML documents (double yuck). BPEL relies on either heavy-weight diagramming tools or raw XML, and nothing in between.

Tools like this fall into the category one of my colleagues identified as doodleware tools. They let you create pretty pictures but collapse under scale. And they don't support all the common facilities offered by good old fashioned code. Is it really worth giving up robust reuse, refactoring, testing, programmability, and versioning/diffing just to see the pretty picture? Ironically, it's pretty easy to generate a similar picture from code, using tools like GraphViz.

I am a strong believer in the mantra

Keep behavior in code.

We have great tools for code (including ways to generate doodles) -- why would you want to abandon what works for something new and shiny? Except, of course, that code won't take you out for a golf outing in Scotland if you choose it.

On the Lam from the Furniture Police at Agile Atlanta

I have two keynotes this year that I'm presenting at one conference or another. The first out of the gate was my talk entitled On the Lam from the Furniture Police at the Code Freeze conference. This talk has multiple parent ideas. The first derives from the book The Productive Programmer. It turns out that when you write a book about something, you get really immersed in the subject matter. That's obvious. What's less obvious is the inability to turn that overwhelming interest off after the book is done. That can either be good or bad. Once I had finished Art of Java Web Development, I pretty much never wanted to see another web framework! But the Productive Programmer was different, I guess because it's such a broad and unsolved problem. For whatever reason, I'm still soaking up Productive Programmer stuff.

The other parent of this talk involves the intersection of agile software development and productivity. While there's a little of that in The Productive Programmer, I wanted to keep the focus mostly on developer productivity in the book. But that's where On The Lam from the Furniture Police fits. This talk discusses how to be productive in corporate environments, including broader topics like agility. Here's the abstract:

When you were hired by your current employer, you may think it's because of your winning personality, your dazzling smile, or your encyclopedic knowledge of [insert technology here]. But it's not. You were hired for your ability to sit and concentrate for long periods of time to solve problems, then placed in an environment where it's utterly impossible to do that! Who decides that, despite overwhelming evidence that it's bad for productivity and people hate it, that you must sit in a cubicle? The furniture police! This keynote describes the frustrations of modern knowledge workers in their quest to actually get some work done, and solutions for how to gird yourself against all those distractions. I talk about environments, coding, acceleration, automation, and avoiding repetition as ways to defeat the mid-guided attempts to sap your ability to produce good work. And I give you ways to go on the lam from the furniture police and ammunition to fight back!

Every time I make changes to this talk, it's to enhance it's agile focus. So it's perhaps not surprising that I'll be giving a preview of this keynote at Agile Atlanta on Tuesday evening, February 3rd. If you're in Atlanta and interested in how productivity and agility intersect, stop by. And if you can't make it there, the next scheduled appearance is in Las Vegas, as the opening keynote of The ServerSide Symposium.

Why You Should Attend RubyRx

I attend a lot of conferences as a speaker, covering Java, .NET, Ruby, and Agility (plus a few other random topics). The most obvious differences between all these conferences are the technical topics, but other surprising differences exist as well. One of the most striking is the presentation style and content of the Ruby conferences. The quality of the presentations at Ruby conferences is nothing short of stunning. They tend to have not only fascinating topics, but really compelling presentation skills. It's as if everyone at Ruby conference have read and internalized Presentation Zen by Garr Reynolds. No bullets in sight, lots of images, cool transitions, and just generally entertaining. It helps that virtually everyone at Ruby conferences use Macs, and therefore Keynote, which is light years ahead of PowerPoint. Keynote alone won't make a great presentation, but it certainly helps.

Not surprisingly, the worst conferences are those that force the speakers to use hideous conference-themed slide decks. Some conferences force you to create your presentation using a hideous template, then have some clueless intern enforce their misguided rules (up to the point of making changes to the slide show). The worst by far is Microsoft's TechEd. First, you must use PowerPoint (not too surprising, I guess). But last year I took a lot of time making slides with lots of images, centered text, and other transition tricks...only to have them disappear before my talk at the hands of someone who didn't have a clue. They literally took slides upon which I had big, centered text to emphasize a point and changed them to a slide with a made-up title and a single bullet point. This ham-handed treatment of my material is why I'm not doing TechEd this year.

But I digress. Ruby conferences on average have stellar presentations. And the RubyRx conference in Raleigh in late February sounds like it's going to be the exemplar of outstanding talks. It combines the expertise of the No Fluff, Just Stuff conference organization with a great lineup of speakers, including Matthew Bass, David Bock, Chad Fowler, Stuart Halloway (talking about Clojure!), Yehuda Katz, Russ Olsen, Jared Richarson, Venkat Subramaniam, Bruce Tate, and Glenn Vanderburg. And I'm going to be there too, like a kid at the adults table.

RubyRx sounds like it's going to be the great early 2009 Ruby conference. Come see what the fuss is about, both technologically and presentationally.

Upcoming Keynote: Smithying in the 21st Century

I have two keynotes that I'm giving at various conferences this year. The first out of the gate was On the Lam from the Furniture Police, which I debutted at the Code Freeze conference in Minneapolis -- I'll have more to say about this keynote in a future blog post when it's about to appear again.

Smithying in the 21st Century comes next. I love metaphorical titles, and this one harks back to the changes in the blacksmith profession just after the turn of the century (in this case, the 20th one). If you were a blacksmith 1890, you had a terrific career path. However, once automobiles came along, the profession gradually diminished to a shadow of what it was.

Here's the abstract:

Blacksmiths in 1900 and PowerBuilder developers in 1996 have something in common: they thought their job was safe forever. Yet circumstances proved them wrong. One of the nagging concerns for developers is how do you predict the Next Big Thing, preferably before you find yourself dinosaurized. This keynote discusses why people are bad at predicting the future, and why picking the Next Big Thing is hard. Then, it foolishly does just that: tries to predict the future. I also provide some guidelines on how to polish your crystal ball, giving you tools to help ferret out upcoming trends. Don't get caught by the rising tide of the next major coolness: nothing's sadder than an unemployed farrier watching cars drive by.

I'm debutting some form of this one twice in 1 day! I'll give a short version of this keynote at the International Association of Software Architect's ITArc Atlanta Regional Conference as 1/2 of the opening keynote on Friday morning. Then, I get on a plane and fly to Milwaukee, where I'm scheduled to give the first real version of it as part of No Fluff, Just Stuff Greater Wisconsin Software Symposium. The research has been a blast, and I'm looking forward to putting it in front of people. There should be some interesting surprises lurking...

Standards Based vs. Standardized (SOA & the Tarpit of Irrelevancy)

This is the second in a series of blog posts where I discuss what I see wrong with SOA (Service Oriented Architecture) in the way that it's being sold by vendors. The first installment is here.

Back in the very early days of Java web development, interacting with the tools sucked. Every servlet engine vendor had their own deployment scenario, and they varied wildly between vendors. I worked on several projects where we moved a web application from one dysfunctional servlet engine to another (this was in the early days, and they all sucked in one way or another). Then, the J2EE standard came along, including the WAR file format. Suddenly, you had a fighting chance (well, after a few iterations of the standard) of deploying to multiple application servers (notice how "servlet engine" became "application server"). And the vendors hated that.

They hated it because the J2EE standard turned their application servers into commodities. And the price of commodity software quickly approaches zero. That's why the application server vendors immediately started building more elaborate stuff on top of J2EE: portlet engines, content management, elaborate load balancers, etc. They knew that something like JBoss would come along and support the J2EE standard, making it impossible to charge $17,000 per CPU to deploy their application server (this number sticks out because one of my clients in the early 2000's paid WebLogic exactly that...for 8 CPUs). The J2EE standard essentially ran the server vendors out of the business of supplying just an application server.

Contrast that with database vendors. They are still able to charge big bucks for their servers, and keep companies (sorry, Enterprises) on the upgrade treadmill. An ANSI standard for SQL exists but (and this is the key part) it is so weak that it's useless. When last I checked, the ANSI standard for SQL didn't even include indexes, and no one would reasonably deploy a database server without indexing. The database vendors dodged the commoditization bullet by ensuring that the SQL standard would remain toothless in perpetuity.

Now, let's apply this to Enterprise Service Buses. One of the bits of marketing mantra touted by all the vendors of these Rube Goldberg machines is "standards". And they're selling these things into Java shops accustomed to the J2EE standard, so companies equate "standards" to "works the same across all players". All these shops are thinking of standards under the J2EE light, not the database server light. But here's the rub:

 ESBs are standards-based but not standardized.

This distinction is important. All the extant ESBs use standards in every nook and cranny, but it's all held together by highly proprietary glue. The glue shows up in the administration tools, the way their BPEL designer works (along with their custom BPEL meta-data), how you configure the thing, how you handle routing, etc. The list goes on and on. These vendors will never allow the kind of standardization imposed by Sun in J2EE. The last thing the vendors want is to see their (crazy money making) babies turned into commodity software again. They'll make noise about creating a true standard, but it won't happen. They want to be more like the database vendors, not the application server vendors.

Even the open source offerings in the ESB space suffer a little from this because they are giving the bits away and selling consulting and training. This is a good business model (look what it's done for JBoss), but they have the same motivation to keep you locked into their version of proprietary glue.

Having proprietary glue is not necessarily a bad thing. It's one of the factors you have to consider anytime you are thinking about turning part of your infrastructure over to a externally developed piece of software. Obviously, no one is going to build their own database server -- it makes good sense to buy an existing one, and fight the nasty battle if and when it comes time to move to another one. BUT, you need to understand the distinction between standards-based and standardized so that you don't buy yourself into a real long term nightmare.

Tactics vs. Strategy (SOA & The Tarpit of Irrelevancy)

This is the first in a series of blog posts where I discuss what I see wrong with SOA (Service Oriented Architecture) in the way that it's being sold by vendors. The first installment is about how the need for SOA arose: tactics vs. strategy.

No company starts out as an Enterprise, they all start as a company, with just a few employees. As such, their IT needs are small, handled by a small group of developers who can chat with each other over lunch about what's going on in the IT "department". As the business needs software, they have some process to get the requirements to the developers, and they write code. Thus, the accounting department comes to the developers and says "Here are our requirements for our new accounting application", and the developers build some version of it. Inside that application are some small parts of something that the entire company cares about, for example, some of the aspects of customers. Meanwhile, the marketing department comes to the developers and says "We need an application, and here are the requirements", and the developers build it. Of course, this application also encapsulates some aspects of a Customer (not the same as accounting, but possibly with some overlap). It is rare indeed that anyone looks around and tries to come up with a comprehensive strategy for application interoperability: you don't have time for that -- you're a small company, and if you don't get your software, you'll go out of business. This goes on for a while.

Then, one day, you wake up, and you're an Enterprise. The CIO looks around with dismay at all the actual entities the corporation cares about because they are scattered in bits and pieces in all these different siloed and packaged applications. And the database stuff is a mess, with shared tables and views and ETL jobs running on cron jobs. And the CIO throws up in his mouth, just a little. The CIO looks at the landscape, and realizes that the technical debt incurred over the last few years can only get worse from here, so he calls out "Help." Big software vendors are highly attuned to people in big companies (sorry, Enterprises) who can write checks saying "Help". They ride in with a solution wrapped around the SOA moniker. More about our friends the Enterprise vendors in a future part of the series.

What I'm concerned about in this post is the overall landscape, which is another way of asking "How did you get where you are now?" You got here because of 2 reasons: first, you took the path of least resistance when you were a company (before you became an Enterprise) because, if you had taken the time to build a comprehensive strategy, you'd have never survived as a company. Second, and more important, what is strategic to the business is always tactical to IT. Business people can go into a conference room and change the entire direction of the company in 2 hours. Or, the business may decide to merge with another company that has truly dysfunctional IT, but other business concerns override that. IT can never move as fast as the business, with means that IT always has to respond tactically to the decisions and initiatives brought forth from the business. No matter how much effort you put into a comprehensive, beautiful, well-designed enterprise architecture, it'll be blown out of the water the first time the business makes a decision unlike the ones that came before. The myth of SOA sold by the big vendors is that you can create this massively strategic cathedral of enterprise architecture, but it always falls down in the real world because the COO (and CEO) can override the CIO (and his sidekick, the CTO). If you can convince your organization to allow IT to set the strategy for what capabilities the business will have long-term, you should. However, your more agile competitors are going to eat your lunch while you build your cathedral.

Any enterprise strategy you implement must realize that you will always be in tactical mode in IT because the business strategy doesn't require physical labor. Any enterprise architecture you develop must allow the business to evolve according to it's wants (and needs). This is what my colleague Jim Webber calls "Guerrilla SOA" and what I call "Evolutionary SOA". More about the details ofp evolutionary SOA in upcoming installments.

Conferences in Emerging Tech Countries

I speak at lots of conferences in the US and Europe, and they pretty much have the same feel (modulo different languages, both programmer and spoken). But I also speak at a few conferences in emerging technology markets, like my recent conference appearance for JAX-Asia, a Java-centric conference that happens over the course of three days in Singapore, Kuala Lumpur, and Jakarta. Attendees at these conferences are very much like attendees at conferences elsewhere: slightly geeked out, interested in learning about new stuff, and inquisitive. They also exhibit a lot of enthusiasm: there aren't as many conferences there as here, so going to a conference is a bigger deal.

I have two observations from my recent trip. I first observed (with the help of my colleague, Erik Doernenburg) that all the conferences in Asia have lots of sponsorships. Generally, attendees pay very little (if anything) to attend the conference; most of the cost is offset by vendor sponsorships deals. But along with the booth in the exhibit area, vendors get one or more keynotes. Now, I don't want to completely denigrate vendor keynotes. I've seen a couple that had actual content, not just blatant marketing. But, for the most part, the mode in Asia seems to be outrageous over-the-top marketing, making them not so much keynotes as "marketing-notes". Because most of the attendees aren't yet jaded and cynical, they stick around for the marketing-note, succumbing to the danger that their marketing bullshit detector is not as well honed as ones in the US and Europe. They might actually believe what this vendor is saying about SOA -- what a frightening concept! To their credit, the conference also books in regular sessions that offset some of the damage caused by toxic marketing-notes. One of my talks at JAX-Asia was my Evolutionary SOA talk, which is pretty much the polar opposite of what most vendors say about SOA. One of the reasons I like speaking at out of the way conferences like JAX-Asia concerns around this very issue. I'm very interested in providing my (and ThoughtWorks) perspective on technology and methodology to offset the message of people just trying to sell something. I genuinely want to advance the state of software development, which has no geographic boundaries.

The second observation from JAX-Asia is the span of sophistication of the attendees. Just like any conference, you get a fair number of wide-eyed innocents, who just discovered Ant last week and want more cool stuff like that. To those attendees, preaching about dynamic languages and other cutting edge stuff is pointless: their primary interest in the conference is shoring up their current development practice, not spelunking into the future of technology. But it's a mistake to assume that all the attendees are beginners. Like any conference, a wide variety of skill sets and experience show up. For example, for JAX-Asia, I did a talk on JRuby, and another comparing Groovy & JRuby. JRuby was one thing: at least a few of the attendees had heard of Ruby. But the comparison talk was brutal on both me and my audience. No one had even heard of Groovy, so discussing the nuances of the differences between a technology represented by a word they had heard ("Ruby"), with another technology for which no one was familiar ("Groovy") was tough. Yet, generalizations are dangerous. After my talks in Jakarta, I was chatting with one of the attendees, who turns out to be the president of the Ruby users group in Jakarta, which has 50 members! To me, this shows two things. One, never assume that the crowd to which you are speaking in homogeneous. Even if the majority are novices, chances are good that there are some experienced developers there as well. And, two, Ruby has managed to penetrate pretty far and wide. A 50 person Ruby users group in Jakarta encourages me greatly because it shows that the foothold that dynamic languages have in the US and Europe is spreading to other parts of the world.

Irrational Artifact Attachment

The lowly whiteboard is one of my favorite tools for design work on projects: you can stand in front of it as a group, you can easily play "what-if" games with emergent designs, and you can argue until everyone agrees (or at least until everyone is equally unhappy). Once you've got it done, a quick snap with a digital camera and you've got a project artifact, ready to post on a wiki or similar until supplantation by actual code. Once you have real code, you are better off allowing the design to continue to emerge from it rather than trying to keep the two in sync. Alternatively, you can use a reverse engineering tool to produce a prettier version of the original diagram from the code.

I prefer this low-tech approach to the more formalized version using tools (from drawing tools all the way up to really formal tools like Rational Rose) because of the proportional relationship between a person's irrational attachment to some artifact to how long it took to produce. If you create a beautiful UML diagram using some tool like Visio that takes 2 hours, you have an irrational attachment to that artifact that's roughly proportional to the amount of time invested. That means that you'll be more attached to a 4 hour diagram than a 2 hour one. By "irrational attachment", I mean that it's harder to listen to reason as to why it's wrong because you know how much time it took to create it (and therefore the required effort to update it).

This applies to pretty much any software artifact. I've seen this effect for design documents and diagrams, architecture, requirements, use cases, and database schemas. This suggests that really elaborate tools favor one-shot designs, where the pressure is on to get it right on the first pass. Of course, you can change it, but how much willpower exists to do that? One of the benefits to the low-ritual approach to much of agile software development revolves around creating "just in time" artifacts, with as little ceremony as possible (this helps explain the dedication to lots of agilists to index cards and sticky notes). Using low-tech tools frees you to throw away what's not right, freeing you to experiment and allow the true nature of the artifact emerge through revision, collaboration, and discussion.

Generative Ambient Music: Bloom

One of the techniques I describe in The Productive Programmer is focus, distancing yourself from the all too common distractions in modern office environments. I suggest that you can wear headphones (or earbuds) while coding as an indicator that others shouldn't bother you. Some developers can code to music (in fact, some developers have a hard time not coding to music), but others fine it distracting. If you are in either group, you should try ambient music. The goal of ambient music is to create non-offensive background music. I find Muzak-style background music incredibly annoying and distracting, partially because they always choose sappy music (or taking good music and sappifying it) and partially because of a personality quirk described here. Ambient music strives to create a sonic background that not only doesn't distract but qualifies as good music. You can search for ambient music as a category on Amazon. Ambient music frequently uses techniques borrowed from Minimalist music, where you create a melody (which is too stong a term for some minimalist music) that repeats with minor variations.

Brian Eno is one of the pioneers of ambient music. He and Robert Fripp recorded the track The Heavenly Music Corporation on the album No Pussyfooting in 1972 by combining some experimental tape loop techniques from Eno, combined with Fripps guitar loops (called Frippertronics). No Pussyfooting is still available, and quite enjoyable. But the interesting part about it for this post concerns how it was created. Once Eno and Fripp had all the equipment set up, they started the tape loops and recorders for the background track and retired to Eno's front room and drank tea while the album recorded itself. No Pussyfooting has some early examples of generated ambient music (they were certainly not the first to experiment with generated music, but the first in a main-stream context). Eno and Fripp created several albums together. Eno has created a large catalog of ambient music (including the classic Music for Airports, recorded by several ensembles including the incredible Bang on a Can All-stars), and Fripp expanded the ideas behind Frippertronics using electronics for a stunning collections of recordings called Soundscapes.

Which brings me finally to Bloom. Bloom is an iPhone application created by Brian Eno and Peter Chilvers. When you launch Bloom, it starts a gentle drone sound as a background, and gives you the option to create ambient music by tapping the screen. When you tap, a note plays based on the location of your tap. You can tap single notes or chords. After about 5 seconds, the note repeats and continues while it gently decays. Using Bloom, you can create your own ambient music. I can play with Bloom for hours. It is one of the best iPhone applications I've found: a trully innovative application that takes great advantage of the iPhone interface.

The other option when you start Bloom is for it to generate music for you randomly. That brings us back around to focus. Using Bloom, you can generate unique, non-distracting music for an entire day. Rather than buy a collection of ambient recordings, you can generate your own ambient music. Bloom lets you set several "moods", changing the tonal range and drone sounds to add just enough variety in the music to keep it just interesting enough to not distract. What used to take tons of studio equipment now runs on the iPhone. This makes a great way to help enhance your focus in noisy environments. Allow Bloom to create music for you that doesn't distract yet effectively drowns out all the other distracting sounds in your cube-ridden environment. Bloom is a great example that the iPhone isn't just another mobile device: it's a new platform for software development.

Comments == Code Smell

I am sometimes asked about my position on code comments, and, like most things, I have strong opinions about it. Two kinds of comments exist:

• JavaDoc-style comments (which encompasses JavaDoc, XMLDoc, RDoc, etc), which are designed to produce developer documentation at a high level (class and method names and what they do)


• In-line comments, generally scattered around the code to indicate a note from developer to developer

Both kinds of comments represent different smells, each with different odors depending on the target.

What makes comments so smelly in general? In some ways, they represent a violation of the DRY (Don't Repeat Yourself) principle, espoused by the Pragmatic Programmers. You've written the code, now you have to write about the code. In a perfect world, you'd never hove to write comments for this purpose: the code will be expressive enough that someone who reads it will understand it. Two things help achieve this result: expressive, readable languages and the composed method pattern.

The language makes a big difference as to the readability of the code. If you write in assembly language, you're pretty much forced to write comments; no one on earth can read the code directly. As languages have matured, you can get much closer to the ideal of self-documenting code (which was explicitly attempted in Literate Programming). Especially in the modern wave of non-ceremonious languages (like Ruby, Groovy, Scala, etc), you can craft extremely readable code. To this end, ThoughtWorks projects generally avoid the more magical features of languages (like the implicit global variables in Ruby, for example) because it hurts readability. One of the side effects of dynamic typing is outstanding method names. The method name is the only vector of information about what the method does (you can't crutch on return or parameter types), so methods tend to be named much better on the dynamic language projects upon which I've worked.

The second key to readable code is not to have too much of it, especially at the method level. In Smalltalk Best Practice Patterns, Kent Beck defines composed method (which I write about extensively in The Productive Programmer). Composed method encapsulates the idea that all your methods should do one and only one thing, making them as small as possible. The side effect of this discipline leads to public methods that mainly consist of calls to a large number of very small private methods, each of which do only one thing. Composed method has lots of beneficial side effects on your code: small methods are easier to test, you end up with really low cyclomatic complexity for your methods, you discover and harvest reusable code chunks more easily, and your code is readable. This last one brings us back to the topic of comments. If you use composed method, you'll find much less need to have comments to delineate sections of code within methods (actual method calls do that), and you'll find that you use better method names.

Now let's talk about how this applies to the two types of comments. First, where are comments indeed useful (and less smelly)? If you are writing an API, you need some level of generated documentation so that people can use your API. JavaDoc style comments do this job well because they are generated from code and have a fighting chance of staying in sync with the actual code. However, tests make much better documentation than comments. Comments always lie (maybe not now, but on a long enough timeline, all comments will become outdated). Tests can't lie or they fail. When I'm looking at work in progress on projects, I always go to the tests first. The comments may or may not be there, but the tests define what's now done and working.

We were on a project where the client insisted on Javadoc comments for every public class and method. We started the project adding those comments, but eventually stopped. When doing agile development, you don't want anything that hampers refactoring. Having comments in place caused a dilemma: do I refactor the method and change the comment (the most work), refactor the method and leave the old comment (with the theory being that I'll change it again later, and would rather just have to update the comment once), or not refactor? No good options here. Having pervasive comments discourages refactoring because it adds significant extra friction. On this project, we abandoned commenting as we went along. The last week of the project we literally did nothing but go back and add comments to the code, which worked well because the code base had settled down by that point. But notice what's lurking in wait: the client wanted all the comments there to make it easier to maintain in the future. But who's to say that whoever maintains that code will keep the comments up to date? You have the same DRY violation as before. If they maintain the tests, they have to change as code changes because they are executable.

The new wave of Behavior Driven Development tools (like JBehave, RSpec, and easyb make this "executable specification" style of comment + test feasible. I expect to see the usage of these tools skyrocket because they give you documentation that has a fighting chance of staying up to date.

Inline comments are almost always a smell. The only legitimate use of inline comments is when you have some very complex algorithm that you need to have some thoughts about beside the code. Otherwise, the presence of inline comments indicates that you've written code that needs explanation, meaning that it cries out for refactoring. I frequently troll code bases upon which I'm working to look for inline comments so that I can refactor the code to eliminate the need for them.

Comments are a great example of something that seems like a Good Thing, but turn out to cause more harm than good. Fortunately, we've figured out how to achieve the same benefits that comments allegedly provide with tests, particularly BDD-style tests

Voices That Matter: Crafting Software Conference

Just a quick note to let my readers know that I'm speaking at the Voices That Matter: Crafting Software Conference coming up in San Francisco in early December. I'll be speaking on Real-world Refactoring, and I'll be doing a pre-conference tutorial called 10 Ways to Improve Your Code. This is based on material from The Productive Programmer and my 90 minute conference talk of the same name, but I'm expanding it to 3 hours. That shouldn't be hard because there's a lot of rich material there, and 90 minutes always made it hard for cramming in that much information Now, each topic can spread its wings more.

Of course, I don't mention every conference for which I'm speaking in my blog (otherwise, it'll be nothing but conference announcements). I bring this one up because you can get a discount: use CSNPKRA as the priority code when you register and, lo and behold, a discount! Don't ever say you never got anything of value from this blog!

Code, Forrest, Code!

The movie Forrest Gump has been re-playing a lot lately, and, as often happens, I see parallels in the software industry (I've learned not to bring these up to my wife so that I don't get the look). Of course, Forrest was mentally challenged, yet he had a wildly improbable life. The real point of the movie is the exploration of destiny vs. free will, as exemplified by the feather that floats freely, is captured and carried to new places in the Curious George book, and then floats free again. But that's not the point of this blog entry.

How did the mentally challenged Forrest achieve so much? Total concentration. If you notice in the scene where he first learns to play ping pong, the guy showing him says "The most important part of this game is to never take your eye off the ball". As soon as he says that, Forrest squints his eyes and starts concentrating hard on the ball, and starts hitting it. Of course, in the movie, he eventually becomes a celebrated table tennis player, going to China, meeting the president, etc. The events in the movie are highly improbable of course, but the writer had to come up with some talent that Forrest had that would enable all his amazing feats. And that was flow. I talk about flow in The Productive Programmer (in the Focus chapter) as defined by Michael Csikszentmihalyi. Flow is a state intimately familiar to all developers. It's that state of total concentration, where time disappears and you get your best work done. In fact, Csikszentmihalyi argues convincingly in his book that flow is the secret to a happy life: finding things that are challenging enough to engage your full concentration but not enough to be persistently frustrating. The ability to get to this state explains why so many developers love their career. Software development offers lots of opportunities to get to this state, offering puzzles that are always just within reach.

But flow doesn't come for free. It's a psychological state, meaning you can't just force yourself into it. My friend Brian Goetz has the best comparison I've heard: flow is like sleep. You can't force yourself to go to sleep, but you can set up the environment to make the state easier to achieve. That's one of the reasons people have so many rituals around bedtime. Flow is the same way. You can't force it, but you can set up the correct environment. Brian explained this to me in the context of interruptions. When he works at home and his family interrupts him, he's grumpy because they've broken his flow. But they don't understand: it's only a 5 minute disruption. But his killer analogy? What if someone awoke you 4 or 5 times a night for just 5 minutes at a time? You'd have a terrible night's sleep.

This matters a lot because perhaps the most hostile environment for flow is cube land. Putting developers (or any knowledge workers, for that matter) in a sea of cubes forces you to constantly deal with an environment that's hostile to getting to the state where you are most productive. Lots of developers have figured out how to survive (just like big city dwellers learn to sleep through noisy environments). One of my favorite lines from The Blues Brothers movie is when Jake comes "home" with Elwood, directly adjacent to an elevated train. Jake asks "How often do those trains run" to which Elwood answers "So often you won't even notice it!". How often do meetings happen in your office? So often you won't even notice that you only have about 2 hours a day to actually get work done. This is by the way one of the secrets to the effectiveness of pair programming. It is possible for 2 developers to get in flow at the same time. Because more than one person is working, external forces are less likely to disturb them, allowing them to stay in flow for long periods of time. This explains why, if you are new to pair programming, you are hammered after 8 hours of pairing: you've been in that deep concentration state for longer than what you are accustomed.

I talk about strategies for getting to and staying in flow in The Productive Programmer. That's the secret to real productivity when coding. Be like Forrest Gump: learn how to achieve that deep, deep concentration state in your environment (or, if you can't change your environment, change your environment). Code, Forrest, Code!