Missing the Point

One more post from the Entwickler Conference, where the Borland faithful gather in Germany. I sat though the technical keynotes, and they preach two things: Rapid Application Development tools and Big Design Up Front with requirements gathering tools (CaliberRM) and UML (Together). I know a lot of vendors are still beating these horses, and Borland is no different. Delphi now includes a EJB-like framework for persistence and workflow called ECO, which makes heavy use of tooling and inheritance. It doesn't use any of the interesting techniques used widely in the Java world (like dependency injection). Looking at ECO gives me Java flashbacks from 3 or 4 years ago. Anyway, the big finale of the technical keynote (which showed that they have created very good products in their categories), David I (Borland evangelist) showed a video of a 5th grader using Delphi and ECO to build a Teachers -> Classes -> Grades application with persistent and a generated front end in 9 minutes. The obvious message: you can build simple things Really Fast with our tools. Or maybe it was "Build Crappy Software Faster!"

Is that the right message to send out to the world? Is raw developer productivity for mindlessly simple applications buying us anything? The only people impressed by this type of demo-ware are non-technical managers, who have budgets. "Wow, if a 9 year old can build an application in 9 minutes, just think our developers will be building our entire corporate infrastructure! We must buy this tool!". I've posted my thoughts on RAD before. It is disappointing to see a company with so many sharp people pushing 2 aspects of software development that I think counter productive.

Not everyone in Borland feels this way. I'm corresponding with several Borlanders about how they can support agility in their Application Lifecycle Management (ALM) tools. The tide is turning. I am not naive enough to think that they should just ditch their product base because of what is still just opinion. I just hate to see what I consider poisonous messages to developers and their managers.

Dexter

As I stated in my last post, I spent the last week at the Entwickler Conference with the Delphi crowd. One of the highlights of the conference was the technical keynote where they showed the next version of Delphi, code-named Dexter. First, a little background about Delphi. From version 1 thought 7, Delphi was written in itself. The IDE is an Object Pascal project, written using the tool itself (including the components available in the tool. This is nothing new, building a tool in itself, but it was an eloquent argument for the capabilities of the language and libraries. Starting in version 8, Borland re-wrote their IDE base, creating Galileo. Galileo was the base host IDE for both Delphi 8 and C#Builder, their competitor to Visual Studio. One of the key benefits of Delphi 1 through 7 disappeared overnight, because they had to start from scratch. Delphi 8 had a different look and a small sub-set of the capabilities of the version 7 IDE. It was slow, buggy, and had serious re-drawing issues (watching it try to launch an application caused so much flickering that it could induce seizures). This, by the way, was richly ironic to the JBuilder crowd, which had been suffering taunts about slowness and ugliness for years from the Delphi crowd. Flash forward a couple of years. C#Builder became a non-product, and was rolled into Delphi 2005, which hosted both languages. It was a little better but still only about 30% as capable of a really good IDE like IntelliJ.

This has changed somewhat with Dexter. I was pretty impressed with what I saw. As a development tool, it is approximately on par with Visual Studio now in all but stability (beta testers tell me it is still pretty scary). Microsoft has the luxury of beta testing VS for 2 years; Borland cannot afford even a fraction of that. Still, the environment looks pretty good. It finally has a reasonable number of refactorings available (Delphi 2005 had 4, and from the reaction of the Delphi faithful last year, you would have thought that Borland had invented cold fusion). In my estimation, it is about 70% of a real IDE now. It is still hosted on Galileo, but it has greatly improved over the last year or so. The same IDE supports both Win32 and .NET development and has all the Borland Secret Sauce components. The next release, due out mid-next year, will even support .NET 2.0. Still not compelling enough to get anyone to consider it as an alternative to Visual Studio for new C# development (unless you are deeply immersed in Borland's Application Lifecycle Management integration strategy) but at least it isn't an embarrassment now.

How Important is Syntax?

I spent the last week at Kool-aid central for Borland's Delphi RAD
development tool. I was speaking at the Entwickler Conference
("Entwickler" is German for "developer") in Frankfurt, talking about
Java and .NET. This conference is ostensibly a general developer's
conference, but Borland is one of the main sponsors and it has
traditionally been the European gathering place for Delphi developers.
The faithful only get one Delphi focused conference per major land
mass, so they all come. The Java and .NET guys can get information
anywhere. Probably 70% of the talks at this conference either focus on
Delphi or overlap significantly (for example, I did an "Updated
Design Patterns in .NET " in C#, but the material also applies to
Delphi.NET).

The interesting part of my trip were the discussions between sessions,
at dinner, and in the halls. Even though Delphi is a niche development
tool with a shrinking market, the people at this conference are
unusually passionate about their tool. I sought to understand this
blazing exuberance. I traverse several different languages and tools
pretty regularly (Java, .NET, and Ruby), and to me they are just tools
-- each has its own strengths and weaknesses. I also used Delphi
pretty much exclusively for about 4 years (and wrote one of the first
Delphi books), so I understand the tool and language. But I've never
gotten so vein-bulging-in-the-forehead excited when someone told me
they thought something else was better.

This crowd has its back against the wall: Borland's clear future
direction with Delphi is to make it Yet Another .NET Language
(YA.NL?). The last couple of versions of Delphi have supported both
Win32 and .NET capabilities (and most of the developers in Germany are
still doing Win32 applications). In my opinion, this marginalizes
Delphi to also-ran status with C# and Visual Studio. Sure, if you are
building in Delphi now and have a bunch of code, you'll stick to Delphi
(maybe). But if you are starting a new project, there is no way you
would pick Delphi over C#. The IDE isn't any better (and it is less
stable) than Visual Studio, and it will not have a version that
supports .NET 2 until mid-next year.

Tiptoeing around the fanatics, I asked a few people if the above
assessment is true. "No! The next version of Delphi will win converts in
droves!". At the end of the day, it boils down to which syntax do you
like (if the tools are at about parity and the framework is the same,
you are down to begin...end pairs vs. curly braces). I've always had an
easy time switching syntaxes, both between similar (Java and C#)
languages and very different ones (Java and Ruby). I asked some of
these folks: "Is the syntax of the language really that important to
you?" I got a resounding "Yes!" from several people. They are willing
to use a third-world development tool just so they get to type Pascal
all day instead of C#. Just like all marginalized peoples, they
overcompensate anytime someone like me desecrates the object of their
devotion.

You see some of the same fervency in the Ruby and Lisp crowds, but they
are genuinely different from their peers. The Delphi vs. C# crowd are
supporting the same platform, with different syntaxes. Would you be
willing to jeopardize your job over the syntax of a programming
language? I still do not get it.

Swiming Up the Waterfall

Sometimes, you write something that you have to keep a lid on for a little while, because you don't want to offend someone who might be reading. This is one of those cases.

A fellow co-worker and I found ourselves in a client site who was a big believer in my most favorite of dysfunctional of methodologies, Waterfall. We were hired to help them design an advanced new system (that, of course, interacted with their still very active Mainframe development) with innovative technology. My coworker and I showed up with great enthusiasm (because we didn't know what was going on yet).

Here is what we were tasked to create: First, we need a Functional Design document (a technical-ish set of requirements) to pass off to the business unit to get approval for the project (which was already approved -- that's why we were there). It took man-weeks of effort to produce this utterly useless document, which was created by emailing Word documents around, because they don't put design documents in version control (What? Did I hear that right? Why not? "We just don't" “No, why?” “If it’s in version control, others can see it”). OK.

Next comes the Technical Design Document, another multi-man-week project, where we take the Functional Design and produce a Technical Design, meaning that we finally get to reveal all the technical decisions we made while writing the Functional Design but couldn't put in that document. And we can't produce "prototypes", because that's a bad word -- business people want to see prototypes run. "Can we call them spikes? Sure, no one knows that that means". A few times, I thought my coworker's head was actually going to explode. We set up our own secret version control system and starting writing our design documents (oh, sorry, Design Documents) using XML so that we could diff them, and converted them to Word documents when the big flurry of Consolidating the Documents happened.

Then we made a decision. Given the amount of time that we had to create the Technical Design Document, we could just write the code, using XP techniques (test-first coding, iterative design, etc.) and produce the design document based on the working, tested code. And that's what we did with part of the system. We kept this under wraps (after all, we're supposed to be designing, not coding) and made good progress. To show the quality of the code, I implemented code coverage using Cobertura so that we could demonstrate that the unit tests were actually exercising the code. With just a couple of days left in the Technical Design phase, we were asked to provide estimtes as to how long it would take to implement our design. That's when we let the cat out of the bag: "Do you want us to tell you how long it would take, or how long we have left? We can give you really good estimates because it's already done". We showed them the implementation and the unit tests, along with code coverage (96% code coverage, 100% branch coverage) . The Technical Design document that was already based on working code.

They weren't upset, but weren't particularly happy either -- just kind of stunned. This way of writing software is so foreign that they have no perspective on it. They've never seen code with unit tests, or code coverage, nor any code that was produced in an iterative way. You are supposed to spend months on design documents, then write the code, right? How is it that you can not do all that? Instead of the elation that we expected, we just got a weak "Good job, guys", and they took their working code and incorporated it where it was supposed to go. We were finished with our tasks, so we left.

But not without planting some seeds. You see, we involved some of their other programmers in what we were doing. We didn't code with them, but explained how we were going about it. And we showed them unit tests (which they had heard of) and code coverage (which was new). The other developers clearly saw the benefit of what we had done. Sometimes, you can't dig a deep hole all at once because you have to move too much dirt. But a river can dig a deep hole a little at a time, eroding away grime and stone bit by bit. Even if we didn't dig a deep hole, maybe we created a rivulet.

Technology Snake Oil, Part 5: The Integration Myth

Integration is a tricky thing. When it works flawlessly, it is a huge productivity boon. But when you trust it and it lets you down, it is a huge time sink. Some examples: I used to work with someone who had a genuine fetish for tool integration. He wasted more time trying to get tools to work together than he did performing useful work. At the time, we were using JBuilder and Visual Source Safe (for my opinions on this piece of technology, check out VSS: Unsafe at Any Speed). The integration between these two tools was spotty at best. I never even tried – it was easy enough to keep the VSS explorer open alongside JBuilder and just bounce between them. But not Mr. Integration. He tried a variety of supposed integration plug-ins, each with serious shortcomings. Finally, exasperated, I told him that I had the perfect integration strategy. “I’ve fixed your JBuilder/VSS integration. First, open both of them. Second, I’ve installed a special hot-key in your copy of JBuilder, Alt-Tab. When you hit it, the VSS explorer pops up, and it works just like the explorer that VSS itself uses – it’s indistinguishable!”

Recently, another example popped up from an unexpected place. It turns out that there is a bug in my beloved IntelliJ and WebLogic, but only when you are doing distributed transactions and messaging. My colleague spent the better part of a day trying to resolve that little nasty bug.

The snake oil hiding in the shadows here is the promise of integration sold by tool vendors. IDE’s try to encompass more and more. Visual Studio has always been successful with this (especially if you use all Microsoft technologies) without achieving a truly great code experience. Visual Studio.NET 2003 is a third world country compared the scarily intuitive IntelliJ, which is the Rolls Royce of IDEs. VS.NET 2005 is better, but still not up to IntelliJ’s standard. Borland tried to take this to the extreme conclusion with its Software Delivery Optimization suite, which tried to bind together requirements gathering, version control, coding, deployment, and monitoring into a single environment. This vision yields an awesome productivity gain as long as the integration is flawless. However, integration at that level is never flawless, meaning that you spend as much time (and considerably more frustration) trying to figure out why something that should be working isn’t, only to find out that it actually is working but the integration is obscuring the results.

Some integration is good – that’s why we have integrated development environments. However, there is a fine line where vendors try to go too far and end up lessening productivity rather than enhancing it. The problem is that tool vendors are trying to create monolithic environments. I suspect that Visual Studio in all its incarnations may be the culprit here – everyone is trying to replicate that environment. Finding the fine line between integration vs. stand-alone tools is tough and going to get tougher, as vendors produce more and more immersive environments to get you to buy into their integration strategy.

Someone has done this exactly right. In the next Technology Snake Oil (Drowning in Tools), I’ll talk about that.

Technology Snake Oil, Part 4b: Prescriptive Approaches in the Large

I wrote recently about Prescriptive Approaches in the Small. More pervasive and dangerous, though, are Prescriptive Approaches in the Large. Everyone wants a repeatable pattern, a guaranteed way of doing things. Recently, at No Fluff, Just Stuff Columbus, OH, one of the questions for the expert panel was: “Our company is looking for a road map, a sure-fire prescription of what technologies, methodologies, and approaches we should take – can you provide this?” Uhhhh, no. Most of the panelists nearly crawled out of their skin to get in line to answer this question (I thought Prag Dave was going to have a heart attack – if you know him, you know this is anathema to him!). There is no prescription for software development done right in your company – every place is different. You cannot come out and say that this approach (whatever it is) is right in every circumstance.

Following this line of reasoning, I think that the PetStore was one of the worst evils inflicted on the young, innocent, fawnlike Java Enterprise Edition. It was supposed to be a catalog – “Look, here are all the things this shiny new technology can do” – and people took it as a prescription. Rather than sit back and determine for what EJB was suitable, developers dove right into using them in their projects because the prescription said so. EJB was (and is) suitable for only very specialized types of applications. If you were writing a real pet store application (even back when PetStore came out), you would be crazy to use EJB!

The pattern repeats over and over, with technology (“Everyone is using Struts, so we should be too”) to methodologies to products. You can bet that the SOA vendors are salivating over the prospect of creating the prescription that says “Everything must be decoupled, document centric messages – here, we have an expensive product that just happens to do this!” As much as it hurts, you have to make determinations for yourself on which technology (methodology, framework, tools, etc.) are best for the situation you are in right now. There is no prescription, no matter what ersatz technology doctors tell you.

Joel is Plain Wrong about XP

Joel Spolsky writes in a recent blog entry:

As I worked through the screens that would be needed to allow either party to initiate the process, I realized that Aardvark would be just as useful, and radically simpler, if the helper was required to start the whole process. Making this change in the spec took an hour or two. If we had made this change in code, it would have added weeks to the schedule. I can’t tell you how strongly I believe in Big Design Up Front, which the proponents of Extreme Programming consider anathema. I have consistently saved time and made better products by using BDUF and I’m proud to use it, no matter what the XP fanatics claim. They’re just wrong on this point and I can’t be any clearer than that.

Joel has an interesting perspective on this subject and has written quite eloquently on this and other topics (I highly recommend his books and blog). However, I think he’s just wrong about this. He has a skewed view on software development, having worked at product companies his whole career. Building products is a lot different from building business applications (which is where my experience lies). First, for products, the requirements and design are pretty much made up of a few people’s vision on what the product needs to do. The market place determines if they were right or not. Second, you generally have to do this in secret, especially if you have competitors who will cherry pick your design. Third, you are pretty much required to do a “big bang” deployment, when you ship to customers.

Most business applications aren’t like this. The requirements are very fluid (quite often, the business stakeholders don’t know what they want even well into the process). When done correctly, the process is collaborative across business units and developers, making for a stronger product. Last, you can roll out iterations (even if just to QA and/or acceptance testing) without the big bang.

Joel is using the straw man arguments against XP. Many misinformed naysayers against XP assume that we never think about what we’re doing up front. Of course we do. This same group assumes that there is no coherent architecture for complex XP projects. Just because we’re doing agile development doesn’t mean that we just sit down and hack out code with no planning or forethought. Believe me – I’ve had lots of experience building business software, and Big Design Up Front just plain doesn’t’ work, even when it is done well (I’m fighting it yet again even as we speak). XP delivers better quality software, in shorter times, closer to the ultimate target, which has a fighting chance of actually usefulness.

Taking this topic to the logical conclusion, I wonder what would happen if a product company did a full-on agile approach? Building the product based on custom feedback from the early iterations. You would need a patient customer base, willing to comment on the work in progress. I can’t help but believe that, just as in the business software world, you would end up with a better product. How many features could you eliminate in something like Word if it had been developed incrementally? So much useless cruft could be removed from commercial software if product teams listened to their customers while developing the software rather than the complaints after it is done.

Some companies are tiptoeing in this direction. Check out JetBrains Early Access Program for innovative tools like Fabrique and MPS (The Meta Programming System). I think these products will be extraordinary if they can get their target audience to give them feedback. Agile product development – what a concept!

My Blog, in Korean

To satisfy overwhelming demand (OK, there is no such demand), Trustin Lee is translating some of my blog entries into Korean and posting them here. To quote:

I'm an Apache Committer who is still on the way of studying English. I'm trying to level-up my English skill translating great blog entries from great authors.

I think that's more than a bit generous, but I'm gladly supporting this effort because, hey, it isn't costing me anything and it spreads my meaningless opinions all over the world. What an unbeatable combination. Seriously, I appreciate the hard work Trusten is doing for this, and I hope that my dribbly writing is worth translating.

Now, I’m really interested to see if he translates this blog entry, creating a kind of recursive blog referencing stack overflow.

Technology Snake Oil

Technology Snake Oil, Part 4a: Prescriptive Approaches in the Small

The world is a much simpler place if you can just coast through it, without having to expend precious mental energy. In fact, it seems widely encouraged in many organizations: if we can prevent our employees from thinking too much, they will stay out of trouble. Unfortunately, to build insanely great software requires thought on many different levels, which doesn’t go well with prescriptive approaches.

You see all sorts of prescriptive approaches in software development, both large and small (this entry covers small, the next Snake Oil piece covers large). Small prescriptive manna floats down in many different guises, mostly in a good context like design patterns and best practices. Many developers take these nuggets out of context and start believing that this Prescription for building all types of things represents the One True Way (without bothering to apply additional thought). Don’t think that I’m denigrating design patterns – they are clearly a good thing, as a way to catalog common patterns. The problem comes in when they are used without thinking.

Here is a classic example that appears all over the Java code I see every day. Java doesn’t really have a struct-type thing, which is just a holder for data. In Java, everything is a class, which is OK. Nevertheless, I see developers who just need to pass composite data around creating a Java class with accessors and mutators that won’t every be used for encapsulation. It has become a common prescriptive practice in Java to create the class, create private member variables, and generate getters and setters for them without remembering why you create accessors and mutators. If this class is never going to have any behavior attached to it, if it is a simple container for information, you don’t need the accessor/mutator pair or the private scoping – make them public. If you really need to add behavior later and/or encapsulation, you can refactor the class into a real class, not just a composite data type.

The real sermon here is to think about why you create things, don’t just blindly create artifacts. Yes, encapsulation is generally a good thing, but isn’t required in every single thing that looks like a class – sometimes it’s really just a struct. You must always be diligent about why you are creating what you create.

Fighting for Your Life

Fighting for Your Life

I was almost voted off the island recently.

Because of some scheduling issues involving me rolling off a project, another one starting, and the slated person still a week away of rolling of his project, I was in the unfortunate position of going to a client's site for just 1 week, at the start of a new project. The first day, I arrived late because of travel details, and we spent the day going over the technical details of what it was we were going to be doing. All fine and good. In fact, one of the overviewers was kind enough to identify something he thought I could do in just a week, something orthogonal to the rest of the work.

The next day I get to work and the client supervisor asked me into his office. "Neal, we just don't think the one week thing is going to do us any good, we like you and all, but you should just go ahead and leave." He was trying to kick me off the island! The last thing he said was "Unless you can think of a good reason we should keep you..." That was my chance. I said "Well, of course, it is your decision, but I have found something I can get done in a week", and went on to tell him about the one-off sub-project we had identified the day before in technical detail. He listened to my case, then called the technical lead into his office and had me repeat it. The tech lead said "Well, we do need that piece, and I think he can get it done in a week". The supervisor turned to me and said "Alright, get back in your office and get to work!". Saved!

And it turns out that I was able to get the work done in a week, and the client requested that I stay on instead of the guy originally slated for the gig. How is that for defending your life?

An Actual, Real, Everyday Use for XSLT

I have played with XSL and XSLT for a while, and even toyed with the idea of basing my whole website on an XML+stylesheet vision. Because XSLT can get complicated, that idea died before its time. However, I recently found a really good use of XSL on a development project on which I was working. This project included the ubiquitous Log4J library, including a largish logging.xml document. My job was to make sure this document is still relevant. The problem with trying to analyze such a document lies in the way that items reference each other. Instead of pulling my hair out bouncing around in the XML file, I created an XSLT stylesheet to format the whole thing into HTML, with links from loggers to appenders and appenders to loggers. The end result looks like this (broken into 2 pieces for space savings):

The stylesheet that generates this info from the XML document is not long or complex:

<?xml version="1.0"?>
<!--  XSL Stylesheet to make it easier to determine the relationshps between -->
<!-- the loggers and appenders used in the application.                      -->
<!-- This stylesheet is automatically linked into the logger.xml document.   -->
<!-- If you are using any reasonable browser, this stylesheet should be      -->
<!-- automatically applied to this file.                                     -->


<xsl:stylesheet version="1.0" xmlns:xsl="http://www.w3.org/1999/XSL/Transform"
   xmlns:log4j='http://jakarta.apache.org/log4j/'>

   <xsl:template match="/">
         <html>
           <head>
             <title>
                 Loggers
             </title>
           </head>
           <body>

             <xsl:apply-templates select="log4j:configuration" />
           </body>
         </html>
   </xsl:template>

   <xsl:template match="log4j:configuration">
       <h1>Project Loggers</h1>
       <h2>Loggers</h2>
       <table border="1">
           <tr>
               <th>Logger</th>
               <th>Priority</th>
               <th>Appender(s)</th>
           </tr>

           <xsl:for-each select="category">
               <xsl:sort select="@name" />
               <tr>
                   <td><xsl:value-of select="@name" /></td>
                   <td><xsl:for-each select="priority" >
                       <xsl:value-of select="@value" />
                   </xsl:for-each></td>

                   <xsl:for-each select="appender-ref">
                       <td><xsl:value-of select="@ref" /></td>

                   </xsl:for-each>
               </tr>
           </xsl:for-each>
       </table>
       <h2>Appender to Logger Relationship</h2>
           <ul>
           <xsl:for-each select="appender">
               <xsl:sort select="@name" />
               <tr>
                   <li><xsl:value-of select="@name" /></li>
                   <xsl:for-each select="param">
                       <xsl:if test="@name = 'File'">
                           [ <xsl:value-of select="substring-after(@value, '${logging.home}/')" /> ]
                       </xsl:if>

                   </xsl:for-each>

                   <ul>
                   <xsl:call-template name="show-categories">
                       <xsl:with-param name="appenderName"><xsl:value-of select="@name" /></xsl:with-param>
                   </xsl:call-template>
                   </ul>
               </tr>
           </xsl:for-each>
           </ul>
   </xsl:template>

   <xsl:template name="show-categories">
       <xsl:param name="appenderName" />
       <xsl:for-each select="//category">
           <xsl:for-each select="appender-ref">
               <xsl:if test="@ref=$appenderName">
                   <li>
                   <xsl:value-of select="../@name" />
                   </li>
               </xsl:if>
           </xsl:for-each>
       </xsl:for-each>

   </xsl:template>

</xsl:stylesheet>

I also added a reference to this stylesheet in the logger.xml document itself:

<?xml-stylesheet href="ShowLoggerRelationships.xsl" type="text/xsl" ?>

The beauty of this is that any reasonable modern browser will automagically apply this stylesheet anytime you double-click on the XML file. This is the ultimate form of documentation -- generated directly from the source in real time. Any changes to the relationships in the logger.xml document are instantly shown when you look at the file. If you still want to see the raw XML, you right-click and "Open With..." instead. This makes the browser view of the XML document useful without hiding or obscuring the functional elements. And is a good example of the DRY principle (Don't Repeat Yourself) from the Pragmatic Programming applied to documentation.

Technology Snake Oil, Part 3: MDA

Model Driven Architecture (or "Cartoons to Code") is the ultimate culmination of the vision of UML. Let's draw the software with diagrams, including decorations to represent logic, then click a button to generate the code. A laudable goal, but far short of reality.

The only real success stories with MDA are for very limited problem domains, like embedded software. Here's why: consider the cash register at McDonalds. The number of items is very limited, and the number of things you can do with them are very limited. This is well suited to a pictorial (in fact, an ideogrammatic) representation. That's because the concepts on display are retricted to morphemes, which in linguistic terms represents the smallest language unit that carries a semantic interpretation. Thus, it's easy to create a McDonalds cash register using just pictures and make it a quite effective solution to the food ordering problem.

For complex problems, you can't break down the problem to simple morphemes -- imagine the semantic complexities of a concept like "loan" or "payment". For a problem domain this rich, you need the richer representation of a full-blown language (which the Egyptians also discovered -- they were the last MDA success story, and they eventually abandoned it!). Trying to cram rich semantics into a pictorial representation so that you can create software that abstracts those semantics is absurd.

In fact, trying to cram the entire world into a strict tree-shaped hierarchy isn't much better (see Programming, Object-oriented). At least OOP has more flexible semantics, and we keep forcing solutions to this hierarchy problem with inventions like aspects, which cut across the tree-shaped world we've created (because stuff in the real world cuts across trees). So, where do we go from here?

Let's solve it the same way mankind (including the Egyptians) did -- create language! Language is ultimately the only way to handle the rich semantics of the world around us because that's what the people stating the problem domain use to state the problem. Which is why I'm so geeked about Language Oriented Programming -- building domain specific languages that are as close as possible to the problem we're trying to solve. LOP done right encapsulates the low-level details of a language like Java, allowing developers and others to work at a higher level of abstraction. We've been mired in curly-brace languages too long -- we need to upgrade our abstraction.

MDA is the wrong direction for abstraction upgrade. It tries to further restrict our expressiveness, believing that the entire world can be graphed and you can create real software this way. Imagine for a moment the MDA diagrams (including Object Constraint Logic) for your average J2EE enterprise application. You can build it in code, with the crude semantics of curly-brace Java in 1/3 the time you could model it to the point where you can click the "And Then a Miracle Occurs" button to generate the application. MDA might work for simple, semantically unique domains, but no where near the real world.

Happpy with What You Have to be Happy With

Recently, I did a 3-hour tutorial workshop at the SOA conference in Singapore entitled "Agile Development and SOA", where I talked about aspects of Agile development as it applies to SOA type projects. Mostly, it was an excuse to talk about Agility in a foreign land.

I asked the attendees what type of methodology they were currently using. The three replies I got were 1. None (what's a methodology?), 2. Waterfall (although she didn't know that name for it), and 3. SDLC (the Software Development Lifecycle (another variant of Waterfall, check out this interesting Wikipedia image of SDLC )). So, basically, everyone is using some form of Waterfall. What's interesting is that I asked to most vocal of the crowd (meaning that she would actually answer without me jumping up and down on the table in front of her -- she was the SDLC victim) if her company was happy with that methodology. She quickly replied "Oh, yes, it works very well for us". Needless to say, I was surprised. So I asked her a few follow-up questions:

• Do you ever miss deadlines? Oh, yes


• Are your user's happy with what you produce? Well, no


• Are your developers frustrated by vague requirements, which leads to user dissatisfaction? Yes


• Do you ever accidentally break code in your "big bang" deployments, which takes a long time to fix? Yes

I asked a few more, but you get the gist. She thought they were very happy with SDLC because they didn't know that something better existed. Perfect crowd for an agility talk, right? It was an uphill battle. No one could believe that this crazy idea could ever work -- waste all that time writing tests? Include the users as part of the development team? Pair Program?!? That cuts your productivity in half! Sigh. I waged a good battle, but don't know how well the war is going. At the end of the talk, I had either convinced them to tip-toe towards Agility (because these were developers -- can you imagine what their boss will say?) or they were just tired of me talking about it and said anything to get me to shut up.

I've blogged about this phenomenon before -- why is it that developers and managers believe that all forward progress in methodology stopped in the mid-70's and all language development froze about the time Objects appeared? Is it just a non-technical person's fatigue at trying to keep up, so an arbitrary milestone was created beyond which "I won't think about it anymore?" Are developers so overwhelmed that they just shut down too, and keep working on what's in front of them even though better things abound?

Technology Snake Oil, Part 2: RAD

Back in the mid-nineties, Rapid Application Development was the answer to all developer woes. RAD tools came out in droves and everyone touted the huge productivity gains made available by drag and drop user interface design and component based development. RAD environments made for great demo-ware -- you could stand in front of a group of Pointy Haired Boss types and build a working application right before their eyes while you were talking about it (I know because I did some of this -- and I still have trouble sleeping sometimes). Of course, the message was clear: buy This Tool, and you too can have developers that can point and click their way to huge productivity gains. And bought it they did.

Then a funny thing happened to the developers on the way to this astounding productivity. They learned the tools and dutifully dragged and dropped their way to building applications...and it was good. Until, that is, they figured out that building the application is the short, easy part -- now they have to maintain the monstrosity they've created. RAD development doesn't scale well, and when you build large and very large applications with it, maintaining those applications is a nightmare of searching for the scattered crumbs of code scattered hither and yon. Working for a while in this environment is what made the Design Patterns book really hit home -- the kind of code we were creating was entirely anti-pattern, and more about "how fast can it be created"? The maintenance nightmares led me, as CTO of a consulting and development company, to figure out new ways to use the RAD tools we had to use (because of our clients) that did follow good design principles. What we found is that the tools fight you when you try to do the right thing. RAD Development tools have a path of least resistance, which makes things very easy while you stay on the path (and use their wizards and other support structures). Stray from the path, and you find obstacles at every turn. But, it's worth fighting through them to get to a place where you can create maintainable code -- and you find that you are no longer doing Rapid Application Development, just Application Development.

RAD is now rightfully deprecated in most serious software development efforts. Even serious developers using RADish tools like Visual Studio are avoiding the RAD pitfalls. Some developers still fight with RAD tools, and think that maintenance is just always extremely difficult in big projects. I'm still shocked by the developers who have still never heard of Design Patterns and who still work on Sisyphean development chores.

Which is why I think the RAD aspect of Ruby on Rails misses the point of what makes Rails interesting. RAD doesn't work for large, complex projects, no matter what technology. The interesting part of Ruby on Rails is the effective use of Ruby to build a domain specific language on top of the underlying Ruby language that makes it easier to do web applications with persistence. It is more advanced than the RAD tools of old because you can drop to the underlying Ruby language and get underneath the framework when you must -- and that's a powerful idea. But, building complex applications in it still requires real thought. Maybe the breathless amazement of the Rails community reflects the absence of effective RAD tools in this space before. Don't get me wrong -- I think that Rails is cool, just not for the reasons touted by so many Rails-flavored Kool-aid carriers.

Technology Snake Oil, Part 1: UML

I was told by someone recently at ThoughtWorks that I was so old that I "added to the company's diversity". Well, I've decided to take this sage old-timers wisdom and put it to some good use. In the development world, we keep getting sold Snake Oil in one guise or another, and I'm going to start pointing out some snake oil. Starting with the most venerated of crappy technology: UML.

Some of us craggy old-timers remember the time before UML, when we had Booch, Rumbaugh, and several other competing diagramming notations. The Holy Grail was going to be the Unified Modeling Language, which would solve all our problems and finally give us software developers a universal language, for everything from use cases through structure to deployment. And that's what we got. A classic example of "be careful what you ask for". UML feels like a compromise between a group of people, and it's so general that it's almost useless. The standards bodies keep bolting on new stuff to try to get it to the point of basic usefulness, but it just makes it worse. Does anyone really use Object Constraint Language to decorate their diagrams so that they can generate code that Just Works? I have never seen a case where you couldn't get it done faster by writing the code, then reverse engineering it into one of the expensive tools. Of course, it's good to design things before you code them, but a white board is much better. Combine the nature of UML with the fact that you can't ever show its diagrams to business analysts or end users because it is too obtuse, and I end up doing most of my architecture and design work in Visio (and I'm no fan of it either). It creates diagrams you can show everyone that capture the essence of what you are doing just before you implement it.

I'm not just a tourist in the UML and RUP world -- the company I worked for in the dim and distant past used RUP, and we foisted Heavy Duty Object-oriented Analysis and Design on clients, and used it ourselves. Why do you think I'm so militantly agile now? It just flat doesn't work for 90% of the types of projects that real developers write. And it's Bastard Child Model Driven Architecture is featured in an upcoming Snake Oil entry...

SOA (Service Oriented Aberration)

While talking to some of the other speakers at the SOA conference in Singapore, it occurred to me that SOA has an interesting side effect that has nothing to do with technology. To make SOA really work, a business has to strictly define what it is they do, to the point that a developer can create services to represent it. That's the rub -- most businesses don't understand their own business to that level of detail. Time and time again, I've noticed that the analysts and developers on a project actually define a client's business processes to a much finer level of detail than the business ever has. To write code for something, you have to really understand it. Some of the clients for which I've worked even commented on the fact that, because of the development process, they understand their business better.

Of course, the opposite is also true (and, alas, more likely) -- companies with poorly defined business processes will try to adapt SOA and fail miserably, creating a case study of SOA failure, when the technology had nothing to do with it. Worse yet, tool vendors of expensive development tools like Enterprise Service Buses will sell their tools to unsuspecting clients as a way to "fix" their business process. Putting a dress on a pig doesn't make the pig any prettier, and it just annoys the pig. SOA, like so many other similar technological breakthroughs (see Programming, Object-oriented and Development, Component-based), won't solve the dysfunction of dysfunctional businesses.

Conspicuously Caucasian

In both trips this year to Singapore, I had the eerie sense of sticking out like a sore thumb -- I was conspicuously Not From Around Here. In most of the other places I've traveled, I could, with a little effort, blend in with the natives. In Europe or Australia, if I dress like the natives, I look like a native. Of course, as soon as I open my mouth, I'm a goner, but I can quietly blend. Not in Singapore! I am obviously not from the neighborhood.

I don't mind being different (I certainly was in high school, and I was from the neighborhood). However, in Singapore, I'm a target for all sorts of commerce attempts. I can't walk past a store front, even in the tony parts of town, without someone trying to sell me something. I'm not only conspicuously Caucasian, I'm American, and therefore both gullible and rich! I can't get the locals price around here for anything. Just another example of how travel broadens you, and why I like travel so much because it provides perspective.

Battle of the Desktop Search Giants

Back in the dim and distant past, I blogged about desktop search in general and Google Desktop Search specifically. In that post, I lamented the search tools of yore that went away for one reason or another. A commenter kindly pointed out that my all time favorite (Magellan) now lives on as X1..

Lo and behold, Yahoo has entered the fray with their own desktop search tool, based on X1. In fact, it is a free version of X1 that doesn't include the enterprise features. It has a slick Windows-based UI and does a lot of cool tricks. So, I displaced Google and installed Yahoo Desktop Search (PCMagazine also gave it their editor's choice).

Well, after using it for a while, I'm un-installing it and going back to Google. Both are in beta, but Google is much more stable IMO. Yahoo was constantly doing odd things, and seemed to be belly up when I needed it the most. While Google has a sparser UI and doesn't handle as many file types, it is (at least for now, for me) more stable. And, for infrastructure software like desktop search, stable wins the day for me. I'm writing this while un-installing Yahoo and re-installing Google. Maybe once Yahoo turns 1.0, I'll re-evaluate. But for now, I'm switching back to Google.

Passion Part 2

As if on cue from my last post, an aphorism from Robert Fripp:

Life is too short to spend time on activities where our passion is not engaged.

I know a lot of people who fundamentally don't enjoy their work. While earning a living is a laudable goal, I think those of us who do really enjoy their work are lucky. Most of my friends who don't enjoy their work envy the passion with which I attack my work. Some of them accuse me of not really working, just enjoying my hobby: that's not true -- it's still work. For my hobby, I would create an entirely different set of software. It still tires me out at the end of the day. But, for the most part, I take joy in the job that I do. I'm lucky.

Enjoying the Passion

Last week was one of those great ThoughtWorks weeks in Chicago. There were lots of people in the office, it was a new hire week (with several people in orientation), and there were lots of gatherings to which I was either invited or managed to latch onto. It's always a mixed crowd at ThoughtWorks, with business analysts, project managers, and, of course, developer geeks. During one of the meals, I started talking about Language Oriented Programming (I know, a big shock to those of you who read my blog and/or been around me lately). The business analysts sitting nearby didn't have a clue as to what we were talking about. At one point, I turned to someone sitting next to me and apologized for the obviously obtuse (for non geeks) discussion. That reminded me of a similar situation that happened in Columbus, OH, when I met fellow ThoughtWorkers (a developer-type and a project manager). Of course, we started immediately talking about very esoteric software engineering topics. At one point, just as this week, I turned to the PM and apologized. She said: "That's OK, I'm just enjoying the passion".

Both events, last week and in Columbus, is a vivid reminder why I do this for a living and why I work where I do. How many people you know are really passionate about what they do for work, and so actively proselytize it to everyone they meet that can understand it? And in fact actively seek out people who can understand it so that they can talk about it some more? Even the non-techies at ThoughtWorks enjoy it, albeit vicariously.

Running Tests to Watch Them Fail

I was reminded again today by a fellow ThoughtWorker why in Java it's a good idea to write the test, run it to watch the test fail, then make it pass: what if you mis-spell the testing method, i.e., "tsetMyMethodToSeeWhatItDoes()"? Because JUnit uses reflection to find the methods to run as tests, it won't pick up "tset".

Which is why I think that NUnit v2 got it righter (is that a word?) by using annotations to mark the test methods instead of naming patterns. If you misspell the annotation, the compiler complains, which, in this case, is A Good Thing. Improvements are already afoot in the Java world -- TestNG already supports annotations (and JavaDoc tags if you are using 1.4) and several other features, and rumor has it that JUnit 4 will support annotations and some of the other TestNG features as well.

Beyond OOP

One of the questions that frequently comes up during the expert panels at No Fluff, Just Stuff is "What Comes Next?" What, in the opinion of the speakers, is the Next Big Thing? Consequently, I think about this probably more than most developers just so that I'll have a lucid answer at these panels.

One of my favorite characteristics of college was the accidental synergy that occurs between classes that you happen to be taking in the same quarter. For example, I took the automata class the same quarter as the compilers class, and there are obvious common elements. Even something as seemingly unrelated as artificial intelligence and philosophy creates a fertile ground for interactions.

So, how are these two things related? Lately, during my constant ruminations about What Comes Next, I've seen an interesting synergy between two separate but related concepts. Language oriented programming, my new favorite geek obsession, is one. It addresses the disconnect between the hierarchical nature of OOP and the rich semantics of language. The other day, I was talking to Obie Fernanadez, a fellow ThoughtWorker, and he is really into RDF and the Semantic Web. He's dealing in hierarchies, but in a different way. I am probably greatly injuring this assertion by my newness to this subject, but the Semantic Web folks maintain that, among other things, the problem with OOP is that it is basically context free. Hierarchies exist, but they don't relate in the rest of the world with other hierarchies -- each inheritance tree is an island. And properties that exist on classes don't have any context outside the class. In the real world, "address" can have separate meanings based on the context -- are we talking a physical place or a inaugural speech? The way our minds work isn't like OOP, where everything exists in a strict hierarchy. We have these "free floating" concepts like "address" that can be attached to the correct context. What's needed is context...and that's what the semantics of language provides that OOP doesn't already have. Now, this week at least, I'm thinking that What Comes Next is related to both language oriented programming, which provides needed context to the structure of programming, and semantic ontologies, which provides global context. These are fundamentally different things, but seems like there is some synergy to be found. More later...

Metaphor Shear

I was reminded today of a term I read about several years ago in the excellent essay "In the Beginning was the Command Line" by science fiction author Neal Stephenson. In it, he talks a lot about the layers of metaphors that make working with a computer what it is today. Without metaphors, you are stuck looking at a bunch of 1's and 0's (side note: every expensive piece of software you own -- Oracle, Application Servers, etc. -- are just 1's and 0's...it's the order they're in that's important). Anyone who has seen one of the original Osborne computers knows just how scintillating starting at blinking lights representing 1's and 0's is. The problems come in when the metaphors suddenly turn into vapor. I'll let the other Neal take it from here:

Anyone who uses a word processor for very long inevitably has the experience of putting hours of work into a long document and then losing it because the computer crashes or the power goes out. Until the moment that it disappears from the screen, the document seems every bit as solid and real as if it had been typed out in ink on paper. But in the next moment, without warning, it is completely and irretrievably gone, as if it had never existed. The user is left with a feeling of disorientation (to say nothing of annoyance) stemming from a kind of metaphor shear--you realize that you've been living and thinking inside of a metaphor that is essentially bogus.

The whole essay is a great read, with lots of insights into fundamental ideas about operating systems (if you think this metaphor is good, wait until you read his description of Windows vs. Mac OS vs. Linux as car dealerships). I think about metaphor shear anytime I'm forced down to the guts of computers or operating systems.

Language Workbench Sample Ported to Java

As I've blogged about recently, I'm updating my Building Domain Languages atop Java conference talk to incorporate information from Martin Fowler's article Language Workbenches: The Killer-App for Domain Specific Languages?. I wanted to show his example domain language sample in my talk but my talk is Java and his sample is in C#. So, as a public service to Java developers everywhere, I ported his sample to Java 5. This is mostly a straight port with a few differences:

• I used generics in a few places to clean up type casting
• The reflection stuff is different because that's where .NET and Java differ the most
• I cheated and created simple classes with public fields rather than properties for the target classes to make the reflection code easier

It is a complete version of the sample (including all the supporting files to make it run + IntelliJ project file). Martin graciously gave me permission to post it on my website and in my blog. Language oriented programming looks to me like the next big revolution in programming paradigms, and I'm spending a lot of time playing with this right now.

Berg's Chamber Symphony

Recently, I had the privilege to see a unique performance of Alban Berg's Chamber Concerto by members of the Chicago Symphony Orchestra, lead by Pierre Boulez, with Daniel Barenboim on piano and Pinchas Zukerman on violin. This concert was special because it re-created a landmark recording of this work by the same ensemble. The other unique thing about this concert: the musical work is only about 30 minutes long, but it was the only piece of the evening. Mr. Boulez and the entire ensemble came out and discussed the work (with examples) for 1 hour, 40 minutes, then took a short intermission, then played the work.


What makes this music so interesting that you can talk about it for 3 times as long as the piece itself? Well, it would take me over an hour to explain! Here are some examples, though. This piece was written in homage to Berg's teacher, Arnold Schonberg, who developed the 12-tone method of composition. The first theme of Berg's work represents Schonberg, Berg, and Anton Webern (another student) with the motif Arnold Schonberg Anton Webern Alban Berg (using the German alphabet). So, the main theme of the work uses the names of the teacher and students. Another example: the second movement is a palindrome: it plays up to the middle (signified by the piano's only involvement in the 2nd movement) the plays in reverse to the end, using different combinations of instruments so that it's not immediately obvious. There are lots of mathematical features in this music: the first two movements have 240 bars each, and the third has 480 bars.


Needless to say, this is a fascinating piece of music, made all the richer if you understand the context of the time in which it was written and all the ideas that went into it. Many people who don't like modern serious music don't have the correct expectations or context. Just as the movie Memento is different than Casablanca, they are both great achievements. The same is true for serious music.

Improving Your AIM

One of my biggest treats of No Fluff, Just Stuff symposia is the conversations you get into before and after. On the way to the airport in Orlando, Glenn Vanderberg, David Bock, and I shared a crowded airport shuttle (with a bunch of vacationers who didn't share our conversation). Glenn talked about some fascinating writing he's doing with feedback loops in Extreme Programming and David talked about the Agility Index Measurement. My contribution? A interested but dazed stare.

David has since blogged about AIM (nice acronym, very managerial) here. He's soliciting for people to help him develop a more or less objective measurement of just how agile a team really is. To quote: "the point is I WANT A WAY to compare the capabilities of teams implementing these [agile] processes." If this topic interests you, wander over to his blog and put your own $0.02 for what categories should be measured.

Riffin' at No Fluff, Just Stuff

One of the early histories of Saturday Night Live talked about the first meeting of the writers. Even though they had never met one another, and were from different parts of the country, they had a common language and thought that the same kind of things were funny. It was compared to jazz musicians, who can jam with one another even if they've never met, because they share the same vocabulary (and, dare I say, are familiar with the same problem domain).

The same thing happened last weekend at No Fluff, Just Stuff in Orlando. One of the speakers became ill at the last minute, and the organizer had to scramble to find speakers to fill his spots. I filled one of the talks, about fallicies in enterprise architecture, after never having seen the talk and having a whole 10 minutes to look at the slides for the first time. However, the talk came off really well -- I managed to talk for the entire ninety minutes, and got good scores. Because the problem domain was familiar, I was able to extemporize the talk, bringing my own experience to bear. This talk wasn't about tools or APIs, but about common fallacies developers face in enterprise situations. Well, I've seen a lot of that. Of course, I'm sure my rendering of the talk is no where nearly as good as the original and that the stories and experiences I bring are completely different from the original speaker. However, it reminded me of a jazz trumpet player showing up to a gig and saying "'Girl from Ipanema', extra solo after the chorus, and watch me for the finish. A-one, and-a Two, and...".

The Next Revolution: Language-oriented Programming and Language Workbenches

If you subscribe to Martin Fowler's blog, you've probably already read about Language Oriented Programming and Language Workbenches. He has very nicely encapsulated and clarified some of what I've been thinking for a while and added some badly needed organization to the topic. I've been doing a No Fluff, Just Stuff talk this year about Building Domain Languages atop Java, and it has been received very well in the cities I've where I've given it. Now, I'm about to revamp it and make it a Language Oriented Programming talk instead, taking the material I already have and adding discussions about LOP in general and workbenches. I'll also start showing JetBrain's (the makers of IntelliJ) cool new MPS meta-programming tool. It's still rough around the edges, but still very interesting. You have to have the early access version of IntelliJ (code named Iridia) and the MPS plug-in, but you can download all these goodies at JetBrains' site.

Centralized Help in Eclipse

As further proof that I embrace all IDE's (besides my beloved IntelliJ), I have just published an article on IBM developerWorks site about building centralized help in Eclipse. One of the really cool things about Eclipse is that everything important, including the help system, is built as a plug-in. This article shows how you can externalize both the built-in help and help you download with plug-ins into a central place. You can also turn the Eclipse help into a kind of wiki or other kind of collaborative document management tool without too much effort. The help system is quite flexible, as this article shows.

Windows eats itself?

I think my computer was trying to tell me something the other day:



So even Windows doesn't trust Windows Explorer anymore. Hmmmmm...

What You Know You Know, What You Don't Know, and What You Don't Know That You Don't Know

Blogs are a great way to show what you know and also a great way to show what you don't know (and can learn). I blogged yesterday about how to chain 2 ant calls together via Bash. Several comments showed that there is definitely more than one way to do this. You can use the "&&" in Windows to chain commands (which I had seen in the past and promptly forgotten). This also works in Cygwin's Bash because most of the stuff that works in the Windows command prompt also works there (Reason #432 to use Bash). However, I didn't know that you can chain targets together on the command line for Ant! I've been using Ant since early 2001 (I know the time frame because I wrote the first Ant article for Java Developer's Journal in June of 2001), but I didn't know about this handy-dandy little trick.

This illustrates, to me, one of the best reasons to have a public forum like a blog. You inadvertently learn all sorts of stuff if you are brave enough to try to teach others what you already know -- including stuff that you didn't know that you didn't know.

Reason # 431 Why You Should Use Cygwin/Bash Instead of the Windows Command Prompt

I'm working on a project right now that has a huge number of classes, and a corresponding huge number of tests. The whole thing builds via Ant (big surprise there). Currently, I'm writing code in a fairly isolated part of the project, and I've been working a few days on getting a new class Just Right. Getting it Just Right, of course, means unit tests and test first coding. The upshot is that I find myself needing to run the build target and then the target that runs the test suite over and over. I like to run both ant targets at the same time, but I need to know if the build doesn't work (e.g., if I get a compiler error, I don't want to run the tests). Rather than monkey with the ant build file to add this transient behavior, bash comes to the rescue. More and more, I do all my command line stuff in bash for reasons like this. Here is the command I invoke 30 or 40 times a day (all on 1 line, it may wrap here):


if ant; then ant runatest -Dtesttorun=com.thoughtworks.logging.AllTests; fi


This runs the default ant task (which builds everything). If the return code is 0, it then runs the "runatest" target. The test runs only if the build was successful. This little trick takes advantage of the fact that ant is smart enough to tell the underlying OS (regardless of the shell) if it passed or failed. The "fi" at the end is the "end if" indicator for bash.

Yes, you can create the same kind of little batchy kind of thing in Windows, but I never think to. It seems somehow more natural to me to think in terms of automation, shell scripts, and command chaining in bash than it does Windows. Maybe it's because bash is far, far more powerful than the command line tools in Windows. I never run anything in a Windows command shell now that I unless I'm forced.

Notes from Singapore, Part 1

When talking to the natives in Singapore, I and a friend (Terry) heard about Durian ("Stinky fruit"), which is a local fruit. I always ask about local foods from the natives to see what I can try that I haven't had before. It is against the law in Singapore to take a durian on public transport. And it is forbidden to take the fruit inside most buildings. However, several locals at the table said they really liked it. My new mission: try some durian. We ate one night at Newton food court, which had fresh durian, so we got one. Tom, who lives in Bali , is a big fan, so he would eat it if we didn't care for it. Well, they cut it open and Tom insisted that we eat it essentially holding our nose -- the flavor and the smell are only loosely related. So I tried it. Frank said it best -- it's like a mixture of vanilla pudding and onion, with a kind of fruit-flesh/fishy texture. I tried it. It was durian-like. OK, I've tried it. The problem is that, even though I only had a bite, I kept trying it -- the taste would not go away. I ate some other stuff. Still there. Drink water, beer, whatever -- durian. After a while you can sort of get rid of the taste until you have the misfortune of burping. Durian. Stronger than ever. It keeps growing. It was well into the next day until I could taste something else. It took Terry even longer. In fact, he developed a semi-permanent association between Tiger beer and durian taste. He may never appreciate Tiger beer again. The other interesting aspect of this fruit: it smells. And it gets stronger and stronger. It was still at our table because Tom (for whom I have new respect mingled with pity) was gradually eating the leftover durian (all the durian except for one bite each from the other victims). Ingo (one of the speakers) kept asking Tom to move it further away, because the smell, while not exactly the same as the taste, is the olfactory equivalent of the taste. After you have smelled it, you smell it everywhere. For the rest of the trip, we could tell anytime we got close to durian. Terry and I would look at each other at the same time: durian. Every open air market you come to sells that stinky stuff. And you always notice it if it's near. You have been warned.

Singapore Slung

After a long, brutal day of travel, I've made it to Singapore for the SDA.NET Conference, where I'm delivering 3 talks on .NET development on behalf of ThoughtWorks. Total travel time from Atlanta, from the door of my condo to the door of the Pan Pacific Hotel in Singapore: 28 1/2 hours, 20 of that in the air. I saw 4 movies and slept through 2 more on the various flights. But, I'm here! And I have a nice view of the convention center from my hotel window. I hope to enjoy my first trip here to this beautiful, clean city. More details later!

XML Hell

If you programmed in or used Windows, you are all too familiar with "DLL Hell", where DLL's get over-written by newer (or older) incompatible versions. Writing applications as piles of DLL's made sense when resources (particularly disk drives) were expensive, but not so much any more. However, this blog entry isn't about DLL Hell. It's about a related phenomena in the Java world - XML Hell.

One of the saving graces when Java first arrived was the freedom once and for all from DLL Hell. Of course, DLL Hell was really just the creeping scourge of versioning, and it came to Java with a vengeance in the form of XML Hell. I've been fighting with this little problem for the last few days. XML parsers and XSLT engines are supposed to be backwards compatible...the operational words there are "supposed to be". I'm working on a project where everything works great under JDK 1.4.x but causes blue smoke whenever you try to build it under Java 5. Turns out the problem is the version of the XSLT processor that comes with Java 5. One of the vendors of a 3rd party library in the application relies on specific behavior in an older XSLT processor that accidentally takes advantage of a bug in the older processor. The bug is fixed in Java 5, and causes everything to blow up at build time. Sigh....

The only platform that has gotten this right is .NET when using Strong Names (which can be a little cumbersome, but the fundamental solution is a good one). That's one area that .NET has definitely leap-frogged Java: versioning libraries. I hope that Java someday addresses this problem.

Management Mind Tricks, Volume 1

Over my career course as a consultant-type person, I've developed some patterns for dealing with irrational pointy-haired boss types and the technical sycophants that sometimes accompany them, and I've passed these on to friends for their own use. One has recently been used successfully and so I thought I'd document it here. Fortunately, the friend who used it also gave it the memorable name that patterns are supposed to have: "The 'Let Them Say Any Crazy Shit They Want' Mind Trick Pattern".

Here's how it works. If you disagree with someone on a technical point, and you know you are right and they are wrong (either from misunderstanding the problem, politics, underwear too tight, whatever), let them make whatever claims they want about their solution. The executor of the pattern calmly reminds them that his way is better for these calmly stated reasons. Let the arguer point to a phantom requirement, or phantom performance problem, or any other specious claim, and the executor remains calm and points out why his way is better. Repeat this pattern until everyone is saying that the executor's way is better. It takes quiet confidence, zen-like calm, and unshakable conviction, but it works. The recent application of this pattern by my friend (in conjunction with another pattern not of my devising) took 8 man-days but, in the end, they are doing it the right way.

This pattern only works if you are technically unassailable, and you have to concede if the arguer ever does make a salient point. However, the zen-like calm in which it is delivered still keeps lingering resentment over disagreements to a minimum. To be used sparingly in situations where nothing else seems to work and you are being asked to do something stupid for no good reason.

Hedonic Adaptation

Have you ever noticed that you are just one more gadget away from pure technology bliss? If only I could get the newest back-lit, bluetooth, high resolution, 64 Mb turnip twaddler, I wouldn't ever need another gadget. But a funny thing happens a week after you get back from the electronics store...now, you need something else. I had recognized this pattern in myself, but recently discovered that, not only am I not the only one, but psychologists have named this: hedonic adaptation. This comes from studies of happiness. For example, lottery winners, right after they win, report that they are very happy. However, they quickly adapt, and, after a while, their sense of happiness and well being returns to their pre-winning levels. The same is true for technological innovations. No matter how dramatically a bit of technology improves our lives, we quickly take it for granted and start looking for the next thing. This helps explain why people in 2000 were not measurably happier than those in 1900, even though technology has made our lives astoundingly easier.

For more information about this, check out this fascinating article Technology and Happiness in Technology Review. And, as much as it seems like it, that new Powerbook with Tiger on it won't make you blissfully happy...Well, maybe just a little. I'm willing to try.

Test-first Mine Sweeper

I don't play many games on computers (too much other stuff to do). So, some of you will be shocked and mortified that I only recently discovered that, in Minesweeper, if you both-button click on a number, it will either highlight the un-clicked cells adjacent to it or go ahead and uncover cells that you would know are already mine-free. It's harder to explain than to do it. Some of you can't believe that anyone that has been using Windows for better than 10 years is just now finding this out, and the others can't figure out what I'm talking about. For those who don't know, dash over and try it...I'll wait.

Back? OK, here's the thing. When I do test-first development, I write the test and go ahead and run it, knowing full well that it will fail. It's sort of a Zen thing, I guess. I know that it takes a little more time, but it's part of the satisfaction of seeing it fail then succeed shortly afterwards. Even this isn't so unusual.

I've found that I do the same thing when I play (what is to me) the New and Improved Minesweeper -- I do test-first clicking. Even if I know that there is only one possible uncovered cell nearby that this number can reach, I still double-button click on it to satisfy myself that it is the only one. It's test-first Mine Sweeper.

I really should get out more.

Google Desktop Search + Java

One of my favorite pieces of software that died while trying the Extreme Sport of transitioning from DOS to Windows was Magellan. It was a file manager for DOS that indexed your hard drive, then showed you the results in a list. Searching for some phrase that was on your drive was instantaneous. Now, you can get the same thing with Google Desktop. My only complaint is that you can't tell it what types of files you want it to index -- it defaults to things like Office documents and emails. But, lo and behold, there is a plug-in API. And someone has written a Google desktop plug-in that indexes Java source files. Plug it in, let it trundle a while to build the index, and you have instant access to interesting stuff you have in your Java files. Great for spelunking around a deep and wide object hierarchy. Because it's indexed, it's as fast as Ctnl-N in IntelliJ for finding classes, but it works over the entire file system, even on really large drives.

It indexes on the fly (it's amazing how fast it picks up new files), waiting on CPU idle time to kick in. The indexes on my machine currently take up about 1.5% of the disk. It changes the way you search for things, though...I'll give up 2% of my hard drive to get instant access to needles in a 30+ GB haystack.

Helpful/Annoying Software

Helpful people are annoying. Robert Fripp Aphorism

I can't stand chatty software that's trying to be helpful. I understand the motivation (Grandma needs help when she runs (or, more likely, walks) the computer) but I can't stand it. The first thing I do when encountering a fresh Windows install is to turn off all the cute animations, tips, offers for tours, etc. Bob or his illegitimate son, Clippy, anyone? My latest rant-inducing episode: I just installed Visual Studio .NET 2005. It very helpfully rewired my operating system to open all .java files in Visual Studio when I double click on them (because they must be Visual J# files, right?) Arrrgh! It's not like I can't change it back, it's just annoying.

As a public service, here is how to turn off those nattering ballon tips that appear over the task tray and never seem to go away. Thanks but no thanks:

Registry Key: HKEY_CURRENT_USER\Software\Microsoft\Windows\CurrentVersion\Explorer\Advanced

Data Type: REG_DWORD [Dword Value] // Value Name: EnableBalloonTips
Setting for Value Data: [0 = Balloon Tips Disabled / 1 = Balloon Tips Enabled]

Exit Registry and Reboot

Also as a public service, if you don't know about it, there is an entire web site dedicated (no, devoted) to fixing annoying software behavior: www.annoyances.org. Talk about a cottage industry with guaranteed growth...

Andy and Joel

I've been catching up on some geek reading lately. You know those books that twinkle at you from the bookshelf, when you know good and well you should read FooBar in Action book or some other varmint-ridden OReilly book, some books entice with their "You don't need to read me, you want to read me" smoky looks. Well, I succumbed recently. I had a trip and decided that instead of taking "normal" reading material, I would take two enticing geek-tomes. My choices? Revolution in the Valley: The Insanely Great Story of How the Mac Was Made by Andy Hertzfeld and Joel on Software: And on Diverse and Occasionally Related Matters That Will Prove of Interest to Software Developers, Designers, and Managers, and to Those Who, Whether by Good Fortune or Ill Luck, Work with Them in Some Capacity by Joel Spolsky. While at first glance, it looks like I picked the 2 books with the longest sub-titles, that wasn't the case.

Both books are great reads, but the interesting juxtaposition between them is the cultural whiplash you get if you read them side-by-side. The Mac book is all about figuring out The Coolest Way to Do This Thing That Hasn't Been Done Before, mostly in Motorola 68x assembly language. It's frightening to think that the first Macs (you know, the ones with a graphical user interface) had only 128K of memory! The overriding concern was uncompromising excellence. Joel's book is much more pragmatic. It has lots of good advice for anyone engaged in software development. In fact, lots of the advice that I foist on people in my Clean Up Your Code talk overlaps with some of Joel's advice. In one of the chapters he talks about the battle within Microsoft between the pragmatists and the visionaries.

Which brings up the interesting question: when should you be pragmatic and when should you be visionary? If the Mac guys had been pragmatic, we would have had a slightly sexier Applet ][e, not the Mac. But, being visionary doesn't mean that you win or even survive (see BeOS, Amiga, NextStep, the list goes on and on). That's the razor's edge we talk as technologists -- figuring out when to be pragmatic (and get paid, keep a job, and other mundane concerns) and when to say "What the f%#k -- I designing a dialog box with rounded corners because it's never been done before!". For more information about this dilemma, see Rails, Ruby on.

Aspects in the Business Layer

While I was at No Fluff, Just Stuff Boston this last weekend, I got into a conversation with a couple of interesting guys (sorry, fellows, I didn't get your names) at lunch who had attended my Enterprise Debugging session. This always happens at No Fluff, Just Stuff events -- the conversations with other speakers and attendees at lunch and in the halls is at least as interesting as the talks. Anyway, we started talking about aspects and how mainstream they are slowly becoming. I stated my position (which I blogged about earlier) that I believe that they will gradually make their way from the boundary layer into the business layer much like OOP did (see my previous blog entry about this theory). Neither of them bought it -- they were both convinced that aspects are forever going to reside in the boundary layer.

Then, I challenged them: name some cross-cutting business concerns. We thought about it for about 30 seconds and came up with several: legal, regulatory, sales & marketing. These are business concepts that cross-cut the traditional business layer, needing just a little attention in lots of places. And that's why aspects exist. This re-affirms my thinking that we will see this type of code more and more as aspects become more mainstream. It took 1 believer and 2 skeptics about a minute to find where aspects fit outside the business layer. Imagine what we would find if we actually put some effort into it!

Asperger's

Who does definition sound like:

This person has an intense and obsessive level of focus on things of interest and is often characterized by special (and possibly peculiar) gifts; one person might be obsessed with 1950s professional wrestling, another with national anthems of African dictatorships, another with building models out of matchsticks. Particularly common interests are means of transport (for example trains), computers, and dinosaurs. These interests are often coupled with an unusually high capacity to retain and recall encyclopedic amounts of information about the favored subject. In general, orderly things have appeal to these individuals, and they often manifests extremely sophisticated reason, an almost obsessive focus, and eidetic memory.

Sound like most of the developers you've ever met? Well, it's the Wikipedia definition of Asperger's Syndrome, a really mild form of Autism. Instead of seeing something like autism vs "normal" mental development, it's really more of a sliding scale. You've often heard that developers have special in-born characteristics -- I think these folks are a little more towards Aspergers than most. This explains a lot about the social skills and concentration abilities of really brilliant developers who don't interact well with the other humanoids. And why we can't get dates.

Update: Coincidentally, the very night that I posted this blog, I went to dinner with someone (very well respected writer and speaker) who has a child with Aspergers. He was telling me that this is a real problem in Silicon Valley -- geeks marrying geeks and having borderline autistic children. Talking to him made me realize that this entry might be construed to trivialize Asperger's Syndrome. That is not my intent at all -- it is a serious condition. However, talking to my friend drove home the point I made above even more strongly.

Leaving DSW

After 11 1/2 years, I'm leaving The DSW Group to take advantage of an extraordinary opportunity at ThoughtWorks. I have had a wonderful run at DSW, and both I and the company have benefited from my years of working there. I leave DSW on very good terms, leaving lots of friends (not just coworkers) behind. While Terry and DSW didn't want me to leave, he understands that I have a unique career opportunity in ThoughtWorks and he wishes me all the best.

Consequently, my email addresses at DSW will no longer be active. If you need to reach me, you can do so at nford <at> thoughtworks.com or at neal.ford <at> gmail.com. Of course, you can keep up with my travels, career, and hobbies still at www.nealford.com and here at my blog.

Thanks again to everyone at DSW for making my employment stay there extremely rewarding on a personal and professional level and I wish them all the best as I move to another chapter in my career.

Meeting Pure Evil -- Socially

Last Saturday night, I bumped into Pure Evil while at a party thrown by a neighbor. I didn't think I was going to meet Pure Evil, and didn't even realize who it was until we had chatted a bit. But there he was, with a wine glass in his hand. He (and I won't mention his name for reasons you'll see in just a bit) is a former executive of several software companies and has been (and is currently) a consultant. He is now engaged in 2 lines of business: the first is to act as expert witness in dubious patent lawsuits for software patents that should have never been granted. The company he works for finds outrageous patents (they have found someone who has allegedly patented ecommerce -- the whole process) and sues major software companies. While ethically a little shady, this is not illegal. And I know that some smart people agree that you can patent software and processes. The ambulance-chaser way they are doing this, however, is a little creepy.

Turns out that this was just the appetizer. His other line of business is to sell small to medium sized business on as much enterprise software as possible. Things like SharePoint, PlumTree, Biztalk, etc. He really, really likes Microsoft back-office software because, as he puts it, "it's very easy to install". I asked him if most of his clients really need that much stuff -- wouldn't a wiki or something lighter weight solve the problem more elegantly? He was dumbfounded at the question because he can't charge big money for the care and feeding of that kind of software. He also professes to like .NET a lot because it is so extremely RAD. I asked him if doing development like that didn't cause problems during the maintenance phase of the application. After all, that's where you pay for rampant RADism, trying to find all that code you've sprinkled throughout your user interface in event handlers. His response: "We're generally through with the contract at that point, or leave for other reasons, so we don't have to worry with that too much". To summarize: sell them as much software as possible (that we don't have to understand very well), slam together applications that quickly become un-maintainable, then get out of Dodge before the repercussions hit. Clearly, this person and his company are trying hard to make sure that consultants are universally loathed.

People like this have the ethics of slave traders. Back in the day, slave trading was legal, and I'm sure it was very profitable...and morally repugnant. Mr. Pure Evil was also bragging about how much money he made last year ($350K). But how does he sleep at night? It made me realize first hand that you can randomly bump into people like this, who care for profit at the expense of ethics and brag about it. Excuse me, I've got to run take a shower...

Robert Fripp's Aphorisms

Robert Fripp's Aphorisms

If you recognize the name in the title, it is unlikely for the reasons I'm going to write about today. Robert Fripp was (and is) the guitarist and de-facto leader of the only surviving original progressive rock band, King Crimson. While the others (like Yes, for example) still tour in a Dinosaurs Roam the Earth mode (still playing their music from the 70's), King Crimson still tours with new music, and it is very new music indeed. Robert Fripp also founded a school for guitarists called Guitar Craft. It is not a permanent school, but one that moves from place to place, holding week-long classes for all levels of guitarist. I haven't been to one, but I've read about them. Really, they are as much about attitude and discipline as about playing music. There have been several graduates that have gone on to careers in music, most notably The California Guitar Trio.

However, this post isn't about that either. One of the rituals of Guitar craft is the recognition of aphorisms, or quotes that have special meaning in a specific context. The dictionary definition of aphorism is "a concise statement of a principle". Like:

The attitude that life owes us something, if not everything, encourages life to thwart our endeavors.

Taken in the context of guitar craft, these aphorisms relate to (at the macro level) life in general and (at the micro level) about the craft of playing the guitar.

We begin where we are.

However, there is another context with which to read these aphorisms. Fripp is legendarily disciplined, and there are numerous published examples of this trait, perhaps best is his own on-line diary (established before the term "blog" was coined). Another is the (unfortunately) out of print biography by Eric Tamm. These aphorisms, being "concise statements of principle", apply broadly to life and particularly leading a disciplined life. And that is something to which everyone should aspire. Reading and pondering these random aphorisms over the years has helped me write (or, more precisely, finish) books. And they have helped me finish arduous endeavors like Ironman.

Process is Intelligence getting to know itself.

Until you understand Fripp's context, some of the aphorisms don't make much sense. However, over years of reading, you understand the context and draw meaning from all of them.

Relaxation is never accidental.

If you want to see some Fripp aphorisms, go to the Discipline Global Mobile web site. At the bottom of the page is a random CGI script that randomly show aphorisms while you are on the site. One of my favorites (I have part of this engraved on the back of my iPod now) is one about the relationship between music and silence (in many ways, music frames a particular kind of silence):

Music is the cup that holds the wine of silence. Sound is the cup, empty; noise is the cup, broken.

In many ways, software development relates to music: a creative endeavor facilitated by dexterity, tools, and discipline. And, correspondingly, many of the aphorisms apply eerily to software development.

Address the process rather than the outcome.
Then, the outcome becomes more likely.

Enjoy!

Mourning Superior Dead Technologies

I was thinking this morning about buying a Macintosh, which lead me to ruminate on the fate of sometimes superior technologies that nevertheless fade away (not that I think that's happening to the Mac -- honest!). I can think of a bunch of examples off the top of my head: NextStep (I only heard about it, never got to use it), Magellan (the greatest navigation aid for a file system ever, never made it to Windows), Sprint (the best Word processor, never made it to Windows), and the list goes on and on. Even something like IntelliJ, which is creepy smart (side note: today, I realized that it will generate default variable names for you, based on the class type, even make good guesses -- for JButton, it offers both jButton and just button) -- I wonder if it can withstand the onslaught of Eclipse?

So, I reached a conclusion: don't mourn dead technologies. Use the best while it's available, but don't lament what could have been. I'll support Apple, IDEA, and the other technology stuff that I think exemplify good design. But, if market forces deign that they not survive, I won't fret -- just move on the find the next best thing. Besides, the really great ones sometimes live again, in different guise: OS X is really just NextStep reborn, Ruby is SmallTalk with some differences. Now, excuse me while I go shopping at the Apple site, with my iPod firmly implanted in my ears...

Software "Engineering"

It seems like every time someone talks about the job/craft/calling of writing software, they tend to end up with a bunch of tortured metaphors (engineering, woodworking, rock climbing, gardening, etc.). None of these analogies hold up past just superficial scrutiny. I think that it's a reflection on how different it ultimately is.

But to use one of these tortured metaphors for a moment, think about bridge building. I had someone at No Fluff, Just Stuff Philly say that her company had starting outsourcing much of their coding to India. I asked her how they judge the quality of the software. She said that she looks over it, and they have a software architect that comes to look at it periodically. Unit tests? None.

What if you decided to build a bridge? You outsourced all the design to guys who say they can build bridges, and you have someone who has built a few bridges look over the design when it comes in. Would you drive over that bridge? What makes bridges safe? It's the engineering science behind the bridge -- the mathematics of structures, derived over a great many years. So, why don't we apply the same rigor to software? We can't. We don't understand enough about it to quantify it in that way (and may never -- that's where the engineering metaphor breaks down again).

So, what's a poor developer to do? Actually, we do have a great option -- testing! It's extraordinarily difficult to build all the parts of the bridge and test them individually, then test it as a whole. However, because software is so soft, you can test everything. And you should. That's as close as we can get now to the rigors of a "real" engineering discipline. So we'd better damn well start doing it!

Toogle

I read about (and tried) an interesting new site today, Toogle. It performs a search for whatever you type in and returns it as an image generated from the words of the search, using different colors to mimic the colors (and therefor reproduce) the item you searched for. Trying to explain it is fruitless -- just go try it. I still haven't figured out how it decides what to return (a search for my name returns the JBuilder 3 Unleashed book, which is ancient history, not my most recent book), but it's still a fun idea.

Slowly Turning XMLese

Quote of the week, appearing on several other blogs (and in the March 14 issue of eWeek), attributed to Chris Maden:

"XML is like violence: if it doesn't solve your problem, you aren't using enough of it"

Nuff said.

Favorite New Eclipse Plug-in

Generally, I'm at the mercy of the client as to which IDE I use for Java development. Consequently, I try to find tools in each IDE that replicate favorite features in my favorite IDE (currently IntelliJ). Right now, I'm working on a project in Eclipse (the 3.1 M5a version, which has pretty good Java 5 support and isn't too buggy). One of the features I miss the most is the "Go incrementally find this thing", where "this thing" is a class name, part of a class name, or even some special versions of a class name (for example, to find TuPopupMenuManager, you can search for TPMM and it will find it). IntelliJ has this, and, no matter how many times you whack the side of the monitor, Eclipse does not...

...Unless you install my favorite new plug-in, GotoFile. It does just what I want it to, just like IntelliJ. And it even has preferences. Works with any version of Eclipse back into the dim and distant past. And, it's the Eclipse feature I use most during the day.

Aspects and Undoing

I was talking to Ramnivas Laddad (the person who knows the most about Aspects than anyone I know) at the Philly No Fluff, Just Stuff conference, hoping to dazzle him with an idea I had recently about a cool use for Aspects. My idea: using aspects to implement the Memento pattern for undo. The plan: use attributes in Java 5 to flag which fields are undoable, then write an aspect that injects the appropriate code into an inner class of the undoable class (side note: I prefer to use inner classes when implementing Memento, because they meet all the requirements of the Memento but don't break encapsulation because the inner class has a reference to the outer one's private fields).

Of course, someone has already had this idea, which Ramnivas kindly pointed out (check out Jon Tirsen's blog entries here and here). My only contribution to this discussion is the use of attributes in Java 5 to flag the undoable fields -- the rest had already been figured out. Just when you think you have a clever idea, inevitably, someone else has beat you to the punch. I do think this is a great use for Aspects, though.

No Fluff, Just Stuff Philly

Just got back from No Fluff, Just Stuff in Philly (my first NFJS of the year). I had a lot of talks this time (4 in Java, 4 in .NET), spread over 2 days. That was too bad because, as usual, there were a bunch of talks from other speakers that I really wanted to see. That's almost always the case.

No Fluff, Just Stuff attendees continue to surprise me with the level of discourse that occurs, both in the sessions and out. You are hard pressed to find someone to talk to that doesn't have something interesting to say. The expert panel was interesting (if for no other reason than to hear Ted Neward bicker with everyone). And the Birds of a Feather that I did with Scott Davis and Eitan Suez was also very interesting (we were supposed to just talk about Web Frameworks and User Interface stuff, but we ended up in a fascinating, wide ranging discussion covering outsourcing, certification, dynamic languages, and, oh yeah, web frameworks).

Fun stuff.