Technology Snake Oil, Part 4b: Prescriptive Approaches in the Large

I wrote recently about Prescriptive Approaches in the Small. More pervasive and dangerous, though, are Prescriptive Approaches in the Large. Everyone wants a repeatable pattern, a guaranteed way of doing things. Recently, at No Fluff, Just Stuff Columbus, OH, one of the questions for the expert panel was: “Our company is looking for a road map, a sure-fire prescription of what technologies, methodologies, and approaches we should take – can you provide this?” Uhhhh, no. Most of the panelists nearly crawled out of their skin to get in line to answer this question (I thought Prag Dave was going to have a heart attack – if you know him, you know this is anathema to him!). There is no prescription for software development done right in your company – every place is different. You cannot come out and say that this approach (whatever it is) is right in every circumstance.

Following this line of reasoning, I think that the PetStore was one of the worst evils inflicted on the young, innocent, fawnlike Java Enterprise Edition. It was supposed to be a catalog – “Look, here are all the things this shiny new technology can do” – and people took it as a prescription. Rather than sit back and determine for what EJB was suitable, developers dove right into using them in their projects because the prescription said so. EJB was (and is) suitable for only very specialized types of applications. If you were writing a real pet store application (even back when PetStore came out), you would be crazy to use EJB!

The pattern repeats over and over, with technology (“Everyone is using Struts, so we should be too”) to methodologies to products. You can bet that the SOA vendors are salivating over the prospect of creating the prescription that says “Everything must be decoupled, document centric messages – here, we have an expensive product that just happens to do this!” As much as it hurts, you have to make determinations for yourself on which technology (methodology, framework, tools, etc.) are best for the situation you are in right now. There is no prescription, no matter what ersatz technology doctors tell you.

Joel is Plain Wrong about XP

Joel Spolsky writes in a recent blog entry:

As I worked through the screens that would be needed to allow either party to initiate the process, I realized that Aardvark would be just as useful, and radically simpler, if the helper was required to start the whole process. Making this change in the spec took an hour or two. If we had made this change in code, it would have added weeks to the schedule. I can’t tell you how strongly I believe in Big Design Up Front, which the proponents of Extreme Programming consider anathema. I have consistently saved time and made better products by using BDUF and I’m proud to use it, no matter what the XP fanatics claim. They’re just wrong on this point and I can’t be any clearer than that.

Joel has an interesting perspective on this subject and has written quite eloquently on this and other topics (I highly recommend his books and blog). However, I think he’s just wrong about this. He has a skewed view on software development, having worked at product companies his whole career. Building products is a lot different from building business applications (which is where my experience lies). First, for products, the requirements and design are pretty much made up of a few people’s vision on what the product needs to do. The market place determines if they were right or not. Second, you generally have to do this in secret, especially if you have competitors who will cherry pick your design. Third, you are pretty much required to do a “big bang” deployment, when you ship to customers.

Most business applications aren’t like this. The requirements are very fluid (quite often, the business stakeholders don’t know what they want even well into the process). When done correctly, the process is collaborative across business units and developers, making for a stronger product. Last, you can roll out iterations (even if just to QA and/or acceptance testing) without the big bang.

Joel is using the straw man arguments against XP. Many misinformed naysayers against XP assume that we never think about what we’re doing up front. Of course we do. This same group assumes that there is no coherent architecture for complex XP projects. Just because we’re doing agile development doesn’t mean that we just sit down and hack out code with no planning or forethought. Believe me – I’ve had lots of experience building business software, and Big Design Up Front just plain doesn’t’ work, even when it is done well (I’m fighting it yet again even as we speak). XP delivers better quality software, in shorter times, closer to the ultimate target, which has a fighting chance of actually usefulness.

Taking this topic to the logical conclusion, I wonder what would happen if a product company did a full-on agile approach? Building the product based on custom feedback from the early iterations. You would need a patient customer base, willing to comment on the work in progress. I can’t help but believe that, just as in the business software world, you would end up with a better product. How many features could you eliminate in something like Word if it had been developed incrementally? So much useless cruft could be removed from commercial software if product teams listened to their customers while developing the software rather than the complaints after it is done.

Some companies are tiptoeing in this direction. Check out JetBrains Early Access Program for innovative tools like Fabrique and MPS (The Meta Programming System). I think these products will be extraordinary if they can get their target audience to give them feedback. Agile product development – what a concept!

My Blog, in Korean

To satisfy overwhelming demand (OK, there is no such demand), Trustin Lee is translating some of my blog entries into Korean and posting them here. To quote:

I'm an Apache Committer who is still on the way of studying English. I'm trying to level-up my English skill translating great blog entries from great authors.

I think that's more than a bit generous, but I'm gladly supporting this effort because, hey, it isn't costing me anything and it spreads my meaningless opinions all over the world. What an unbeatable combination. Seriously, I appreciate the hard work Trusten is doing for this, and I hope that my dribbly writing is worth translating.

Now, I’m really interested to see if he translates this blog entry, creating a kind of recursive blog referencing stack overflow.

Technology Snake Oil

Technology Snake Oil, Part 4a: Prescriptive Approaches in the Small

The world is a much simpler place if you can just coast through it, without having to expend precious mental energy. In fact, it seems widely encouraged in many organizations: if we can prevent our employees from thinking too much, they will stay out of trouble. Unfortunately, to build insanely great software requires thought on many different levels, which doesn’t go well with prescriptive approaches.

You see all sorts of prescriptive approaches in software development, both large and small (this entry covers small, the next Snake Oil piece covers large). Small prescriptive manna floats down in many different guises, mostly in a good context like design patterns and best practices. Many developers take these nuggets out of context and start believing that this Prescription for building all types of things represents the One True Way (without bothering to apply additional thought). Don’t think that I’m denigrating design patterns – they are clearly a good thing, as a way to catalog common patterns. The problem comes in when they are used without thinking.

Here is a classic example that appears all over the Java code I see every day. Java doesn’t really have a struct-type thing, which is just a holder for data. In Java, everything is a class, which is OK. Nevertheless, I see developers who just need to pass composite data around creating a Java class with accessors and mutators that won’t every be used for encapsulation. It has become a common prescriptive practice in Java to create the class, create private member variables, and generate getters and setters for them without remembering why you create accessors and mutators. If this class is never going to have any behavior attached to it, if it is a simple container for information, you don’t need the accessor/mutator pair or the private scoping – make them public. If you really need to add behavior later and/or encapsulation, you can refactor the class into a real class, not just a composite data type.

The real sermon here is to think about why you create things, don’t just blindly create artifacts. Yes, encapsulation is generally a good thing, but isn’t required in every single thing that looks like a class – sometimes it’s really just a struct. You must always be diligent about why you are creating what you create.

Fighting for Your Life

Fighting for Your Life

I was almost voted off the island recently.

Because of some scheduling issues involving me rolling off a project, another one starting, and the slated person still a week away of rolling of his project, I was in the unfortunate position of going to a client's site for just 1 week, at the start of a new project. The first day, I arrived late because of travel details, and we spent the day going over the technical details of what it was we were going to be doing. All fine and good. In fact, one of the overviewers was kind enough to identify something he thought I could do in just a week, something orthogonal to the rest of the work.

The next day I get to work and the client supervisor asked me into his office. "Neal, we just don't think the one week thing is going to do us any good, we like you and all, but you should just go ahead and leave." He was trying to kick me off the island! The last thing he said was "Unless you can think of a good reason we should keep you..." That was my chance. I said "Well, of course, it is your decision, but I have found something I can get done in a week", and went on to tell him about the one-off sub-project we had identified the day before in technical detail. He listened to my case, then called the technical lead into his office and had me repeat it. The tech lead said "Well, we do need that piece, and I think he can get it done in a week". The supervisor turned to me and said "Alright, get back in your office and get to work!". Saved!

And it turns out that I was able to get the work done in a week, and the client requested that I stay on instead of the guy originally slated for the gig. How is that for defending your life?

An Actual, Real, Everyday Use for XSLT

I have played with XSL and XSLT for a while, and even toyed with the idea of basing my whole website on an XML+stylesheet vision. Because XSLT can get complicated, that idea died before its time. However, I recently found a really good use of XSL on a development project on which I was working. This project included the ubiquitous Log4J library, including a largish logging.xml document. My job was to make sure this document is still relevant. The problem with trying to analyze such a document lies in the way that items reference each other. Instead of pulling my hair out bouncing around in the XML file, I created an XSLT stylesheet to format the whole thing into HTML, with links from loggers to appenders and appenders to loggers. The end result looks like this (broken into 2 pieces for space savings):

The stylesheet that generates this info from the XML document is not long or complex:

<?xml version="1.0"?>
<!--  XSL Stylesheet to make it easier to determine the relationshps between -->
<!-- the loggers and appenders used in the application.                      -->
<!-- This stylesheet is automatically linked into the logger.xml document.   -->
<!-- If you are using any reasonable browser, this stylesheet should be      -->
<!-- automatically applied to this file.                                     -->


<xsl:stylesheet version="1.0" xmlns:xsl="http://www.w3.org/1999/XSL/Transform"
   xmlns:log4j='http://jakarta.apache.org/log4j/'>

   <xsl:template match="/">
         <html>
           <head>
             <title>
                 Loggers
             </title>
           </head>
           <body>

             <xsl:apply-templates select="log4j:configuration" />
           </body>
         </html>
   </xsl:template>

   <xsl:template match="log4j:configuration">
       <h1>Project Loggers</h1>
       <h2>Loggers</h2>
       <table border="1">
           <tr>
               <th>Logger</th>
               <th>Priority</th>
               <th>Appender(s)</th>
           </tr>

           <xsl:for-each select="category">
               <xsl:sort select="@name" />
               <tr>
                   <td><xsl:value-of select="@name" /></td>
                   <td><xsl:for-each select="priority" >
                       <xsl:value-of select="@value" />
                   </xsl:for-each></td>

                   <xsl:for-each select="appender-ref">
                       <td><xsl:value-of select="@ref" /></td>

                   </xsl:for-each>
               </tr>
           </xsl:for-each>
       </table>
       <h2>Appender to Logger Relationship</h2>
           <ul>
           <xsl:for-each select="appender">
               <xsl:sort select="@name" />
               <tr>
                   <li><xsl:value-of select="@name" /></li>
                   <xsl:for-each select="param">
                       <xsl:if test="@name = 'File'">
                           [ <xsl:value-of select="substring-after(@value, '${logging.home}/')" /> ]
                       </xsl:if>

                   </xsl:for-each>

                   <ul>
                   <xsl:call-template name="show-categories">
                       <xsl:with-param name="appenderName"><xsl:value-of select="@name" /></xsl:with-param>
                   </xsl:call-template>
                   </ul>
               </tr>
           </xsl:for-each>
           </ul>
   </xsl:template>

   <xsl:template name="show-categories">
       <xsl:param name="appenderName" />
       <xsl:for-each select="//category">
           <xsl:for-each select="appender-ref">
               <xsl:if test="@ref=$appenderName">
                   <li>
                   <xsl:value-of select="../@name" />
                   </li>
               </xsl:if>
           </xsl:for-each>
       </xsl:for-each>

   </xsl:template>

</xsl:stylesheet>

I also added a reference to this stylesheet in the logger.xml document itself:

<?xml-stylesheet href="ShowLoggerRelationships.xsl" type="text/xsl" ?>

The beauty of this is that any reasonable modern browser will automagically apply this stylesheet anytime you double-click on the XML file. This is the ultimate form of documentation -- generated directly from the source in real time. Any changes to the relationships in the logger.xml document are instantly shown when you look at the file. If you still want to see the raw XML, you right-click and "Open With..." instead. This makes the browser view of the XML document useful without hiding or obscuring the functional elements. And is a good example of the DRY principle (Don't Repeat Yourself) from the Pragmatic Programming applied to documentation.

Technology Snake Oil, Part 3: MDA

Model Driven Architecture (or "Cartoons to Code") is the ultimate culmination of the vision of UML. Let's draw the software with diagrams, including decorations to represent logic, then click a button to generate the code. A laudable goal, but far short of reality.

The only real success stories with MDA are for very limited problem domains, like embedded software. Here's why: consider the cash register at McDonalds. The number of items is very limited, and the number of things you can do with them are very limited. This is well suited to a pictorial (in fact, an ideogrammatic) representation. That's because the concepts on display are retricted to morphemes, which in linguistic terms represents the smallest language unit that carries a semantic interpretation. Thus, it's easy to create a McDonalds cash register using just pictures and make it a quite effective solution to the food ordering problem.

For complex problems, you can't break down the problem to simple morphemes -- imagine the semantic complexities of a concept like "loan" or "payment". For a problem domain this rich, you need the richer representation of a full-blown language (which the Egyptians also discovered -- they were the last MDA success story, and they eventually abandoned it!). Trying to cram rich semantics into a pictorial representation so that you can create software that abstracts those semantics is absurd.

In fact, trying to cram the entire world into a strict tree-shaped hierarchy isn't much better (see Programming, Object-oriented). At least OOP has more flexible semantics, and we keep forcing solutions to this hierarchy problem with inventions like aspects, which cut across the tree-shaped world we've created (because stuff in the real world cuts across trees). So, where do we go from here?

Let's solve it the same way mankind (including the Egyptians) did -- create language! Language is ultimately the only way to handle the rich semantics of the world around us because that's what the people stating the problem domain use to state the problem. Which is why I'm so geeked about Language Oriented Programming -- building domain specific languages that are as close as possible to the problem we're trying to solve. LOP done right encapsulates the low-level details of a language like Java, allowing developers and others to work at a higher level of abstraction. We've been mired in curly-brace languages too long -- we need to upgrade our abstraction.

MDA is the wrong direction for abstraction upgrade. It tries to further restrict our expressiveness, believing that the entire world can be graphed and you can create real software this way. Imagine for a moment the MDA diagrams (including Object Constraint Logic) for your average J2EE enterprise application. You can build it in code, with the crude semantics of curly-brace Java in 1/3 the time you could model it to the point where you can click the "And Then a Miracle Occurs" button to generate the application. MDA might work for simple, semantically unique domains, but no where near the real world.

Happpy with What You Have to be Happy With

Recently, I did a 3-hour tutorial workshop at the SOA conference in Singapore entitled "Agile Development and SOA", where I talked about aspects of Agile development as it applies to SOA type projects. Mostly, it was an excuse to talk about Agility in a foreign land.

I asked the attendees what type of methodology they were currently using. The three replies I got were 1. None (what's a methodology?), 2. Waterfall (although she didn't know that name for it), and 3. SDLC (the Software Development Lifecycle (another variant of Waterfall, check out this interesting Wikipedia image of SDLC )). So, basically, everyone is using some form of Waterfall. What's interesting is that I asked to most vocal of the crowd (meaning that she would actually answer without me jumping up and down on the table in front of her -- she was the SDLC victim) if her company was happy with that methodology. She quickly replied "Oh, yes, it works very well for us". Needless to say, I was surprised. So I asked her a few follow-up questions:

• Do you ever miss deadlines? Oh, yes


• Are your user's happy with what you produce? Well, no


• Are your developers frustrated by vague requirements, which leads to user dissatisfaction? Yes


• Do you ever accidentally break code in your "big bang" deployments, which takes a long time to fix? Yes

I asked a few more, but you get the gist. She thought they were very happy with SDLC because they didn't know that something better existed. Perfect crowd for an agility talk, right? It was an uphill battle. No one could believe that this crazy idea could ever work -- waste all that time writing tests? Include the users as part of the development team? Pair Program?!? That cuts your productivity in half! Sigh. I waged a good battle, but don't know how well the war is going. At the end of the talk, I had either convinced them to tip-toe towards Agility (because these were developers -- can you imagine what their boss will say?) or they were just tired of me talking about it and said anything to get me to shut up.

I've blogged about this phenomenon before -- why is it that developers and managers believe that all forward progress in methodology stopped in the mid-70's and all language development froze about the time Objects appeared? Is it just a non-technical person's fatigue at trying to keep up, so an arbitrary milestone was created beyond which "I won't think about it anymore?" Are developers so overwhelmed that they just shut down too, and keep working on what's in front of them even though better things abound?

Technology Snake Oil, Part 2: RAD

Back in the mid-nineties, Rapid Application Development was the answer to all developer woes. RAD tools came out in droves and everyone touted the huge productivity gains made available by drag and drop user interface design and component based development. RAD environments made for great demo-ware -- you could stand in front of a group of Pointy Haired Boss types and build a working application right before their eyes while you were talking about it (I know because I did some of this -- and I still have trouble sleeping sometimes). Of course, the message was clear: buy This Tool, and you too can have developers that can point and click their way to huge productivity gains. And bought it they did.

Then a funny thing happened to the developers on the way to this astounding productivity. They learned the tools and dutifully dragged and dropped their way to building applications...and it was good. Until, that is, they figured out that building the application is the short, easy part -- now they have to maintain the monstrosity they've created. RAD development doesn't scale well, and when you build large and very large applications with it, maintaining those applications is a nightmare of searching for the scattered crumbs of code scattered hither and yon. Working for a while in this environment is what made the Design Patterns book really hit home -- the kind of code we were creating was entirely anti-pattern, and more about "how fast can it be created"? The maintenance nightmares led me, as CTO of a consulting and development company, to figure out new ways to use the RAD tools we had to use (because of our clients) that did follow good design principles. What we found is that the tools fight you when you try to do the right thing. RAD Development tools have a path of least resistance, which makes things very easy while you stay on the path (and use their wizards and other support structures). Stray from the path, and you find obstacles at every turn. But, it's worth fighting through them to get to a place where you can create maintainable code -- and you find that you are no longer doing Rapid Application Development, just Application Development.

RAD is now rightfully deprecated in most serious software development efforts. Even serious developers using RADish tools like Visual Studio are avoiding the RAD pitfalls. Some developers still fight with RAD tools, and think that maintenance is just always extremely difficult in big projects. I'm still shocked by the developers who have still never heard of Design Patterns and who still work on Sisyphean development chores.

Which is why I think the RAD aspect of Ruby on Rails misses the point of what makes Rails interesting. RAD doesn't work for large, complex projects, no matter what technology. The interesting part of Ruby on Rails is the effective use of Ruby to build a domain specific language on top of the underlying Ruby language that makes it easier to do web applications with persistence. It is more advanced than the RAD tools of old because you can drop to the underlying Ruby language and get underneath the framework when you must -- and that's a powerful idea. But, building complex applications in it still requires real thought. Maybe the breathless amazement of the Rails community reflects the absence of effective RAD tools in this space before. Don't get me wrong -- I think that Rails is cool, just not for the reasons touted by so many Rails-flavored Kool-aid carriers.

Technology Snake Oil, Part 1: UML

I was told by someone recently at ThoughtWorks that I was so old that I "added to the company's diversity". Well, I've decided to take this sage old-timers wisdom and put it to some good use. In the development world, we keep getting sold Snake Oil in one guise or another, and I'm going to start pointing out some snake oil. Starting with the most venerated of crappy technology: UML.

Some of us craggy old-timers remember the time before UML, when we had Booch, Rumbaugh, and several other competing diagramming notations. The Holy Grail was going to be the Unified Modeling Language, which would solve all our problems and finally give us software developers a universal language, for everything from use cases through structure to deployment. And that's what we got. A classic example of "be careful what you ask for". UML feels like a compromise between a group of people, and it's so general that it's almost useless. The standards bodies keep bolting on new stuff to try to get it to the point of basic usefulness, but it just makes it worse. Does anyone really use Object Constraint Language to decorate their diagrams so that they can generate code that Just Works? I have never seen a case where you couldn't get it done faster by writing the code, then reverse engineering it into one of the expensive tools. Of course, it's good to design things before you code them, but a white board is much better. Combine the nature of UML with the fact that you can't ever show its diagrams to business analysts or end users because it is too obtuse, and I end up doing most of my architecture and design work in Visio (and I'm no fan of it either). It creates diagrams you can show everyone that capture the essence of what you are doing just before you implement it.

I'm not just a tourist in the UML and RUP world -- the company I worked for in the dim and distant past used RUP, and we foisted Heavy Duty Object-oriented Analysis and Design on clients, and used it ourselves. Why do you think I'm so militantly agile now? It just flat doesn't work for 90% of the types of projects that real developers write. And it's Bastard Child Model Driven Architecture is featured in an upcoming Snake Oil entry...

SOA (Service Oriented Aberration)

While talking to some of the other speakers at the SOA conference in Singapore, it occurred to me that SOA has an interesting side effect that has nothing to do with technology. To make SOA really work, a business has to strictly define what it is they do, to the point that a developer can create services to represent it. That's the rub -- most businesses don't understand their own business to that level of detail. Time and time again, I've noticed that the analysts and developers on a project actually define a client's business processes to a much finer level of detail than the business ever has. To write code for something, you have to really understand it. Some of the clients for which I've worked even commented on the fact that, because of the development process, they understand their business better.

Of course, the opposite is also true (and, alas, more likely) -- companies with poorly defined business processes will try to adapt SOA and fail miserably, creating a case study of SOA failure, when the technology had nothing to do with it. Worse yet, tool vendors of expensive development tools like Enterprise Service Buses will sell their tools to unsuspecting clients as a way to "fix" their business process. Putting a dress on a pig doesn't make the pig any prettier, and it just annoys the pig. SOA, like so many other similar technological breakthroughs (see Programming, Object-oriented and Development, Component-based), won't solve the dysfunction of dysfunctional businesses.