Ruby Matters: Meta-programming, Synthesis, and Generation

In my last Ruby Matters blog post, I talked about meta-programming in Ruby, contending that Ruby gives you "places to put your stuff". I always wondered about meta-programming in Smalltalk and how that compares to Ruby, and Where to Put Stuff in Smalltalk. The final piece of the puzzle came after I talked to Glenn Vanderburg (the Chief Scientist of Relevance). I was puzzled as to why the Gang of Four book (which had examples in both C++ and Smalltalk) didn't have more meta-programming. Lots of the design patterns are almost trivially easy to implement with meta-programming, but they didn't do that in the Smalltalk examples. They used the same structural approach as C++. It looks more and more to me that the Design Patterns book was really just a way to solve problems in C++ that should have been easier to solve in a more powerful language like Smalltalk. Which is why I was puzzled about the lack of of more meta-programming solutions in Smalltalk. Glenn enlightened me. One of the overriding characteristics of Smalltalk is the way code is stored, in an image file, which allows for really smart tools. The program and the environment all reside in the binary image file. There are no source files as we know them today, just the image.

A comparison about implementation details is in order, on how Ruby differs from Smalltalk. Let's talk about has_many in Ruby on Rails. Typical Rails code looks like this:

class Order < ActiveRecord::Base
  has_many :lineitems
end

For those not familiar with Ruby, this is a method, defined as a class-level initializer (just like an instance code block in Java, a chunk of curly-brace code in the middle of a class definition, which Java picks up and executes as the class is instantiated). So, ultimately, this is the Ruby equivalent of a static method call, which gets called as the class is created.

Let's talk about Smalltalk, which has first-class meta-programming. You could easily build has_many in Smalltalk, implemented as a button you click in the browser which launches a dialog with properties that allow you to set all the characteristics embodied in the Ruby version. When you are done with the dialog, it would go do exactly what Ruby does in Rails: generate a bunch of methods, add them to the class (stuff like the find_* and count_* methods). When you are done, all the methods would be there, as instance methods of your class.

OK, so at this point, the behavior is the same in Smalltalk as in Rails. But there is one key difference: The Smalltalk version using code generation. It's a sophisticated version of a code wizard, generating the code using meta-programming techniques. The Ruby version uses code synthesis: it generates the code at runtime, not build time. Building stuff at runtime means more flexibility. But that is a minor point compared to this one: In the Smalltalk version, you use the dialog and properties to generate all the methods you need. The original impetus for the has_many intent lives only while you are running the dialog. Once you are finished, you are left with lots of imperative code. In the Ruby version, the intent stays right where you put it. When you read the class again, 6 months from now, you can clearly see that you still mean has_many. Smalltalk has the same meta-programming support, but the intent of code synthesis remains forever. That's why it's important to have a place to put your stuff. It isn't accidental in Rails that many of the DSL characteristics appear as class methods rather than code you call in the initialize method. Placing them as part of the class declaration declares intent in a big way, and keeps the code very declarative.

To summarize the similarities and differences:

1. Both Ruby and Smalltalk give you a place to put your meta-programmed stuff


2. Both give you a place to put the declaration of intent (the tool in Smalltalk, the shadow meta-class in Ruby)


3. A time in the lifecycle of the class when things happen. In the Smalltalk version, it's a one-time deal, as you use the tool to generate the code. In Ruby, the synthesis takes place at class load time. This leaves the clean, declarative code right where you put it, rather than generating a bunch of much less clear imperative code.
   

Glenn made an excellent point here: the Smalltalk version is a great example of accidental complexity, not essential complexity. Software is full of essential complexity: writing software is hard. But we end up subjecting ourselves to lots of accidental complexity in our tools and languages. And it should be stamped out. The Ruby version eliminates accidental complexity by providing a great abbreviation for the intent of has_many. This blog does a great job of illustrating the differences between abbreviations and abstractions.

Smalltalk had (and has) an awesome environment, including incredible tool support. Because the tool is pervasive (literally part of the project itself), Smalltalkers generally shied away from the kind of meta-programming described above because you have to build tool support along with the meta-programmed code. This is a conscious trade off. One of the best things about the Ruby world (up until now) is the lack of tool support, meaning that the tools never constrained (either literally or influentially) what you can do. Fortunately, this level of power in Ruby is pretty ingrained (look at Rails for lots of examples), so even when the tools finally come out, they need to support the incredible things you can do in Ruby.

Next up, "design patterns" in Ruby.

Thanks to Glenn for supplying me Smalltalk information, as a sounding board for this, and all the interesting bits, really.

Ruby Matters: A Place to Put Your Stuff

Why are so many people into Ruby? I get this question a lot because I speak at a lot of Java conferences and talk about Ruby and JRuby. In this and several more blog entries, I'm going to explore this question in depth.


For myself and many of my peers, it was a rite of passage to learn all the intricate tricky stuff in C++. Once I got over my fascination with C++ and got tired of banging my head against all of the arcane nonsense in the language (I used to know the details of all the different ways you could use the word const, but I've fortunately purged that knowledge), I asked one of my colleagues why he continued to love it so much. "Knowing C++ makes me part of the C++ priesthood. When people ask me questions, I'm purposely cagey when I give them answers because I think they should work as hard as I did to figure this stuff out. I don't want to give my advantage away." I shuddered and was glad I wandered to greener pastures.

After a brief visit to the refugee camp of Delphi, I made it to Java and thought I was in heaven. Finally, a language that made some sense. I liked the locked down nature of the language because I thought that Java helped prevent mistakes, especially for large projects. This is before I had embraced unit testing, so that seemed like a good thing. I would gladly give up flexibility for perceived safety. This was when I used to think that one of the most important duties of a computer language was to prevent mistakes first and support building interesting stuff second. Interesting stuff must be hazardous to large projects, and I was an Enterprise Developer. After more than a decade in Java, I started playing with Ruby because I'm a bit of a language geek and I was taking the advice in The Pragmatic Programer to learn a new language every year. But, at the time, Ruby was just a cool little scripting language with nice syntax.

When Ruby on Rails came out, I started looking at it with great interest because it was so vastly different from the Java frameworks with which I was so familiar (this was just after I had written a book comparing them). And the more I looked at Ruby, the more puzzled I got. Wow, there's some funky stuff in there, and I couldn't understand what some of the code meant. So, I dug deeper. And slowly, I gained enlightenment. At first, I thought that it was just more of the same "obfuscated code" syndrome from C++. But I later understood that it wasn't.

The epiphany really came when I understood what some of the Rails code was doing with meta-programming, which made me look closer at the core Ruby stuff. Meta-programming was supposed to be scary stuff that only wild hackers did. But it made problems melt away, in very few lines of code. If you look at how attr_accessor works, it's so simple. I thought for a long time that attr_accessor was a keyword or something infra-structural that I shouldn't bother looking at (before I realized that lots of stuff in Ruby looks like keywords but they aren't).

I have come to understand that meta-programability is an extremely important part of a language. So many things that we solve in languages like Java with complex structures and hierarchies are much more elegantly solved with meta-programming. This realization led me to start looking around at other languages, like Smalltalk and Python.


That's when it hit me: one of the things that makes Ruby so powerful is that it already gives you a place to put your stuff. If you don't think this is any different than Java, check out whytheluckystiff's Seeing Meta-classes Clearly article. I'll wait... Yes, there is a shadow meta-class behind every object in Ruby, giving you the syntax and location to put much of the interesting meta-programming stuff in Ruby. You can do lots of the same meta stuff in, for example, Groovy, but you have to create a Groovy builder object to do the equivalent of eval. And, doing class_eval would take more building a place to put it. I was talking to a Python programmer recently, talking about the same subject, and I got the same response: "Yeah, Python can handle that, but you'll have to build a place to put it." One of the elegant features of Ruby is that it comes pre-defined with places to put your important stuff, rather than forcing you to build a place to put your stuff. Which is why Paul Graham says that Lisp is the most powerful language available; because you are writing code in the abstract syntax tree, you are living in the place where stuff gets put.

In the next post, I'll talk about the special case of Smalltalk, tools, and code generation vs. synthesis.

Martin and Me and DSLs

Back in June of this year, Martin Fowler and I did a keynote at The ServerSide Symposium in Barcelona on a subject for which we both have some interest (and dare I say passion): Building Domain Specific Languages. I saw cameras at the back of the room, but it didn't actually dawn on me that they were filming the whole thing...and that they would put in on the web. But, sure enough, here it is. It took them two months to post it and I think I know why: some poor person had to go and transcribe Martin and me yammering away for an hour. I hope they got paid well!

This keynote pretty much summarizes much of the thinking that I've been doing about DSLs over the past couple of years, and it is nice to see the agreements and disagreements that Martin and I have on some nuances of this topic. Clearly, though, we both think this remains a Big Deal.

Lights, Cameras, Geeks!

At the Agile 2007 conference this year, they asked for submissions for videos featuring agile development themes call the Agile Advert contest. ThoughtWorks submitted a couple (created in ThoughtWorks UK) and won the top prizes. You can see them on YouTube under Developer Abuse and Being Agile is our favourite thing. Hilarious!

Pair Programming Spellchecking

My colleague Jay Fields has a nice post about pair programming, when it's good, and when it's not so good. I wanted to add to and amplify his remarks a bit. Basically, anytime you pair two developers of vastly different skill sets (what Jay calls a "spell checking pair"), that's not pair programming: it is mentoring. That doesn't mean that it is wasted time, but it is a different activity.

I find that mentoring is tiring for both the advanced and novice developers: tiring for the advanced developer because they have to go slower than their natural pace and tiring for the novice because they are having to peddle faster to keep up. However, mentoring is a great way to greatly increase the skill level of the novice. If you are going to mentor, it should be for just part of a day. After the mentoring is done, switch pairs with more matched levels.

A lot of companies seem to think that's what pair programming is: take a intern from college and pair them with an ace to build up their skill level. All that does is stress both.

We are generally careful when engaging with clients to pair program with their developers to determine if the project is enablement or delivery. If it is delivery focused, we tell them that the more senior developer may well leave their pair in the dust, and should be reorganized. You can't usually do both enablement and delivery at the same time except in rare circumstances.

Dependency Injection in One Sentence

Jim Weirich issued a challenge to describe dependency injection in once sentence. My friends at Relevance had a good version. Here's mine:


Dependency Injection enables a vitally important but nevertheless weak, limited, syntactically confounding, and dauntingly complex form of one of the kinds of meta-programming that should exist in the language.


This is fun -- can we do annotations(C#)/attributes(Java) next?

No Fluff, Just Stuff eXchange: London 2007

Way back in the late nineties, I was working for The DSW Group, which was a Borland partner. We did Borland training classes worldwide, and that's ultimately how I got into the business of speaking at conferences. At the time, I was all geeked up about Delphi and at the first few BorCons I spoke exclusively about it and C++ (in the form of C++Builder). However, in the late 90s, I stopped having that lovin' feeling for Delphi and started really dedicating my efforts in the rising tide of Java. But what was interesting to me was the adoption level in different countries. I was still traveling to BorCons around the world, talking about JBuilder. And some countries I would have huge audiences. And in others, you could see tumbleweeds blowing through the aisles. I remember the first few Java talks I did at the Entwickler Conference in Frankfurt, no one even knew what "Java" was.

It was interesting because the UK seemed to be one of the last holdouts for Java. Even into the early 2000s, I would go to Borland conferences in London and be in the small room, while the Delphi groups were packed. Of course, these were Borland conferences, so the crowds were self selecting, but it seemed to me like Java was a slow burn in the UK. Then, of course, overnight, it was the cool thing to do.

London is now a first-class Java city, with lots of Java work going on (Java is very popular in the financial sectors, and London sure has lots of that). The only thing missing from the London Java scene finally appears this August: No Fluff, Just Stuff. What better place to make the cross-ocean debut of the premiere Java conference. In the past, people have literally flown from the UK and mainland Europe to see No Fluff, Just Stuff shows in the US, in random cities like Columbus, OH. Now, you can ride the tube to see a No Fluff, Just Stuff show!

The traveling circus that is No Fluff, Just Stuff invades London on August 29th, for 3 days. It's the normal cast of speakers from the US (including Venkat Subramaniam, Ted Neward, Brian Sletten, David Geary, and I've managed to stow away as well) mixed with some great guys from the UK (my ThoughtWorks colleague Erik Doernenburg, Graeme Rocher of Groovy/Grails fame, and some wild cards).

To help pump up the enthusiasm for this event, No Fluff, Just Stuff and Skills Matter have a cool promotion. Every attendee who registers before August 17th with the special code of NFJS-NEF666 will receive a Nintendo Wii. Not "entered into a drawing for a Wii" -- "will receive a Wii". Wow.

Come see us in London -- it should be a blast.

Polyglot Programming at erubycon

In less than a week, I'll be doing a special version of my Polyglot Programming keynote at erubycon in Columbus, OH. In it, I talk about how to make Ruby palatable to conservative enterprises, and some arguments against "conventional wisdom".

If you are into Enterprise Ruby, there are still slots available at this conference. This conference will have a great combination of speakers who are actively pushing Ruby and Rails into the Enterprise. Lots of us see bright futures for dynamic languages and their potential in traditional IT; come see a bunch of like minded speakers (and attendees) who are on the cutting edge of making it happen.

The Rich Web Experience

The web is changing. Just a couple of years ago, we had reconciled ourselves to treating the browsers as a graphical dumb terminal. Ajax and related technologies have changed all that. And anytime there are seismic changes in the technology landscape, developers have to learn the whole new world. The preferred way to do this quickly is to attend a highly focused conference.

That's what The Rich Web Experience is all about: rich application development that running in the browser. This conference is beyond just a single technology's APIs. It also covers security, usability, and other topics. I'm speaking on testing and debugging the web tier, including a bunch of best practices with Selenium.

Come join us in San Jose for what should be a rocking good gathering of cool web based technologies.

Travel Broadens You

Not too long ago, I was in India to speak at JAX India. The conference was held at a science center in downtown Bangalore, and the speakers were staying in a hotel several miles away. One of the interesting things about India is that you can hire car services for cheap, so the conference organizers had a platoon of cars at our disposal, for trips to and from the conference center, which took about 20 minutes (it would be about 5 minutes if there was no traffic, but apparently that state never exists in Bangalore).

While driving to the conference center a couple of times, we passed a part of town where there were several cows lounging in the road, in amongst the traffic. No one even noticed: the traffic flowed right around them as if they were any old standard barrier in the road. Of course, it struck me as very odd, because I almost never see cows on the main roads of Atlanta.

It made me realize that people can get used to and habituate anything. During the conference, I talked to Java developers about some of the odd things in Java. Java has its share of strange behaviors (generics, anonymous inner class syntax, the recursive Enum definition) that look to strangers as odd as a cow standing in the road. Because Java developers are accustomed to seeing these cows all the time, they don't even notice anymore. But it strikes tourists as very odd and surprising.

Here is a concrete example. Groovy makes it really easy to interact with Java objects, including those that follow the standard JavaBean specification for get and set methods to form properties. In Groovy, when you create a new class, you can create your properties using the get and set style of coding, yet when you call the property, you can leave the get and set off.

class Foo {
  def name

    def getName() {
        return name.toUpperCase()
    }           

    def setName(value) {
        name = value
    }
}

foo = new Foo()
foo.name = "Homer"
println foo.name 

Now, when I refer to what looks like a regular public field (to a tourist), I get the uppercase version because of the automatic mapping to a getter. To a Java developer, this auto-magic mapping from the property methods to actual properties seems as normal as can be. However, to a Java tourist, it looks as strange as a cow in the road.

This suggests 2 things to me. While every language has its own bizarre quirks, it is the seemingly irrational stuff that turns off tourists to the language. Do you really want to go into a history of the JavaBean specification to the poor Python programmer who's vacationing in JavaLand so that they can understand this weird but expected behavior? And will they possibly stay awake for it? Probably not, so they'll just say something like "Wow, Java's broken" and move on. I think this hampers attempts to expand Groovy's meme outside the Java world. I know there are efforts to port Groovy to the .NET platform, but will it possibly attract any .NET developers because of its unabashed Java-ness?

The other thing this suggests to me is the advice in The Pragmatic Programmer to learn a new language every year. Just as physical travel broadens you, traveling to a strange language broadens you too, so that when you see cows in the road back home, you understand that it's not normal for everyone. Stranger in a strange language, anyone?

Fair and Balance Essays?

An interesting review of the 2007 No Fluff, Just Stuff Anthology appeared on Amazon the other day. Generally, I don't bother to reply to Amazon reviews (because it is after all someone's opinion, and I can't dispute someone's else's opinion), but this one has some interesting points. First and foremost, I would like to thank the reviewer for giving us 4 out of 5 stars, so his complaints should be kept in perspective (he did at the end of the day enjoy the book, and I don't want to gloss over that by way of indicating that we don't appreciate that immensely).

The reviewer mostly takes me to task for not forcing the authors to be more balanced against what he calls "agitation for the new age software movement". If you look closely at the cover of the book, you'll notice I'm not listed as the editor, but as the compiler (as in "Compiled by Neal Ford"). This distinction is important and planned. I did not edit this book in the normal sense (of vetting what the authors want to write), I merely compiled the essays. Frankly, it would be an awful job trying to get this group of authors to bend to my will! All the authors of this anthology are quite passionate about their subjects (which is what makes the book interesting, in my opinion).

The reviewer takes Brian to task for not providing the WS-* case against REST, but I don't think that was the purpose of the essay. Where in the vast ocean of information about WS-* do you even see a mention of REST? Unless you are specifically comparing two technologies, you frequently don't, well, compare them. I notice that David Geary (whose excellent essay was praised by the reviewer) didn't compare and contrast it with Struts, which is the de-facto market leader.

But the more interesting issue for me is the clear disdain the reviewer has for what he calls "new age dogma": REST vs. SOAP, dynamic languages, and Agile development. It's no secret that many of the No Fluff, Just Stuff speakers do prefer agile development and looser contracts. While not suitable for all applications, this is the cutting edge of software development right now. It is contingent on "thought leaders" to point out the latest trends in software development, so that the attendees and readers know what's on the horizon. Software and software development continues to evolve at a furious pace. While no one can predict the future (except maybe Bruce Tate), it is interesting to see where the people who were really into Java in 1996 are spending their time now (a hint: mostly with loose contracty type stuff).

Oh, and the reason there is more information about IntelliJ than Eclipse? Because most of the authors use IntelliJ (because we think it's the best tool available), I got inundated with cool IntelliJ tips and tricks (and ended up cutting a bunch of them). We had to ask over and over to even get tips for Eclipse, which is not to say that Eclipse is bad, it just doesn't generate as much passion than IntelliJ. There's that word again: passion. All the writers of this volume are passionate about technology (which is extraordinary in and of itself), and want to write about it for very little remuneration.

Given the amount of other open source in the book, why wouldn't we prefer Eclipse over IntelliJ it they were essentially the same? As a group, we tend to choose things that we think are best of breed, whether web framework or IDE. Hopefully, that's at least some of the appeal of both the anthology and the No Fluff, Just Stuff tour.

PragMactic-OSXer

New Mac OS X users are like recently reformed smokers: they can't stand to see it when someone indulges in their old vice (in this case, Windows). I made the total plunge about 1 1/2 years ago, and more and more of my friends are doing it too. People who attend No Fluff, Just Stuff often note that most of the speakers are using Macs: the answer I frequently give is "because we can". It is crystal clear to me that I am simply more productive on the combination of Mac hardware and Mac OS X. I'm paid for how much I can produce in a given amount of time, so it makes sense for me (even if I have to buy it myself) to pick the sharpest tool I can find.

However, just picking up a Mac won't instantly make you more productive. In fact, if you just scratch the surface, you have no idea that Mac OS X is a deep ocean. There is always a learning curve when you switch something as important as an operating system. To that end, several of the No Fluff, Just Stuff speakers and myself have started a blog devoted to making the best use of the Mac. It is called "PragMactic-OSXer". If you are switching, or are just curious to see how the other half lives, come over for a visit (or attach your blog reader to it.)

TSS DSL Keynote Recap

Martin Fowler and I just finished delivering the keynote on Language-Oriented Programming at The ServerSide Symposium in Barcelona, Spain. He and I have never presented together before, but we are both passionate about this subject, so it sounded like fun. And it was. I created a skeleton slide show in Keynote, to which he added and made suggestions, and we paired for about 1/2 hour the night before to make sure that it represented not only a consensus view but also good leaping off points for the presentation. We approached it like a public conversation: we had pretty minimal slides, and we took turns talking about the subject (along with minor disagreements) along the way.

There was a little bit of a frenzy just before it started. For some reason, some projector switching boxes just don't seem to like the Mac. My laptop worked fine in all the other breakout rooms (that only had one projector). In the main room, my machine could see that it was attached to an external monitor (I could see the resolution, the refresh rate, and every other detail about the external projectors), but for some reason they refused to show the image. I worked on this with an increasingly frantic IT guy from the hotel. We tried a Dell laptop and it worked fine. He kept insisting that there was something wrong with my machine, and I kept demonstrating to him that I could see his equipment just fine. I've encountered this problem before (see this entry, and it always turns out to be the switching box for the projectors, which for some bizarre reason has an affinity for Windows). Finally, as a last resort (time was running out), I exported the presentation from Keynote to a PDF file, copied it to a thumb drive, and moved it to the Windows laptop. I lost all the animations and other effects, but the slides looked OK. By this time, it was time to start speaking, so I opened the slide show full screen on the laptop. Unbeknownst to me, someone had set the machine to operate in kiosk mode when Acrobat is full screen, so the slide show starting playing without me doing anything. Finally, rather than try to figure out how to turn that off in front of 300 people, I just left it with the Acrobat Reader border showing and launched into the talk. As much as I love the Mac, it seems that the rest of the world still harbors ill-will for anything non-Windows. And it's not major stuff anymore -- it's the strange little things like switching boxes for projectors.

For anyone who saw the presentation, that was why it was in non-full screen Acrobat Reader view. Despite this hiccough, I thought the talk went reasonably well. Martin and I both have lots to say on this topic. When we started, I feared that we wouldn't have enough material, but he and I are both prodigious talkers, so we ended up having to rush a little to get to the end. I feel like we presented a pretty good summary view of how we both feel about this important subject.

Speaking at TSS Barcelona Next Week

I spoke at the The ServerSide Symposium in Las Vegas earlier this year. Next week, I'm traveling to Barcelona (for the first time) to speak at the European edition. Martin Fowler and I are pairing on the keynote, about Domain Specific Languages (a topic for which he and I share considerable enthusiasm). I'm also doing a regular talk on Selenium.

Coalmine Canary Tests

I was pairing the other day and we started talking about the scope of tests. I'm a huge advocate of writing the simplest, just slightly more than trivial tests when doing test-driven development, especially in a dynamic language (we were using Ruby). My pair, Kristy, and I got into a discussion of around 2 areas: just how simple should they be and should you keep them around after you've written the code? Lots of developers use these ultra simple tests, then make the test more robust once they've gotten the first incarnation to pass. I'm a fan of what I call "Coalmine Canary" tests. Back in the old days, coal miners used to take caged canaries down in the mines with them. The birds were much more sensitive to the gas that can build up in mines; if the bird died or starting struggling for breath, the coal miners knew it was time to high-tail it out of the mine. In the same way, very simple tests help you judge the safety and semantics of the code under test. We created a very simple test (basically, verify a method that included a semi-complicated ActiveRecord query returned a non-zero number of rows). We decided to keep it around, because it verified that we were calling the infrastructure correctly (in other words, that the semantics of the method were correct). Sure enough, as we started adding more code to the class, we introduced bugs. The first test we ran after making changes was the Canary test. If it failed, we knew that we had a fundamental problem with our infrastructure and didn't even bother looking at the more complex, nuanced behavior we have added until the Canary test passed again.

Having (and keeping) the simple Coalmine Canary test is a great sanity check on your code. If it fails, you've got serious problems, and you'd better fix those before you do any more mining in your code.

Neal and David Talkin' 'Bout Agile

As a sneak peak to the workshop David and I are doing at The Agile Experience, No Fluff, Just Stuff caught us late last year in a video podcast about agility, how companies react to it, and it's future. It's a pretty accurate representation of how I feel about agility (minus the faux pas of saying that the Agile Manifesto was signed in Colorado, not Utah).

Are Open Classes Evil?

When doing my JRuby talk at No Fluff, Just Stuff, one of the consistent questions I get is the morality question about open classes: are they evil? Open classes in dynamic languages allow you to crack open a class and add your own methods to it. In Groovy, it's done with either a Category or the Expando Meta-class (which I think is a great name). JRuby allows you to do this to Java classes as well. For example, you can add methods to the ArrayList class thusly:

require "java"
include_class "java.util.ArrayList"
list = ArrayList.new
%w(Red Green Blue).each { |color| list.add(color) }

# Add "first" method to proxy of Java ArrayList class.
class ArrayList
  def first
    size == 0 ? nil : get(0)
  end
end
puts "first item is #{list.first}"

Here, I just crack open the ArrayList class and add a first method (which probably should have been there anyway, no?). When you define a class, Ruby checks its classpath to see if another class of the same name has already been loaded. If it has, it adds the new behavior to the class.

It it's too frightening to add a method to the entire class, Ruby gives you the option of adding it to an object instance instead. Consider this:

# Add "last" method only to the list object ... a singleton method.
def list.last
  size == 0 ? nil : get(size - 1)
end
puts "last item is #{list.last}"

Here, I add the last method just to the instance of the list class (i.e., this list object). That way, you don't add it to the entire class.

Many Java and C# developers are shuddering in horror right now. The consensus seems to be that this is just too dangerous. And, like all advanced language features, it can be abused. But here is a counter argument. How many Java and C# developers have a StringUtils class in their project? Pretty much everyone. Why? Because the language designers for both languages no only won't allow you to add your own methods to String, they won't even allow you to subclass it to create your own String class. Thus, you are forced by the language design to switch from object-oriented coding to procedural coding, passing Strings around like you are writing in C again.

Open classes allow you to make your code much better, when used responsibly. One common argument against this feature from paranoid developers is that they don't trust "junior" developers with this kind of power. So, tell the junior developers on the project not to do it! And, if you really hate the additions that have been made, you can also reopen a class and remove methods at runtime, using either remove_method (which removes it from a class) or undef_method (which removes it from the entire inheritance tree).

You can add new methods to classes in Java if you want to use Aspects, but:

• uses an entirely different syntax from Java


• you give up tool support in your IDE for the new methods (in fact, it probably won't even compile it properly without some plugins and such)


• it's so much trouble that you just suffer through the ugly, non-OOP StringUtils class

In JRuby, you can add methods to existing classes:

• using the natural syntax of the language


• it is supported as much as anything else (we don't have IntelliJ for Ruby...yet), but nothing chokes either


• it is so easy that it is natural

I think this level of power makes developers used to non-dynamic languages queasy. But in the right hands, it can make you code much more expressive, and not pollute your namespaces with lots of made up class names with Util and Helper tagged on the end. Like all advanced features, when used correctly, it makes your code much better. Are open classes evil? No more evil than any advanced language feature.

Cruising

This is old news to some, but Ruby now has a great continuous integration story: cruisecontrol.rb. Written by a bunch of Really Smart Guys, it has some innovative features (convention over configuration, no XML, DSL-style tasks, and many more). I mistakenly said in my aboutGroovy Podcast that cruisecontrol.rb supplants the other versions of CruiseControl, but that was clearly wrong: Enterprise CruiseControl was announced recently, which provides consulting, 24x7 support, and new features.

CruiseControl is an example of the best kind of tool: one extracted from real need on a project. The original CruiseControl was developed because developers on a project were feeling build pain and came up with a great solution. Tools are always better when they are derived from real-world experience. Another great example of this is Ruby on Rails. DHH didn't wander over to an ivory tower and think about what a framework should look like. He built some web applications and extracted a framework. Getting developers together to invent frameworks and tools never works as well as taking working stuff and distilling frameworks and tools from it. The classic easy target of how not to do it is EJB 1 and 2: get a bunch of vendors in a room and figure out what developers need (or, more likely, what do we have to sell that we can convince developers they need).

CruiseControl is an ongoing illustration of what happens when you get clever people together who can not only spot a problem but come up with a top notch solution to fix it, then distill the results.

The Importance of Language

Language is important.

One of the purposes of my last two blog posts was to use hyperbole to make a point. And, judging from the hue and cry, I was successful in irritating a lot of people. To do that, I used 2 purposefully inflammatory terms: "communist" and "bureaucracy". As many people pointed out, communism as a form of government is clearly a bad thing, as proven in very expensive social experiments in the last century. Bureaucracy, though, merely has a negative connotation because it tends to be misused. In very large groups of people, some level of bureaucracy is necessary to preserve order. It is in the arbitrary application of bureaucracy that earns it a negative connotation.

When you read the first sentence of this post, you probably though I was referring to computer languages, not English. That's a perfect example of how spoken languages are sometimes ambiguous in useful ways. In fact, poetry is an art form that takes good advantage of the imprecise nature of our spoken languages. It is in fact hard to create a useful human language with very strict, unambiguous rules (as Esperanto showed). I'm sure lawyers would love to have a language where nothing is ambiguous; it seems like a large part of contract negotiations concerns removing ambiguities and trying to craft real, objective meaning from our incredibly malleable language.

Computer languages are important in just the same way as spoken languages. When creating a bureaucratic, legalistic system, strong contracts and unambiguous language matters. Very large systems benefit from bureaucracy. Using Java, C#, or some other statically typed language for building incredibly large applications takes advantage of the useful bureaucracy inherent in the language. I used to think that this serious hampered the ability to build large, complex systems in dynamically typed languages. You necessarily remove some of the useful ambiguity from such systems.

Some of the richness of spoken language hinges on purposeful vagueness. This means that syntax matters (in both spoken and computer languages). Ultimately, computer languages help us operate Turing machines, which means that any Turing-complete language will allow you to do anything you want, contingent on effort. As Paul Graham aptly states, you could build all of the features of Lisp using C, but the shortest route would be to build a Lisp interpreter in C, then write your Lisp code on top of that.

That suggests that the choice of language depends on the useful tension between inherent capabilities, expressiveness, rules, and the productivity because of the interplay between these factors. This helps explain why there are thousands of computer languages, each finding their own combination of those factors. Paul Graham has a great essay about languages, power, and suitability called Revenge of the Nerds.

Computer languages are just tools, and I try not to get religious about tools (my inflammatory posts for effect to the contrary). I personally have changed my views over the last few years about the efficacy of dynamically typed languages like Ruby, even for large enterprisey applications, as long as you have strong testing to back you up. The flexibility of the meta-programming in Ruby allows you to solve hard problems in elegant ways that are either too cumbersome or just flat impossible in a more limited language (even if those limitations are on purpose). Rails is a great example of a framework builder pushing the really powerful features of the language and in the process creating a framework that you don't need to know all those things to operate. Limiting the power of the language hurts your best programmers the most. The interesting question is: given the productivity difference between the best and worst developers, how much to you want to limit your best developers at the expense of "protecting" the weaker ones? And, before you think that bureaucratic languages make bad developers safe, consider the great quote by Glenn Vanderburg (based on his experiences in the real world) : "Bad developers will move heaven and earth to do the wrong thing."

Static Typing & Bureaucracy Redux

As I expected, there is serious dissension in the ranks over my post claiming that "static typing is communist bureaucracy". Some of the areas worth rebutting:

• Static typing helps tools
  
  Yes, it does (although dynamic typing doesn't preclude tool support -- Smalltalk had the one of the first refactoring tools built in, and NetBeans is adding some pretty good support for JRuby). This is a problem that will be solved. However, I reverse this question: tools make static languages merely palatable. Iterating over collections in Java or C# is ridiculously painful. Using the each method with a closure in Ruby or Groovy is a joy. Tool support in languages cover up for deficiencies in languages and libraries. Graeme Rocher (creator of the Grails framework) stated this quite elegantly in the GroovyOne panel I was on, stating that you don't need as much tool support in well designed languages because they tend to be more consistent (what Martin Fowler calls "Humane Interfaces").
  
  Granted, tooling makes things easier: no question there. But, are you willing to give up the flexibility of a dynamically typed language just to get better tool support? In my aboutGroovy podcast, I said (and meant it) that I would rather write Groovy in VI than Java in IntelliJ (and I think my position on IntelliJ is pretty clear).
  


• Static analysis tools are useful.
  
  Yes, indeed, anything you can find with an automated tool is low-hanging fruit. But, again, it's not a pure trade-off. You can do some of the same kinds of things in dynamic languages (although clearly not as much), but I would still rather have the flexibility of the dynamic language plus tests to verify my code. In the perfect world, I'd have the flexibility and power of the dynamic language and the tooling possible and available for static ones. This is one of the places that Groovy comes in handy: if your static analysis tool works off byte code (like FindBugs), you can run it on a Groovy codebase.
  


• Static typing creates better contracts and calling semantics for libraries.
  
  Dynamic languages tend to eschew strong contracts like interfaces. What I've found it that it forces developers to create much better method and parameter names, because that becomes the primary way that you consume methods. When done right, this is an effective way to communicate. Granted, this is not universally the case even in the Ruby libraries, but removing props sometimes forces people to make better use of what's left. The best example is the town of Drachten, in the Netherlands, who removed all their traffic signs and lights, forcing people to interact in the intersections. It made their streets much safer because they removed signage.
  

Sure, in a perfect world, all the facilities that we lean on in statically typed languages would be available in dynamic languages. But they're not. And, having worked extensively in both static and dynamic languages, I prefer the flexibility and expressiveness of the dynamic languages currently extant to the static ones, even with all the tool support. Still, testing is the most important verifier of code. I think it is professionally irresponsible to write code that isn't tested. If you are going to write that level of testing anyway (especially if you do it test-first), a dynamic language allows you to get more done in less time with a lot less typing (a lot of which is required to make static typing work).

Static Typing is Communist Bureaucracy

In a conversation that came up at No Fluff, Just Stuff Denver this last weekend, a fundamental realization came to me. We were talking about dynamic languages like Ruby and Groovy, and a lot of the Java developers just couldn't get over the idea of dynamic typing. They feared that utter chaos would reign if we discard typing. First, I asked them about the significance of testing: "Do we all agree that testing is important?" to which they replied "Of course". And, in the course of the expert panel, we had discussed the appropriateness of code coverage, what it tells you, what it cannot tell you, etc. The general consensus was that code coverage is a Good Thing, and that we should all aspire to 100% as much as possible. After all, it's the tests that tell you if your code if correct or not.

Once we had agreed that testing + code coverage is a good thing, I realized: Static typing is Communist Bureaucracy. "To get bread, you must stand in the line over there, but first you must get a blue stamp on your booklet." If you have pervasive testing, static typing == more typing. The static typing is nothing but a requirement to type extraneous code to satisfy a compiler that isn't telling you anything interesting anymore.

I've said it before and I'll say it again: testing is the engineering rigor of software development. Testing destroys the need to static typing. Dynamic typing speeds up development because there is physically less to type. And testing verifies the correctness of your code, not just that it compiles.

Justifying Automation

One of the topics that comes up in my Productive Programmer talk about automation is how to justify it. Sure, everyone knows that automation is a Good Thing. And, if you can demonstrate that you can automate something as fast as you can do it by hand once, that's a total no-brainer. But what about the grayer areas where you aren't sure how long its going to take to automate something, and you aren't sure if you can justify the time it takes to the Powers That Be?

I have a couple of strategies for this that I'm including in The Productive Programmer book. First, make sure you timebox your efforts. Even if the end result of the timebox is that you create another one (because you've made such good progress), that prevents the automation from turning in to a week-long exercise when you thought it would just take 1/2 a day. Timeboxing also cuts down on the inevitable yak shaving that goes on anytime you start experimenting with something you haven't done before.

Automation is all about risk mitigation. For the thing you are thinking about automating, ask yourself the following questions:

• How long does it take to do it by hand now?


• How many times are you going to have to do it before the end of the project?


• What is the consequence of doing it wrong one time (i.e., forgetting a crucial step)?

If you take the product of the first item (how long does it take now) times the second (how many times are you going to have to do it), that tells you what kind of time investment you have now. If that time investment is longer than what it would take you to automate it, no brainer again: automate it. The third item is about risk mitigation: if it's disastrous if you miss a step, that should make you lean towards automating it. For example, I was on a project not long ago where (for bizarre historical reasons) they didn't want to separate the compiled versions of their tests between unit, functional, and integration. Their solution was to create hand-crafted test suites. But, that means that we have to remember to add them by hand every time we created new ones. That's a big risk: I'm never going to remember that! Instead, we automated it by creating a little Groovy script that ran at build time to auto-generate the test suites based on the root directories. The risk of not automating this problem was wasted time: we wrote a test and forgot to include it by hand, meaning that we don't get the benefit of the test and thus wasted the time it took to write it.

I'm curious if others have tried-and-true strategies for justifying automation...

From Architect to Wrangler

One of the nice things about ThoughtWorks is the ability to create your own title, which leads to some interesting business cards (like a title of "Seat Warmer"). But, the restriction is that you only get one set of business cards, so choose wisely. Instead of going nuts on my first set, I chose the ultra-conservative "Application Architect", which is a reasonable approximation for what I do on projects.

But now I've changed. The title of "Architect" for software developers has gotten so diluted that its meaningless anymore. In fact, its almost pejorative because so many "paper" architects give it a bad name. I've gotten "Oh, dude, I'm so sorry" looks from people when I tell them I'm an architect, assuming that I've had a head injury or something and can't do real development work anymore. After long formulating and advice from others, I've now changed my title to the probably more apt "ThoughtWorker/Meme Wrangler". As you've probably noticed, I love the "meme" meme (after all, you are reading this on the "memeagora" blog). While it's a little cutsy, at least I'm not treated as a tragic has-been by coworkers and clients.

The interesting side effect of this happens because I put my title on my presentation slides. In the US, but particularly overseas, people ask me "What does mee-mee wrangler mean?" So, I get to explain it.

But that's way better than the now deprecated "Architect" title. It's a shame that, because we have no real industry-wide certifications, the nominally most advanced title has been co-opted by so many people divorced from reality. I've had to defend decisions made for SOA initiatives in front of "Architectural Review Boards" by people who last wrote code in COBOL. Can they really make good decisions about modern technology if they never touch it?

Git along, now, little neuron...

Keeping Knowledge in Plain Text

The book The Pragmatic Programmer is absolutely worth re-reading every few years, because every page has stuff on it that's still relevant. One of the important lessons is "Keep knowledge in plain text". To paraphrase, you shouldn't keep important information in a proprietary piece of software, because if the software dies (because the company goes out of business) or it relies on a changing protocol (more about this in a second), your knowledge is trapped. If you keep it in plain text, you can read (and manipulate it) using a lowly text editor.

Two things made me realize this in a Big Way recently. The first was my ToDo list. I used to keep all my ToDos in a good (but fatally proprietary) piece of software named LifeBalance. It's the best of that category of software, but using it meant that I had to have a Mac OS X, Windows, and Palm client to access it. It became its own project just to keep it in sync. And, unless LifeBalance works instantaneously, adding items takes time. I was reading the LifeHacker book (distillation of the awesome LifeHacker web site) earlier this year, and one of the first items is an entry about a text-based ToDo list manager called ToDo.txt (which has its own website). It's called "Task tracking for command line lovers". The nice thing about ToDo.txt is that it uses a plain text file for storage. And, the author has written a bunch of bash/batch files to help manage it. That means you can append stuff of your own (using QuickSilver, in my case) or you can use the tools. The command line tools also let you prioritize, archive, and a bunch of other stuff. It's a good marriage between command line and plain text.


The other driving home of the important "Keep knowledge in plain text" lesson concerns my blog. For years, I just wrote my blog entries in a text editor in HTML and uploaded them by hand to Blogger. Last year, I fell for a seductive blog editor/manager called MarsEdit. It works fine, allowing you to create drafts, view your blog, handle multiple blogs, etc. The way it did this was via using one of Blogger's APIs for managing blogs.


Then something terrible happened: Blogger changed their API. Suddenly, MarsEdit didn't work anymore. Posting new entries wasn't a problem because I could go back to doing it by hand, but what amount my 9 months of existing, historical blog entries it held? Trapped! Now, I can't use it, and I can't easily free all my information. I violated the rule!


I just spent a couple of hours fixing this mess, re-syncing my original directory structure by hand (at least I was able to use wget to slurp my whole blog down to my local machine, making it much easier). I've felt the pain of violating the rule, and I promise not to do it again. My last step was removing MarsEdit from my machine (which, BTW, has now updated itself to work with the new API -- unlike our president, I remember the phrase "Fooled me once, shame on you, fooled me twice, shame on me").

Of course, Dave and Andy already knew this, and they told me, but I wasn't listening. Sometimes, you have to suffer for good advice to take hold.

Don't Be a Hater -- Think Different

People still fear what they don't understand or what's not familiar. When I arrived at the SOA conference in Kuala Lumpur a couple of days ago, I was met by the hotel's AV guy to hook up my Mac to the projector. I used the normal elaborate procedure I usually do to sync my Mac with a projector: I plugged the projector into my machine. But, unusually, no signal. I tried Display Preferences, changing the resolution, the number of colors: nothing. He was convinced that my exotic, bizarre machine/OS was the problem, and I kept trying to convince him otherwise. Finally, I persuaded him to go get another cable, which he reluctantly did (I don't know any Malay curse words, but I'm pretty sure I've now heard some). You can probably guess what happened next: new cable, perfect signal, at all my original settings.

Which reminded me of another conversation I had a couple of years ago with a friend in Singapore. We were sitting around talking about computers, and he made an off-hand comment that he didn't like Macs because he thought they were unintuitive, but he thought Windows made perfect sense. So, I asked him "So, you're telling me that the operating system where you click on the Start button to turn it off is the more intuitive on? I think you're just used to it", to which he replied: "You're right, I'll shut up now". I saw him again last night in Singapore, and we were catching up on all sorts of stuff. And you know what he told me? He just got a new machine with Vista on it, used it for 2 weeks, and returned it to Dell. He told me that he can't wait until his laptop finally dies because he's going to get a Mac. The reason for the change? He's now addicted to his iPod, and it's made him realize that Design Matters.

I can't resist beating everyone on the head with the moral: Don't succumb to FUD because something is different. And, in the tortured grammar of the old Apple campaign, Think Different. Because Vista is so different from previous versions, it may be the best thing that ever happened to Apple. If you're contemplating a change anyway, why not look at the entire field so that you end up with the OS equivalent of an iPod, not a Zune.

Pontificating about SOA

What else is there to do about SOA? Back in December, Mark Richards (Enterprise Architect at IBM) and I sat down and recorded a video PodCast about SOA (which we informally named "Taking Back SOA"). The results has now appeared at the No Fluff, Just Stuff web site. Mark and I have a very pragmatic approach to SOA, and we both hate the level of hype that appears in that whole space. He and I had a great time recording this video, which proves once and for all that IBMers and ThoughtWorkers can get along after all!

No, I Won't Stop Talking about IntelliJ!

During my No Fluff, Just Stuff talk on The Productive Programmer, I got a comment at the Minneapolis show to "stop showing us IntelliJ shortcuts -- we all use Eclipse!" And, Venkat Subramaniam blogged about the whole Eclipse vs. IntelliJ debate that came up at the expert panel in that same city. Venkat quite eloquently gives his reasons for choosing IntelliJ: it simply makes him more productive. And the same is true for me. I've used all the major Java IDE's in anger: NetBeans, JBuilder (I used to be considered a JBuilder expert, but I got over it), Eclipse, and IntelliJ. For me, IntelliJ works best. Period. In fact, I think that IntelliJ would land in my top five pieces of software of all time list. I also edited the No Fluff, Just Stuff Anthology chapter on IntelliJ tips and tricks, just out in treeware.

There are actually 2 IDE Tips and Tricks chapters in the anthology. When I sent out call for contributions from the authors and friends for IntelliJ tips, I got a flood of them, all very cool (and a few that I didn't already know about, like the Key Promoter). Ted Neward (who edited the Eclipse chapter) and I sent out the same call for Eclipse tips and tricks, and we got none. We sent out another call, and a few trickled in. And that is reflected in the book. I'm sure we're going to get grief over the IntelliJ chapter having more cool stuff. But that's really the whole point: like the preponderance of Macs on the No Fluff, Just Stuff tour, the people who use IntelliJ are passionate about it, because it exudes excellence. Like Venkat says, many people using Eclipse are in a bad arranged marriage. It's hard to drum up passion for an arranged marriage.

I feel like it is my duty to try to turn people on to things that make their life better. That's part of what being a speaker is about. So, I'm not apologetic about proselytizing IntelliJ. Every-time I find a tool that I think will make developer's lives easier, I'm going to talk about it. And, if I develop a passion about it, so be it.

PodCasting About Groovy

A couple of weeks ago, I recorded a PodCast for AboutGroovy.com, talking about (obviously) Groovy, DSLs, testing, and how to insinuate Groovy into your company's infrastructure. Basically, Scott Davis and I had the kind of conversation we have all the time at No Fluff, Just Stuff. The only difference is that he was recording it. It was fun and the content will come as no surprise to anyone who knows me.

Being a Consultant Makes You Tough (or Stupid)

Since working for ThoughtWorks these 2 years, I've gotten much tougher in the face of being able to work on short or bizarrely interrupted sleep. Case in point: I left Minneapolis (after speaking at the sold-out No Fluff, Just Stuff Twin Cities Software Symposium) on Saturday evening, dashing from my last session at 4:45 straight to the airport. 8 hours later, I was in Amsterdamn, with a 2 hour layover. 12 hours after that, I'm in Kuala Lumpur, arriving at 6:15 AM. I probably got about 8 hour of sleep, but never more than 2 hours at a time. I'm now exactly on the other side of the world from where I live: 12 time-zones away. I cleared customs, got my suitcase, and headed for the car service the conference had arranged. Traffic was awful, and combined with the distance of the airport from downtown KL, I arrived at the hotel at 8:35 AM...and the conference starts at 9 AM with me facilitating. Quick shower, dash upstairs, and I'm ready to go at 9 AM sharp. I then proceeded to talk on and off for 8 hours. I managed to stay up until after 8 PM. It's now the next day, and I feel fine. I'm apparently adjusted to the new time zone (I sleep soundly last night, interrupted only by my phone ringing at 11 PM (or, 11 AM in the states)) which I promptly ignored.

Two years ago, I could have never done this. Just part of the implicit training all the travel and distributed work instills. But there are benefits as well. Here is a picture taken just outside the hotel's meeting rooms.

The Elusive Right Click

Since he got his Mac, Venkat Subramaniam and I have been playing around with pushing the envelope on human computer interaction and productivity as hard as we can. He finds a cool new keyboard shortcut and can't wait to tell me, I learn a new QuickSilver bit of magic and can't wait to tell him. The Holy Grail of this kind of pursuit is finding a way to generate a mouse right-click using only the keyboard. Of course, Mac OS X is pretty friendly to use the track pad, but that's not enough for us: total keyboard control is the goal.

And yesterday I stumbled upon the Holy Grail. I was in the process of shaving a yak, and was waiting for something to finish, so I decided to solve this problem Once and For All. I did a bunch of Googling and experimentation and stumbled upon the solution. Go to Universal Access and turn on Mouse Keys. This the the accessibility option that allows you to drive the mouse strictly from the keyboard, using the embedded keypad. Once you turn that on, you can use the FN-I key to click the mouse, which means that you can use FN-CTRL-I to right click. It's not perfect (it clicks where the mouse current resides, not where the keyboard focus lives) and it precludes using the embedded numeric keypad for doing 10-key entry (which I never do anyway). You can turn it on and off by clicking the OPTION key five times (this is a setting on Universal Access as well).

But we're happy. Venkat got as excited as I've ever seen him when I showed him this morning; I was looking forward to show it to him all day. Ah, simple pleasures!

Don't Misquote Me to Create a Straw Man!

OK, I can't let this pass.

The ServerSide wrote an article, summarizing both mine and Venkat Subramaniam's talks at The ServerSide Symposium last week. It's a reasonable summary, with a few details left out (that's why it's a "summary").

But then, Cedric Beust takes the summary and constructs a agile development straw man to beat up. And, like many straw man arguments, he takes lots of stuff out of context and makes his own conclusions not supported by the original talk.

The first thing he gets wrong is the fault of the summary. Early in the talk, I clearly state that the best way to get good software is to interact directly with the users. And I give an example from my project history of the best project experience I every participated in, where the developers literally sat in the middle of the users. Every time we had a question, we just got up and resolved it directly with the users. It was project heaven.

But most corporate environments won't give you such direct access to the users, because the users are after all there to get work done. Thus, you must have user proxies. And that's where the business analysts come it. Perhaps at ThoughtWorks, our business analysts are different from the "typical" ones (I know our project managers are). The best business analysts are direct proxies for the users. They understand the users concerns and a little about technology. During the talk I also made the point that good business analysts prevent "paving the cowpath", suggesting innovative business solutions to problems rather than "that's the way we've always done it".

When I say that business analysts participate in the estimation process, that's because we can't get the users directly.

Do you know the difference between a good developer and a bad developer? A good developer can give you an estimate of the time needed to complete a task. And the better that developer is, the more accurate their estimate will be. If I ask a developer "How long is it going to take to do this?" and he answers with "It's very complicated", I want him off my team.

Another point I made in my talk is that, regardless of developer, you can't ask developers "how long will this take". Every project is different: in a 2 man project with no meetings and direct access to users, each developer will get more done in less time. In a project with 16 developers and a 2-hour status meeting every day, each developer will be much less effective (this has been known since Brook's The Mythical Man Month). But that is constant is the relative complexity of the story you are trying to complete. Relative to the other stories, not the entire universe. This allows developers to hone their estimation skills because it cuts down on the number of variables they have to hold in their head. This is the closet you can get to an objective measurement of the work required to complete a piece of software.

We don't need agile consultants and book authors to sell us vague and unproven metrics along with rigid guidelines that your team must embrace without any questions.

Ironically enough, we agree wholeheartedly about this, but he doesn't seem to know that this is exactly the point of my talk at The Serverside Symposium. The whole point of the talk was that we need real metrics like passed tests, real reconciliation between estimates and actuals, and binary completion criteria. The article makes that pretty clear, but that doesn't contribute to the weak straw man, so Cedric conveniently left that out.

What's most frustrating about Cedric's rant is the implicit assumption that "agile" is this vague concept that no one seems to be able to define. Yet, at ThoughtWorks, every single project is agile to some degree (and where we control everything (like fixed bid projects), it's pretty close to Extreme Programming). And we have a really high success rate, way above the industry average. I've been around for a while (one of my colleagues once told me that I was old enough to "add diversity to the company"), and I've used a bunch of different methodologies. One of the reasons I'm at ThoughtWorks is because we use proven techniques for building software, and we're damn good at it, making it a much less frustrating place to work.

DSL Podcast on No Fluff, Just Stuff

As my regular reader(s) know, I'm big on Domain Specific Languages, and have been talking and writing about this style of development for a while. During the course of last year, we recorded some PodCasts at some of the No Fluff, Just Stuff shows. One of those shows was Chicago, where I recorded a PodCast about my feeling about DSLs and this approach to building software using this technique. The PodCast has now appeared on the No Fluff, Just Stuff web site, for your listening pleasure.

Anthologization Redux

Well, it's that time of year again: the No Fluff, Just Stuff Anthology, Volume 2 (The 2007 Edition) is at the printers, undergoing the conversion from thought to dead trees.


Just like the blurbage on the back says:

Once again, some Sixteen of the world's best trainers and speakers are writing chapters on things they care passionately about. You'll find topics from the latest conferences including Groovy, JavaScript, Continuations, Web services and REST, JVM Byte Code, and Agility

These essays are a summary of the latest thinking in the industry, and range from the philosophical to the tutorial, covering the topics that the writers felt were the most important for readers today. If you feel like the neatest technology and latest ideas are passing you by, this book can help bring you back you to speed.

It's all good stuff, without any fluffy filler, as these essays are based on presentations given at the incredibly popular "No Fluff, Just Stuff" symposium series. Twenty-six times a year, the symposium visits a city and the speakers and attendees share ideas and perspectives. The speakers are all internationally known experts in their field.

So, to out and buy one as soon as possible (in a book store near you in early April). Or, order it directly from the Prags or at Amazon. In fact, I recommend that you buy two in case you loose the first.

Of course, I was intimately involved in its creation, but this is a really entertaining read (the main reason I like editing the Anthology is because it means I can read the essays as soon as possible).

Enjoy!

Garrulous Software

A particularly annoying characteristic of software these days is excessive chattiness. This occurs at both the user and developer level. In fact, one of the things I prefer on the Mac vs. Windows is the lack of meaningless messages the Mac generates (No, I do NOT want to clean off my desktop right now!). Pestering messages at the OS level distract from your attention and diminish your productivity. In The Humane Interface, Jef Raskin discusses the locus of attention, which is the work you are trying to perform. Anything that moves your locus of attention had better be a) relevant to your work and b) urgent and important. In fact, I think its a good idea to turn off the automatic notifications of email arrival, especially if it interrupts real work. Experts on "flow" maintain that it takes up to 20 minutes to get back to that deep concentration state, and it gets harder as the day goes on to overcome the inertia required to get back to that flow state.

The problem in the development world is much worse. It seems like the norm now to generate tons of meaningless noise. Think about application server startup, scrolling messages into the console faster than you can read it. 99% of that information is just the application server saying, over and over again like an idiot, "I'm OK...I'm OK...I'm OK.............I'm OK" (that last one was an EJB loading). The only thing that allows you to see errors is the massive stack traces that come from errors in the J2EE world. In fact, this problem is so bad that developers have taken to dubious practices to make their own messages to stand out from the flood of useless information. Admit it...you've put a Systen.out.println() in your J2EE code to debug it (a whole other subject). And, to make yours stand out from the noise, you put 30 or 40 "~"'s in front of it. Of course, your coworker is doing the same thing, and they use 30 or 40 "#"'s for theirs. Before long, application servers look like they have Tourette's Syndrome, spewing mostly meaningless noise.

My pair and I encountered this very problem recently. We were working with client code that makes heavy (handed) use of Log4J to spew out messages to the console. Unfortunately, that makes for ugly unit test results: you can't see the interesting stuff for all the immense noise being generated by the code under test. In this case, it wasn't as simple as turning off the logging level (the code under test was creating an instance of something that had its own logging stuff buried deep inside). So, after a little thought, we figured out how to shut it up.

In out test case, we unplug the normal output stream, like so:

protected void setUp() throws Exception {
  super.setUp();
  originalOut = System.out;
  System.setOut(new PrintStream(new DumbOutputStream()));
}

And, in tearDown, we put it back:

protected void tearDown() throws Exception {
  super.tearDown();
  System.setOut(originalOut);
}

Our "DumbOutputStream" isn't stupid, just quiet:

 private class DumbOutputStream extends OutputStream {
   public void write(int b) throws IOException {
     //do nothing, because I'm dumb.
   }
 }

This effectively shut off the fountain of noise, leaving the testing framework's output (which occurs on System.err, not System.out) alone. Sometimes, you have to work hard just to shut something up.

Inside Edition: Star Wars

Warning: this blog post is a complete waste of your time!

On the Relevance site, I saw a link to this fascinating analysis of the connections between the first Star Wars trilogy (episodes IV-VI) and the second one (episodes I-III). As much as I'm loathe to admit it, I've done some thinking along these same lines, especially since 2 things have recently happened: 1) I got a new HDTV and 2) HBO has turned into the "All Star Wars Station" (but only for episodes I and III -- they must not have rights to II). You should probably read the analysis before you go on (or it won't make much sense). I'll wait.

In fact, the only flaw I can find in the analysis is the fact that the Jawas weren't at first going to sell R2D2 to the Skywalkers -- it was only after the other similar droid blew up (maybe R2D2 did something to him in the hold of the Jawas' ship?) But it kills the idea that the reason they ended up at the Skywalkers was some sort of deal with the Jawas (unless they double crossed him). See, I told you this was a waste of time, right?

Actually, I distinctly remember interviews with Lucas back in the late 70's where he talked about a trilogy of trilogies (it was supposed to go episodes IV-VI, I-III, and then VII - IX). He has since retracted that statement (and claims he never made it). Whatever. Given how much work it is, I completely understand.

But I always thought that it would be cool in episode IX for C3P0 and R2D2 to suddenly reveal that they are in fact the keepers of the force, much like some of Asimov's robot stories where the robots turn out to be the guardians of mankind while pretending to serve him. And, if you take C3P0 out of the equation (in light of the analysis above), then R2D2 can be said to be the guardian of the good part of the force. That would be cool!

When I want to waste some more of your time, remind me to tell you about the plot idea I have for the origin of the Borg...

More Sony eReader Pix

By popular demand, I'm posting a few more pictures of my Sony eReader, to convey a better impression of the resolution modes I described in my previous post.

The first shows the smallest eBook resolution:



The second shows the largest eBook size (Mr Magoo Mode):



The third shows a Pragmatic Press PDF (Best of Ruby Quiz), in the best viewing mode (rotated to landscape, zoomed in to text width):



I hope that satisfies everyone's curiosity! If you want to feel and touch one, I saw them on sale at my local Borders in Atlanta last weekend.

Sony eReader

Those who know me know that I'm a gadget nut, and I've gotten a few gadgets lately that fall into the category of essential for my on-the-road lifestyle and of interest to software developers and other geeks as well. One of them is the Sony eReader. This is the first real application of digital ink, which has been The Next Big thing for several years While it looks like a monochrome LCD display (with no back-lighting), digital ink acts more like real ink in that it doesn't consume power while it's being displayed. Most LCDs need a constant (very low level) charge to keep their dark bits aligned. With digital ink, you just zap them once and they stay in place until you zap them again. The obvious use of them in a digital book reader makes perfect sense: it consumes no power while you are actually reading the page, just when you turn the pages.

The Sony eReader was announced at the beginning of 2006, and was originally going to be available in the Spring of last year. Fast forward to November, when they actually shipped (I got mine late November, and I was on the list of "ship it too me right off the assembly line". Here's a picture:



The Good
Reading eBooks on it is an awesome experience, pretty much all I hoped for. The screen is highly readable as long as you have light (it does not have a back light). Essentially, if you have enough light to read a regular book, you can read this one. Even in low light it isn't bad because you have 3 zoom levels for eBooks: small, medium, and large (Mr. Magoo could read the large without his glasses). The page flips are slow because of the digital ink, but it's not really noticeable. I quickly forgot that I was reading something on a new fangled device and just enjoyed it, and have subsequently read more than a dozen books on it. The battery life is stunning: I used it for an entire week in Singapore, reading a couple of hours a day, without recharging it once (and that included 2 marathon reading binges on the 26 hour flights there and back). The eBooks (and even PDFs) take up very little room. It comes with about 100 Mb of storage; I immediately put a 2 Gb Memory Stick in it. I guesstimate that I can probably get 1000 books on this puppy.

The Bad
One of the great appeals for me was the ability to put PDFs on it. I carry (or would like to carry) a fairly sizable collection of reference books with me, in addition to whatever I'm reading at the moment, and lots of books (especially technical ones) exist on PDF. Alas, the reading experience with PDFs leaves something to be desired. Unlike eBooks, the eReader can't reflow PDFs, meaning that you can basically see them in 1 of 2 sizes: regular (the entire page) or zoomed (just the text on the page, eliminating the margins). The text of most PDFs is still too small for portrait mode even in the zoomed state, so I generally flip it to landscape. One of the annoying bugs bites you here: when you flip pages on PDFs in landscape mode, it re-sets itself to regular mode but still thinks it's in zoomed. That means to be able to read the page, you have to flip it back to regular (for real), then flip it back to landscape. That means that every time you flip pages, it's a 3-press operation. Which wouldn't be so bad except for the glacially slow refresh rate for the digital ink (which is hardly noticeable for a single page flip). Once they get this bug fixed, PDF's will be reasonable on this device, but painful for now.

The Ugly
The selection of eBooks is pretty small right now, but growing fast. As you would expect, there are a fair number of science fiction titles, including Neal Stephenson's whole catalog (which I have dully downloaded and and synced onto my eReader). The worst part of that whole experience, though, is the Connect software that comes with it and manages both syncing with the device and shopping at their on-line bookstore. Strike 1: Windows only, so I have to crank up Parallels to access it. Strike 2: Think iTunes written by a high school student who's just learned VB. It's awful and clunky. The should have picked a reasonable starting place (like Eclipse RCP) rather than reinvent a buggy version of an application that needs a file system view, a HTML-centric content pane (or pain, in this case), and standard menu stuff. Strike 3: buggy, crashing at the drop of a hat. One of the few options on the menus is "Update", which I access frequently to try to get a better version. While it works mostly, it also locks up on a regular basis. At least the syncing part pretty much works and is fast. It really shows, though, how much you take something like iTunes + iPod for granted until you see an amateurish version.

The Bottom Line
The above notwithstanding, I love this little thing, especially for eBooks. If they would get the PDF bug mentioned above fixed, it would be Good Enough for PDFs too. Of course, what I would really like is the ability to reflow PDFs, but thats much tougher. If anyone reading this knows of a PDF to eBook converter, please let me know...I've searched in vain. The Sony eReader has some pain points (like the PDF bug) for a version 1.0 product, but it still a great addition to my collection of gadgets.

Why I Hate Christmas Music

Most of my family thinks I'm a curmudgeon because I hate Christmas music. Generally, I don't bother to go into the full details of why I hate it because it is long and drawn out, and chances are slim that they would understand anyway. But here's why.

Music has a special place in my life. Like a lot people in the software industry, I have an intense relationship with music. Ever since I discovered music in a big way when I was in the 7th grade, it has been a constant companion. Literally. I always have some music playing in my head, always in the background. There is no way for me not to have music playing in my head. I wake up with music, spend my whole day with it, and go to sleep at night with it. And therein lies the problem.

I hate catchy tunes because the stick in my head and won't go away. What most people like about pop music is the very thing that makes me dislike it: the hook. The problem is that those little snippets of music get stuck in my head and persist. Hours later, I find the same track playing over and over, an increasingly irritating background noise. After a while, I have almost a sore spot in my conscience, a groove worn deep by the same little track playing over and over. The catchier the tune, the harder it is to exorcise. It's like a cut on the roof of your mouth: constant low level irritation.

This explains several things about my music taste. It explains why I have 1000+ CDs and counting: I must have variety. It also explains why I like so much music that most people find disturbing and discordant. I crave complex music all the time. My wife, Candy, complains that I don't like any "happy" music because I have such a taste for the unusual: the anti-catchy tune.

And that also explains why I don't like Christmas music. By definition, all Christmas music features catchy tunes, designed for humming and singing along. When I was a kid, I enjoyed it as much as the next person. But is has a cumlative effect. Every year, the grooves in my head worn deep from the past get worn deeper. I find myself in a constant bad mood around Christmas because the music is inescapable. This last year, they were playing it on the airplane as they taxied, at exactly the time you are forbidden to use some other music player to drown it out. My head was filled with those tunes for hours afterwards, boring their way through my gray matter. I defensively wore my iPod in every store that I went during the holiday season

Christmas music isn't the only thing that affects me. At the client where I am currently stationed, some fool has his cell phone play "When the Saints Come Marching In" as his ring tone. For the first few weeks I was here, I would wonder why, at the end of the day, my head was full of that irritating tune, playing over and over, drilling its way through my head. It's like inadvertanly stepping in something sticky, and now every step pulls irritatingly on your shoe. I figured that it must be a cell phone somewhere, so I started listening for it. Sure enough, someone in the vast cube farm over my right shoulder gets about 10 calls a day, polluting the air around me with that insidious song. I wondered why coming to work here put me in such a bad mood: now I know. Because I know it's there now, I try to fight it off everytime it rings with some other music that is soothing (to me at least) that I can summon from my mental reserves. It's not cool here to were my iPod or I would, all day, every day. And the worst part is, I can't really complain because I can't find this person (the office is large enough so that I can't home in on the signal before they answer it) and they would think I'm crazy. Just like my family thinks I'm a scrooge.

Technology Snake Oil Part 11: Customizable Rube Goldberg Machines

I've found myself more and more talking to people about SOA because of my background with distributed computing. Over and over, in these discussions with CIO's, CTO's, and CCYAO's, they keep talking (without realizing it) about Customizable Rube Goldberg machines.

Rube Goldberg is a cartoonist whose name made it all the way to the dictionary:

Rube Golderg - n. A comically involved, complicated invention, laboriously contrived to perform a simple operation.


Rube Goldberg was a Pulitzer Prize winning cartoonist, sculptor, and author who lived from 1883 - 1970. He is well remembered for his cartoons (thus the dictionary entry), some of which everyone has seen (maybe without realizing it). Here is Rube Goldberg's "Simplified Pencil Sharpener":



Many SOA architectures look just like this diagram, with similar unintentionally ironic names. As I wrote before in Entropic Software, software tends towards complexity. It seems doubly so that enterprise architecture tends towards (frequently needless) complexity. Every SOA pursuit I've ever seen could stand some significant simplification. Of course, most consultants won't say that; they view monstrously complex software as job security. That is perhaps one of the differences from mercenary consultants who are only in it for the buck (complexity == $$$ + long periods of time == job security). If, on the other hand, you have someone who is trying to implement what you need, the tendency should be towards simpler architecture and systems. This isn't a slam against big consulting firms (I know some guys who work for the biggest who really care about what they do, and they would agree with my assertion).

The Snake Oil here comes from C(x?)O's who believe that Rube Goldberg-like complexity is inevitable, and this myth is perpetrated by mercenary consultants. Like Rube's "How to Tee a Golf Ball without Bending Over":



The key to successful distributed computing lies with extreme simplification. The most scalable distributed architecture in the universe (as far as we know) is based on a very simple protocol: HTTP. The key to successful SOA endeavors is two-fold: simplicity and testing.

You can read more about Rube Goldberg and see a whole gallery of his work at www.rube-goldberg.com.

Writing Assignments

I haven't been blogging a lot lately because I've so tied up with other writing stuff. I have a huge number of writing projects ongoing. Ironically enough, I've been writing more than ever, just on longer-term stuff than blogging (and, for various reasons, my primary blog writing time, which is sitting in airports, has been down lately). Here are the projects:

• I'm editing the No Fluff, Just Stuff Anthology, Volume 2: The 2007 Edition as we speak. Just like last year, this one is filled with contributions from No Fluff, Just Stuff authors (14 of them this year) ranging from Web Services, language stuff, testing, agility, persistence, and a whole bunch more. We've also retained the very popular The Authors Speak chapter, where the authors talk about books, favorite tools, and (new this year) favorite travel horror stories. We've also added a couple of collaborative chapters, one on Eclipse Tips and Tricks and another on IntelliJ Tips and Tricks, with contributions by authors who use those environments. This one is bigger than last year's, weighing in at over 350 pages. It'll be out in March of 2007.
  


• 
  The Productive Programmer, for Pragmatic Press, is the book David Bock and I have been working on for about a year. We keep coming up with good ideas and our writing time keeps getting OBE (Overtaken By Events). But, he and I are really starting to crank out some pages because we really want it out in the first half of 2007. Look for it to pop up on the Pragmatic Press website soon as a beta book.
  


• 
  I'm also involved in a project with 3 other authors (Zak Tamsen, Jeremy Stell Smith, and Joe O'Brien) on writing DSLs in Ruby. Frequent readers of my blog know that this is a big topic for me, and my co-authors have come up with some awesome examples. This book is also for Pragmatic Press, and we've been cranking away at the pages recently, after spending several months discussing and generating examples. This book will blow some people's minds!
  


• 
  I'm also contributing to another anthology which shall go unnamed at the moment. Thankfully, that was just a couple of articles (one solo, one collaboration) that is done and off my plate for now. I wrote about Polyglot Programming, greatly expanding on the ideas presented in my blog entry of the same name.
  

Think I've got enough writing pending? Whew! Hopefully, if it doesn't kill me, I'll have 2 books and 2 anthologies out in 2007.

Finding the balance between short-term, idea fountain type writing in blogs and longer term, more cohesive writing for books is an increasing challenge for those of us who write a lot. Some of my colleagues (like Jay Fields) have essentially written a book, one blog entry at a time. If he published his blog between covers, he'd have one of the influential DSL books on the market. I've been trying to talk him into doing just that. It just goes to show how something as staid as technical publishing is not immune to seismic shifts because of the very topics that appear in their books.

ClearType

My friend and college Karan scolded me the other day when I told him about ClearType, which every user of Windows should use on their laptops. He was scolding me because I hadn't blogged about it, and he and I see each other rarely these days (as I'm no longer in Chicago). So, this is for you, Karan. I'll try to be better in the future when I find stuff like this.

ClearType is one of the great innovations produced by Microsoft R&D. Microsoft has a massive R&D effort world wide. Last year, I attended the Microsoft Tech Summit, and some of the coolest stuff we saw came from the head of R&D's presentation. ClearType is a utility that you download from Microsoft's site (here). It does sophisticated font smoothing, making fonts look much better on LCD displays. I'm not nearly so sensitive to fonts as some of my more aesthetically inclined colleagues (hey, David F., I talkin' about you), who can tell you in minute detail about kerning, anti-aliasing, and a bunch of other arcana. So, it's a big deal that I can tell that ClearType makes a noticeable difference. It makes the fonts look less jagged, and in the latest versions, it allows you to adjust the effect to get the best possible display.

If you are using Windows (still), and especially if you are using a laptop, you should download ClearType and give it a try.

Ruby, Meet the Enterprise

A lot of people (myself included) are convinced that Ruby is going to be the next major language (regardless of the platform it runs upon: native, JRuby, or the CLR). And, consequently, Ruby will have a major impact on enterprise development. In fact, I think that SOAR (SOA + Ruby) will be hugely popular in the coming years. As I've stated before, Ruby is the revenge of the Smalltalk crowd. It has a similar feel to developers but isn't hamstrung (like Smalltalk was) by proprietariness and expensive tools. Where Smalltalk failed, Ruby will succeed. Ruby is perfected positioned to become a major player in the SOA space: loose typing == flexibility, which is what SOA is all about.

Interestingly enough, if you look at the guys who were really into Java in 1995 and 1996, they are almost all heavily involved (and mostly billable now) in the Ruby world. Of course, this isn't an accurate barometer, but there is no good way to tell what is the Next Big Thing. Looking at people who've identified it before is as good a way as any.

Anticipating Ruby's move into the Enterprise world, next year features erubycon, the first Enterprise Ruby Conference. This is highly important to the Ruby community who wants to promote Ruby in the enterprise. Having a conference dedicated to this topic helps validate both Ruby's readiness for the enterprise and vice versa. I predict that, like RubyConf, this conference will be small this year, and in 5 years it'll be massive.

Dynamacism

The Pragmatic Programmer contains the outstanding piece of advice that you should learn a new language every year. It not only gives you more tools to apply to problems, but it encourages you to think in different idioms. Every language has its own set of patterns and styles (the original Kernighan and Ritchie book on C, known affectionately as the "K&R" book) was as much about how C should look as what it did. People still talk about the "K&R style"; they defined the idioms of C along with the language. Learning new idioms and styles makes you a better developer, thus the efficacy of the Prags advice.

Here's a recent example that I perpetrated. After a stint in .NET and Ruby on Rails, I'm back on a Java project for a while (and coding again in my beloved IntelliJ, even if it is on Windows). I ran across some code that caused my refactoring alarm to go off. Here's a snippet:

  
public void record(String toBeRecorded) {
    MessageWriter messageWriter = null;
    try {
        messageWriter = new MessageWriter(_destination);
        messageWriter.write(toBeRecorded);
    } catch (RuntimeException rx) {
        throw rx;
    } finally {
        if (messageWriter != null)
            messageWriter.close();
    }
}
public void record(List<String> listToBeRecorded) {
    MessageWriter messageWriter = null;
    try {
        messageWriter = new MessageWriter(_destination);
        for (String s : listToBeRecorded)
            messageWriter.write(s);
    } catch (RuntimeException rx) {
        throw new RuntimeException("Cannot write list of messages");
    } finally {
        if (messageWriter != null)
            messageWriter.close();
    }
}

This is an overloaded method that does basically the same stuff before and after the important stuff (writing a single message or a list). Because I've been in languages with closures for a while, I refactored it thusly:

public void record(final String toBeRecorded) {
    recordMessage(new Runnable() {
        public void run() {
            _messageWriter.write(toBeRecorded);
        }   
    });                                                   
}
public void record(final List<String> listToBeRecorded) {
    recordMessage(new Runnable() {
        public void run() {
            for (String s : listToBeRecorded)
                _messageWriter.write(s);
        }
    });
}
private void recordMessage(Runnable recordingStrategy) {
    try {
        _messageWriter = new MessageWriter(_destination);
        recordingStrategy.run();
    } catch (RuntimeException rx) {
        throw new RuntimeException("Cannot write list of messages");
    } finally {
        if (_messageWriter != null)
            _messageWriter.close();
    }
}

This is basically just the Command Design Pattern. But before I was using closures on a daily basis, I never would have thought to implement it this way. I would have created some additional classes in their own files, created a formal implementation of the Command pattern, and a lot of other excessive ritual. Of course, if this got much more complex, it might need that ritual but, in this case, I was able to consoliate all that open and close code in a single place (which made things DRYer as well).