Wednesday, September 05, 2007

Ruby Matters: Meta-programming, Synthesis, and Generation

In my last Ruby Matters blog post, I talked about meta-programming in Ruby, contending that Ruby gives you "places to put your stuff". I always wondered about meta-programming in Smalltalk and how that compares to Ruby, and Where to Put Stuff in Smalltalk. The final piece of the puzzle came after I talked to Glenn Vanderburg (the Chief Scientist of Relevance). I was puzzled as to why the Gang of Four book (which had examples in both C++ and Smalltalk) didn't have more meta-programming. Lots of the design patterns are almost trivially easy to implement with meta-programming, but they didn't do that in the Smalltalk examples. They used the same structural approach as C++. It looks more and more to me that the Design Patterns book was really just a way to solve problems in C++ that should have been easier to solve in a more powerful language like Smalltalk. Which is why I was puzzled about the lack of of more meta-programming solutions in Smalltalk. Glenn enlightened me. One of the overriding characteristics of Smalltalk is the way code is stored, in an image file, which allows for really smart tools. The program and the environment all reside in the binary image file. There are no source files as we know them today, just the image.

A comparison about implementation details is in order, on how Ruby differs from Smalltalk. Let's talk about has_many in Ruby on Rails. Typical Rails code looks like this:

class Order < ActiveRecord::Base
has_many :lineitems

For those not familiar with Ruby, this is a method, defined as a class-level initializer (just like an instance code block in Java, a chunk of curly-brace code in the middle of a class definition, which Java picks up and executes as the class is instantiated). So, ultimately, this is the Ruby equivalent of a static method call, which gets called as the class is created.

Let's talk about Smalltalk, which has first-class meta-programming. You could easily build has_many in Smalltalk, implemented as a button you click in the browser which launches a dialog with properties that allow you to set all the characteristics embodied in the Ruby version. When you are done with the dialog, it would go do exactly what Ruby does in Rails: generate a bunch of methods, add them to the class (stuff like the find_* and count_* methods). When you are done, all the methods would be there, as instance methods of your class.

OK, so at this point, the behavior is the same in Smalltalk as in Rails. But there is one key difference: The Smalltalk version using code generation. It's a sophisticated version of a code wizard, generating the code using meta-programming techniques. The Ruby version uses code synthesis: it generates the code at runtime, not build time. Building stuff at runtime means more flexibility. But that is a minor point compared to this one: In the Smalltalk version, you use the dialog and properties to generate all the methods you need. The original impetus for the has_many intent lives only while you are running the dialog. Once you are finished, you are left with lots of imperative code. In the Ruby version, the intent stays right where you put it. When you read the class again, 6 months from now, you can clearly see that you still mean has_many. Smalltalk has the same meta-programming support, but the intent of code synthesis remains forever. That's why it's important to have a place to put your stuff. It isn't accidental in Rails that many of the DSL characteristics appear as class methods rather than code you call in the initialize method. Placing them as part of the class declaration declares intent in a big way, and keeps the code very declarative.

To summarize the similarities and differences:

  1. Both Ruby and Smalltalk give you a place to put your meta-programmed stuff

  2. Both give you a place to put the declaration of intent (the tool in Smalltalk, the shadow meta-class in Ruby)

  3. A time in the lifecycle of the class when things happen. In the Smalltalk version, it's a one-time deal, as you use the tool to generate the code. In Ruby, the synthesis takes place at class load time. This leaves the clean, declarative code right where you put it, rather than generating a bunch of much less clear imperative code.

Glenn made an excellent point here: the Smalltalk version is a great example of accidental complexity, not essential complexity. Software is full of essential complexity: writing software is hard. But we end up subjecting ourselves to lots of accidental complexity in our tools and languages. And it should be stamped out. The Ruby version eliminates accidental complexity by providing a great abbreviation for the intent of has_many. This blog does a great job of illustrating the differences between abbreviations and abstractions.

Smalltalk had (and has) an awesome environment, including incredible tool support. Because the tool is pervasive (literally part of the project itself), Smalltalkers generally shied away from the kind of meta-programming described above because you have to build tool support along with the meta-programmed code. This is a conscious trade off. One of the best things about the Ruby world (up until now) is the lack of tool support, meaning that the tools never constrained (either literally or influentially) what you can do. Fortunately, this level of power in Ruby is pretty ingrained (look at Rails for lots of examples), so even when the tools finally come out, they need to support the incredible things you can do in Ruby.

Next up, "design patterns" in Ruby.

Thanks to Glenn for supplying me Smalltalk information, as a sounding board for this, and all the interesting bits, really.


Kirit Sælensminde said...

There is another aspect of this that is at least worth mentioning.

The Smalltalk way of generating the code is done in your time. You as a developer crystallise how the design is to be implemented by clicking on that button.

In Ruby this crystallisation happens each time the code is run.

The trade off here is not only in flexibility, but also in risk. It is much harder to break parts of a Smalltalk implementation by changing the meta-programming. But of course a bug fix has to be manually applied in Smalltalk but in Ruby you get it for free.

Another aspect that is important for some purposes is when all of this happens. Smalltalk won't have any run-time overhead because the hard work has already been done. With Ruby this work must be done each time the code is run.

This is one thing that meta-programming is increasingly used for in C++ as it allows algorithms to be specified at a high level, but parts of them to be executed within the compiler rather than at run-time.

AkitaOnRails said...

Hello Neal, great post. I think you nailed it.

I've took the liberty to translate this post literally into brazilian portuguese, I hope you don't mind but coincidentally I was in the middle of a very hot discussion regarding Ruby and Smalltalk.

One of the readers insist that you probably know nothing about Glorp or Magritte. Mainly Glorp which, he says, "there is no need for something like a has_many because things like this are meta-programmed directly in runtime" and he goes on saying, "which ressembles Django-like models". Whatever that means ... can you get it?

He also claims Smalltalk superiority because he implies that it doesn't have to do 'low level stuff' like the new Ambition gem that was just released to get to the same results. He probably is criticizing the use of the ParseTree level to inject higher abstractions.

He concludes saying that "because everything in Smalltalk is more unified - including having an internal IDE - makes extensions that much more easy than Ruby, basically making DSL-like features unnecessary".

I'm eager to know what do you think about those claims.

Neal Ford said...

One of the readers insist that you probably know nothing about Glorp or Magritte.

That is true -- I know nothing about OR mapping frameworks in Smalltalk. But posing this as strictly a discussion about OR mapping misses the point entirely. I'm talking about the meta-programming technique, not this implementation.

because everything in Smalltalk is more unified - including having an internal IDE - makes extensions that much more easy than Ruby, basically making DSL-like features unnecessary
This is also true, tragically so in the Smalltalk world. DSLs are about abstractions, revealing intent. You can build the same features in tools (ala my example) but you lose the intent of what you were doing. I'm sure that lots of Smalltalkers will disagree with my assertions. That's fine: it isn't a zero-sum game. But to provide tooling like Smalltalk does entails conscious trade-offs. I like the trade-offs made by Ruby so far, and don't like the ones imposed on Smalltalk by the environment. You mileage may vary. But, eventually Ruby will get sophisticated tools, but it would be very difficult to re-wire Smalltalk to provide some of the affordances in Ruby.

Esteban said...

I believe you are missing the point about Smalltalk "meta programming".
In some way, the entire Smalltalk system is meta, because all the classes and methods are "generated" by other objects of the system.
But many meta programming is done by aggregating and/or composing objects, or instantiating new ones.

Hardly ever you hear a Smalltalker talking about "generating code", we use objects, we don't generate code, the method "has" source (aka code), which if lost can be "regenerated" from its internals.
Some developer tools modify the bytecode of the methods to add functionalities (spies, traces, etc) without modify its source.

The only "code generation" technique I see from time to time is the instantiation of some sort of "spec" (windowing, ORM, etc.), which ends instantiating objects.

The "semantic" is not lost if the entities are properly reified.
For example, in one of the available ORM frameworks, the has_many is a message sent to a spec/definition:

define: #children as: (SortedCollection of: Person).


Neal Ford said...

Some developer tools modify the bytecode of the methods to add functionalities (spies, traces, etc) without modify its source.

Which is exactly my point. I didn't mean to imply (impune?) Smalltalk by saying that its meta-programming (which is, I agree, hopelessly overloaded itself) is somehow deficient or weak. In fact, it is probably better thought out than Ruby's. My point (and the only real point of comparison) is that the tools in Smalltalk handled much of this type of coding, whereas we don't have these tools in Ruby. The end result is close to the same, but the Domain Specific Language aspects are preserved in Ruby because we don't have the tool.