This is the third in a series of blog posts where I discuss what I see wrong with SOA (Service Oriented Architecture) in the way that it's being sold by vendors. For those who are behind, you can read the first installment and second installments.
While rank and file developers go to conferences to soak in deep technical content, their peripherally technical managers (the ones who wrote some rockin' good Cobol code back in the day, but now they make decisions about modern enterprise architecture) go to different conferences in Palm Springs. At those conferences, they have a 2-hour morning session, run by a big tool vendor, then play golf for the balance of the afternoon. And what the vendors show them is poison.
Mostly what they see these days are tools that support SOA and ESBs. And in particular, the favorite demo-ware application is their BPEL (Business Process Execution Language) designer. This designer allows you to wire together services by drawing lines between boxes. The lines can include transformations and other sexiness. And it demos great. "Look, just draw a couple of lines here and here, click on the Run button and voila! Instant SOA".
Then, the manager brings it back home and notifies the developers that this is the new tool of choice. When developers start using it, they realize the awful truth: they've been sold a hairball generator. Tools like this work great on really small scales, when it's easy to see all the connections between things. But, as things get complicated, they start suffering from the hairball effect: all the lines start running together, and you can't create a diagram that makes sense to anyone anymore. Perhaps maybe you can fight through this, by creating workflows in chunks, and zooming in and out.
Then reality arrives. Because you create workflows using these tools, you are coding, in the worst possible language (a graphical representation). Thus, you are defining behavior, just like you do when you write source code. But the behavior you define lacks all the benefits you get from writing it in code.
reuse: you can't really reuse portions of your workflow because their is no method or subroutine functionality (you might get lucky with a sub-workflow). Mostly, people achieve "reuse" by copy and pasting, which you never do in code.
refactoring: no refactoring, making it harder to identify common workflow chunks for reuse. When you don't have refactoring, you don't watch for opportunities for refactoring as much.
limited programmability: you don't get if statements and for loops, you get whatever this particular BPEL designer supports. You get flow-chartly looking stand-ins for real decision statements, but they are much more brittle than the facilities offered in modern languages.
testing: you can't write unit, functional, or integration tests for these workflows. The only real testing option is user acceptance, meaning that the entire universe must be up and running. If you have no unit testing, you also don't have mock objects or other testing techniques common in code.
hard to diff: lets say you fought the beast and get a non-trivial workflow up and running, and everything is great. In six months, you change it in non-trivial ways, and all is good. Then it comes time to see what's different. BPEL tools don't have diff facilities, so you can either visually diff the diagrams (yuck) or diff 2 10,000 line XML documents (double yuck). BPEL relies on either heavy-weight diagramming tools or raw XML, and nothing in between.
Tools like this fall into the category one of my colleagues identified as doodleware tools. They let you create pretty pictures but collapse under scale. And they don't support all the common facilities offered by good old fashioned code. Is it really worth giving up robust reuse, refactoring, testing, programmability, and versioning/diffing just to see the pretty picture? Ironically, it's pretty easy to generate a similar picture from code, using tools like GraphViz.
I am a strong believer in the mantra
Keep behavior in code.
We have great tools for code (including ways to generate doodles) -- why would you want to abandon what works for something new and shiny? Except, of course, that code won't take you out for a golf outing in Scotland if you choose it.
14 comments:
"Then, the manager brings it back home and notifies the developers that this is the new tool of choice."
Even if the new tool of choice, accidentally turns out to be right, this is still the underlying problem. So I would tend not to focus on why the decision is wrong and instead focus on why the decision process is wrong.
But don't you get consulting and training assignments from these same golfing IT managers? How do they react when you tell them and/or their staff what your wrote in this post?
Peter -
Sometimes, but I'm not willing to do what I consider bad for their business just to make a buck. That makes me a lousy mercenary, but I sure sleep better.
It's been my experience that, long term, it's better to tell it like I see it. Ultimately, my value as a consultant is the guy who isn't shy about expressing my opinions. If that annoys someone, then we would have had a poor professional relationship anyway.
I agree with everything you say about BPM. I would like to add that the BPM engine is one more thing to setup and one more thing to break. If you are using something simple like JBPM in a limited fashion within a single application, it is not so bad. However, when you start using a commercially licensed BPM for SOA, that is when you really run into difficulties. When our team tried to use one of these BPM engines, one developer out of six was able to get it to work on his machine. It is also a huge memory hog. Anyway, once managers and architects decide on this sort of thing, everybody else is just stuck with it. All we can do is make the best of it. I think it is a fad that will eventually go away.
This is slightly off-topic. A few disclaimers up front: I work for IBM on the Information Server product, specifically on the parallel execution engine. This isn't a BPEL tool per se, but it has a number of similarities to what you've described. In particular, you write data flows using a graphical design tool that get executed by an engine. I've worked on, and with, this flow-based programming system for over a decade.
Being a developer, and an emacsish kind of guy, my instincts are all with you when you talk about code vs. diagrams. In general, I just have no patience for GUIs. But - the problems you mention regarding graphical design tools are not insoluable, and in fact Information Server (IS) addresses some of them:
* Coding with graphics: When I start thinking about taking some bolus of data, manipulating it, and putting it someplace else, I inevitably wind up standing at a whiteboard drawing boxes and arrows with annotations. So even a script-enabled weenie like me thinks graphically for this kind of problem. I don't agree that it's the "the worst possible language."
* Reuse/refactoring: IS has "shared containers", which are parameterizable subflows, resuable across multiple jobs. If you edit the container, the flows that use it pick up the new behavior (unless you explicitly incorporated the container by copying, which is not the default). BTW, if you've never encountered reuse in code via cut'n'paste, as opposed to subroutines, you haven't seen much real code. Refactoring is, in my experience, far less commonly practiced than we'd all like. It's one of the first things I do when attacking a glob of crufty old code.
Now, that's a small subset of your objections, so clearly we have a ways to go. But the problems aren't inherent in the graphical design paradigm:
* Large diagrams: Simple, fast pan and zoom controls (think Google Earth) would help a lot, along with the ability to cut the window into separately scrollable panes. But consider textual code: can you see all of that at once? Or do you wind up scrolling and bouncing back and forth between files a lot? (Emacs makes that really easy; it's painful to watch the way most programmers browse text...)
* Diffs: Yes, a good visual diff tool would be nice. There's no reason a tool couldn't highlight (change in line or background color?) the sections of a flow that have changed between versions.
* Testing: It's possible to compose the heart of a flow as a container and then substitute alternate data sources (such as imported flat files) to drive unit tests. But the tools don't (currently) make this easy.
Now, in my heart of hearts, i am a scripter, not a picture-drawer. Information Server's parallel engine has a low-level scripting language, and we in development use it heavily for unit and regression testing. But scripts containing multiple flow paths (several prep steps leading up to a join, etc.) are awkward to visualize as text; what I'd really like is a thorough round-tripping tool between the graphical design tool and the scripting language. Then I have the best of both worlds. Why should we be content with one-way tools?
Great post Neal. Couldn't agree more. Thanks for sharing.
Hi Neal,
I couldn't agree more. I have a conflict of interest with these tools and see that many of them do have these failings. Looking at companies like CogHead that have gone out of business recently demonstrates the fact that there is a significant problem with design tools like these being touted as the holy grail.
I have been involved with a project that spent nearly a year getting a BPM enabled system and we had in-house consulting by the very large vendor. It was a decision by management to use this technology. The BPM system that was deployed was a disaster, took down half of the org's database servers and cost them a lot of money and the developers a lot of sleep. In the end it was replaced with 4 days of coding C#.
So I am disenchanted with that aspect with BPM. The thing is I have written a BPM engine that is quite useful, but I take the approach that the workflows that are created should only be used for specific things such as user-customisable extensions such as 'What actions should you take after an order has shipped' and ways to add extensibility for those that cannot code and need simple communication customisation after. The reality is that these systems should always be 'replacable by code' where needed, and are only ever sold as part of the solution.
A project should be a balanced score card of technologies and not a polarised view of one technology or vendor. You should be able to tell a client that your product is not the right choice if you know the right answer.
-Glenn, the insane swimming aussie.
@Jerry Callen:
"When I start thinking about taking some bolus of data, manipulating it, and putting it someplace else, I inevitably wind up standing at a whiteboard drawing boxes and arrows with annotations. So even a script-enabled weenie like me thinks graphically for this kind of problem. I don't agree that it's the "the worst possible language."
Yes, many of us think graphically about design. Unfortunately, planning and actually producing are still different activities (to some degree).
Visual tools inevitably break down for the latter once you get above a certain threshold of complexity.
And while some of these issues can be mitigated (you provide some good suggestions), these will require huge investments of time and money in tooling support. Traditional code is text. I have a myriad choices of tools to view, transform, create and otherwise manipulate text. Equivalent tools for "visual code"? Not so much.
Nice analysis of the state of things. I don't think I really believe the miss-selling story entirely. Sure, it must happen, but ESBs do fill an important role in extracting integration logic and promoting looser coupling. Maybe not as big a role as the $ would imply but where is the alternative?
@Thomas Kjeldahl Nilsson:
Yes, many of us think graphically about design. Unfortunately, planning and actually producing are still different activities (to some degree).
I'd like to be able to do as much as the design and production using a good graphical tool. I have no trouble at all envisioning an improved version of the DataStage Designer that would make even an emacs/scripting weenie like me very happy for nearly everything I want to do with DataStage. We aren't there - yet. But I have hopes. Flow-based programming is an excellent fit to most business apps, and it really wants to be expressed graphically.
Now, I just wrote, "nearly everything." The major exception is, of course, the situations where I want to be able to generate a flow diagram using another program. In that case, I need to be able to target some sort of textual representation. What would really please me is to be able to target a textual representation that is round-trip-able with the graphical tool. This facilitates the initial prototyping of the generator, and also facilitiates debugging.
Visual tools inevitably break down for the latter once you get above a certain threshold of complexity.
Our current tools do. I don't agree that this breakdown is inevitable.
And while some of these issues can be mitigated (you provide some good suggestions), these will require huge investments of time and money in tooling support.
Well, yeah! I want to be paid to build those new tools! :-)
Remember when Google Maps introduced the "pan by grab and drag" model to map viewing, supplanting scroll bars? It was a major, "Oh, WOW!" moment, at least for me. I think we can produce tools for flow-based programming that are equally pleasurable to use.
Traditional code is text. I have a myriad choices of tools to view, transform, create and otherwise manipulate text. Equivalent tools for "visual code"? Not so much.
Right. We're not there yet, and as I say, I think we'll always want a round-trip-able text representation as an escape hatch. But just because we aren't there yet doesn't mean it isn't possible and desirable.
Hi --
Good post and comments.
Stepping back, at the mountaintop level, the problem is also about process. A lot of programmers and all vendors in the world believe every problem known to humankind can be put into little order-system, linear and mechanical ‘data processing’ models. Problem is, they can't. The overweening process era is waning.
Fact is, emerging enterprise business problems inhabit scalable networks as complex adaptive systems. Process engineering is categorically the wrong approach. Under this new complex network nature and outcome focus, SOA, BPEL, BPR collapse under their own process hubris.
-j
http://1id.com/=jheuristic
this is a great post.
@Jason Callen
There's nothing wrong with UML or BPM, as long as clean the whiteboard when you're done (take a photo if you need to "save" it!).
Personally I find a good whiteboard a great way to think visually about design. And sometimes I even find I want to draw the process design with "a tool" (Omnigraffle my current tool of choice) in order to put it on a wiki or maybe show others who weren't there when we drew it up while discussing it on the whiteboard.
Implementation, as others have said, is another matter.
Amen brother! I'm a 47 yr old architect that wrote some rockin' C++ at a shipyard back in the day. Anyhow, you're right about the peripheral management and even executive staff. No one wants to deal with detail, just network diagrams, security, and server infrastructure. The application development suffers. No training (saving money) ... I can't remember the last time I went to Developmentor for a week of SDK training or BPEL for that matter LOL. I wish we could get through to the real decision makers .. and tell them "stop buying things because you found it on an adsense link or saw it in an eWeek magazine in your bathroom. Good posting.
The article is very relevant and raises very interesting problems. But I don't agree 100%. The problem we have is that people think this brand new SOA BPEL-design tool should replace all code. I have made the same mistake myself, and learned from those mistakes. BPEL is a domain specific language for web-service centric BPM. You have to think about how you use a DSL. SQL is a DSL but you don't try to code your whole solution in SQL. You use it for some parts, where it's relevant.
Another example is Spring Web Flow, for those who are familiar with it. It is actually a BPM-tool in a certain form, but you don't use it to describe anything else than user-interface workflows.
What I'm trying to say here is that use BPEL wisely, and it will help you. Use code to describe how to do things, eg. your detailed business logic, use BPEL (or BPM) to desribe the flow and data. If you have many branches in you BPEL, you should concider redesigning.
Regarding reuse; If you find yourself copying BPEL-code you have designed your integration/service wrong. You should reuse on a Web Service level. A very importan principle of Service Oriented design is that a service should do ONE thing, and do that thing WELL. So if you follow that principle you will partition your problem into several services, and some integration components. Some of the services will be reusable, some wil not.
Regarding testability; I agree that it's hard to test your SOA/BPEL-code. This is a weakness, but if you follow the principle I described over you will be able to test the various componetns (services) by them selves using tools such as SoapUI. Also if you write your low level business logic in Java for example, this is totally testable.
Post a Comment