Easy external DSLs for Java applications

Or JRuby for fun and profit

I’ve been developing software for some time now, and have recently found myself applying ideas from various esoteric areas of computer-science to every day tasks of building common-place applications (often these concepts are quite old, indeed some were thought of in 1958).

One powerful idea that I’ve been playing with recently is that of embedding a DSL (domain specific language) into your basic application framework – and writing most of the features of the application in that DSL.

This is simply an implementation of the concept of raising the level of abstraction. The point, of course, being that when writing code in a DSL implemented in such a fashion, one can express ideas in terms of high-level abstractions that represent actual concepts from the problem domain. In other words, it is like using a programming language that has primitives rooted in the domain.

A lot of people have been writing about this kind of software design – and most implement these ideas in a dynamic language of their choice. How does one go about doing the same in a language like Java? That is what this article is about. And I cheat in my answer. Consider the following design stack –

Creating DSLs in JRuby

I propose that you only implement basic and absolutely required pieces of functionality in Java – the things that rely on, say, external systems that expose a Java interface, or some EJB-type resource, or some other reason that requires the use of Java. The functionality you develop here is then exposed through an interface to the layers above. You can also add APIs for other support services you might need.

The layer above is a bunch of JRuby code that behaves as a facade to the Java API underneath. This leaves you with a Ruby API to that underlying Java (and other whatever, really!) stuff – and makes it possible to code against that functionality in pure Ruby. The JRuby interpreter runs as part of the deployable and simply executes all that Ruby code transparently. As far as your Ruby code is concerned, it doesn’t even care that some of the calls are internally implemented in Java. Sweet!

We can stop here. At this point, we are in a position to write the rest of our application in a nice dynamic language like Ruby. For some people, a nice fluent interface in Ruby suffices – and keeps all developers happy. This is depicted on the top-left part of the diagram.

However, we can go one step further, and implement a DSL in Ruby that raises the level of abstraction even more – as referred to earlier. So on top of the Ruby layer, you’d implement a set of classes that allow you to write code in simple domain-like terms, which then get translated into a form executable by the Ruby interpreter. This is shown in the top right part of the diagram. Ultimately, potentially any one (developers, QA or business analysts) could express their intent in something that looked very much like English.

So where to write what?

How much to put in your Java layer depends on the situation – some people (like me) prefer to write as little as possible in such a static language. Others like the static typing and the associated tool support, and prefer to put more code here. When merely shooting for a little bit of dynamism through the DSL engine in the layers above, most of the code could be written in Java, and a fluent API in the dynamic language could be enough. When shooting for rapid feature turn-around and a lot more flexibility, most of the code could be in the DSL or in the external dynamic language.

The answer to this question really depends on things like the requirements, team structure, skill-sets, and other such situational factors.

OK, so where’s the code?

My intention with this post was to stay at a high level – and talk of how one could structure an application to make it possible to embed a scripting engine into it, and to give an overview of the possibilities this creates. In subsequent posts, I will talk about how actual DSLs can be created, tested, and also how a team might be structured around it.

Story points – handling unknowns

I’m a big fan of usingstorypoints to estimate effort on software development projects. It is a good way of keeping things simple and it ensures that teams can be efficient and quick about their estimation tasks.

One question that often comes up is how a team ought to handle stories (or epics) that they just don’t know enough about at a given moment. My answer depended on the kind of scale the team was using – if they were using T-Shirt sizing, then the scale ought to have an Unknown Size (translating to a scale of say – XS, S, M, L, and UNKNOWN).

If they’re using a numbered scale, then I suggest a twist on the UNKNOWN level. I’ve recently switched to using a geometric scale (as opposed to a Fibonacci scale) – and to that I’ve added an UNKNOWN level of 1,000,000 points. So now, my scale is – 10, 20, 40, 80, 1000000.

What this does is it radiates a little bit more information about the total estimate for the project. Lets say you have 200 stories, of which 10 are unknown. Lets also say that the rest 190 stories add up to 8450 points. Now, the total (because of the new level for the unknown items) becomes 10,008,450 points. It is still clear that we have about 8450 points of estimated work, but it also makes clear that we have 10 items that are unknowns.

If the unknown level was only twice or thrice or even ten times the size of a Large story, then the total for the project obfuscates how many items are unknown – and gives an impression that is not quite accurate. This large one-million-point level for unknown stories fixes that. And it also effectively broadcasts the idea that a subset of stories just don’t have the same level of detail and clarity as the rest.

Gotham Ruby Conference 2007

I attended GoRuCo 2007 yesterday, and it was a fine conference – where I met a bunch of old friends I hadn’t seen in a while. Here’s what the agenda looked like – I enjoyed the talks on Adhearsion, JRuby, and BNLs. And the conference took place in Google’s offices in NYC, and that was neat.

It was cool how many people knew of ThoughtWorks, and were interested in talking about what it was like to work there. (Probably less than the number that were interested in Google in the same way, but hey…) We ended the conference by finishing up with the afterparty in the wee hours of the morning. It was fun! Amazingly, both Deepthi and I won swag at the event – I got this neat laptop bag, and Deepthi got Rails Recipes. Oh, and here are some pictures.

You might want to read some live-blogging from the event.

Advantages of story-trees

There are several benefits of using story trees, and they’re primarily in two areas –

Backlog (or Master Story List) creation:
Mind maps, which can be used to create the story trees in the first place, make for an excellent tool for facilitating brainstorming sessions within the team
– They’re also a fundamentally intuitive tool for thinking about things, and this in itself make things very easy

During development:
– It is easy for a developer to look at a story title in the context of its tree and understand the workflow it belongs too, and the business context in which it was created
– It is an easy, visual way to see what stories are connected to what other ones, and how the dependencies might turn out. It also helps in managing those dependencies across those stories, because of its visual nature.
– Speaking of how things are related, it is also easy to see areas or pieces of functionality that are impacted by change in a particular area. Not always, but this is useful many times.
– By color coding the nodes and leaves of the tree, say based on status, priority, size (t-shirt sizes or story points) or whatever else makes sense, one can render different view of the story tree, which can communicate different things in a very easy to understand manner
– It is a very useful tool with which to have meaningful conversations with the customers. When asked things like where scope-growth was occurring, or where they might look to trim scope, the tree becomes a useful talking aid. It can be color-coded to show times ranges when certain stories were added, etc.
– Finally, story trees are useful because they always pitch the topic of conversation (say about a set of stories) in the context of the entire graph of requirements and business work-flows. This perspective is often lost, and is extremely useful when having certain kinds of conversations (the important ones).

The joys of large-screen monitors

In my ideal world, I’d have two 30″ Apple Cinema Displays, but my less than ideal world ain’t so bad either. My souped-up 24″ iMac serves me well for my day to day work, quite well indeed.

I’m positive that my productivity gains are large – I always have emacs open with at least 4 frames, or eclipse with several views open without anything important having to stay annoyingly minimized. I also have a browser open with lots of tabs, a bunch of terminals open with iTerm, and a bunch of other what-nots, and managing all this is easy. This multiple desktop manager makes it even easier. Finally, if all those windows distract you, use Think!

Anyway, if you want a quick and relatively cheap way (in the long run) to become more productive, get a large screen monitor. Or three.

Requirements management, user stories, mind-maps, and story-trees

For those who’ve been following along, I recently started a new project team that’s working on pretty much a green-field type of a project. I’ve been using mind-maps to kick things off, in various areas of the teams’ initial effort, and have now extended it as a way to build the functional backlogs that the teams will work on.

Essentially, what that means is this – imagine that we, along with an analyst (or a team of analysts) sit down to brainstorm an area of the application. We always start with the high-level ‘project’, lets call it Todos2Go. Just to retain context, lets also go ahead and list out the high-level functional areas of the application. These include modules we already discussed and elaborated, they certainly include new modules we want to include in our system. So in our example, Todos2Go would start with the basic todo-list creation module. It might also have a collaboration module, a search module with advanced search options, a calendar integration piece, an RSS generation module, an email interface module, a tagging and tag-browsing piece, and others. The analyst team doesn’t worry about trying to capture everything at any given point of time, knowing that they will be revisiting these requirements (the mind map) whenever they need to. So at this point, here’s what we have –

todos2go.png

Remember, we’re only capturing functionality from a business requirements perspective right now. Things like config-management, architecture, etc. are somewhat orthogonal areas. They’re captured in a different mind map. Also, we’re not concerned with scheduling right now – as in, when a particular piece might get built. Those decisions will be made based on business priorities, technical dependencies, and developer estimates. So, onward! Lets continue following the analyst team as they keep using the mind map to go further and further into details.

The analyst team picks one module that they’re going to focus on for the moment. They decide that since they’re just starting the project, they might as well pick the one they know best (from previous versions of the software, or from market research) – a basic task-management feature – the Todolists module. So they start to discuss the things the user might want to do, and/or how they would like to break the functionality down for development. Here’s something they might come up with –

Todos2Go-Todolists.png

The idea is simple. Keep breaking things down as you go down a path of functionality. In other words, go ahead and do plain ol’ functional decomposition. Again, remember to view things from a business requirements perspective. This means that ‘design database schema’ is probably not something that ought to appear on this mind map. The numbers next to the items are simply ID numbers, for tracking purposes, and will be of use later on. This view is what I’ve started to call Story Trees, thanks to a colleague – Roger Marlow.

Also, since most people use Excel to track projects (so do I, and I will continue to do so until Mingle launches), here’s how I capture this meeting artifact (it’s usually a whiteboard thing) in a spreadsheet –

todos2go-storylist-excel.png

The Parent ID column is simple. It is just the ID of the bubble in the mind map that is a ‘parent’ of a given bubble. This way, a simple script can be written to reconstruct the mind map if needed. The reconstructed version of the mind map can sport color-coding based on status, priority, or whatever else one might consider useful. More on this script (or my version of it) in another blog post.

So, I do hope this was useful. This is a simple meeting technique for project managers to use to facilitate requirement gathering workshops or business analysis sessions. Again, as I mentioned in one of my earlier posts, I do not use software/laptop/projector in my meetings because it is just not a good, interactive way to run a meeting. I use a whiteboard, flip-charts, and Post-Its. Later, I use software to capture it.

Coming soon – more benefits of Story Trees.

P.S. – Please pardon and ignore the horrid use of the generic term ‘user’ in my story list. We typically use personas, the focus of this post however is the story trees.

From freedom languages to Java (and back again?)

Recently, I started working with Java again. I had little choice in the matter, really, since its for an upcoming product in the mobile application development tools space, and I’m focusing on the Java Micro Edition area. I’ll have more to say on this skunkworks initiative another time. (Watch this space, and all that).

I’ve been using mostly other languages in the recent past, Ruby, a little Python, Common Lisp, a little Haskell. But mostly Ruby. And it seems that having stayed Java-free for about two years has made me really rusty. That apart, this time around Java started out feeling annoying, and morphed into being mostly amusing. In an annoying way. The question I constantly have to tell myself to refrain from asking (out loud, and to the world in general) is – “Why can’t the bloody runtime figure this out for itself? Why do I have to type this extra (vestigial) code?”

In any case, working on the new Java Micro Edition platform again is nice – reminds me of a project I did at college – and of simpler times… 🙂