conjure – simple mocking and stubbing for Clojure unit-tests

Siva and I were pairing on a unit-test that involved writing something to HBase. When Siva said that mocking the call to the save-to-hbase function would make testing easier (a simple thing using JMock, he said), I decided to write a quick mocking utility for Clojure.

Then later, we realized that we wanted to go one step further. The row-id that was used as the key to the object in HBase was generated using system-time. That meant that even if we wanted to confirm that the object was indeed saved, we had no way of knowing what the row-id was. One solution to such a problem is to inject the row-id in (instead of being tightly coupled to the function that generated the row-id). Instead, I wrote a stubbing utility that makes this arbitrarily easy to do.

So here they are – mocking and stubbing – packaged up as the conjure project on github.

The set up

Imagine we had the following functions –

(defn xx [a b]
  10)

(defn yy [z]
  20)

(defn fn-under-test []
  (xx 1 2)
  (yy  "blah"))

(defn another-fn-under-test []
  (+ (xx nil nil) (yy nil)))

Also imagine that we had to test fn-under-test and another-fn-under-test, and we didn’t want to have to deal with the xx or yy functions. Maybe they’re horrible functions that open connections to computers running Windoze or something, I dunno.

Mocking

Here’s how we might mock them out –

(deftest test-basic-mocking
  (mocking [xx yy]
    (fn-under-test))
  (verify-call-times-for xx 1)
  (verify-call-times-for yy 1)
  (verify-first-call-args-for xx 1 2)
  (verify-first-call-args-for yy "blah"))

Pretty straightforward, eh? You just use the mocking macro, specifying all the functions that need to be mocked out. Then, within the scope of mocking, you call your functions that need to be tested. The calls to the specified functions will get mocked out (they won’t occur), and you can then use things like verify-call-times-for and verify-first-call-args-for to ensure things worked as expected.

Stubbing

As mentioned in the intro to this post, sometimes your tests need to specify values to be returned by the functions being mocked out. That’s where stubbing comes in. Here’s how it works –

(deftest test-basic-stubbing
  (is (= (another-fn-under-test) 30))
  (stubbing [xx 1 yy 2]
    (is (= (another-fn-under-test) 3))))

So that’s it! Pretty simple. Note how within the scope of stubbing, xx returns 1 and yy returns 2. Now, for the implementation.

Implementation

The code is almost embarrassingly straight-forward. Take a look –

(ns org.rathore.amit.conjure.core
  (:use clojure.test))

(def call-times (atom {}))

(defn stub-fn [function-name return-value]
  (swap! call-times assoc function-name [])
  (fn [& args]
    (swap! call-times update-in [function-name] conj args)
    return-value))

(defn mock-fn [function-name]
  (stub-fn function-name nil))

(defn verify-call-times-for [fn-name number]
  (is (= number (count (@call-times fn-name)))))

(defn verify-first-call-args-for [fn-name & args]
  (is (= args (first (@call-times fn-name)))))

(defn verify-nth-call-args-for [n fn-name & args]
  (is (= args (nth (@call-times fn-name) (dec n)))))

(defn clear-calls []
  (reset! call-times {}))

(defmacro mocking [fn-names & body]
  (let [mocks (map #(list 'mock-fn %) fn-names)]
    `(binding [~@(interleave fn-names mocks)]
       ~@body)))

(defmacro stubbing [stub-forms & body]
  (let [stub-pairs (partition 2 stub-forms)
        fn-names (map first stub-pairs)
        stubs (map #(list 'stub-fn (first %) (last %)) stub-pairs)]
    `(binding [~@(interleave fn-names stubs)]
       ~@body)))

It’s just an hour or so of work, so it’s probably rough, and certainly doesn’t support more complex features of other mocking/stubbing libraries. But I thought the simplicity was enjoyable.

frumiOS – a simple object-system for Clojure

I’ve nearly stopped blogging, because all my spare time goes into writing Clojure in Action. But I was a bit bored this weekend, and wrote this little library that can be used to write traditional looking Object-Oriented (TM) code in Clojure.

Why would you do that, when you can use a rifle-oriented programming style instead? Think of it like using the rifle as a club… On the other had, the implementation makes plenty use of closures and macros, so it is probably a rifle-oriented program 🙂

The implementation is hosted on github, in a project called frumios. And if you reall want to see it now, click below.

(ns org.rathore.amit.frumios.core)
 
(declare new-object find-method) 
 
(defn new-class [class-name parent methods]
  (let [klass ((comp resolve symbol name) class-name)]
    (fn [command & args]
      (cond
	(= :parent command) parent
	(= :name command) klass
	(= :method-names command) (keys methods)
	(= :methods command) methods
	(= :new command) (new-object klass)
	(= :method command) 
          (let [[method-name] args]
	    (find-method method-name methods parent))
	:else (throw (RuntimeException. (str "Unknown message: " command)))))))
 
(def OBJECT (new-class :org.rathore.amit.frumios.core/OBJECT nil {}))
(def this)
 
(defn new-object [klass]
  (let [state (ref {})]
    (fn thiz [command & args]
      (cond
        (= :class command) klass
        (= :set! command) (let [[k v] args]
			    (dosync (alter state assoc k v))
			    nil)
        (= :get command) (let [[key] args]
			   (state key))
        :else (let [method (klass :method command)]
		(if method 
		  (binding [this thiz]
		    (apply method args))))))))
 
(defn find-method [method-name instance-methods parent-class]
  (let [method (instance-methods method-name)]
    (or method
	(if-not (= #'org.rathore.amit.frumios.core/OBJECT parent-class)
	  (find-method method-name (parent-class :methods) (parent-class :parent))))))
 
(defn parent-class-spec [sexprs]
  (let [extends-spec (filter #(= :extends (first %)) sexprs)
        extends (first extends-spec)]
    (if (empty? extends)
      'org.rathore.amit.frumios.core/OBJECT
      (do 
	(if-not (= 1 (count extends-spec))
	  (throw (RuntimeException. "defclass only accepts a single extends clause")))
	(if-not (= 2 (count extends))
	  (throw (RuntimeException. "the extends clause only accepts a single parent class")))
	(last extends)))))
 
(defn method-spec [sexpr]
  (let [name (keyword (second sexpr))
	remaining (next sexpr)]
    {name (conj remaining 'fn)}))
 
(defn method-specs [sexprs]
  (let [method-spec? #(= 'method (first %))
	specs (filter method-spec? sexprs)]
    (apply merge (map method-spec sexprs))))
 
(defmacro defclass [class-name & specs]
  (let [parent-class-symbol (parent-class-spec specs)
        this-class-name (keyword class-name)
	fns (method-specs specs)]
    `(def ~class-name 
        (new-class ~this-class-name (var ~parent-class-symbol) ~(or fns {})))))

But first, examples –

(ns frumios-spec)

(use 'org.rathore.amit.frumios.core)

(defclass animal
  (method sound []
    "grr")

  (method say-something []
    (str (this :sound) ", I say!"))

  (method move []
    "going!"))

(defclass cat
  (:extends animal)

  (method sound []
    "meow"))

There, that defines a simple class hierarchy. Let’s examine these classes –

frumios-spec> (cat :parent)
#'frumios-spec/animal

frumios-spec> (animal :parent)
#'org.rathore.amit.frumios.core/OBJECT

frumios-spec> (animal :method-names)
(:move :say-something :sound)

frumios-spec> (cat :method-names)
(:sound)

Now, let’s define a couple of instances –

(def a (animal :new))
(def c (cat :new))

What can we do with these instances? Let’s explore –

frumios-spec> (c :class)
#'frumios-spec/cat

frumios-spec> (c :set! :name "Mr. Muggles")
nil

frumios-spec> (c :get :name)
"Mr. Muggles"

That’s the basic stuff, how about calling methods?

frumios-spec> (a :move)
"going!"

frumios-spec> (a :sound)
"grr"

frumios-spec> (c :sound)
"meow"

Notice how cat overrides the sound method. OK, how about a method that calls another method? It calls for the this keyword. Here it is in action –

frumios-spec> (a :say-something)
"grr, I say!"

frumios-spec> (c :say-something)
"meow, I say!"

Notice how in the second call, (this :sound) resolved itself to the overridden sound method in the cat class. That’s subtype polymorphism, common to languages such as Java and Ruby. We could use it to implement something like the template pattern. We can do fairly arbitrary things with frumiOS –

(defclass person
  (method greet [visitor]
    (println "Hi" visitor ", I'm here!"))

  (method dob []
    (str "I was born on " (this :get :birth-date)))

  (method age []
    2)

  (method experience [years]
    (str years " years"))

  (method bio []
    (let [msg (str (this :dob) ", and have " (this :experience (this :age)) " of experience.")]
      (println msg))))

Let’s play with it –

frumios-spec> (def kyle (person :new))
#'frumios-spec/kyle

frumios-spec> (kyle :greet "rob")
Hi rob , I'm here!
nil

The bio method makes two calls using the this construct, one nested inside the other. It works as expected –

frumios-spec> (kyle :set! :birth-date "1977-01-01")
nil

frumios-spec> (kyle :bio)
I was born on 1977-01-01, and have 2 years of experience.
nil

So there it is. I’m sure it doesn’t do lots of stuff a real object-system does. But at 70 lines of Clojure code, you can’t expect a whole lot more. Silly as this is, I had fun writing it! Click here to see how the frumiOS is implemented.

Software development agility – bottom-up design and domain-specific languages (DSLs)

It’s been nearly three months that I’ve been working at Runa. And in that time I’ve worked with the team to get a lean and agile process in place, and while we’re not there yet, things are getting more and more focused every day.

I’ve also been working to significantly re-factor the code that existed, as we get ready to get the first beta release out the door. In this post, I’d like to talk about one aspect of our design philosophy which I’m glad to say has been working out really well – bottom-up design (and an associated DSL).

Before we decided to change the system, it had been designed in the usual top-down manner. Runa operates in the online retail space – we provide web-based services to online merchants, and while we have a grand vision for where we want our platform to go, our first offering is a promotions service. It allows merchants to run campaigns based on a variety of criteria. Merchants are a fussy lot, and they like to control every aspect of campaigns – they want to be able to tweak all kinds of things about it.

In true agile fashion, our initial release allows them to select only a few parameters. Our system, however, needs to be extensible, so that as we learn more about what our merchants need, we can implement these things and give it to them. Quickly. And all of this needs to be done in a changing environment with lots of back-and-forth between the business team and the dev team.

So here’s what we came up with – it is an example of what we call a campaign template –

campaign_template :recapture_campaign do
  title is 'Recapture'
  subtitle is 'Reduce cart abandonment rate'
  description is 'This campaign presents offers to shoppers as they abandon their shopping cart.'

  #adding criteria here
  accept required time_period_criteria with defaults start_date('1/1/08'), end_date('12/31/10')
  accept required product_criteria with no defaults

  hardcode visit_count_criteria with number('1')

  #more criteria
  reject optional url_referrer_criteria with no defaults

  inside context :view_badge do
    never applicable
  end

  inside context :abandon_cart do
    allow only customer_type_criteria with customer_type('visitor')
  end

  inside context :cart do
    allow only user_action_criteria with user_action('accepted_recapture')
  end

end

A campaign template behaves like a blue-print of an actual campaign. Sort of like the relationship between a class and an object. In a sense, this is a higher-order description of just such a relationship. The (computer) language now lets us speak in the domain of the business.

There are a couple of reasons why our core business logic is written like this –

a) It lets us communicate easily with the business. Whenever a question about a rule comes up, I bring up the associated template up on the screen, and make them read the ‘code’. Once they agree thats what they mean, it just works, because it is real code.

b) Since this is in essence a way to script the domain model, it has forced a certain design upon it. All the objects evolved in a bottom-up manner, and each does a very specific thing. It lends to a very highly de-coupled design where objects collaborate together to achieve the higher goal, but each is very independent of the other.

c) This notation makes several things easier. One, the actual business rules are described here, and they just work. The other thing is that we’re able to use this same representation for other things – for example, our merchant GUI is completely auto-generated off these template files. Menu items, navigation, saving, editing, error-reporting, everything is generated.

This allows very fast turn around time for implementing new concepts, or making changes to existing ones.

It’s an internal DSL written in Ruby, and does whatever it can without any extra parsing, as you can probably imagine. I will write about the specifics of how this is implemented in future posts. For the moment, I would like to stress, however, the importance of the bottom-up approach. Because our domain model is made up of many small objects (instead of a few larger ones), each representing a tiny aspect of the domain, we’re able to put them together and fashion more complex beasts out of them. And the combinations are limitless, bounded only by business rules. This is where the power of this approach lies.

The DSL itself is almost only a convenience!

Easy external DSLs for Java applications

Or JRuby for fun and profit

I’ve been developing software for some time now, and have recently found myself applying ideas from various esoteric areas of computer-science to every day tasks of building common-place applications (often these concepts are quite old, indeed some were thought of in 1958).

One powerful idea that I’ve been playing with recently is that of embedding a DSL (domain specific language) into your basic application framework – and writing most of the features of the application in that DSL.

This is simply an implementation of the concept of raising the level of abstraction. The point, of course, being that when writing code in a DSL implemented in such a fashion, one can express ideas in terms of high-level abstractions that represent actual concepts from the problem domain. In other words, it is like using a programming language that has primitives rooted in the domain.

A lot of people have been writing about this kind of software design – and most implement these ideas in a dynamic language of their choice. How does one go about doing the same in a language like Java? That is what this article is about. And I cheat in my answer. Consider the following design stack –

Creating DSLs in JRuby

I propose that you only implement basic and absolutely required pieces of functionality in Java – the things that rely on, say, external systems that expose a Java interface, or some EJB-type resource, or some other reason that requires the use of Java. The functionality you develop here is then exposed through an interface to the layers above. You can also add APIs for other support services you might need.

The layer above is a bunch of JRuby code that behaves as a facade to the Java API underneath. This leaves you with a Ruby API to that underlying Java (and other whatever, really!) stuff – and makes it possible to code against that functionality in pure Ruby. The JRuby interpreter runs as part of the deployable and simply executes all that Ruby code transparently. As far as your Ruby code is concerned, it doesn’t even care that some of the calls are internally implemented in Java. Sweet!

We can stop here. At this point, we are in a position to write the rest of our application in a nice dynamic language like Ruby. For some people, a nice fluent interface in Ruby suffices – and keeps all developers happy. This is depicted on the top-left part of the diagram.

However, we can go one step further, and implement a DSL in Ruby that raises the level of abstraction even more – as referred to earlier. So on top of the Ruby layer, you’d implement a set of classes that allow you to write code in simple domain-like terms, which then get translated into a form executable by the Ruby interpreter. This is shown in the top right part of the diagram. Ultimately, potentially any one (developers, QA or business analysts) could express their intent in something that looked very much like English.

So where to write what?

How much to put in your Java layer depends on the situation – some people (like me) prefer to write as little as possible in such a static language. Others like the static typing and the associated tool support, and prefer to put more code here. When merely shooting for a little bit of dynamism through the DSL engine in the layers above, most of the code could be written in Java, and a fluent API in the dynamic language could be enough. When shooting for rapid feature turn-around and a lot more flexibility, most of the code could be in the DSL or in the external dynamic language.

The answer to this question really depends on things like the requirements, team structure, skill-sets, and other such situational factors.

OK, so where’s the code?

My intention with this post was to stay at a high level – and talk of how one could structure an application to make it possible to embed a scripting engine into it, and to give an overview of the possibilities this creates. In subsequent posts, I will talk about how actual DSLs can be created, tested, and also how a team might be structured around it.