s-expressions

Amit Rathore blogs about software development

conjure – simple mocking and stubbing for Clojure unit-tests

Posted by Amit Rathore on January 24, 2010

Siva and I were pairing on a unit-test that involved writing something to HBase. When Siva said that mocking the call to the save-to-hbase function would make testing easier (a simple thing using JMock, he said), I decided to write a quick mocking utility for Clojure.

Then later, we realized that we wanted to go one step further. The row-id that was used as the key to the object in HBase was generated using system-time. That meant that even if we wanted to confirm that the object was indeed saved, we had no way of knowing what the row-id was. One solution to such a problem is to inject the row-id in (instead of being tightly coupled to the function that generated the row-id). Instead, I wrote a stubbing utility that makes this arbitrarily easy to do.

So here they are – mocking and stubbing – packaged up as the conjure project on github.

The set up

Imagine we had the following functions -

(defn xx [a b]
  10)

(defn yy [z]
  20)

(defn fn-under-test []
  (xx 1 2)
  (yy  "blah"))

(defn another-fn-under-test []
  (+ (xx nil nil) (yy nil)))

Also imagine that we had to test fn-under-test and another-fn-under-test, and we didn’t want to have to deal with the xx or yy functions. Maybe they’re horrible functions that open connections to computers running Windoze or something, I dunno.

Mocking

Here’s how we might mock them out -

(deftest test-basic-mocking
  (mocking [xx yy]
    (fn-under-test))
  (verify-call-times-for xx 1)
  (verify-call-times-for yy 1)
  (verify-first-call-args-for xx 1 2)
  (verify-first-call-args-for yy "blah"))

Pretty straightforward, eh? You just use the mocking macro, specifying all the functions that need to be mocked out. Then, within the scope of mocking, you call your functions that need to be tested. The calls to the specified functions will get mocked out (they won’t occur), and you can then use things like verify-call-times-for and verify-first-call-args-for to ensure things worked as expected.

Stubbing

As mentioned in the intro to this post, sometimes your tests need to specify values to be returned by the functions being mocked out. That’s where stubbing comes in. Here’s how it works -

(deftest test-basic-stubbing
  (is (= (another-fn-under-test) 30))
  (stubbing [xx 1 yy 2]
    (is (= (another-fn-under-test) 3))))

So that’s it! Pretty simple. Note how within the scope of stubbing, xx returns 1 and yy returns 2. Now, for the implementation.

Implementation

The code is almost embarrassingly straight-forward. Take a look -

(ns org.rathore.amit.conjure.core
  (:use clojure.test))

(def call-times (atom {}))

(defn stub-fn [function-name return-value]
  (swap! call-times assoc function-name [])
  (fn [& args]
    (swap! call-times update-in [function-name] conj args)
    return-value))

(defn mock-fn [function-name]
  (stub-fn function-name nil))

(defn verify-call-times-for [fn-name number]
  (is (= number (count (@call-times fn-name)))))

(defn verify-first-call-args-for [fn-name & args]
  (is (= args (first (@call-times fn-name)))))

(defn verify-nth-call-args-for [n fn-name & args]
  (is (= args (nth (@call-times fn-name) (dec n)))))

(defn clear-calls []
  (reset! call-times {}))

(defmacro mocking [fn-names & body]
  (let [mocks (map #(list 'mock-fn %) fn-names)]
    `(binding [~@(interleave fn-names mocks)]
       ~@body)))

(defmacro stubbing [stub-forms & body]
  (let [stub-pairs (partition 2 stub-forms)
        fn-names (map first stub-pairs)
        stubs (map #(list 'stub-fn (first %) (last %)) stub-pairs)]
    `(binding [~@(interleave fn-names stubs)]
       ~@body)))

It’s just an hour or so of work, so it’s probably rough, and certainly doesn’t support more complex features of other mocking/stubbing libraries. But I thought the simplicity was enjoyable.

Posted in Uncategorized | Tagged: , , , , , | 2 Comments »

frumiOS – a simple object-system for Clojure

Posted by Amit Rathore on December 10, 2009

I’ve nearly stopped blogging, because all my spare time goes into writing Clojure in Action. But I was a bit bored this weekend, and wrote this little library that can be used to write traditional looking Object-Oriented (TM) code in Clojure.

Why would you do that, when you can use a rifle-oriented programming style instead? Think of it like using the rifle as a club… On the other had, the implementation makes plenty use of closures and macros, so it is probably a rifle-oriented program :-)

The implementation is hosted on github, in a project called frumios. And if you reall want to see it now, click below.

(ns org.rathore.amit.frumios.core)

(declare new-object find-method) 

(defn new-class [class-name parent methods]
  (let [klass ((comp resolve symbol name) class-name)]
    (fn [command & args]
      (cond
	(= :parent command) parent
	(= :name command) klass
	(= :method-names command) (keys methods)
	(= :methods command) methods
	(= :new command) (new-object klass)
	(= :method command)
          (let [[method-name] args]
	    (find-method method-name methods parent))
	:else (throw (RuntimeException. (str "Unknown message: " command)))))))

(def OBJECT (new-class :o rg.rathore.amit.frumios.core/OBJECT nil {}))
(def this)

(defn new-object [klass]
  (let [state (ref {})]
    (fn thiz [command & args]
      (cond
        (= :class command) klass
        (= :set! command) (let [[k v] args]
			    (dosync (alter state assoc k v))
			    nil)
        (= :get command) (let [[key] args]
			   (state key))
        :else (let [method (klass :method command)]
		(if method
		  (binding [this thiz]
		    (apply method args))))))))

(defn find-method [method-name instance-methods parent-class]
  (let [method (instance-methods method-name)]
    (or method
	(if-not (= #'org.rathore.amit.frumios.core/OBJECT parent-class)
	  (find-method method-name (parent-class :methods) (parent-class :parent))))))

(defn parent-class-spec [sexprs]
  (let [extends-spec (filter #(= :extends (first %)) sexprs)
        extends (first extends-spec)]
    (if (empty? extends)
      'org.rathore.amit.frumios.core/OBJECT
      (do
	(if-not (= 1 (count extends-spec))
	  (throw (RuntimeException. "defclass only accepts a single extends clause")))
	(if-not (= 2 (count extends))
	  (throw (RuntimeException. "the extends clause only accepts a single parent class")))
	(last extends)))))

(defn method-spec [sexpr]
  (let [name (keyword (second sexpr))
	remaining (next sexpr)]
    {name (conj remaining 'fn)}))

(defn method-specs [sexprs]
  (let [method-spec? #(= 'method (first %))
	specs (filter method-spec? sexprs)]
    (apply merge (map method-spec sexprs))))

(defmacro defclass [class-name & specs]
  (let [parent-class-symbol (parent-class-spec specs)
        this-class-name (keyword class-name)
	fns (method-specs specs)]
    `(def ~class-name
        (new-class ~this-class-name (var ~parent-class-symbol) ~(or fns {})))))

But first, examples -

(ns frumios-spec)

(use 'org.rathore.amit.frumios.core)

(defclass animal
  (method sound []
    "grr")

  (method say-something []
    (str (this :sound) ", I say!"))

  (method move []
    "going!"))

(defclass cat
  (:extends animal)

  (method sound []
    "meow"))

There, that defines a simple class hierarchy. Let’s examine these classes -

frumios-spec> (cat :parent)
#'frumios-spec/animal

frumios-spec> (animal :parent)
#'org.rathore.amit.frumios.core/OBJECT

frumios-spec> (animal :method-names)
(:move :say-something :sound)

frumios-spec> (cat :method-names)
(:sound)

Now, let’s define a couple of instances -

(def a (animal :new))
(def c (cat :new))

What can we do with these instances? Let’s explore -

frumios-spec> (c :class)
#'frumios-spec/cat

frumios-spec> (c :set! :name "Mr. Muggles")
nil

frumios-spec> (c :get :name)
"Mr. Muggles"

That’s the basic stuff, how about calling methods?

frumios-spec> (a :move)
"going!"

frumios-spec> (a :sound)
"grr"

frumios-spec> (c :sound)
"meow"

Notice how cat overrides the sound method. OK, how about a method that calls another method? It calls for the this keyword. Here it is in action -

frumios-spec> (a :say-something)
"grr, I say!"

frumios-spec> (c :say-something)
"meow, I say!"

Notice how in the second call, (this :sound) resolved itself to the overridden sound method in the cat class. That’s subtype polymorphism, common to languages such as Java and Ruby. We could use it to implement something like the template pattern. We can do fairly arbitrary things with frumiOS -

(defclass person
  (method greet [visitor]
    (println "Hi" visitor ", I'm here!"))

  (method dob []
    (str "I was born on " (this :get :birth-date)))

  (method age []
    2)

  (method experience [years]
    (str years " years"))

  (method bio []
    (let [msg (str (this :dob) ", and have " (this :experience (this :age)) " of experience.")]
      (println msg))))

Let’s play with it -

frumios-spec> (def kyle (person :new))
#'frumios-spec/kyle

frumios-spec> (kyle :greet "rob")
Hi rob , I'm here!
nil

The bio method makes two calls using the this construct, one nested inside the other. It works as expected -

frumios-spec> (kyle :set! :birth-date "1977-01-01")
nil

frumios-spec> (kyle :bio)
I was born on 1977-01-01, and have 2 years of experience.
nil

So there it is. I’m sure it doesn’t do lots of stuff a real object-system does. But at 70 lines of Clojure code, you can’t expect a whole lot more. Silly as this is, I had fun writing it! Click here to see how the frumiOS is implemented.

Posted in Uncategorized | Tagged: , , , , | 2 Comments »

Runa is hiring developers

Posted by Amit Rathore on August 27, 2009

We, Runa (runa.com), are looking for great developers to join our small team. We’re an early stage, pre-series-A startup (presently funded with strategic investments from two large corporations) playing in the e-commerce space. We’re creating a new product in the small-to-medium online-retailing segment, and if we’re successful, it will be a very large disruption.

Techie keywords: clojure, hadoop, hbase, rabbitmq, erlang, ruby, rails, javascript, amazon EC2, unit-testing, functional-testing, selenium, agile, lean, XP

If you’re interested, email me at amit@runa.com

If you want to know more, read on!

What we do

Runa aims to provide small-to-not-so-large online retailers with tools/services that companies like amazon.com use/provide. These smaller guys can’t afford to do anything on that scale, but by using our SaaS services, they can make more money while providing customers with greater value.

The first service we’re building is what we call Dynamic Sale Price.

It’s a simple concept – it allows the online-retailer to offer a sale price for each product on his site, personalized to the individual consumer who is browsing it. By using this service, merchants are able to -

  • increase conversion (get them to buy!) and
  • offer consumers a special price which maximizes the merchant’s profit

This is different from “dumb-discounting” where something is marked-down, and everyone sees the same price. This service is more like airline or hotel pricing which varies from day to day, but much more dynamic and real-time. Further, it is based on broad statistical factors AND individual consumer behavior. After all, if you lower prices enough, consumers will buy. Instead, we dynamically lower prices to a point where statistically, that consumer is most likely to buy.

How we do it

Runa does this by performing statistical analysis and pattern recognition of what consumers are doing on the merchant sites. This includes browsing products on various pages, adding and removing items from carts, and purchasing or abandoning the carts. We track consumers as they browse, and collect vast quantities of this click-stream data. By mining this data and applying algorithms to determine a price point per consumer based on their behavior, we’re able to maximize both conversion (getting the consumer to buy) AND merchant profit.

We also offer the merchant comprehensive reports based on analysis of the mountains of data we collect. Since the data tracks consumer activity down to the individual product SKU level (for each individual consumer), we can provide very rich analytics. This is a tool that merchants need today, but don’t have the resources to build for themselves.

The business model

For reference, it is useful to understand the affiliate marketing space. Small-to-medium merchants (our target audience) pay affiliates up to 40% of a sale price. Yes, 40%. The average is in the 20% range.

We charge our merchants around 10% of sales the Runa delivers. Our merchants are happy to pay it, because it is a performance-based pay, lower than what they pay affiliates, and there is zero up-front cost to the service. In fact, the above mentioned analytics reports are free.

We’re targeting e-commerce PLATFORMS (as opposed to individual merchants); in this way, we’re able to scale up merchant-acquisition. We have 10 early-customer merchants right now, with about 100 more planned to go live in the next 2-3 months. By the end of next year, we’re targeting about 1,000 merchants and 10,000 merchants the following year. Our channel deployment model makes these goals achievable.

At something like a 5% to 10% service charge, and a typical merchant having between 500K to 1M in sales per year, this is a VERY profitable business model. That is, of course, if we’re successful… but we’re seeing very positive signs so far.

And we haven’t even talked about all the other things on our product-roadmap!

Technology

Most of our front-end stuff (like the merchant-dashboard, reports, campaign management) is built with Ruby on Rails. Our merchant integration requires browser-side Javascript magic. All our analytics (batch-processing) and real-time pricing services are written in Clojure. We use RabbitMQ for all our messaging needs. We store data in HBase. We’re deployed on Amazon’s EC2.

We need to be extremely scalable and fast. This is for two reasons -

  • The prices are offered to consumers in real-time, as they browse. There can be no delay.
  • The load on our service grows quickly – each time we sign on a merchant, we’re hit with the traffic from all their customers

It is a very challenging problem, and a lot of fun to solve.

Here are a few blog postings about what we’ve been up to -

We’ve also open-sourced a few of our projects -

  • swarmiji – A distributed computing system to write and run Clojure code in parallel, across CPUs
  • capjure – Clojure persistence for HBase

Culture at Runa

We’re a small team, very passionate about what we do. We’re focused on delivering a ground-breaking, disruptive service that will allow merchants to really change the way they sell online. We work start-up hours, but we’re flexible and laid-back about it. We know that a healthy personal life is important for a good professional life. We work with each other to support it.

We use an agile process with a lot of influences from the Lean and Kanban world. We use Mingle to run our development process. Everything, OK mostly everything :) is covered by automated tests, so we can change things as needed.

We’re all Apple in the office – developers get a MacPro with a nice 30” screen, and a nice 17” MacBook Pro. We deploy on Ubuntu servers. Aeron chairs are cliché, yes; but, very comfy.

The environment is chilled out… you can wear shorts and sandals to work… Very flat organization, very non-bureaucratic… nice open spaces (no cubes!). Lunch is brought in on most days! Beer and snacks are always in the fridge.

We’re walking distance to the San Antonio Caltrain station (biking distance from the Mountain View Caltrain/VTA lightrail station).

What’s in it for you

  • Competitive salaries, and lots of stock-options
  • Cutting edge technology stack
  • Fantastic business opportunity, and early-stage (= great time to join!)
  • Developer #5 – means plenty of influence on foundational architecture and design
  • Smart, fun people to work with
  • Very comfortable, nice office environment

OK!

So, if you’re interested, email me at amit@runa.com

Posted in Uncategorized | Tagged: , , , , , | Leave a Comment »

Clojure, the REPL and test-driven development

Posted by Amit Rathore on July 28, 2009

I’ve been using Clojure for nearly a year now, and something strange has been happening… I still think unit-tesitng is extremely important, but for some reason I don’t seem to be writing the same number of tests any more. I’m ashamed to say it, but there it is. And it gets stranger – this new lower test count doesn’t seem to matter.

It seems to me that my Clojure code works right the first time more often than my Ruby or Java code ever did. And I seem to find less defects in the Clojure code over time, too.

This is not just a fanboy speaking, though I am a huge fan of Clojure. I think that the reasons I’m observing this is due to a an important characteristic of the language. Instead of just talking about it, let me first walk you through an example.

This is something I had to do recently – we wanted to build a kind of reverse index for an HBase table. The row ids of this table are time-stamps. The idea was that this “reverse index” would allow us to answer the question of what the first time-stamp for a given day was. In other words, we needed to convert a list of time-stamps into a lookup of day vs. the first time-stamp of that day. Eg.

Input: 


[“112323123” “1231231231” “123123123” “ 1231231123” ....]

Output:


{“2009-07-01” “123123123”
 “2009-07-02” “123131213”
 “2009-07-03” “123123122”}

(Note: I plucked the numbers out of the air, they aren’t accurate. But the idea is that the input is a long stream of timestamps, and possibly hundreds could correspond to each day.)

So I get started… thinking to myself – I know how to convert a timestamp to a day. From there, it’s easy to write a function that returns a hash containing the day vs. timestamp (Since I already had a function day-for-timestamp, it was easy) -


(defn day-vs-timestamp [time-stamp]
  {(day-for-time-stamp time-stamp) time-stamp})

So now, all I have to do is map the above function across the input. This gives me a list of hash-maps, each with one key-value pair. To ensure that I’m doing this in order of oldest first, I sort the input as well. Inside of a let form, all of this looks like – 


(let [all-pairs (map day-vs-timestamp (sort input-list))]

Now, I have this list of hashes, each with one key (the day) and one corresponding value (the time-stamp itself). I want to combine these into one single hash-map which would be the final answer. But I have to deal with the issue of duplicate keys – when I find a duplicate key, I want to keep the first value associated with the key since it would be the oldest.

Clojure has a merge-with function which does just this – it accepts a function with 2 arguments (which are the two values in case a duplicate key is found) and the returning value is used in the merged hash-map.


(apply merge-with #(first [%1 %2]) all-pairs)

That’s basically it.

Combining everything -


(defn day-vs-timestamp [time-stamp]
  {(day-for-time-stamp time-stamp) time-stamp})

(defn lookup-table [input-timestamps]
  (let [all-pairs (map day-vs-timestamp (sort input-list))]
    (apply merge-with #(first [%1 %2]) all-pairs)))

When I write code like this – I often ask myself, what exactly should I test? I end up writing a few happy path tests that prove my code works. And then a couple of tests that test border cases and negative paths. And I sometimes do it test first.

But the REPL has spoilt me. What I used TDD for when coding with Ruby (and still do), I often do at the REPL. I build tiny functions that work – these are often single lines of code. Then I combine these into other functions, often no more than two lines of code each, sometimes three. And it all just works – leaving me wondering what to cover with tests.

The main reason I still write tests is for regression – if something breaks in the future, I catch it quickly. However, the other thing – the test *driven* design aspect of TDD – has been somewhat replaced by the REPL. And its very much more dynamic than a set of static tests. It really brings out the rapid, in rapid application development – especially when combined with Emacs and SLIME.

One main difference with Clojure vs. Ruby (say) is that Clojure is functional (I use very little of Clojure’s constructs for state). And in the functional world, I just don’t have to worry about state (obviously), and this tremendously simplifies code. I think in terms of map, filter, reduce, some, every, merge, etc. and the actual logic is in tiny functions used from within these other higher level constructs. The idea of first-class functions is also key – I can build up the business logic by writing small functions that do a tiny thing each – and combine them using higher-order functions.

This is one reason why we’re so productive with Clojure. We’ve moved to Clojure for 90% of our work. That said, we still use Ruby for parts of our code-base, and it’s still my favorite imperative language :)

Posted in Uncategorized | Tagged: , , , , , | 8 Comments »

Capjure: a simple HBase persistence layer – updated with documentation

Posted by Amit Rathore on May 25, 2009

This is just to let folks know I’ve (finally) written up some documentation on how to use capjure. Hope it helps!

Posted in webscale | Tagged: , , , , | 1 Comment »

Startup logbook – v0.2 – distributed Clojure system in production

Posted by Amit Rathore on May 2, 2009

This past weekend, we pushed another major release into production. We’ve been working on several things and have made a few pushes since the last time I wrote about this – but this release has a bunch of interesting Clojure related stuff.

Long-running processes

The main thing of note is that the majority of our back-end is now written in Clojure. You might recall that our online-merchant customers send us a lot of data, and we run a ton of analytics on that data. Our initial plans involved Ruby, but as we started using Clojure, it turned out that it is very well suited for this job as well (long running, message-driven processes that crunch numbers).

The raw data sits in HBase, and every night a “master” process starts up which kicks-off the processing of the previous day’s worth of data. The job of this master is only to coordinate the work (it doesn’t actually do any real work), it does this by breaking work into chunks and dispatching messages that each assign work to any worker process that picks it up. The master is single threaded for simplicity, but failure tolerant – it checkpoints everything in a local MySQL database, and if it crashes, it is automatically re-spawned and it recovers from where it left off.

clojure-in-production-v0.2.png

An elastic cloud of worker processes run in anticipation of the master handing out this work. The worker processes use the MySQL database to keep track of their progress as well. The rest is rather domain-specific. We use intermediate representations of the raw data, which is also stored in HBase, before finally storing the summarized version again in HBase.

Swarmiji

We use an in-house distributed-programming framework called Swarmiji to make such distributed programs very easy to write and run. Swarmiji implements a flavor of staged event-driven architecture (SEDA) to allow server processes that exhibit scalable, predictable throughput. This is especially true in the face of over-load, which we can certainly expect in our environment.

The reason I wrote this framework was that I wanted to create distributed, parallel programs which exploited large numbers of machines (like in a data-center) – without being limited by clojure’s in-JVM-threads-based model. So each worker process in Swarmiji gets deployed as a shared-nothing JVM process.

I will write up a post introducing Swarmiji in the next few weeks – once its a bit more battle-tested, and I’ve added a few more features (mainly around process management).

Posted in webscale | Tagged: , , , , , | 3 Comments »

Using messaging for scalability

Posted by Amit Rathore on April 12, 2009

I don’t understand why this whole debate about scalability is so focused on choice of the programming language.

At Runa, we provide a service to our immediate customers who are all online retailers. Their website integrates with our services in a couple of ways (via Javascript inserts and via traditional http calls) – and each time a shopper browses any of these online market-places, our services are called. Lots of times – a dozen times per retailer page-view on average.

Our load also grows in large step-functions. Each time we get a new customer, our services get called by another set of users (as all our customers’ customers get added on). We need our services to keep up with all this demand.

Finally, lets take an example of one of the things we provide via these services – dynamic pricing for products. Obviously, the response of such calls needs to be in real-time – since the price has to be shown next to the product being browsed.

So we have a load as well as a response-time concern – as most typical web-scale services do.

Our approach has been to favor simplicity – very early on we introduced a messaging backbone.

messaging_for_scalability.png

Despite the fact that this picture looks a bit more complex than it would without the RabbitMQ portion, this has allowed us to do a few things -

  • For those service calls that don’t need immediate responses – (for instance, our client websites send us data that we analyze later, or we need to send an email notification) – we just drop these onto an appropriate queue. An asynchronous processor picks up the message, and does the needful.
  • For those services that need responses immediate responses, the call is handled synchronously by one of the application servers.
  • For those services that are heavier in terms of the computation required, we split the request into pieces and have them run on separate machines. A simple federation model is used to coordinate the responses and they’re combined to return the result to the requester.
  • With the above in place, and by ensuring that each partial service handler is completely stateless, we can scale by simply adding more machines. Transparently. The same is true for all the asynchronous processors.

As an aside – I’ve written a mini-framework to help with those last couple of bits. It is called Swarmiji – and once it works (well) in production, I will open-source it. It may be useful for other folks who want a sort of message-passing based parallel programming system in Clojure.

So anyway, with this messaging system in place, we can do a lot of things with individual services. Importantly, we can try different approaches to implementing them – including trying different technologies and even languages.

IMHO, you can’t really have a conversation about scalability without context of how much load you’re talking about. When you’re just getting started with the system, this is moot – you don’t want to optimize early at all – and you can (mostly) ignore the issue altogether.

When you get to the next level – asynchronicity can get you far – and this side-steps the whole discussion of whether the language you’re using is efficient or not. Ruby is as good a choice as any in this scenario – Python or Java or most other languages will leave you in the same order of magnitude of capability. The key here is high-level design, and not doing too many obviously stupid things in the code.

When you do get crazy large (like Google or whatever), then you can start looking at ways to squeeze more out of each box – and here it may be possible that using the right language can be an important issue. I know Google even has a bunch of people working on compilers – purely to squeeze more out of the generated code . When you have tens of thousands of computers in your server farms, a 2% increase is probably worth a lot of money.

Still, this choice of language issue should be treated with caution. I’m personally of the opinion that programmer productivity is more important than raw language efficiency. That is one reason why we’re writing most of our system in a lisp (Clojure) this time around. The other thing is that these days runtimes are not what they used to be – Ruby code (and Python and Clojure and Scala) can all run on the JVM. So you can get the benefits of all those years of R&D basically for free.

Finally, a word on our messaging system. We’re using RabbitMQ – and it is *fast*, reliable and scalable. It truly is the backbone of our system – allowing us to pick and choose technology, language, and approach. It’s also a good way to minimize risk – a sub-system can be replaced with no impact to the rest of the pieces.

Anyway – take all of this with many grains of salt – after all, we’re not the size of Google (or even Twitter) – so what do I know?

Posted in webscale | Tagged: , , , , | Leave a Comment »

Capjure: a simple HBase persistence layer

Posted by Amit Rathore on March 31, 2009

Or how to get some free help putting stuff into HBase. For Clojure programmers.

OK, so maybe this one can even be called simplistic. Still, it works for my needs right now – so I thought it might help others. I wrote about my use-case in a previous post about HBase schema design – and this is the little helper utility I mentioned towards the end of it.

How to use it

Download here. There are two vars that need appropriate bindings for capjure to work – *hbase-master* and *primary-keys-config*

*hbase-master*

This one is obvious. It must be bound to an hbase name-node or a master server. For example, I might bind it to hbase.test-site.net:60000. Watch out for the fact that Amazon’s EC2 instances need lots of configuration to open such ports to the world. This has no real relevance to this post, just thought I’d mention it.

*primary-keys-config*

This one is a bit more involved – and I’m sure I’ve done a bad job of simplifying this usage pattern. Still, lets consider the example from the previous post. When you have an array of several hashes as a value in your JSON object that is being persisted (eg. for the :cars key) -

  :cars => [
    {:make => 'honda', :model => 'fit', :license => 'ah12001'},
    {:make => 'toyota', :model => 'yaris', :license => 'xb34544'}],

it will be converted into 

{
  "cars_model:ah12001" => "fit",
  "cars_make:ah12001" => "honda",
  "cars_model:xb34544" => "yaris",
  "cars_make:xb34544" => "toyota"
}

To make this happen, capjure needs to know what to use as a primary-key. Or something like that :) Here, we have decided upon the :license attribute of each hash. Capjure then removes that property from the child-hashes being saved, and sticks the value into the key part of the flattened data-structure as shown above.

This is accomplished by -

(def encoders (config-keys
  (config-for :cars :license  (fn [car-map]
					       (car-map :license))))

Similarly, other primary-keys can be configured. And because the actual value used is the value returned by the function defined (as above), it can be as complex as needed. For example of the values have spaces, you can encode it using some scheme.

A similar configuration is needed for this process to be reversed during reading out of HBase. The 

(def decoders (config-keys
  (config-for :cars :license  (fn [value]
					       		value)))

In this case, we just use an identity function because the reverse mapping is straight-forward (in other words, we didn’t do anything fancy during the previous flattening operation). What happens is that a key-value pair (key being the one specified (:license)) and the value as whatever is returned by the function is added to the flattened object being re-hydrated.

Similarly, other configuration parameters can be added for other sub-objects that have primary-keys.

Together, the encoders and decoders form the *primary-keys-config*. Thus, if you do the following -

(def keys-config {:encode encoders :decode decoders})

then keys-config should be used as the value that *primary-keys-config* gets bound to.

Methods of interest

Once this is done, objects can be pushed into and out of HBase quite trivially -

(binding [*hbase-master* "hbase.test-site.net:60000" *primary-keys-config* keys-config]
	(capjure-insert some-json-object "hbase_table_name" "some-row-id"))

and -

(binding [*hbase-master* "hbase.test-site.net:60000" *primary-keys-config* keys-config]
	(read-as-hydrated "hbase_table_name" "some-row-id"))

Other convenience methods

Capjure provides other convenience methods like -

row-exists? [hbase-table-name row-id-string]
cell-value-as-string [row column-name]
read-all-versions-as-strings [hbase-table-name row-id-string number-of-versions column-family-as-string]
read-cell [hbase-table-name row-id column-name]
rowcount [hbase-table-name & columns]
delete-all [hbase-table-name & row-ids-as-strings]

and others. Everything is based off (uses it underneath) the HBase client API. Thanks to Dan Larkin for clojure-json.

Limitations

I’m no expert in persistence systems – and I’m sure this one has plenty of issues. The main limitation is that the object that capjure can persist can only be so deep. Specifically, the object should be a hash that contain symbols (or strings) as keys, and the values can either be strings (or other primitives), arrays of such primitives, a hash with one level of key-values, or an array of hashes that are one level deep.

Feedback welcome

Please contact me if you have suggestions and stuff. Again, the code is on github.

Posted in webscale | Tagged: , , , | Leave a Comment »

HBase: On designing schemas for column-oriented data-stores

Posted by Amit Rathore on March 8, 2009

At Runa, we made an early decision to optimize. :)

We decided not go down the scaling a traditional database system (the one in question at the time was MySQL) route and instead to use a column-oriented data-store, that was built for scaling. After a cursory evaluation (which was done by me – and it involved a few hours of browsing the web (pseudo research),  checking email (not research),  and instant-messaging with a couple of buddies (definitely not research) – we picked HBase.

OK, so the real reason was that  we knew a few companies are using it, and that there seemed to be a little bit more of a community around it. In fact, there are a couple of HBase user-groups out here in the bay area. Someone recently asked me about CouchDB, and I’ve noticed a lot more buzz about it now that I’m watching for it… I’ve no reason why we didn’t pick it instead. They’re both Apache projects… maybe we’ll have use of CouchDB someday.

HBase schemas:

So anyway, we picked HBase. Now, having done so, and also having spent all my life using relational databases, I had no idea how to begin using HBase – the very first question sort of stumped me – what should the schema look like?

Here’s what I figured out – and I’m sure people will flame me for this – you don’t really need to design one. You just take your objects, extract their data using something like protocol buffers, yaml/json, or even (ugh!) XML – and then stick those into a single column in an HBase table.

Our simplistic HBase mapper:

We use JSON – but we’re doing something slightly different. Think of our persistable object represented by a fairly shallow JSON object – like so:

:painting => {
  :trees => [ "cedar", "maple", "oak"],
  :houses => 4,
  :cars => [
    {:make => 'honda', :model => 'fit', :license => 'ah12001'},
    {:make => 'toyota', :model => 'yaris', :license => 'xb34544'}],
  :road => {:name => '101N', :speed => 65}
}

OK, bizarre example. Still – the way you would persist this in HBase with the common approach would be to create an HBase table that had a simple schema – a single column family with one column – and you’d store the entire JSON message as text in each row. Maybe you would compress it. The row-id could be something that makes sense in your domain – I’ve seen user-ids, other object-ids, even time-stamps used. We use time-stamps for one of our core tables.

The variation we’re using is instead of storing the whole thing as a ‘blob’, we’re splitting it into columns – so that the tree-like structure of this JSON message is represented as a flat set of key-value pairs. Then, each value can be stored under a column-family:column-name.

The example above would translate to the following flat structure -

{
  "road:name" => "101N",
  "road:speed" => 65,
  "houses:" => 4,
  "trees:cedar" => "cedar",
  "trees:maple" => "maple",
  "trees:oak" => "oak",
  "cars_model:ah12001" => "fit",
  "cars_model:xb34544" => "yaris",
  "cars_make:ah12001" => "honda",
  "cars_make:xb34544" => "toyota"
}

Now it is ready to be inserted into a single row in HBase. The table we’re inserting into has the following column-families -

"road"
"houses"
"trees"
"cars_model"
"cars_make"

This can then be read back and converted into the original object easily enough.

Column-names as data:

Something to note here is that there are now column-family:column-name pairs (that together constitute the ‘column-name’) contain data themselves. An example is ‘cars_model:ah12001′ which is the name of the column, whose value is ‘fit’.

Why do we do this? Because we want the entire object-graph flattened into one row, and this allows us to do that.(what is the primary key?)

The thing to remember here is that in HBase (and others like it), each row can have any number of columns (constrained only by the column-families defined) and rows can populate values for different columns, leaving others blank. Nulls are stored for free. Coupled with the fact that HBase is optimized for millions of columns, this pattern of data-modeling becomes feasible. In fact you could store tons of data in any row in this manner.

Final comments:

If you’re always going to operate on the full object graph, then you don’t really need to split things up this way – you could use one of the options described above (xml, json, or protocol buffers). If different clients of this data typically need only a subset of the object graph (say only the car models or only the speed limits of roads, or some such combination), then with this data-splitting approach, they could only load up the required columns.

This idea of using columns as data take a little getting used to. I’m sure there are better ways of using such powerful data-stores – but this is the approach we’re taking right now, and it seems to be working for us so far.

Clojure HBase helper:

I’ve written some Clojure code that helps this transformation back and forth: hash-tables into/out of HBase. It is open-source – and once I clean it up, I will write about it.

Hope this stuff helps – and if I’ve described something stupid, then please leave a correction. Thanks!

P. S. – I met a Googler today who said BigTable (and by inference HBase) is not a column-oriented database. I think that is incorrect – at least according to wikipedia. I read it on the Internet, I must be right :)

Posted in webscale | Tagged: , , | Leave a Comment »

Clojure: understanding dynamic vars and laziness

Posted by Amit Rathore on February 27, 2009

I spent a frustrating day figuring out why something wasn’t working as expected in a particular situation, even thought it was working just fine elsewhere. It turned out the problem was my flawed understanding of how Clojure’s dynamic vars behaved with respect to laziness. Thanks to Chouser and Chousuke on the #clojure channel in IRC, I think I have a slightly better grasp of the thing.

Update: Here’s a summary for those who don’t want to read the whole thing. What was happening is this – my assumption was that dynamic vars get captured in lazy objects – just like locals get captured in closures. They don’t – so if you later rebind the vars, when the lazy objects are realized, the vars will evaluate according to the latest binding. To baaad results :)

Here’s the chat transcript if you want to read the whole thing – it might help:

[10:24am] erohtar: chouser: u there?

[10:24am] Chouser: yep

[10:24am] erohtar: chouser: so i was thinking about our conversation yesterday about vars

[10:24am] erohtar: chouser: so vars in clojure arent the same as dynamically scoped variables in common lisp?

[10:25am] Chouser: I don’t know CL well enough to say for sure, but my understanding is that they’re pretty much equivalent.

[10:26am] Chouser: a possible difference being how threads interact with them — not sure what CL does there.

[10:26am] erohtar: chouser: the thing is… in common lisp, if i bing var-a to x, any thing down the call stack will see x…

[10:27am] erohtar: chouser: in clojure, it doesnt seem to behave like a closure… (laziness, during realization) is that correct? if var-a was bound to y elsewhere, realization of the old lazy stuff would now see y

[10:28am] erohtar: chouser: is that right?

[10:28am] Chouser: CL doesn’t do lazy seqs, right? so lets ignore that for now

[10:29am] erohtar: chouser: ok…

[10:29am] Chouser: let’s just look at how a closure works.

[10:30am] erohtar: chouser: i think i understand how a closure works, but when a closure is going to be evaluated lazily… what happens to a var that has since been rebound?

[10:31am] Chouser: (def var-a 2), (let [add (binding [var-a 5] (fn [n] (+ var-a n)))] (add 3))

[10:31am] Chouser: ok, here’s a little example that I would expect to work the same if translated to CL

[10:31am] Chouser: var-a has a root binding of 2.

[10:31am] erohtar: chouser: that should eval to 8, yes?

[10:31am] Chouser: no

[10:32am] erohtar: chouser: hmmm

[10:32am] Chouser: within the dynamic scope of the binding, a closure is created that refers to var-a

[10:32am] erohtar: chouser: ok… and that is 5?

[10:33am] Chousuke: nah, it’s var-a

[10:33am] erohtar: oh

[10:33am] Chouser: at the moment the closure is created, var-a has a thread-local binding of 5, but that doesn’t really matter because nothing is asking for var-a’s *value*, just grabbing var-a itself.

[10:33am] erohtar: i see… ok…

[10:34am] Chouser: so that closure is returned to outside the binding, and named ‘add’

[10:34am] Chousuke: I guess it’s a bit surprising though.

[10:34am] Chouser: and what does the closure have bundled inside? it knows it wants one arg n, and it has references to the vars var-a and +

[10:35am] erohtar: and it dynamically uses the value 2, for var-a when it needs it?

[10:35am] Chouser: right

[10:35am] erohtar: arent closures supposed to capture their environment?

[10:35am] Chouser: their lexical environment, yes.

[10:35am] erohtar: not the values?

[10:36am] Chouser: which is why this works: (def var-a 2), (let [m 7, add (binding [var-a 5] (fn [] (+ var-a m)))] (add))

[10:37am] Chouser: hm, not that’s not quite right, since m is still available when ‘add’ is called. But anyway, it wouldn’t have to be.

[10:38am] erohtar: thinking about this

[10:38am] Chouser: if you think about how dynamic vars are used in clojure.core, you’ll see it must be this way.

[10:38am] Chouser: let’s look at *in*

[10:38am] Chouser: er, *out*

[10:38am] Chousuke: ,(let [x 1 add (fn [] (+ 4 x))] [(binding [x 2] (add)) (binding [+ -] (add))])

[10:38am] clojurebot: java.lang.Exception: Unable to resolve var: x in this context

[10:38am] Chousuke: hmm

[10:39am] erohtar: ok – talking about *out* – is that the std-out ?

[10:39am] Chouser: somewhere there’s something like (def *out* System/out)

[10:39am] Chouser: right

[10:39am] Chousuke: ,(let [x 1] (let [add (fn [] (+ 4 x))] [(binding [x 2] (add)) (binding [+ -] (add))]))

[10:39am] erohtar: right

[10:39am] clojurebot: java.lang.Exception: Unable to resolve var: x in this context

[10:39am] Chouser: that’s the root binding of *out*

[10:39am] erohtar: ok, with u so far

[10:39am] Chousuke: hmm, I guess binding let-bound variables does not work

[10:40am] Chouser: then you’ve got functions like prn that use *out*

[10:40am] erohtar: yes… i think they need to be vars

[10:40am] erohtar: right

[10:40am] kotarak: let-bound locals are not Vars

[10:40am] Chousuke: hmm

[10:40am] Chousuke: ,(let [x 1] #’x)

[10:40am] clojurebot: java.lang.Exception: Unable to resolve var: x in this context

[10:41am] Chouser: we can pretend prn is defined something like (defn prn [& args] (.write *out* args))

[10:41am] erohtar: when u create a lazy something, that uses prn, what happens?

[10:41am] Chousuke: indeed.

[10:41am] Chouser: now, if prn took the value of the *out* when it was defined, there would be no way to get prn to write to any other stream — it would always use the root binding, since that’s what was in play when prn was defined.

[10:42am] Chouser: but that’s not how it works.

[10:42am] Chouser: prn has a reference to the Var *out*, not its value.

[10:42am] erohtar: so something like – (map #(prn %)
[1 2 3])

[10:42am] erohtar: chouser: i completely agree

[10:42am] Chouser: so instead, prn will use the dynamic value of *out* when you call it.

[10:42am] erohtar: chouser: hmmm

[10:42am] Chouser: ,(with-out-str (prn 5 10))

[10:42am] clojurebot: “5 10\n”

[10:43am] Chouser: got it?

[10:43am] erohtar: i do

[10:43am] erohtar: im thinking about my confusion – and trying to figure out the question to ask next

[10:44am] erohtar: so lets take my example -

[10:44am] erohtar: (map #(prn %)
[1 2 3])

[10:45am] erohtar: now, somewhere else, i rebind *out* to str… and in there, if i use the above object, will it also use the new binding?

[10:45am] erohtar: it will…

[10:45am] erohtar: i think im getting it now

[10:46am] erohtar: ok – final thought -

[10:46am] Chouser: (def q (map #(prn %)
[1 2 3])), (with-out-str (doall q))

[10:47am] Chouser: so yes, even though you created the lazy seq and closure outside the with-out-str, since its not computed or realized until the doall which is within the with-out-str, all those prn’s will use the binding provided by with-out-str

[10:47am] erohtar: got it…

[10:48am] erohtar: so my final thought – what if instead of *out* we’re talking about a connection to a database… bad use of vars, right?

[10:48am] Chouser: now lazy seqs can be a bit tricky because they cache their results and can effectively contain multiple closures.

[10:49am] Chouser: hmm….

[10:49am] erohtar: if i read from a db a list of objects using a var that holds database config…. and then i rebind the config to write into another database

[10:49am] Chouser: if you’re going to open a connection to a database, do a batch of work in a single thread on it, and then close the connection — then that would be a fine use.

[10:50am] erohtar: since the initial read may be lazy… when it realizes… it will ‘read from the wrong place’”?

[10:50am] Chouser: right

[10:50am] Chouser: basically you want to be very careful about passing closures (and therefore lazy seqs) past ‘binding’ boundaries.

[10:50am] erohtar: chouser: on that thought – what i did was, i read from a db, and to write into the other, i created agents, binding the new db config in each

[10:51am] erohtar: chouser: it didnt work…

[10:51am] erohtar: chouser: ok… im beginning to understand …

[10:51am] erohtar: i need to rethink what im doing

[10:52am] erohtar: and avoid using vars for some of this stuff, since im doing a lot of agent related stuff

[10:52am] Chouser: yes, I’d recommend trying to have the behavior of as much of your code as possible depend only on the args passed directly in.

[10:53am] erohtar: chouser: this whole mess started when i wanted to avoid passing db-config around everywhere

[10:53am] Chouser: this is better for testing, reasoning about behavior, easier to work with threads, closures, laziness, etc.

[10:53am] Chouser: heh, yeah.

[10:53am] erohtar: yup – i hear you

[10:53am] Chouser: well, another thing that may be helpful — just guessing since I haven’t seen your code…

[10:53am] erohtar: thanks a ton for your time – ur incredibly helpful

[10:54am] erohtar: ok?

[10:54am] Chouser: would be to try to keep as much of your code purely functional as possible.

[10:54am] erohtar: yes – i understand

[10:54am] Chouser: so avoid writing a function that takes some args, does some computation, and then writes to a db. Instead, have a fn that takes args, does computation, returns result.

[10:55am] Chouser: then perhaps you don’t have to pass in db config at all — whoever calls this fn can to the db write itself.

[10:55am] erohtar: i think it mostly is… except for this stuff… where i ended up inadvertently depending on “global” db-config

[10:55am] erohtar: well, this is a persistence layer

[10:56am] erohtar: i take an object in, break it up etc., and put it into hbase

[10:56am] erohtar: the code is open-sourece… would u like to see?

[10:56am] Chouser: hm, sure. I may need to ramp up on hbase soon.

[10:56am] erohtar: http://github.com/amitrathore/capjure/blob/b939a7038ebb4aed0d068d6e86291cc881ecf72b/src/org/rathore/amit/capjure.clj

[10:57am] erohtar: its kind a messy – learning clojure and hbase while doing this…

[10:57am] Chouser:

[10:57am] Chousuke: erohtar: first thought: use some newlines.

[10:58am] danlarkin: Noooooooooooo -jure

[10:58am] erohtar: danlarkin: haha – u’ve told me that already

[10:58am] Chouser: danlarkin: hey, at least he’s got a name

[10:58am] Chousuke: erohtar: one newline after each def/defn would make it a lot nicer

[10:58am] danlarkin: erohtar, Chouser:

[10:58am] erohtar: ok – more newlines, check

[10:59am] Chousuke: well, the def group is fine I guess.

[10:59am] Chousuke: but (defn foo) (defn bar) without an empty line in the middle is a bit weird

[11:00am] erohtar: ok – those were sort of just quick utility functions… but i’ll put newlines

[11:00am] Chousuke: newlines never hurt anyone

[11:00am] Chouser: erohtar: this is what you get for posting code. unsolicited advice.

[11:01am] erohtar: haha

[11:01am] erohtar: its fine – maybe i’ll learn something too

[11:01am] Chouser: but while we’re at it — I much prefer to have the code arranged so that ‘declare’ is only needed in mutually-recusive situations.

[11:01am] Chouser: though others may disagree with me

[11:02am] erohtar: right – i struggled with that one – cause i dont like to have to order the functions in “reverse”

[11:02am] • Chouser nods

[11:02am] erohtar: see – capjure-insert is the only one that most people will call

[11:03am] Chousuke: erohtar: you could add more newlines and some kind of “headlines” for sections of code

[11:03am] erohtar: u bind the config stuff – *hbase-master* and *primary-keys-config* – and then u call capjure-insert with the params

[11:03am] Chousuke: so people can easily skip the helper function blocks

[11:03am] erohtar: most of the remaining functions are functional

[11:03am] erohtar: only the ones dealing with hbase have (obvious) side-effects

[11:04am] erohtar: and it works fine… and its in production too… ran into issues trying to use TWO hbase configs (to move data between them)

[11:04am] Chousuke: that’s a lot of functions for one namespace too, though

[11:05am] Chouser: I really don’t think more namespaces would be better.

[11:05am] Chousuke: if possible maybe it’d make sense to put the “side-effecting” functions (that deal with hbase) in a separate file or namespace?

[11:05am] erohtar: cause lazy lists of data from one hbase, seem to be getting messed up when i rebind the config to start writing into the other hbase

[11:06am] erohtar: do u know what i mean? (trying to get discussion back to lazy closures and bindings )

[11:06am] Chousuke: ah

[11:06am] Chousuke: right, I can see why that happens

[11:06am] Chousuke: the laziness gets evaluated in the context of the new hbase.

[11:06am] erohtar: well, im only starting to see it thanks to u guys…

[11:06am] erohtar: yea

[11:06am] erohtar: drove me crazy

[11:06am] erohtar: and i did all this only so i wouldnt need to pass the db config around all the functions

[11:06am] erohtar: like u said, there are a lot of them…

[11:07am] Chousuke: which means you will have to move away from using global vars for config and instead create something you explicitly pass around to the functions.

[11:07am] erohtar: so now what… ? pass the config around everywhere?

[11:08am] Chousuke: maybe only to a few core functions.

[11:08am] erohtar: yea

[11:08am] Chousuke: which then would bind the dynamic vars.

[11:09am] erohtar: well, if i pass it in, i dont need dynamic vars, rite

[11:09am] Chouser: or don’t use lazy seqs

[11:09am] Chouser: use doall or stuff them in a vector first

[11:09am] erohtar: hmmm

[11:09am] cp2 is now known as c|p.

[11:09am] erohtar: that might be one solution

[11:09am] Chousuke: right, if they are side-effecting that might be better.

[11:09am] Chousuke: though maybe not memory-efficient

[11:10am] erohtar: yea

[11:10am] erohtar: i think i will get rid of the dynamic vars

[11:10am] Chousuke: it’s a shame though

[11:10am] erohtar: what is the pattern for this kind of ’static’ config being passed around?

[11:10am] Chousuke: they make the code much easier to write

[11:10am] Chousuke: I wonder if there’s a better solution

[11:11am] Chousuke: some way to “bind” the current config to when the lazy-seq is first created

[11:11am] erohtar: in typical OO, u can create an instance variable to hold the config – i wonder if in clojure u can create something like that… and since its immutable, it should be fine, rite?

[11:11am] Chouser: yes

[11:11am] Chousuke: maybe do (let [config *global*] (return-some-lazy-seq-qith config))?

[11:11am] erohtar: but there is no way to do that, rite?

[11:11am] Chousuke: would that fix the value?

[11:12am] erohtar: yea

[11:12am] Chouser: stuff all your config into a map, and pass that single thing into functions that need to use it

[11:12am] erohtar: i thought of doing something like that… but other functions couldnt access those values…

[11:12am] erohtar: i thought of simulating an ‘immutable object oriented system’ -

[11:12am] brianh joined the chat room.

[11:12am] erohtar: like how u do in javascript… but immutable

[11:13am] Chousuke: I did in clojurebot so that I pass a map of config stuff around with every function

[11:13am] erohtar: return a closure with the vars bound with a let… and then closures for all the other functions also

[11:13am] Chousuke: it’s a bit tedious though

[11:13am] erohtar: i dunno if im being clear

[11:13am] erohtar: very tedious

[11:14am] erohtar: macros can help

[11:14am] erohtar: essentially what id be doing is creating an immutable OO system

[11:14am] erohtar: something like that

[11:14am] erohtar: does that make sense?

[11:14am] Chousuke: erohtar: but your problem is only with lazy seqs right?

[11:14am] erohtar: lazy seqs and dynamic vars

[11:14am] Chousuke: erohtar: don’t you only need to fixate the config when returning a seq

[11:14am] erohtar: the interplay

[11:14am] erohtar: yes

[11:15am] Chouser: erohtar: clojure.zip does something like that

[11:15am] erohtar: chouser: like what?

[11:15am] Chouser: each node is a object with a several functions attached as metadata

[11:16am] erohtar: chouser: ah, i see

[11:16am] erohtar: got it

[11:16am] Chouser: most of the api fns are like (children my-node), where children calls fns from my-node’s metadata, passing in my-node

[11:16am] erohtar: i see -

[11:17am] erohtar: thats rather cool – so u can basically have my-node equivalent contain the config data

[11:17am] Chouser: sure

[11:18am] erohtar: alright folks, thanks a lot for this conversation – im going to experiment and see how to get this done with the least amount of work

[11:18am] Chouser: the only reason zip nodes have fns in metadata is so it can be polymorphic.

[11:18am] erohtar: yes- makes sense

[11:18am] erohtar: (then everyone can tell me how my code sucks )

[11:18am] Chouser: if you’re going to only have one kind of db config (seems likely) then you don’t need the metadata piece.

[11:19am] erohtar: yes – just need to take care of the fact that the config will be rebound… so just closures that capture the config, and the functions that use it

[11:19am] erohtar: something like that

P.S. erohtar is my handle on IRC… it’s the reverse of my last name. In case you were wondering.

Posted in Uncategorized | Tagged: , , , | Leave a Comment »

« Previous Entries