Capjure: a simple HBase persistence layer

Or how to get some free help putting stuff into HBase. For Clojure programmers.

OK, so maybe this one can even be called simplistic. Still, it works for my needs right now – so I thought it might help others. I wrote about my use-case in a previous post about HBase schema design – and this is the little helper utility I mentioned towards the end of it.

How to use it

Download here. There are two vars that need appropriate bindings for capjure to work – *hbase-master* and *primary-keys-config*

*hbase-master*

This one is obvious. It must be bound to an hbase name-node or a master server. For example, I might bind it to hbase.test-site.net:60000. Watch out for the fact that Amazon’s EC2 instances need lots of configuration to open such ports to the world. This has no real relevance to this post, just thought I’d mention it.

*primary-keys-config*

This one is a bit more involved – and I’m sure I’ve done a bad job of simplifying this usage pattern. Still, lets consider the example from the previous post. When you have an array of several hashes as a value in your JSON object that is being persisted (eg. for the :cars key) -

  :cars => [
    {:make => 'honda', :model => 'fit', :license => 'ah12001'},
    {:make => 'toyota', :model => 'yaris', :license => 'xb34544'}],

it will be converted into

{
  "cars_model:ah12001" => "fit",
  "cars_make:ah12001" => "honda",
  "cars_model:xb34544" => "yaris",
  "cars_make:xb34544" => "toyota"
}

To make this happen, capjure needs to know what to use as a primary-key. Or something like that :) Here, we have decided upon the :license attribute of each hash. Capjure then removes that property from the child-hashes being saved, and sticks the value into the key part of the flattened data-structure as shown above.

This is accomplished by -

(def encoders (config-keys
  (config-for :cars :license  (fn [car-map]
					       (car-map :license))))

Similarly, other primary-keys can be configured. And because the actual value used is the value returned by the function defined (as above), it can be as complex as needed. For example of the values have spaces, you can encode it using some scheme.

A similar configuration is needed for this process to be reversed during reading out of HBase. The

(def decoders (config-keys
  (config-for :cars :license  (fn [value]
					       		value)))

In this case, we just use an identity function because the reverse mapping is straight-forward (in other words, we didn’t do anything fancy during the previous flattening operation). What happens is that a key-value pair (key being the one specified (:license)) and the value as whatever is returned by the function is added to the flattened object being re-hydrated.

Similarly, other configuration parameters can be added for other sub-objects that have primary-keys.

Together, the encoders and decoders form the *primary-keys-config*. Thus, if you do the following -

 
(def keys-config {:encode encoders :decode decoders})

then keys-config should be used as the value that *primary-keys-config* gets bound to.

Methods of interest

Once this is done, objects can be pushed into and out of HBase quite trivially -

 
(binding [*hbase-master* "hbase.test-site.net:60000" *primary-keys-config* keys-config]
	(capjure-insert some-json-object "hbase_table_name" "some-row-id"))

and -

 
(binding [*hbase-master* "hbase.test-site.net:60000" *primary-keys-config* keys-config]
	(read-as-hydrated "hbase_table_name" "some-row-id"))

Other convenience methods

Capjure provides other convenience methods like -

 
row-exists? [hbase-table-name row-id-string]
cell-value-as-string [row column-name]
read-all-versions-as-strings [hbase-table-name row-id-string number-of-versions column-family-as-string]
read-cell [hbase-table-name row-id column-name]
rowcount [hbase-table-name & columns]
delete-all [hbase-table-name & row-ids-as-strings]

and others. Everything is based off (uses it underneath) the HBase client API. Thanks to Dan Larkin for clojure-json.

Limitations

I’m no expert in persistence systems – and I’m sure this one has plenty of issues. The main limitation is that the object that capjure can persist can only be so deep. Specifically, the object should be a hash that contain symbols (or strings) as keys, and the values can either be strings (or other primitives), arrays of such primitives, a hash with one level of key-values, or an array of hashes that are one level deep.

Feedback welcome

Please contact me if you have suggestions and stuff. Again, the code is on github.

About these ads

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s