s-expressions

Amit Rathore blogs about software development

Posts Tagged ‘startup’

Why Datomic?

Posted by Amit Rathore on March 29, 2013

Cross-posted from Zololabs.

Many of you know we’re using Datomic for all our storage needs for Zolodeck. It’s an extremely new database (not even version 1.0 yet), and is not open-source. So why would we want to base our startup on something like it, especially when we have to pay for it? I’ve been asked this question a number of  times, so I figured I’d blog about my reasons:

  • I’m an unabashed fan of Clojure and Rich Hickey
  • I’ve always believed that databases (and the insane number of optimization options) could be simpler
  • We get basically unlimited read scalability (by upping read throughput in Amazon DynamoDB)
  • Automatic built-in caching (no more code to use memcached (makes DB effectively local))
  • Datalog-as-query language (declarative logic programming (and no explicit joins))
  • Datalog is extensible through user-defined functions
  • Full-text search (via Lucene) is built right in
  • Query engine on client-side, so no danger from long-running or computation-heavy queries
  • Immutable data – audits all versions everything automatically
  • “As of” queries and “time-window” queries are possible
  • Minimal schema (think RDF triples (except Datomic tuples also include the notion of time)
  • Supports cardinality out of the box (has-many or has-one)
  • These reference relationships are bi-directional, so you can traverse the relationship graph in either direction
  • Transactions are first-class (can be queried or “subscribed to” (for db-event-driven designs))
  • Transactions can be annotated (with custom meta-data) 
  • Elastic 
  • Write scaling without sharding (hundreds of thousands of facts (tuples) per second)
  • Supports “speculative” transactions that don’t actually persist to datastore
  • Out of the box support for in-memory version (great for unit-testing)
  • All this, and not even v1.0
  • It’s a particularly good fit with Clojure (and with Storm)

This is a long list, but perhaps begins to explain why Datomic is such an amazing step forward. Ping me with questions if you have ‘em! And as far as the last point goes, I’ve talked about our technology choices and how they fit in with each other at the Strange Loop conference last year. Here’s a video of that talk.

Posted in Uncategorized | Tagged: , , , , , | 3 Comments »

The technical priorities of a startup

Posted by Amit Rathore on July 30, 2012

There should be none. I mean, sure, there should be some, but just enough to get over the hump of creating testable hypotheses. The hypotheses, should in turn, be iterations on your product – and they should be in the hands of your users.

There are just so many other things that cause a startup to fail, that optimizing the technology stack is just the wrong place to spend any resources. This doesn’t mean you shouldn’t use promising technologies if they can solve clearly anticipated problems. It does mean that you don’t want to waste resources in premature scaling. This, of course, applies to all aspects of your startup – tech, sales, support, etc. 

As far as tech is concerned, pick the best tools for the job, and then move forward quickly. Don’t worry about being perfect (in fact, don’t be). Get traction first – no one cares how amazing your backend is.

Posted in Uncategorized | Tagged: , , | Leave a Comment »

The tech stack of the startup

Posted by Amit Rathore on July 28, 2012

This post is about startups and technology. Of course, nothing will help you if you don’t have a market, or traction, or a business model, or an actual product, or good people. Those are conversations for another day, but now, here’s what I’d use if I’m starting a startup today:

Programming languages:

Data stores:

Data processing:

Machine learning:

Messaging:

Version control:

Builds:

Cloud:

Functional testing:

Project management:

Am I missing stuff?

Posted in Uncategorized | Tagged: , , | 2 Comments »

Runa is hiring developers

Posted by Amit Rathore on August 27, 2009

We, Runa (runa.com), are looking for great developers to join our small team. We’re an early stage, pre-series-A startup (presently funded with strategic investments from two large corporations) playing in the e-commerce space. We’re creating a new product in the small-to-medium online-retailing segment, and if we’re successful, it will be a very large disruption.

Techie keywords: clojure, hadoop, hbase, rabbitmq, erlang, ruby, rails, javascript, amazon EC2, unit-testing, functional-testing, selenium, agile, lean, XP

If you’re interested, email me at amit@runa.com

If you want to know more, read on!

What we do

Runa aims to provide small-to-not-so-large online retailers with tools/services that companies like amazon.com use/provide. These smaller guys can’t afford to do anything on that scale, but by using our SaaS services, they can make more money while providing customers with greater value.

The first service we’re building is what we call Dynamic Sale Price.

It’s a simple concept – it allows the online-retailer to offer a sale price for each product on his site, personalized to the individual consumer who is browsing it. By using this service, merchants are able to -

  • increase conversion (get them to buy!) and
  • offer consumers a special price which maximizes the merchant’s profit

This is different from “dumb-discounting” where something is marked-down, and everyone sees the same price. This service is more like airline or hotel pricing which varies from day to day, but much more dynamic and real-time. Further, it is based on broad statistical factors AND individual consumer behavior. After all, if you lower prices enough, consumers will buy. Instead, we dynamically lower prices to a point where statistically, that consumer is most likely to buy.

How we do it

Runa does this by performing statistical analysis and pattern recognition of what consumers are doing on the merchant sites. This includes browsing products on various pages, adding and removing items from carts, and purchasing or abandoning the carts. We track consumers as they browse, and collect vast quantities of this click-stream data. By mining this data and applying algorithms to determine a price point per consumer based on their behavior, we’re able to maximize both conversion (getting the consumer to buy) AND merchant profit.

We also offer the merchant comprehensive reports based on analysis of the mountains of data we collect. Since the data tracks consumer activity down to the individual product SKU level (for each individual consumer), we can provide very rich analytics. This is a tool that merchants need today, but don’t have the resources to build for themselves.

The business model

For reference, it is useful to understand the affiliate marketing space. Small-to-medium merchants (our target audience) pay affiliates up to 40% of a sale price. Yes, 40%. The average is in the 20% range.

We charge our merchants around 10% of sales the Runa delivers. Our merchants are happy to pay it, because it is a performance-based pay, lower than what they pay affiliates, and there is zero up-front cost to the service. In fact, the above mentioned analytics reports are free.

We’re targeting e-commerce PLATFORMS (as opposed to individual merchants); in this way, we’re able to scale up merchant-acquisition. We have 10 early-customer merchants right now, with about 100 more planned to go live in the next 2-3 months. By the end of next year, we’re targeting about 1,000 merchants and 10,000 merchants the following year. Our channel deployment model makes these goals achievable.

At something like a 5% to 10% service charge, and a typical merchant having between 500K to 1M in sales per year, this is a VERY profitable business model. That is, of course, if we’re successful… but we’re seeing very positive signs so far.

And we haven’t even talked about all the other things on our product-roadmap!

Technology

Most of our front-end stuff (like the merchant-dashboard, reports, campaign management) is built with Ruby on Rails. Our merchant integration requires browser-side Javascript magic. All our analytics (batch-processing) and real-time pricing services are written in Clojure. We use RabbitMQ for all our messaging needs. We store data in HBase. We’re deployed on Amazon’s EC2.

We need to be extremely scalable and fast. This is for two reasons -

  • The prices are offered to consumers in real-time, as they browse. There can be no delay.
  • The load on our service grows quickly – each time we sign on a merchant, we’re hit with the traffic from all their customers

It is a very challenging problem, and a lot of fun to solve.

Here are a few blog postings about what we’ve been up to -

We’ve also open-sourced a few of our projects -

  • swarmiji – A distributed computing system to write and run Clojure code in parallel, across CPUs
  • capjure – Clojure persistence for HBase

Culture at Runa

We’re a small team, very passionate about what we do. We’re focused on delivering a ground-breaking, disruptive service that will allow merchants to really change the way they sell online. We work start-up hours, but we’re flexible and laid-back about it. We know that a healthy personal life is important for a good professional life. We work with each other to support it.

We use an agile process with a lot of influences from the Lean and Kanban world. We use Mingle to run our development process. Everything, OK mostly everything :) is covered by automated tests, so we can change things as needed.

We’re all Apple in the office – developers get a MacPro with a nice 30” screen, and a nice 17” MacBook Pro. We deploy on Ubuntu servers. Aeron chairs are cliché, yes; but, very comfy.

The environment is chilled out… you can wear shorts and sandals to work… Very flat organization, very non-bureaucratic… nice open spaces (no cubes!). Lunch is brought in on most days! Beer and snacks are always in the fridge.

We’re walking distance to the San Antonio Caltrain station (biking distance from the Mountain View Caltrain/VTA lightrail station).

What’s in it for you

  • Competitive salaries, and lots of stock-options
  • Cutting edge technology stack
  • Fantastic business opportunity, and early-stage (= great time to join!)
  • Developer #5 – means plenty of influence on foundational architecture and design
  • Smart, fun people to work with
  • Very comfortable, nice office environment

OK!

So, if you’re interested, email me at amit@runa.com

Posted in Uncategorized | Tagged: , , , , , | Leave a Comment »

Startup logbook – v0.2 – distributed Clojure system in production

Posted by Amit Rathore on May 2, 2009

This past weekend, we pushed another major release into production. We’ve been working on several things and have made a few pushes since the last time I wrote about this – but this release has a bunch of interesting Clojure related stuff.

Long-running processes

The main thing of note is that the majority of our back-end is now written in Clojure. You might recall that our online-merchant customers send us a lot of data, and we run a ton of analytics on that data. Our initial plans involved Ruby, but as we started using Clojure, it turned out that it is very well suited for this job as well (long running, message-driven processes that crunch numbers).

The raw data sits in HBase, and every night a “master” process starts up which kicks-off the processing of the previous day’s worth of data. The job of this master is only to coordinate the work (it doesn’t actually do any real work), it does this by breaking work into chunks and dispatching messages that each assign work to any worker process that picks it up. The master is single threaded for simplicity, but failure tolerant – it checkpoints everything in a local MySQL database, and if it crashes, it is automatically re-spawned and it recovers from where it left off.

clojure-in-production-v0.2.png

An elastic cloud of worker processes run in anticipation of the master handing out this work. The worker processes use the MySQL database to keep track of their progress as well. The rest is rather domain-specific. We use intermediate representations of the raw data, which is also stored in HBase, before finally storing the summarized version again in HBase.

Swarmiji

We use an in-house distributed-programming framework called Swarmiji to make such distributed programs very easy to write and run. Swarmiji implements a flavor of staged event-driven architecture (SEDA) to allow server processes that exhibit scalable, predictable throughput. This is especially true in the face of over-load, which we can certainly expect in our environment.

The reason I wrote this framework was that I wanted to create distributed, parallel programs which exploited large numbers of machines (like in a data-center) – without being limited by clojure’s in-JVM-threads-based model. So each worker process in Swarmiji gets deployed as a shared-nothing JVM process.

I will write up a post introducing Swarmiji in the next few weeks – once its a bit more battle-tested, and I’ve added a few more features (mainly around process management).

Posted in Uncategorized | Tagged: , , , , , | 3 Comments »

Startup logbook – v0.1 – Clojure in production

Posted by Amit Rathore on January 28, 2009

Late last night, around 3 AM to be exact, we made one of our usual releases to production (unfortunately, we’re still in semi-stealth mode, so I’m not talking a lot about our product yet – I will in a few weeks). There was nothing particularly remarkable about the release from a business point of view. It had a couple of enhancements to functionality, and a couple of bug-fixes.

What was rather interesting, at least to the geek in me, was that it was something of a technology milestone for us. It was the first production release of a new architecture that I’d been working on over the past 2-3 weeks. Our service contains several pieces, and until this release we had a rather traditional architecture – we were using ruby on rails for all our back-end logic (eg. real-time analytics and pricing calculations) as well as the user-facing website. And we were using mysql for our storage requirements.

This release has seen our production system undergo some changes. Going forward, the rails portion will continue to do what it was originally designed for – supporting a user-facing web-UI. The run-time service is now a combination of rails and a cluster of clojure processes.

When data needs to be collected (for further analytics down the line), the rails application simply drops JSON messages on a queue (we’re using the excellent erlang-based RabbitMQ), and one of a cluster of clojure processes picks it up, processes it, and stores it in an HBase store. Since each message can result in several actions that need to be performed (and these are mostly independent), clojure’s safe concurrency helps a lot. And since its a lisp, the code is just so much shorter than equivalent ruby could ever be.

Currently, all business rules, analytics, and pricing calculations are still being handled by the ruby/rails code. Over the next few releases we’re looking to move away from this – to instead let the clojure processes do most of the heavy lifting.

We’re hoping we can continue to do this in a highly incremental fashion, as the risk of trying to get this perfect the first time is too high. We absolutely need to get the feedback that only production can give us – so we’re more sure that we’re building the thing right.

The last few days have been the most fun I’ve had in any job so far. Besides learning clojure, and hadoop/ hbase pretty much at the same time (and getting paid for doing that!), it has also been a great opportunity to do this as incrementally as possible. I strongly believe in set-based engineering methods, and this is the approach I took with this as well – currently, we haven’t turned off the ruby/rails/mysql system – it is doing essentially the same thing that the new clojure/hbase system is doing. We’re looking to build the rest of the system out (incrementally), ensure it works (and repeat until it does) – before turning off the (nearly legacy) ruby system.

I’ll keep posting as I progress on this front. Overall, we’re very excited at the possibilities that using clojure represents – and hey, if it turns out to be a mistake – we’ll throw it out instead.

Posted in Uncategorized | Tagged: , , , , , , | 5 Comments »

Design for throwability

Posted by Amit Rathore on January 16, 2009

Build one to throw away, you will anyhow. — Fred Brooks

Fred Brooks said what he did many, many years ago but it is probably just as true today. How many times have you and your team gotten a few months into a project only to realize all the design mistakes you made? Ask any engineer, and they’ll tell you they would build it right the second time.

This is just reality, the nature of discovering the complexity of the domain or the technology or the usage pattern or whatever else you didn’t know about when you started.

On the other hand, there’s this [what Joel Spolsky says about rewriting software] -

It is the single worst strategic mistake that any software company can make — Joel Spolsky

So… what gives? The answer, IMHO, is basically two things -

1. Understand and internalize the idea of the strangler application

2. Architect your system in such a way to support strangling it later

In essence, this means that because it would be a bad idea to rewrite the entire system from scratch, it must be built in a way so as to enable swapping out components of it as they are rewritten (or perhaps heavily refactored).

The architecture must draw from an approach called  concurrent set-based engineering (CSBE) – and indeed, sometimes each logical component would have more than one implementation. At Runa, two components of our system actually have two implementations each. And in each case, they’re both running in production – in parallel.

The way we accomplish this is through very loose-coupling. Additionally, because we take a very release-driven approach to our software process, our architecture evolves according to our current needs… and we refactor and extend things as new requirements are prioritized. At all times, despite our super-short release-cycles, our goal is to always have a version of the system in production. Whenever our pipeline tells us that a peice of the existing design may not work in the long term, we start to work on the replacements – more than one, and in parallel.

We run them in what I’ve been calling shadow-mode. This implies that its not quite part of the official system, but is running in order to prove some design hypothesis. Once everyone involved is satisfied with the results, we pick the most suitable sub-system and decommission  the other contenders (including the old one). At Runa, we achieve much of our inter-component loose-coupling via messaging (our current choice is RabbitMQ).

To summarize – we design everything  with one over-arching goal in mind  – the thing will be thrown away someday, and be replaced with another. As I said before, this enforces a few things -

1. Loose coupling

2. Clear interfaces between components

3. Good automated system testing!

About that last point – because we have many moving parts, functional testing becomes even more important. We currently use Selenium for true functional testing (Runa is a web-based service) – and a variety of other home-grown tools for custom systems testing. Not only do automated system tests tell us that the collaborating set of components are working right, but they also allow us to change things with impunity – knowing that we’ll know if things break.

This thinking is what I’ve been jokingly calling Design For Throwability – and it’s been working rather nicely. It’s essentially a design philosophy that embraces CSBE – and is especially useful for small startups where everything is changing quickly – almost by definition.

Posted in Uncategorized | Tagged: , , , | Leave a Comment »

Software development agility – bottom-up design and domain-specific languages (DSLs)

Posted by Amit Rathore on November 10, 2008

It’s been nearly three months that I’ve been working at Runa. And in that time I’ve worked with the team to get a lean and agile process in place, and while we’re not there yet, things are getting more and more focused every day.

I’ve also been working to significantly re-factor the code that existed, as we get ready to get the first beta release out the door. In this post, I’d like to talk about one aspect of our design philosophy which I’m glad to say has been working out really well – bottom-up design (and an associated DSL).

Before we decided to change the system, it had been designed in the usual top-down manner. Runa operates in the online retail space – we provide web-based services to online merchants, and while we have a grand vision for where we want our platform to go, our first offering is a promotions service. It allows merchants to run campaigns based on a variety of criteria. Merchants are a fussy lot, and they like to control every aspect of campaigns – they want to be able to tweak all kinds of things about it.

In true agile fashion, our initial release allows them to select only a few parameters. Our system, however, needs to be extensible, so that as we learn more about what our merchants need, we can implement these things and give it to them. Quickly. And all of this needs to be done in a changing environment with lots of back-and-forth between the business team and the dev team.

So here’s what we came up with – it is an example of what we call a campaign template -

campaign_template :recapture_campaign do
  title is 'Recapture'
  subtitle is 'Reduce cart abandonment rate'
  description is 'This campaign presents offers to shoppers as they abandon their shopping cart.'

  #adding criteria here
  accept required time_period_criteria with defaults start_date('1/1/08'), end_date('12/31/10')
  accept required product_criteria with no defaults

  hardcode visit_count_criteria with number('1')

  #more criteria
  reject optional url_referrer_criteria with no defaults

  inside context :view_badge do
    never applicable
  end

  inside context :abandon_cart do
    allow only customer_type_criteria with customer_type('visitor')
  end

  inside context :cart do
    allow only user_action_criteria with user_action('accepted_recapture')
  end

end

A campaign template behaves like a blue-print of an actual campaign. Sort of like the relationship between a class and an object. In a sense, this is a higher-order description of just such a relationship. The (computer) language now lets us speak in the domain of the business.

There are a couple of reasons why our core business logic is written like this -

a) It lets us communicate easily with the business. Whenever a question about a rule comes up, I bring up the associated template up on the screen, and make them read the ‘code’. Once they agree thats what they mean, it just works, because it is real code.

b) Since this is in essence a way to script the domain model, it has forced a certain design upon it. All the objects evolved in a bottom-up manner, and each does a very specific thing. It lends to a very highly de-coupled design where objects collaborate together to achieve the higher goal, but each is very independent of the other.

c) This notation makes several things easier. One, the actual business rules are described here, and they just work. The other thing is that we’re able to use this same representation for other things – for example, our merchant GUI is completely auto-generated off these template files. Menu items, navigation, saving, editing, error-reporting, everything is generated.

This allows very fast turn around time for implementing new concepts, or making changes to existing ones.

It’s an internal DSL written in Ruby, and does whatever it can without any extra parsing, as you can probably imagine. I will write about the specifics of how this is implemented in future posts. For the moment, I would like to stress, however, the importance of the bottom-up approach. Because our domain model is made up of many small objects (instead of a few larger ones), each representing a tiny aspect of the domain, we’re able to put them together and fashion more complex beasts out of them. And the combinations are limitless, bounded only by business rules. This is where the power of this approach lies.

The DSL itself is almost only a convenience!

Posted in Uncategorized | Tagged: , , , , , , , , | Leave a Comment »

First week at Runa

Posted by Amit Rathore on August 24, 2008

During my last week at ThoughtWorks, my colleagues were asking me what I would be doing at my new job, and what technologies I might be working on.

I told them that the job involved Rails for the consumer-focused piece, and a combination of other technologies at the backend – Ruby, Java (along with Hadoop), Erlang, and possibly others depending on the task at hand. I wasn’t sure if they were impressed or not, but they did joke about how I would actually end up working on PHP and ColdFusion. I indignantly denied any such thing.

So of course, my first week at the new job saw me working on PHP and ColdFusion.
:|

Basically, since a part of Runa is a set of services for merchants, the consumers of these services could be using PHP, ColdFusion, or any other platform. Hence the need for us to ensure our code works with all of them, and indeed, for us to develop libraries for every major e-commerce platform out there. Anyway, that’s what I worked on through Week 1.

Heh, just thought this was funny!

Posted in Uncategorized | Tagged: , , | Leave a Comment »

Startup School 2008

Posted by Amit Rathore on April 19, 2008

I attended this year’s Startup School – and all eight of the talks were really awesome. I got to hang out with a ton of people who had either already started their companies, or people that were looking to do so. The energy and the buzz was fantastic. Most importantly, however, I got to see three of my heroes -

paul_graham.jpg

Paul Graham – talking about the idea of being a benevolent startup.

jeff_bezos.jpg

Jeff Bezos – talking about Amazon Web Services as a way forward for startups.

peter_norvig.jpg

Peter Norvig – talking about extracting data from the web, and leveraging it in startups.

Posted in Uncategorized | Tagged: , , | Leave a Comment »

 
Follow

Get every new post delivered to your Inbox.

Join 1,585 other followers