Lean Software Development For Startups

Here’s what I will be talking about at the Lean & Kanban 2009 conference in Miami this year.

Lean Software Development For Startups

(Or Why Agile Isnʼt Enough And How To Do More With Less)

Abstract:

If youʼre in a startup, then you know that statistically, the odds are heavily against you. Pretty much the only inherent characteristic of a startup that can be counted upon to help, is that of its small size. If the company can be nimble and agile, then it can hope to
gain some traction against its larger rivals.

In such an environment, using an Agile methodology is a given. Without some form of a hyper-iterative software process, it is impossible for a startup to create a successful product. Or even to determine what that product is!

In todayʼs climate, exacerbated as it is due to competition, lower capital requirements for software companies, the compression of Internet time, and the recessionary economic conditions, it is no longer enough to just use an Agile method. To stay
competitive, indeed to just survive, something more is required.

Lean Thinking provides just such an advantage.

A startup needs to ground its philosophy in Lean Thinking, Theory of Constraints, Critical Chain, Queueing Theory, Systems Thinking, and the like. It will obviously gain from the long-term focus, throughput-based accounting, and value-based constructs that these provide.

This presentation is about how Lean ideas when applied to standard Agile processes can make an organization super-productive even in the extreme short-term. Specifically, it draws on my experiences from having run multiple projects using this philosophy during my consulting career at ThoughtWorks, and more recently as a founding member of an Internet startup called Cinch.

—-

So, that’s what the talk will be about. I have a 45 minute slot – so I’m looking to finish speaking in about 20-25 minutes and the rest could be for a discussion. Hope to see you there!

What iteration-less agile really means

I’ve written about the idea of an iteration-less process before, and we’ve been using the spirit of the idea at the startup I work at (Cinch) for the past few weeks.

The idea is simple, and has two parts. First, eliminate the concept of formal iterations (use a pull-system to let work flow through the pipeline – use Kanban cards if you like). Second, replace the rhythmic heart-beat of iterations with actual releases to customers (internal or external).

These releases do not have to be spaced out by a week or two or any arbitrary span of time. They ought to be, in fact, the shortest amount of time that the software system can be re-stabilized after undergoing change (through adding features or fixing defects). Hence, if it takes 2 days to do a release, do it then. If, for some reason, it takes 6 days (instead of the old five), then so be it. However, the focus of the team should be on getting quality software into the hands of the final users. After all, everything else is inventory.

Pretty simple, eh? We have combined this with the idea of the estimation-less process, for even greater effect. I have been hearing from some people that I have worked with in the past, and they’ve reported excellent results through just such a process. So, anyway – that’s what iteration-less means. 🙂

P.S.

In a past life I had spoken about this with Jeff Sutherland when I attended Deep Lean (in 2007) – and he agreed that this made sense. He didn’t recommend it for any but the most experienced team – and I certainly agree with that. If your team is new to agile or is lacking in some other way, getting it going with the standard approach to agile is probably the best thing to do.

However – if your team can pull this off, you might want to give it a shot and see how much productivity increases over time.

You don’t need story-points either

Or an estimation-less approach to software development

I’ve had too many conversations (and overheard a few) about re-estimating stories or “re-baselining” the effort required for software projects. The latest one was about when it might be a good idea to do this re-estimation, and how, and what Mike Cohn says about it, and so on. The logic appears to be that as you gain more knowledge about the software being built, you can create better estimates and thereby have better predictability. This post talks about estimates, how they are just another form of muda (waste) and what might be a better way.

First things first

Lets start with fundamentals again. In order to stay profitable and prosperous, a software products company must be competitive. If you do enough root-cause analysis, the only thing that can ultimately ensure this is raw speed. Hence, the only metric that matters is cycle-time and organizations must always try to reduce it.

OK, so now, the issue simplifies itself – all we have to do is look for constraints, remove them, and repeat – all the while measuring cycle-time (and thereby throughput). Simple enough! And in the first few passes of doing so, I first realized that if you must estimate things, use story points. Then I realized that estimation in general is a form of muda – and converting from story-points to real-hours at the start of each iteration is a really stupid waste of time.

Even less muda

Now, having said all that, we can move on to the point of this post. I’ve come to realize there is a better way. One which achieves the goal of faster throughput, with less process and less waste. Eliminate estimation altogether. You don’t even need story points anymore. Here’s how it works –

Take a list of requirements. Ensure they always stay current – you know, by talking with the stakeholders and users and all the other involved parties. Ensure it is always in prioritized order – according to business value. Again, not hard to do – you just need to ensure that the stakeholders are as involved with the product development as the rest of the team is.

Next, pick a duration – say a couple of weeks. If in your business, two weeks is too short (you’re either a monopoly or you’re on your way out) then pick what makes sense. Pick up the first requirement and break it down into stories. Here’s the trick – each story should not be more than 1-3 days of work to implement. In other words, each story, on average, should be about 2 days of development time. Simple enough to do, if the developers on your team are as involved in the process of delivering value to your customers as the business is. Pick the next requirement and do the same thing, keep going until your two weeks seems full. Do a couple more if you like. Ensure that this list stays prioritized.

Then, start developing. There is no need for estimation. Sure, some stories will not really be 2 days worth – break them down when you realize it. Sure, you need good business-analysts and willing developers. You’ve hired the best, right? Next, as business functionality gets implemented to the point where releasing it makes sense, release it! Then, repeat.

What (re) estimation?

In this scenario, there really is no reason to do formal-estimation at all. Sure, super-high-level sizing is probably still needed to get the project off the ground, but beyond that, there is no need to do anything other than deliver quality software, gather real-user feedback, and repeat.

Now come the questions of how to say how much work is done, or when a certain feature will be delivered. The dreaded “how much more work is left” questions. The answer in this new world is simply = 2 days * number of stories in question. (Remember the average story size?) Many people like to track things, but measuring actuals is also muda.

The important thing is of course to not look too far ahead. In today’s business environment, the world changes at the speed of the Internet, so luckily this works to our advantage. Whenever something changes, you just re-prioritize the list of stories. This process is not only much leaner, it is also much simpler.

Let’s come back to the issue of re-estimation. People cite the fact that during the start of a project, there isn’t enough knowledge to truly estimate the high-level requirements. Then, as the project proceeds and they learn more, they seem to want to re-estimate things. I ask, why bother?

The key here is to recognize that “accurate” estimates don’t deliver any business value. It is only important that complex requirements are broken down into incremental stories and they are implemented and released quickly.

The real benefits

The fact that this saves time and effort is actually just a good side-effect. The real value comes from being able to truly stay in control of the development process. Tiny stories allow you to change them when required, pull them from the backlog when needed, or to add new ones whenever business demands it. It also lets you move faster because it’s easier to write code in small incremental chunks, testing is easier, and pushing small releases out to users is easier.

And finally, by imposing the discipline that each story be small, it ensures that people think about the requirement deeply enough before coding begins. It also forces the team to break down requirements into incremental pieces and totally avoids never-ending stories that always have a little bit left to complete.

So there

This allows the whole process to be streamlined. It allows the team to focus on just what the users cares about (working software at their fingertips), and not on all the meaningless artifacts mandated by many methodology books (and indeed unenlightened management). Also, in this manner we handle the issue of estimation and re-estimation. In a zen-line fashion, we do it by not doing it.

Against all Oddities

I just remembered, over this past weekend, that a long time ago I had submitted an article for the ThoughtWorks Anthology. I had actually gotten accepted for the book – with a sort of caveat. Martin Fowler told me that I’d have to improve the essay – to make it actually helpful to people reading it. Great advice, of course!

Except that I’m way too lazy and couldn’t be bothered.

So here it is, in it’s original unedited glory. Against All Oddities by Amit Rathore. I’m also too lazy to convert it into HTML, let Google do that, I say!

Why I don’t use real hours for sprint planning

OK, so several of my friends and colleagues pointed me to Mike Cohn’s recent blog post about why he doesn’t use story points for sprint planning. I left a couple of comments on his blog, but I couldn’t wait until he moderates them… so I thought I’d post my thoughts here.

In my opinion, using either story-points or real-hours for sprint planning makes one assumption. The assumption is that it is somehow very important to get a sprint “just right”. In other words, it is important for a team to estimate a sprint’s worth of work well enough so as to make it fit perfectly within a sprint.

I believe that it’s not important at all. Here’s why.

First of all, the length of a sprint is picked by the team in a mostly arbitrary manner. Some pick 30 days, some pick 1 week. For the most part, it could be any number in between there. So if the duration is not that critical, then why should fitting work into it be? In any case, when a team happens to under-estimate, the undone work gets pushed into a future sprint. And when a team over-estimates, the stakeholders gleefully add more work to the sprint. So… what was the point of all the real-hours-based estimation?

Next, lets talk about predictability. Take an example of a basket-ball team. They can make a general statement about their long-term average, but can they predict what they’ll score in the next game? No… but even more critical, is it even important to try to make this prediction? The answer is not really, it is only important that they do their darned best. A software team can certainly spend time breaking stories down into tasks and then even more time estimating each in real hours, and then bargaining with the stakeholders to make sure whatever they pick all fits in… but it’s just more important that they work on the highest priority cards, with as less distraction and road-blocks as possible.

Whether they fit it all in, into that one sprint or not, is a mere detail.

Now, it’s not to say that having dates is not important. It is, but you can’t be scope-bound AND time-bound. Pick one. My default choice is time-bound with variable-scope. In other words, I like to set a release-date, and then ensure that the team is burning down the highest priority items at all times, as that date approaches. When the date is nigh, release! We’re all writing production-ready software, aren’t we? Then, repeat.

This approach then makes it possible to simply drive the whole project off the product-backlog… without strict sprint-backlogs. It allows velocity to be measured in story-points (from the product-backlog) and prediction is a matter of total-story-points divided by velocity-in-story-points. Simple. If anyone asks what will be done this sprint, the answer is the highest priority stories on the backlog, as many as the team can, here’s our best guess (average-velocity-in-story-points). This approach is actually faster than sitting and re-estimating individual stories in real-hours, because well, the team doesn’t have that distraction anymore.

In fact, from my experience, when the team gets comfortable doing this, they even ask… OK, so if the length of the sprint was set fairly arbitrarily, and as long as we work on the most important stuff at all times, then why are we bothering with the concept of sprints at all? These teams go beyond what I described above and eliminate the waste of iterations. Fun, eh?

If I talk about this stuff to people, sooner or later, someone brings up the happy concept of rhythm – how it is important for the team to have a pulse, a beat to which they work – sprint-planning, development, sprint-close, retrospective, repeat. It gives them satisfaction. It probably does, but I think we can do better. I prefer my teams to not just close an iteration based on some rhythm, but even push the software into production each time. Let the rhythm be set not by an arbitrarily picked number, but through customers getting their hands on the next release of the software. They still do the retrospectives regularly, and have team outings all the time. But the rhythm of the team is set through actual releases. When teams get better and better at this, and realize the critical importance of cycle-time, they do all they can to reduce it.

Anyway, I can talk about this stuff for a long time. Back to sprint-planning. My take is – do it like agile architecture/design. Think of it as evolutionary. We’ve eliminated the big-up-front-design, it is time to eliminate the big-up-front-sprint-plan. Prioritize the back-log constantly, always ensure that the team is working on the highest priority item. Ensure there are as less distractions and road-blocks as possible. And release quickly, as early and as often as possible – make a rhythm out of it.

And if the stake-holder asks when a particular story will be done, see where it stands in the prioritized backlog. The answer to the question is, of course, after all the ones above it are completed. Ask also if they would like to re-prioritize. That’s all there is to it!

How Lean and Agile are different, not that it matters

This post is an attempt to clear the air a little bit around this issue. First off I want to make it clear that, in my opinion, the theoretical distinctions between the two concepts are not important enough for teams to change anything they’re doing to produce good software. In other words, who cares what you call it?

However, it is useful to understand what each concept is. This allows you to have clear conversations about issues and solutions, and it also makes it easier to see what needs to be done at every level to improve the overall throughput of the team or organization (or, indeed, of the entire value-stream.) The way I see it, Agile and Lean are similar in many ways, and they really complement each other the rest of the way.

Agile

Agile is a framework that focuses on iterative-development and feedback to build out a software product. Agile focuses on quality by making things like test-driven-development, continuous integration, and refactoring essential pillars of the process. By definition, agile fosters the creation of an adaptable and evolving team process – it accommodates the fact that each project is different, and processes need to fit the situation. Agile also emphasizes communication and collaboration over documentation – thus, it enables teams to move quicker by resolving things face-to-face. It also considers the customer a part of the team, thereby ensuring maximum feedback reaches the ultimate users, and allowing them in turn to change the course of the work product.

Agile methods usually measure things within the software development world – number of stories (scope), estimates (time), velocity (speed). This in turns drive costs and budgets.

Overall, Agile can be thought of as a framework for the software-construction (planning, development/QA, release) part of creating a product.

Lean

Lean, on the other hand, is a management philosophy. Ultimately, it focuses on throughput (of whatever is being produced) by taking a strictly systems-level view of things. In other words, it doesn’t focus on particular components of the value-stream like code-construction or QA, but on whether all the components of the chain are working as efficiently as possible so as to generate as much overall value as possible. Value, of course, includes things like high-quality, and optimized for time and resources.

Lean is based on several things – queuing theory, the theory of constraints, concurrent engineering (set-based development) and delaying commitment to the last responsible moment. Careful metrics (only the ones that truly measure throughput: this often means going one level up) is an important part of lean – this allows everyone to be objective about everything. Speaking of which, Lean software recommends measuring things across the value chain – and as extended as that chain can be. It recommends measuring return on investment (ROI), customer satisfaction, customer usage patterns, market share, and so on. This in turn drives budgets and costs within the software organization.

Where they’re similar

Both Lean and Agile focus on people – over pretty much everything else. They both focus on inspecting and adapting – in order to improve the work-product and efficiency in producing it. In other words, feedback is critical – from people, from customers, from stake-holders, and from the product itself. They’re both quality focused, and they encourage early discovery and more importantly, prevention of defects.

The area where they complement each other most, is in the breadth of their world-view. Agile is usually focused very much within the software development team or organization, Lean focuses on the entire system that includes as many workers, partners, customers, external stakeholders as possible.

How they fit

Having made these remarks, I’d like to present this picture – it is a way I like to think of the way Agile and Lean fit together. It is sort of a stack – Lean is a foundational framework, Agile sits on top of it. I’ve also taken a shot at depicting some of the building blocks within each.

How Lean and Agile fit together

It doesn’t really matter

I’ve found that after incorporating more and more aspects of Lean into your Agile processes, the distinctions start to blur. In the end, Lean subsumes everything, and ultimately, the only thing that matters is whether the organization delivers what customers want and like, how fast they can deliver it, and how much money the organization and its customers can make together.

Lean Software Development: How to find bottlenecks – Metrics that Matter

In the past, I’ve written about why measuring actuals in the way a lot of people do, isn’t particularly effective. I’ve also written that I think the only way to truly improve the quality of a development team is by reducing its cycle-time. Which begs the question – what metrics ought one to collect in order to reduce cycle-time? This post focuses itself on one set of numbers that can be useful in order to pursue this reduction.

Queue-sizes as symptoms of constraints

Typical software teams have multiple roles – customers, analysts, developers, testers, and so on. To develop the best possible software, these folks should work together in a collaborative way. Despite this mode of interaction, there often are localized work products that each function works on – story cards and narratives, the code that implements the desired functionality, the manual, ad-hoc, and exploratory testing that needs to get done outside of what automation covers.

If we consider the entire team to be a system comprising of the various functional components, then the work of creating working software flows through it in the form of a story card that changes state from requirement-analysis to elaboration to development to testing. This shouldn’t be considered as a waterfall process – this is just how the work flows. An agile team would iterate through these phases very quickly and in a highly collaborative manner.

To be able to determine the health of such a system, which by the way can also be thought of as a set of queues that feed into each other, one needs to know how much work-in-progress is actually inside it at any given moment of time. To tell if the system is running at steady-state, the dashboard of the project team must include number of story-cards in each queue.

Let’s take an example –

queues.png

So the moment we see that the number of cards in a particular area begin to grow in an out-of-proportion manner, we know there is a bottle-neck, or in other words – we have found a constraint. By resolving it – maybe by adding people (always a good idea, right?), or reducing the batch-size (the number of stories picked up in an iteration), or by reducing the arrival rate (stretching out the iteration), or removing other waste that might be the cause (found through root-cause analysis) – one can improve the throughput of the software development team (system).

A Finger Chart

Analyzing carefully collected data is the only way to reliably diagnose a problem in a system – probably something familiar to all developers that were ever tasked with improving the performance of a piece of software. After collecting data, visualizing it helps even more. In the case of trying to discover a bottleneck through the data above, a finger chart can help. It looks like this –

finger_chart.png

The reason it’s called a finger chart is because it is supposed to look like the lines formed between stacked fingers. The width of each color represents the size of the queue. Now, finding the bottleneck is as simple as looking for a widening band. There’s your trouble area, the one you need to optimize.

Systems thinking

This technique may be useful in your team situation. However, it never does to miss the forest for the trees. In many cases, much more bang for the buck can be got by simply looking at an extended value-stream, as opposed to a localized one. In other words, the largest constraint may lie outside the team boundary – for example from having multiple stake-holders that often set conflicting goals for the team. Or that several critical team-members are involved in multiple projects and are therefore spread too thin to be productive on any. It behooves the team members, and indeed the management, to try and resolve issues at all levels by taking a systems approach to improvement.

In any case, despite the fact that queueing theory teaches us (among much else) that optimizing locally at various points ultimately results in lowered throughput of the complete system, it is useful at times to fix what one can. There are certainly several areas where things can be improved, and several ways of handling each. The finger chart above helps in one such area, and by mounting it on a team wall, it becomes an information radiator that lets everyone in on the health of the system.