How Lean and Agile are different, not that it matters

This post is an attempt to clear the air a little bit around this issue. First off I want to make it clear that, in my opinion, the theoretical distinctions between the two concepts are not important enough for teams to change anything they’re doing to produce good software. In other words, who cares what you call it?

However, it is useful to understand what each concept is. This allows you to have clear conversations about issues and solutions, and it also makes it easier to see what needs to be done at every level to improve the overall throughput of the team or organization (or, indeed, of the entire value-stream.) The way I see it, Agile and Lean are similar in many ways, and they really complement each other the rest of the way.

Agile

Agile is a framework that focuses on iterative-development and feedback to build out a software product. Agile focuses on quality by making things like test-driven-development, continuous integration, and refactoring essential pillars of the process. By definition, agile fosters the creation of an adaptable and evolving team process – it accommodates the fact that each project is different, and processes need to fit the situation. Agile also emphasizes communication and collaboration over documentation – thus, it enables teams to move quicker by resolving things face-to-face. It also considers the customer a part of the team, thereby ensuring maximum feedback reaches the ultimate users, and allowing them in turn to change the course of the work product.

Agile methods usually measure things within the software development world – number of stories (scope), estimates (time), velocity (speed). This in turns drive costs and budgets.

Overall, Agile can be thought of as a framework for the software-construction (planning, development/QA, release) part of creating a product.

Lean

Lean, on the other hand, is a management philosophy. Ultimately, it focuses on throughput (of whatever is being produced) by taking a strictly systems-level view of things. In other words, it doesn’t focus on particular components of the value-stream like code-construction or QA, but on whether all the components of the chain are working as efficiently as possible so as to generate as much overall value as possible. Value, of course, includes things like high-quality, and optimized for time and resources.

Lean is based on several things – queuing theory, the theory of constraints, concurrent engineering (set-based development) and delaying commitment to the last responsible moment. Careful metrics (only the ones that truly measure throughput: this often means going one level up) is an important part of lean – this allows everyone to be objective about everything. Speaking of which, Lean software recommends measuring things across the value chain – and as extended as that chain can be. It recommends measuring return on investment (ROI), customer satisfaction, customer usage patterns, market share, and so on. This in turn drives budgets and costs within the software organization.

Where they’re similar

Both Lean and Agile focus on people – over pretty much everything else. They both focus on inspecting and adapting – in order to improve the work-product and efficiency in producing it. In other words, feedback is critical – from people, from customers, from stake-holders, and from the product itself. They’re both quality focused, and they encourage early discovery and more importantly, prevention of defects.

The area where they complement each other most, is in the breadth of their world-view. Agile is usually focused very much within the software development team or organization, Lean focuses on the entire system that includes as many workers, partners, customers, external stakeholders as possible.

How they fit

Having made these remarks, I’d like to present this picture – it is a way I like to think of the way Agile and Lean fit together. It is sort of a stack – Lean is a foundational framework, Agile sits on top of it. I’ve also taken a shot at depicting some of the building blocks within each.

How Lean and Agile fit together

It doesn’t really matter

I’ve found that after incorporating more and more aspects of Lean into your Agile processes, the distinctions start to blur. In the end, Lean subsumes everything, and ultimately, the only thing that matters is whether the organization delivers what customers want and like, how fast they can deliver it, and how much money the organization and its customers can make together.

Lean Software Development: How to find bottlenecks – Metrics that Matter

In the past, I’ve written about why measuring actuals in the way a lot of people do, isn’t particularly effective. I’ve also written that I think the only way to truly improve the quality of a development team is by reducing its cycle-time. Which begs the question – what metrics ought one to collect in order to reduce cycle-time? This post focuses itself on one set of numbers that can be useful in order to pursue this reduction.

Queue-sizes as symptoms of constraints

Typical software teams have multiple roles – customers, analysts, developers, testers, and so on. To develop the best possible software, these folks should work together in a collaborative way. Despite this mode of interaction, there often are localized work products that each function works on – story cards and narratives, the code that implements the desired functionality, the manual, ad-hoc, and exploratory testing that needs to get done outside of what automation covers.

If we consider the entire team to be a system comprising of the various functional components, then the work of creating working software flows through it in the form of a story card that changes state from requirement-analysis to elaboration to development to testing. This shouldn’t be considered as a waterfall process – this is just how the work flows. An agile team would iterate through these phases very quickly and in a highly collaborative manner.

To be able to determine the health of such a system, which by the way can also be thought of as a set of queues that feed into each other, one needs to know how much work-in-progress is actually inside it at any given moment of time. To tell if the system is running at steady-state, the dashboard of the project team must include number of story-cards in each queue.

Let’s take an example –

queues.png

So the moment we see that the number of cards in a particular area begin to grow in an out-of-proportion manner, we know there is a bottle-neck, or in other words – we have found a constraint. By resolving it – maybe by adding people (always a good idea, right?), or reducing the batch-size (the number of stories picked up in an iteration), or by reducing the arrival rate (stretching out the iteration), or removing other waste that might be the cause (found through root-cause analysis) – one can improve the throughput of the software development team (system).

A Finger Chart

Analyzing carefully collected data is the only way to reliably diagnose a problem in a system – probably something familiar to all developers that were ever tasked with improving the performance of a piece of software. After collecting data, visualizing it helps even more. In the case of trying to discover a bottleneck through the data above, a finger chart can help. It looks like this –

finger_chart.png

The reason it’s called a finger chart is because it is supposed to look like the lines formed between stacked fingers. The width of each color represents the size of the queue. Now, finding the bottleneck is as simple as looking for a widening band. There’s your trouble area, the one you need to optimize.

Systems thinking

This technique may be useful in your team situation. However, it never does to miss the forest for the trees. In many cases, much more bang for the buck can be got by simply looking at an extended value-stream, as opposed to a localized one. In other words, the largest constraint may lie outside the team boundary – for example from having multiple stake-holders that often set conflicting goals for the team. Or that several critical team-members are involved in multiple projects and are therefore spread too thin to be productive on any. It behooves the team members, and indeed the management, to try and resolve issues at all levels by taking a systems approach to improvement.

In any case, despite the fact that queueing theory teaches us (among much else) that optimizing locally at various points ultimately results in lowered throughput of the complete system, it is useful at times to fix what one can. There are certainly several areas where things can be improved, and several ways of handling each. The finger chart above helps in one such area, and by mounting it on a team wall, it becomes an information radiator that lets everyone in on the health of the system.

Lean software development: Why reduce cycle-time?

Or why reducing cycle-time is like Aikido

Why, indeed?

To cut a long story short, the answer to why reducing cycle-time matters is that a short cycle-time is the only thing that can truly differentiate between a great development team and a mediocre one. A team that has reduced its cycle-time to say, about a day, has mastered many things in the process. In other words, it takes a lot of things for a team to be able to deliver software in a super-short cycle, but it is those very things that become its biggest advantages.

Let’s talk about some of these – both what it takes, as well as the advantages that these translate to.

Understanding delivery

To be able to even talk about a short cycle-time, it is important to first establish what the starting point is and what finish-line is. The first one is relatively easy. It is simply the point in time when a requirement has been recognized as something that would add value to the software being developed. This recognition might come from a customer who made a request, or it might come from an internal product-management function, or a combination of the two.

The finish line is more fuzzy – at least it is with many traditional software development teams. Some people call a feature done when it moves out of development, some call it done when it moves out of QA, some when it moves out of user-acceptance testing, some when it moves into the backlog of the depoloyment team, and so on. Sometimes, there are multiple levels of done-ness, so to speak – development-done-but-not-integrated, integrated-but-not-tested, done-but-not-verified, done-but-not-user-accepted, all-but-deployed, and so on and so forth.

The first thing that a focus on reducing cycle-time brings about is a clear understanding of what being done means. And here, like in many places, the common-sense answer is the best. Something is done, when a customer can use it. Period.

The aim, then, of reducing cycle-time simply becomes that of getting a feature into the hands of a customer as fast as possible.

Identifying waste

If the goal is to get something into the hands of the customer as fast as possible, then anything that gets in the way of doing that becomes waste. This gives a simple definition to what qualifies as an inefficiency. Cause and effect – the goal of delivering quickly to a customer gives rise to the next goal – removing obstacles from the path of this delivery. In other words – removing waste from the team’s software development process.

If weekly or daily status meetings that last an hour or longer dont seem to be helping anyone move towards this delivery, then in the spirit of removing waste, these meetings should be cancelled. The best way for an integration defect to get fixed usually isn’t filling out a bug-report in the tracking system and having it run through a workflow involving the enterprise-architect, two tech-leads, the QA manager, and the project-manager. Again, to eliminate waste, process needs to be changed. It might just become scheduling time to get the two concerned developers to talk and fix the issue. So on and so forth.

Avoiding rework, upping quality, and root-cause analysis

If a feature has to be worked on again because of defects, it represents a large waste of time and effort. This wasted time doesn’t help in getting software in the hands of the customers, therefore, an effort should be made to reclaim it. The only way to do that would be to – get it right the first time. To do that, one has to ensure that no (or as few as possible) bugs are found during the final testing, instead, everything ought to be caught and fixed earlier.

This means two things – one, that testing needs to be done early in the game and continously through all stages of the game. The second is that each time a non-trivial issue is found – a root-cause analysis is done to ensure that the same kind of thing doesn’t occur again. This is the equivalent of the ‘stop-the-line’ philosophy of lean manufacturing.

Getting better at the difficult stuff

One reason why certain teams don’t truly deliver software frequently is because deployment is hard. They’re not good at it, or it is an arduous process with lots of moving parts etc. These are all just excuses – if something is difficult to do – it shouldn’t be put off and worked-around. Instead, it should be tackled head-on – and be thoroughly analysed for root-causes – and all issues should be fixed so the problem actually goes away.

So basically, time should be taken to look into and simplify the difficult parts of the process, and the team should ultimately become good at those things. There can be no excuses about this – the debt that inevitably piles up from not doing this boldly, simply must be avoided at all costs – else it will kill even the possibility of speedy delivery.

Poly-skilled people

When a team-member is an expert at something, and only that one person is an expert at it, she becomes a single-point of failure. If she takes some time off, getting something that involves her area of expertise into the hands of the customer becomes impossible. What if she should leave the team? What if there is more than one thing in the pipeline at a given point of time that needs the expert?

To be able to truly deliver software to customers in the shortest time possible, such bottlenecks must be resolved. When there is work for two in a certain area and only one ‘expert’, then another team-member should be able to help out. If a QA person becomes unexpectedly unavailable for a period of time, a developer or anyone else ought to be willing and able to step in to fill the gap.

Being poly-skilled in a lean team is extremely important – because the final delivery of software is the culmination of the team’s effort – not of a subset of team-members or of only one or two functions. To be able to perform as efficiently as possible, sometimes, having poly-skilled team-members becomes a life-saver.

A culture of excellence

There are other things a team must be able to do (indeed, has to!) to be able to reduce cycle-time to a minimum. These include taking on just enough work, taking on work in smaller chunks, limiting the size of its queues, using pull scheduling systems, and so forth. It also entails always looking for the next bottleneck so it can be resolved, the next piece of waste that can be trimmed from the process, and the next optimization that would improve throughput of the team.

Each team-member should have a caring attitude for the customer – and also an attitude that asks a lot of questions. One set of questions are directed towards the customers themselves – so as to gain a real understanding of what they’re trying to achieve – so that the best, most creative, and efficient solution can be found. Another set of questions are directed at the team and its processes – there should be no sacred-cows when it comes to trying to improve things. Anything that wants to stay on as part of the team’s process should be able to withstand stringent questioning and devil’s advocacy. By doing all this, things become better for the customer, thereby for the company and the thus for the team as well.

This culture of excellence judges its results by one metric alone – throughput. It realizes that any step taken to improve things needs to be questioned, yes, but if put into practice, its effects are to be measured. Measurements are extremely important to a lean software development team – but it only cares about the one measure that matters – that of delivery to the customer. This is why no optimizations should be made at a local level (say within QA or deployment), at the expense of the global work-product. In other words this culture is always looking to optimize the whole system, and not a part of it. Cycle-time is the easiest and most effective measure of this throughput.

A short cycle-time as a secret-weapon

This matters most to me, on a personal level, because I work for a small company today – and probably always will. If you’re competing with large software corporations with deep pockets that throw large development teams at projects – indeed specifically at ones that related to the product your little start-up is undertaking – then the only advantage you have is that of agility.

This is where the Aikido thing comes in – to succeed in a world dominated (for better or worse) by large companies that usually have wasteful processes, traditional project-management philosophies, old technology stacks, but often lots of cash and people – you have to use their ‘strenghts’ and weaknesses against them.

If your little start-up can turn out a feature in a day or two, what customer will be able to resist? Sure, you still have to get over other obstacles in the path of business success (like building what the customer actually wants) – but at least one thing will be possible. Because of the small size of your team, you’ll be able to adopt all kinds of agile and lean practices quickly – and deliver features into the hands of customers so fast that you could run circles around your larger competitors.

Reducing cycle-time and doing all that it entails is all you have to do.

P.S. If large corporations are able to successfully pull this off as well – then… who knows what innovation is possible?

Lean Software Development and the Theory of Constraints

I’ve been trying to apply lean software development principles on my project for a little over a year now. So, when I recently re-read Mary Poppendieck’s Lean Software Development – An Agile Toolkit, I thoroughly enjoyed it and found I was nodding to myself a lot. I highly recommend reading this book because it explains in very simple terms what this whole lean process thing is about, and how it relates to software development. A knowledge of why Agile processes work and what they’re aimed at would help, but is not essential.

For those who don’t have the patience (or time) to read the full book – here’s my version of a summary –

– Strive to eliminate waste. Waste comes in many forms in the world of software development, all of these is waste – partially done work, extra processes that add no value, extra features that have little impact on business value, task switching because of multiple objectives or other reasons, waiting for something, having to go somewhere (to meet someone or fetch something) to get information, defects, and ‘management’ activities.

– Use a technique called value-stream mapping to understand where the process you’re using delivers value and (more importantly) to determine where it adds none.

– Amplify learning through feedback. This is easily done through short iterations and negotiating scope for each of them.

– Think about set-based development techniques.

– Delay commitment – decide as late as possible. At the same time, think about concurrent development.

– Deliver as fast as possible – this enhances learning, and delivers value at the same time. Use concepts from pull-systems and queuing theory.

– Understand the cost of delay – use a product model. Have an accountant on your team who can help with this, and make trade-off decisions in light of what the model says.

– Empower the team!

– Think about integrity – both conceptual integrity, and perceived integrity.

– Testing – TDD, automated functional testing, continuous integration – is key. Refactoring is key.

– Always optimize at a level higher than the perceived issue. Employ systems thinking.

– Embrace the idea of Customer Collaboration vs. Contract Negotiation.

I’ve found that as I started to apply lean software development methods, our processes started to become more and more natural, and that the team became more and more effective. As expected, the team members themselves began to look for new and innovative ways to optimize their workflow, and my job as a project manager became easier and easier from that perspective.

Speaking of optimization, I recall being introduced to the writings of Eli Goldratt by a really smart project manager I worked with in the past – Matt Gelbwaks. Goldratt writes about the Theory of Constraints – his books The Goal and Critical Chain are a really good start to the whole discipline. He has a really simple algorithm to optimize a system, based on throughput accounting

1. IDENTIFY the system’s constraint(s)
2. Describe how to EXPLOIT the system’s constraint(s)
3. SUBORDINATE everything else to the above decision
4. ELEVATE the system’s constraint(s)
5. If in the above process, a constraint has been broken, GO BACK to step 1 but do not allow inertia to cause a system’s constraint.

Applying the Theory of Constraints to improve my teams’ processes, while using lean software development ideas to drive agile practices has proved absolutely invaluable to me over the past year.

I’d be happy to hear from you if you tried similar things on your project… and about how they worked out…