Adopting lean software development: What is a user story?

I recently got an email from a reader (Dmitry Lobanov) who had some questions about the process stuff I’ve written about here. With his permission, I’ve reproduced the contents of his email and have responded to his queries. It may help other folks that are in early stages of adopting lean/agile methods –

Hello Amit. You keep interesting blog with a lot of useful information about project management. To my opinion the most attractive idea in your blog is user stories decoupling.

Let me tell how I found it interesting and then I’ll ask a few questions, Ok?

In our project we use scrum with some XP techniques. I found link to your blog in an overview of Mingle at Thoughtworks website. While reading I found your idea of decoupling stories into 2 story points tasks very interesting. And I think, that it makes sense. We tried to compare two approaches (yours and card poker) while estimating amount of time required for 1 specific user story. Well, it is not 100% your approach in fact, because we still tried to use estimation and ideal days, but we took your idea concerning 2 story points tasks. But we are not ready to forget about estimation yet 🙂

First we estimated using card poker, and then repeated estimation using your approach. And we found the following advantages of your approach:
– your approach gave us complicated tree of concrete task. That means that we could just take them and start working on this user story, we know what and how should be done. In other words we have got a plan! Card poker didn’t force us to decouple user story into atomic tasks and just gave us list of generalized tasks (which themselves can be user stories).
– card poker estimation gave us some number of story points (we used ideal days according to scrum guidelines), let it be X ideal days. And then we used your approach for estimation: number of atomic tasks multiplied by 2 story points (we still use ideal days 🙂 ), let it be Y ideal days. And we have been amazed, that Y was 10 times greater than X! 10 times! That’s too much.

This experiment shows us that standard estimation lacks of accuracy a lot… well, we suspected it, but only after experiment we saw it clearly. And using card poker (i.e. bad decoupling) we constantly have problems with developers got stuck in development process, and as a result they do 5-days task for 2 weeks or even more. We have one user story, which had been estimated as 8 days, but two developers work on it for 2 sprints already. It’s no good at all.
So in the near sprint we’ll try decoupling user stories into 2 story points tasks and watch the result.

I have a couple of questions about decoupling and sprint planning, and it would be nice if you provide your point of view concerning them.

In your post about requirements management (http://epistemologic.com/2007/04/08/requirements-management-user-stories-mind-maps-and-story-trees/) you touched only initial planing. But what to do with bugs and technical tasks? I’ll explain. Look, our project has been in development for 2 years, we develop payment acceptance system. And our system has been in use for a bit less than 2 years (it took approximately 3 moths to issue first release). So it changes constantly, we issue new release every 2-3 weeks. But when the development started, team suffers a lot from a lack of project management experience. I came to project few months ago (I’m not a project manager, just developer, who wants to improve project). Now situation seems to improve. But we have a LOT of bugs. As you understand, these bugs can’t be attached to specific user story, moreover we hadn’t user stories before, in fact we introduced scrum only 2-3 moths ago.

For example we need to fix one bug (one of many) concerning working with hardware. And we clearly understand, that we need to refactor part of our system, which works with hardware. It could take a lot of time, we understand that we need to add hardware manager, change hardware interfaces and so fouth in order to unify working with hardware. And it should be done, because if we just fix bug not changing anything, then literally tomorrow we’ll have to fix similar bug, and then another one, e.t.c. How this task should be registered? It is not a user story, because product owner (or stake-holders) wouldn’t see the result, they wouldn’t be able to “touch” it, it can’t be demonstrated to them. And it is not a defect for particular user story, because it is the behaivour of legacy code, we haven’t corresponding user story. What should we do?

I see only one solution. We should wait till stakeholders give us user story concerned this functionality. For example, stakeholders want us to add support for a new type of device. Hence we can include refactoring task to a bunch of tasks under this user story.

But our system has been in development for a long time, and we added support for a lot of devices already. So it can take a lot of time till suitable user story will appear. And we should just wait and fix bugs?

And how to register bugs if we don’t have corresponding user story?

Another one situation we faced yesterday. If we have user story, which involves a lot of changes in architecture. That is user story contains a lot of task, but these task can’t be accomplished separately, and they are not user stories themselves, because they are technical tasks (refactoring and researches). It’s clear that we can’t show them to stakeholders, but we can’t accomplish user story not accomplishing these tasks. And we think, that we won’t be able to accomplish all these tasks in one sprint. And I don’t know how we can decouple 1 user story intro several user stories. What should be done?

Correct me please if I am wrong in my interpretation of user stories. I think that user story is “something” useful to stakeholder. Stakeholders can “touch” user story, they can take a look at it, they can play with new functionality and so fourth. Hence anything that can’t be demonstrated to stakeholders is not a user story, is it so? Stakeholders are not interested in tasks, they want to know only 2 things: What and When, they don’t want to know How.

And I have another one question concerning terminology. What does “epic story” mean in terms of user stories? How does it compare to user stories?

Looking forward to your reply.

Best regards,
Dmitry Lobanov

OK – so here’s my take on the questions raised:

User stories: I’ve had to deal with the question of what a user story is quite a few times over the years. There are many theoretical definitions – and I don’t care about most of them. In my opinion, the only thing that matters is this – a user-story should add business value. That raises the obvious – shouldn’t everything you do while working on a project satisfy this criteria? Yes, it should – and this is usually something that business stake-holders understand quite well.

There are several kinds of stories. The first is the obvious “feature” stories that have a GUI etc. This is the category of stories that can be “touched” and is “tangible”. Justifying this one is easy – indeed, these are the ones most commonly requested by the business.

Then there are the “technical” stories that don’t have a UI – e.g. – “store the uploaded data in a compressed format because the space used up by the system right now is costing us too much money”. Again, the value is obvious.

The next category is bugs – fixing these delivers clear value and are no trouble to manage as such. See below for more information.

One other type of story is the spike. These are easy to deal with since it is obvious what value they deliver – namely an experiment to determine if an approach might work.

The final variation is stories that deal with paying down technical debt. These are harder to justify because the business is usually not technical enough to understand these. There is no easy answer to how to sell these – but there is usually a very tangible benefit that can be attained by playing these cards. Usually this benefit is deferred – allowing the team to build new features faster, or reduce the number of bugs being discovered in a subsystem. When put in these terms the business is usually quick to understand – and then it boils down to a question of ROI. The cost of implementing these cards should be less than the savings they represent. It is important for everyone on the team to see the whole – and make these decisions together. The key is to sell the business on the value these stories delivers as opposed to the technical details. And by the way, this should be done whenever the debt starts to weigh heavier than what it should. A clean code-base is a happy code-base.

How to deal with bugs: The short answer is I usually treat bugs as stories. They don’t need to be associated with a specific user story. A bug is a bug – it doesn’t matter how it came to be – (of course, it is often related to a story – and most often the same developer ends up fixing such bugs) – and I track them in the same system used for stories. I also let the same prioritization process determine which ones get fixed and when. Further, some bugs are so tiny that they may even get fixed without there being any record of them at all. Some are so large that they might actually result in multiple stories – especially if a redesign is required. Keeping in mind that it is cheaper to fix bugs sooner than later, bugs can be managed just like any other story.

So – to recap, I don’t care for the theoretical definition of user-stories – that they should be tangible and what not. Instead, I recommend that folks do what makes sense for their situation. If I had to give only one tip around stories, it would be this – keep them small. This mnemonic might help.

Finally – about epic stories. An epic is merely a feature (or some technical aspect of the system) that is way too large to be completed as a single user-story. It is epic, because it is a large story 🙂 I always keep playable stories down to less than 2 days in length – as far as possible. Epic stories, therefore, must be managed by breaking them down into incremental chunks of functionality. This is possible about 95% of the time – sometimes however when some spike is involved or some hairy technical refactoring is needed – longer stories can be played.

I hope these thoughts answer some of the questions. I’m sure other folks have different experiences and different solutions to these issues – and that’s fine too, since agile is all about adapting.

Lean Software Development For Startups

Here’s what I will be talking about at the Lean & Kanban 2009 conference in Miami this year.

Lean Software Development For Startups

(Or Why Agile Isnʼt Enough And How To Do More With Less)

Abstract:

If youʼre in a startup, then you know that statistically, the odds are heavily against you. Pretty much the only inherent characteristic of a startup that can be counted upon to help, is that of its small size. If the company can be nimble and agile, then it can hope to
gain some traction against its larger rivals.

In such an environment, using an Agile methodology is a given. Without some form of a hyper-iterative software process, it is impossible for a startup to create a successful product. Or even to determine what that product is!

In todayʼs climate, exacerbated as it is due to competition, lower capital requirements for software companies, the compression of Internet time, and the recessionary economic conditions, it is no longer enough to just use an Agile method. To stay
competitive, indeed to just survive, something more is required.

Lean Thinking provides just such an advantage.

A startup needs to ground its philosophy in Lean Thinking, Theory of Constraints, Critical Chain, Queueing Theory, Systems Thinking, and the like. It will obviously gain from the long-term focus, throughput-based accounting, and value-based constructs that these provide.

This presentation is about how Lean ideas when applied to standard Agile processes can make an organization super-productive even in the extreme short-term. Specifically, it draws on my experiences from having run multiple projects using this philosophy during my consulting career at ThoughtWorks, and more recently as a founding member of an Internet startup called Cinch.

—-

So, that’s what the talk will be about. I have a 45 minute slot – so I’m looking to finish speaking in about 20-25 minutes and the rest could be for a discussion. Hope to see you there!

What iteration-less agile really means

I’ve written about the idea of an iteration-less process before, and we’ve been using the spirit of the idea at the startup I work at (Cinch) for the past few weeks.

The idea is simple, and has two parts. First, eliminate the concept of formal iterations (use a pull-system to let work flow through the pipeline – use Kanban cards if you like). Second, replace the rhythmic heart-beat of iterations with actual releases to customers (internal or external).

These releases do not have to be spaced out by a week or two or any arbitrary span of time. They ought to be, in fact, the shortest amount of time that the software system can be re-stabilized after undergoing change (through adding features or fixing defects). Hence, if it takes 2 days to do a release, do it then. If, for some reason, it takes 6 days (instead of the old five), then so be it. However, the focus of the team should be on getting quality software into the hands of the final users. After all, everything else is inventory.

Pretty simple, eh? We have combined this with the idea of the estimation-less process, for even greater effect. I have been hearing from some people that I have worked with in the past, and they’ve reported excellent results through just such a process. So, anyway – that’s what iteration-less means. 🙂

P.S.

In a past life I had spoken about this with Jeff Sutherland when I attended Deep Lean (in 2007) – and he agreed that this made sense. He didn’t recommend it for any but the most experienced team – and I certainly agree with that. If your team is new to agile or is lacking in some other way, getting it going with the standard approach to agile is probably the best thing to do.

However – if your team can pull this off, you might want to give it a shot and see how much productivity increases over time.

You don’t need story-points either

Or an estimation-less approach to software development

I’ve had too many conversations (and overheard a few) about re-estimating stories or “re-baselining” the effort required for software projects. The latest one was about when it might be a good idea to do this re-estimation, and how, and what Mike Cohn says about it, and so on. The logic appears to be that as you gain more knowledge about the software being built, you can create better estimates and thereby have better predictability. This post talks about estimates, how they are just another form of muda (waste) and what might be a better way.

First things first

Lets start with fundamentals again. In order to stay profitable and prosperous, a software products company must be competitive. If you do enough root-cause analysis, the only thing that can ultimately ensure this is raw speed. Hence, the only metric that matters is cycle-time and organizations must always try to reduce it.

OK, so now, the issue simplifies itself – all we have to do is look for constraints, remove them, and repeat – all the while measuring cycle-time (and thereby throughput). Simple enough! And in the first few passes of doing so, I first realized that if you must estimate things, use story points. Then I realized that estimation in general is a form of muda – and converting from story-points to real-hours at the start of each iteration is a really stupid waste of time.

Even less muda

Now, having said all that, we can move on to the point of this post. I’ve come to realize there is a better way. One which achieves the goal of faster throughput, with less process and less waste. Eliminate estimation altogether. You don’t even need story points anymore. Here’s how it works –

Take a list of requirements. Ensure they always stay current – you know, by talking with the stakeholders and users and all the other involved parties. Ensure it is always in prioritized order – according to business value. Again, not hard to do – you just need to ensure that the stakeholders are as involved with the product development as the rest of the team is.

Next, pick a duration – say a couple of weeks. If in your business, two weeks is too short (you’re either a monopoly or you’re on your way out) then pick what makes sense. Pick up the first requirement and break it down into stories. Here’s the trick – each story should not be more than 1-3 days of work to implement. In other words, each story, on average, should be about 2 days of development time. Simple enough to do, if the developers on your team are as involved in the process of delivering value to your customers as the business is. Pick the next requirement and do the same thing, keep going until your two weeks seems full. Do a couple more if you like. Ensure that this list stays prioritized.

Then, start developing. There is no need for estimation. Sure, some stories will not really be 2 days worth – break them down when you realize it. Sure, you need good business-analysts and willing developers. You’ve hired the best, right? Next, as business functionality gets implemented to the point where releasing it makes sense, release it! Then, repeat.

What (re) estimation?

In this scenario, there really is no reason to do formal-estimation at all. Sure, super-high-level sizing is probably still needed to get the project off the ground, but beyond that, there is no need to do anything other than deliver quality software, gather real-user feedback, and repeat.

Now come the questions of how to say how much work is done, or when a certain feature will be delivered. The dreaded “how much more work is left” questions. The answer in this new world is simply = 2 days * number of stories in question. (Remember the average story size?) Many people like to track things, but measuring actuals is also muda.

The important thing is of course to not look too far ahead. In today’s business environment, the world changes at the speed of the Internet, so luckily this works to our advantage. Whenever something changes, you just re-prioritize the list of stories. This process is not only much leaner, it is also much simpler.

Let’s come back to the issue of re-estimation. People cite the fact that during the start of a project, there isn’t enough knowledge to truly estimate the high-level requirements. Then, as the project proceeds and they learn more, they seem to want to re-estimate things. I ask, why bother?

The key here is to recognize that “accurate” estimates don’t deliver any business value. It is only important that complex requirements are broken down into incremental stories and they are implemented and released quickly.

The real benefits

The fact that this saves time and effort is actually just a good side-effect. The real value comes from being able to truly stay in control of the development process. Tiny stories allow you to change them when required, pull them from the backlog when needed, or to add new ones whenever business demands it. It also lets you move faster because it’s easier to write code in small incremental chunks, testing is easier, and pushing small releases out to users is easier.

And finally, by imposing the discipline that each story be small, it ensures that people think about the requirement deeply enough before coding begins. It also forces the team to break down requirements into incremental pieces and totally avoids never-ending stories that always have a little bit left to complete.

So there

This allows the whole process to be streamlined. It allows the team to focus on just what the users cares about (working software at their fingertips), and not on all the meaningless artifacts mandated by many methodology books (and indeed unenlightened management). Also, in this manner we handle the issue of estimation and re-estimation. In a zen-line fashion, we do it by not doing it.

Against all Oddities

I just remembered, over this past weekend, that a long time ago I had submitted an article for the ThoughtWorks Anthology. I had actually gotten accepted for the book – with a sort of caveat. Martin Fowler told me that I’d have to improve the essay – to make it actually helpful to people reading it. Great advice, of course!

Except that I’m way too lazy and couldn’t be bothered.

So here it is, in it’s original unedited glory. Against All Oddities by Amit Rathore. I’m also too lazy to convert it into HTML, let Google do that, I say!

Why I don’t use real hours for sprint planning

OK, so several of my friends and colleagues pointed me to Mike Cohn’s recent blog post about why he doesn’t use story points for sprint planning. I left a couple of comments on his blog, but I couldn’t wait until he moderates them… so I thought I’d post my thoughts here.

In my opinion, using either story-points or real-hours for sprint planning makes one assumption. The assumption is that it is somehow very important to get a sprint “just right”. In other words, it is important for a team to estimate a sprint’s worth of work well enough so as to make it fit perfectly within a sprint.

I believe that it’s not important at all. Here’s why.

First of all, the length of a sprint is picked by the team in a mostly arbitrary manner. Some pick 30 days, some pick 1 week. For the most part, it could be any number in between there. So if the duration is not that critical, then why should fitting work into it be? In any case, when a team happens to under-estimate, the undone work gets pushed into a future sprint. And when a team over-estimates, the stakeholders gleefully add more work to the sprint. So… what was the point of all the real-hours-based estimation?

Next, lets talk about predictability. Take an example of a basket-ball team. They can make a general statement about their long-term average, but can they predict what they’ll score in the next game? No… but even more critical, is it even important to try to make this prediction? The answer is not really, it is only important that they do their darned best. A software team can certainly spend time breaking stories down into tasks and then even more time estimating each in real hours, and then bargaining with the stakeholders to make sure whatever they pick all fits in… but it’s just more important that they work on the highest priority cards, with as less distraction and road-blocks as possible.

Whether they fit it all in, into that one sprint or not, is a mere detail.

Now, it’s not to say that having dates is not important. It is, but you can’t be scope-bound AND time-bound. Pick one. My default choice is time-bound with variable-scope. In other words, I like to set a release-date, and then ensure that the team is burning down the highest priority items at all times, as that date approaches. When the date is nigh, release! We’re all writing production-ready software, aren’t we? Then, repeat.

This approach then makes it possible to simply drive the whole project off the product-backlog… without strict sprint-backlogs. It allows velocity to be measured in story-points (from the product-backlog) and prediction is a matter of total-story-points divided by velocity-in-story-points. Simple. If anyone asks what will be done this sprint, the answer is the highest priority stories on the backlog, as many as the team can, here’s our best guess (average-velocity-in-story-points). This approach is actually faster than sitting and re-estimating individual stories in real-hours, because well, the team doesn’t have that distraction anymore.

In fact, from my experience, when the team gets comfortable doing this, they even ask… OK, so if the length of the sprint was set fairly arbitrarily, and as long as we work on the most important stuff at all times, then why are we bothering with the concept of sprints at all? These teams go beyond what I described above and eliminate the waste of iterations. Fun, eh?

If I talk about this stuff to people, sooner or later, someone brings up the happy concept of rhythm – how it is important for the team to have a pulse, a beat to which they work – sprint-planning, development, sprint-close, retrospective, repeat. It gives them satisfaction. It probably does, but I think we can do better. I prefer my teams to not just close an iteration based on some rhythm, but even push the software into production each time. Let the rhythm be set not by an arbitrarily picked number, but through customers getting their hands on the next release of the software. They still do the retrospectives regularly, and have team outings all the time. But the rhythm of the team is set through actual releases. When teams get better and better at this, and realize the critical importance of cycle-time, they do all they can to reduce it.

Anyway, I can talk about this stuff for a long time. Back to sprint-planning. My take is – do it like agile architecture/design. Think of it as evolutionary. We’ve eliminated the big-up-front-design, it is time to eliminate the big-up-front-sprint-plan. Prioritize the back-log constantly, always ensure that the team is working on the highest priority item. Ensure there are as less distractions and road-blocks as possible. And release quickly, as early and as often as possible – make a rhythm out of it.

And if the stake-holder asks when a particular story will be done, see where it stands in the prioritized backlog. The answer to the question is, of course, after all the ones above it are completed. Ask also if they would like to re-prioritize. That’s all there is to it!

A Project Delivery Sanity Test

I’ve been working at ThoughtWorks as a software consultant for many years. During this time, I have done development as well as a project manager work. My most recent exploits were at a large project ($100 million+ program) as the PM on a sub-team of about 40 people.

After working with many clients and projects and also having worked on several side projects during the past few years, I like to think I’ve seen projects that span many characteristics – size, complexity, team composition, geography, domain, and technology. Based on the observations I’ve made over this time I’d like to present a list of things – a checklist if you will – of the most important issues that may determine the ultimate success or failure of a project.

1. A Delivery Focus

Being delivery focused begins with releasing software the moment it is written (almost). Releasing means to get the software into the hands of real, live users. Nothing else counts – developer-done doesn’t count, deploying on QA environments doesn’t count, staging doesn’t count.

Until the software reaches at least a single real user, all the code written (and all the design documents, passing tests, whatever else) is just inventory. (This implies that the goal of making money from the software can’t be achieved, thereby reducing the throughput to zero.)

The biggest benefit of getting code early into production (after creating a value-stream that might actually have some output) is that it brings the kind of focus on doing the right things that nothing else can.

I’d like to make a related point here. One of the few metrics that makes sense on software projects is cycle-time. The team must endeavor to reduce this cycle-time – and should be able to take an idea that needs to be implemented and push it into the hands of users in as little time as possible.

Another important issue of note here is that of usability. Once the software is out there, in the hands of the customers, are they happy using it? Or is it an exercise in frustration? If the users are captive (say internal users of an IT application), then the software still gets used (albeit it is the cause of multiple cuss-words a day). If it is a web-app meant for the general internet population, then the product will tank. Either way, the only way to truly understand and fix usability problems is by releasing it to the actual users.

a) Is your project in production? Are you satisfied with how long something takes to get into the hands of your users?

b) Is the feature-set being designed with a usability focus?

2. Clear Priorities

Speaking of bringing focus to software development, the most important thing is to ensure that you’re building the right thing! This relates to what feature next (and what feature never), what to improve, and what to remove. Setting a priority can not be arbitrary, it must come from a deep understanding of the market, the domain, and from listening feedback from real users.

This is not to say that one should do whatever any fickle user says, but it is important to listen, and include the feedback with the other things that go into prioritizing goals and features.

Without a source for rock-solid priorities, a development team will flounder. Rock-solid means that at any given time, anyone on the team can say what the next few most important things are. It also means that it is rock-solid until it is changed (which can happen at any time, based on changing reality and discovery).

I also like to think of a similarity between priorities and constraints. There can be only one top priority. When there are too many top priorities, there aren’t any.

Do you have a clear set of priorities (at any given point in time)? Does everyone on the team understand them, and how they are set?

3. Stakeholder Involvement

The one thing you can’t outsource is your product-owner function. Many teams seem to forget this, as product-owners spend so little time with the team, that they might as well not be present.

In today’s busy world of everyone playing multiple roles and having too much to do, it is common for people to multitask their days away. The effect of this on developers has been documented and studied – I have seen pretty much the exact same thing happen to busy product-owners.

The reality, however, is that if the bread and butter of an organization is the software their users buy, nothing can be more important than ensuring that it is built right and will garner the customer satisfaction and loyalty that is a must for it to succeed. Nothing.

A side-effect of this is some form of seagull-product-ownership. Here’s an example – the product-owner appears during a planning meeting (or the like), and makes comments that alter priorities (which is OK in itself). This changes what the team has understood so far and sends them into a flurry of activity to try and get things done which don’t necessarily flow with what they were doing until then.

A product-owner must, well, own the product. There can be no substitutes to spending several hours (at least) with the team each week – and truly understanding what the development issues are, and providing the team with critical information that is driving the priorities of the project.

Is your team’s product-owner involved enough with the project?

4. Business Analysis

This item is a larger issue in more complex domains. Remember, to truly complete a story it should be pushed into production. Small, incremental stories that make business sense are thus critical to running things in this manner. To help with analyzing requirements down into suitable chunks of work, they should be created with the INVEST principles in mind.

On teams delving with sufficiently complex domains, having business analysts that both understand the domain and are able to create such stories is a must. Again, the important thing here is the role – and that someone needs to do it. Having a team of poly-skilled people that include this skill-set is enough as well.

For what it’s worth, a good ratio of BA people to developers is about one BA for every one or two pairs of developers.

Does your project run off of well-written user stories? Is it sufficiently staffed up from a BA standpoint?

5. Team Size and Structure

From my experience, this one is a bit counter-intuitive to most people. People seem to think that for large projects (cost, number of requirements, whatever) large teams are required and even justified. There is plenty of research and literature out there that speaks to the negative impact of having too many people on a team. Even empirical observation allows one to reach the same conclusion – after all, how many smooth and successful large projects have you seen?

Ultimately, I think, this issue boils down to one of competent people. I find it somewhat difficult to believe that your average IT shop can attract not a dozen or two, but a couple of hundred developers to a project – and all of them are super-smart, super-motivated, and super-interested in the given domain.

The other issue that hits hard is how the team gets ramped up. Often there are too many people right at the beginning. Teams need to be ramped up slowly at the start – while high-level architecture is solidified (through code, being pushed to production!), and patterns are laid out, things of that sort. As these things happen and everyone understands the code-base well, the team can begin to add new members. This should be done slowly, as each new person needs to reach the same level of deep understanding. Once there are several people who know the existing system well, the ramp up may accelerate a bit.

However, I’m yet to see a project team of about 10 to 20 developers that couldn’t outperform a team of a couple of hundred. After all, you can easily scale up hardware, get more office-space, buy more snacks and food, but you can’t always scale up the rock-star developers that are required for a project to be successful. In other words, it’s probably Ok to hire less of those awesome developers you know if you paid them twice as much as your finance department thinks they’re worth. You’ll end up ahead anyway.

A related issue here is that once a project does get saddled with a large team, how is it sub-divided? Are the divisions made along functional lines (modules) or are they put into fluid workflow teams as the need arises? Is it one large team working on different stories with the whole code-base being collectively owned? Or is it bickering factions that jealously guard their areas of code from any change?

People who have suffered through artificial team separation along ‘module’ lines know the pain this can cause. Throw in the typical office politics into the mix, and the project ends up producing a system that conforms highly to Conway’s Law.

Finally, most people on projects know when these things have become issues – just listen in on any lunch conversation – or for the full details, hang out with the team at the bar.

Is your project over-staffed? Is the team sub-divided along the right lines?

6. Planning and Tracking

I’ve seen many teams ‘do Agile’, while not being agile in any fashion at all. Being agile is about focusing only on what’s important – and ignoring the rest. It means doing just enough of something so that it works as desired and produces the right kind of output. It means not doing anything just for the sake of it, and it means questioning any kind of process-thing which doesn’t add value to the software being produced.

Being agile allows and aids you in being lean.

How does this relate to planning and tracking? Because those teams that truly understand what being agile and lean is all about, focus on the activity of planning, and then react responsibly to changes that follow. Iterations are used to gather more and more knowledge about the system being built (while delivering to production at all times) so that better decisions can be made and better priorities can be set. Really mature teams don’t even need iterations.

What agile isn’t is a set of smaller periods of waterfall-type software development. An iteration planning meeting doesn’t produce a plan that must be met with perfect accuracy. Not meeting an iteration goal doesn’t mean it was a failure. Estimates still need to be treated as estimates, and iteration capacity still shouldn’t be filled to the maximum (basic queueing theory).

Planning on many teams becomes an exercise in craziness. This is especially true when the level of detail that people get into becomes so great, that they might as well just get the work done right then and there. Planning should be high-level goal setting, and as long as each individual team member understands the desired outcome for anything he/she picks up to work on, the intention of the planning meeting has been met.

There is often a tendency to re-estimate work in ‘real hours’, and then to track the progress and completion of each sub-task throughout the length of the iteration. This is a waste of time, and simply doesn’t need to be done. Might as well spend the time on taking the team bowling and spare them the wasteful effort of this sort of planning. At least the team will have a good time, and will get time to gel on a more personal level.

A more insidious issue here is the false sense of security or concern tracking things this way often produces. Graphs, pie-charts, trend-lines – all show the amount of work done, amount outstanding, tasks in progress, how many builds were green, how many bugs were found and how many were fixed, how many hours were worked overall, etc., etc. Instead, how about how many customers liked the product (is it even in their hands yet?), how many releases were made to the customer (boy, are we responsive!), how old a discovered bug is (why isn’t it fixed yet?), how little work-in-progress there is (unreleased software is just inventory), how fast the cycle-time is (turnaround time – from idea to profit) and so on? Metrics do matter, but ensure that they’re the right ones.

Does your project plan at the right level, and track and measure the right things?

7. Technical Architecture

Technical architecture can be a strategic advantage. This is most true if the design of the code allows new features to be added (or bad ones to be taken out) quickly and safely, and also allows maintenance to be done without breaking half of the system. It is an advantage if the turn-around time of a new feature or a fix is so short that the team can run circles around its competitor.

This is a system-wide issue (what isn’t?). If every part of the architecture except one is very well designed, is maintainable and extensible, etc., then that one module will become the constraint of the system when it comes to features in that area. The code-base needs to be kept clean and trimmed of fat at all times, across all areas. (All disclaimers apply – in the end it boils down to the cost of change vs. ROI).

Developer productivity is a related issue here, but is very important. The write_test-write_code-deploy-test-see_results-repeat-checkin cycle is how each developer spends his/her entire day. If the deploy-test-see-results part of this cycle is not down to less than a minute or so (at worst), you can expect major productivity losses from the lack of flow. Build times must be carefully monitored and everything must be done to ensure that developers aren’t waiting around for something like a build result.

Does the technical architecture of your project support a quick response (new feature, bug-fix, enhancement)? Does your project have long build times?

8. Testing, QA practices

Automated tests are the most important part of a code-base. Period.

No one can get a design right the first time, no one can get all the requirements down pat the first time. Things always need to be changed, and software spends four-fifths of its life in maintenance mode. The only thing that can automatically tell a developer that he has broken something is the test-suite. Without this, even making a simple change is a nightmare. Enough said.

QA people are also an important part of this. They are the independent verifiers of the developers’ understanding of the requirements. Testing early and often is the key, as they say, and QA folks need to be part of the development team. They need to be in there while a story is being written up, before it is played, and soon after it leaves the developer’s workstation. And everywhere else with their ad-hoc testing. And they should automate, automate, automate. Why should they do things which the computer is suited for perfectly – repeating a step-by-step procedure and ensure the output is correct? Enough said.

A good ratio of QA to developers is about one QA person for every one or two pairs of developers.

Is your team staffed sufficiently from a QA standpoint? Do you have automated GUI testing in place?

9. Understanding software development

This one is actually the simplest to state, but I’ve noticed that a lot of people have trouble accepting it. Here it is – people who run software projects should know what software development is all about.

If your project is being run by someone who doesn’t really understand the nuances of software development, and heaven forbid thinks he/she does, then you’re in trouble. Generally speaking, of course. I won’t say too much more about this – I just happen to think that people who either write the software or those who manage the people who write the software should know what the heck it’s all about. It just saves a lot of silly mistakes, absurd conversations, rework, and generally helps getting things done right, quicker.

Does your project team have managers that truly understand software development?

So there it is – my first cut at crystalizing many of the issues I’ve seen in the past few years on various projects into a list of nine items. There are other important issues to be watchful for, be sure of that. Also, I don’t think that the issues on any project that is in trouble can be traced to one single point above. It will always be a mix of several things, some listed above, some not.

I hope this is useful though!