Announcing the Bay Area Lean Software Development Meetup

Hi folks, I’d like to announce the Bay Area Lean Software Development Meetup. The first meeting is scheduled for the end of the this month – on the 30th of April.

I’m going to present on the topic that I’m presenting at the Lean & Kanban 2009 conference – and the topic is Lean Software Development for Startups.

This is just the first meetup, so it will be general – exchanging ideas about what different folks in the valley are doing with their software processes… and things like that.

Join the group – and attend, it should be fun!

The first meetup of the Bay Area Clojure User Group

We held the first meeting of the BACUG (ugh, that’s a tedious acronym) last week, on Thursday the 5th of February. I was expecting about 6-8 people to show up, but we had about 12 people, quite a nice turn out!

After introductions and such, Chris Turner presented his unit-testing library for clojure called Clojure Spec. We talked about testing in general, and because he uses a lot of macro-writing macros in his code, we talked about macros in general and the difficulty in debugging macros. It was a good discussion – though I did throw in (tongue-in-cheek) that we’re using Test Is at Cinch. Thanks Chris, it was a good presentation.

After that talk, I gave an introduction to how we’re presently using Clojure in our startup. Since we’re using HBase as our persistent data-store, and processing that data using a bunch of clojure processes, we talked about data-modeling for HBase-like systems (I will write about my implementation of that data-model in another post).

An interesting thread came up – led by Joe Mikhail who works at Google and obviously does a lot of Map/Reduce/BigTable stuff – around how at least in the beginning, multi-threaded clojure processes can be used in place of Hadoop when processing HBase data. And he did mention that using systems like Terracotta, one can scale up such solutions. We’re going to look into that next week.

The remainder of the meeting went by in a buzz of talking about different languages, technologies, and general geek topics. The one thing of interest here was the point that software transactional memory is no panacea (surprise!) to the whole concurrency thing. Here’s an ACM article (there are several actually, just keep turning the pages) that throws some light on the issue. Here’s another.

Overall, it was a very good meeting – we’re hoping that the next one would be attended by more people, and especially some folks that have used other Lisps in the past – we’re all curious about what such folks think about Clojure. Indeed, and we want to learn from their experience in using idiomatic Lisp.

I’m thinking that we’ll do another Clojure meetup in about 6 weeks or so… join up, and stay tuned!

P.S. – A shout of thanks to my employer Runa. for hosting this meeting.

An organizational pattern: Make it about the people. Really.

This post is inspired by two bodies of literature. One is about Lean Product Development, and the other is by a man named Ricardo Semler (and his two books – Maverick, and the Seven Day Weekend). These two bodies of work re-inforce each other strongly. If an organization could be born that truly, at a very fundamental level, implement the tenets of what they’re proposing – then what an ideal place it might be!

I think both these schools of thought start with their emphasis on the importance of people. How cliche! But seriously, these guys *really* mean it. Toyota has long focused on empowering the workers at the frontline – they get to decide a lot of what the manufacturing process is and how it’s changed/improved and so on. In fact, they’re counted upon to innovate and change the standard processes every single day – it might as well be part of their job description. Semler similarly talks of letting the guys that actually do the real job of producing things that the company sells decide things like processes, work-schedules, even pay-scales.

Believe me when I say I have read a lot about Lean product development. The flavor of agile that I use with my teams is heavily influenced by several Lean principles – both on large software teams (40+ people) and in really tiny ones (3 people). All of this basically involves letting the people self-organize and come up with the best plan and process to get the job done. It has worked admirably – and it made my job as a manager easier as well – all I had to do was to remove roadblocks and facilitate what they wanted to do, rather than plan and track and all that. I understand it, I get it.

However, on first hearing how Semler takes these ideas to their (extreme) conclusion – I was a tad less enthusiastic. It was hard to fully trust the idea that people could be OK knowing what everyone else in the organization makes (in terms of compensation), or that if given the choice to set their own work-schedules (the seven day work-week/weekend) they’d not end up with a chaotic schedule. Or even that letting people attend any meeting that was going on was a good idea, or even more so – letting people walk out of meetings that were boring.

However, none of my initial resistance stood up to the five-whys test. The five-whys is simple – all 4 year old children know it. You keep asking why until no more satisfactory answers are available (apparently you do this five times). Most of the time however, you can’t really get past the second or the third one.

I would love to see an organization that were built on these principles – take the seven-day weekend. A team would be able to decide what days of the week they worked. Who said that only Saturdays and Sundays were allowed as holidays? If you can work on a Sunday to get that important deal, you could certainly take Tuesday off to relax and go shopping. Organizations supposedly do this today – with the idea of ‘comp’ time. But usually it is difficult to get it approved, and it isn’t a matter of course. It should be! Why couldn’t someone work a full day on Saturday instead of on Wednesday, every week? My initial reaction was that it would be too chaotic! How would the team get anything done if everyone was taking different days off – it would just be a mess! But really, if your team were given this option, you wouldn’t randomly disappear without coordinating it with the team, would you? Wouldn’t you and the team figure out what the best schedule was for everyone involved? I’m certain they would, and it would be different for each team, and it would work just fine. Everyone would also get some respite from the fast-paced life we lead these days, and have a day to themselves that isn’t necessarily on a weekend when life still stays rather fast-paced.

Let’s now talk about letting people set their own salaries. This is actually based on top of another very important principle – to really empower people you can’t hide information from them. In this case, the information is of the financial type. If the books were open about how much money the company makes, what its costs are, and what industry standards compensation-levels are for the roles of each person involved, then surely the team would make informed choices. Further, this will require each person to know what other people make – if not specific amounts but the ranges based on say experience levels and the job role. People are mature enough to understand these basics, and as long as these decisions are being made as a team, this can work just fine. There will be situations when someone wants to pay themselves way outside their range, but the team will correct for them – either they will make an exception for reasons they understand, or the person will find that pay level in a different organization. It happens today – but the power and information would lie with the team members.

Speaking of teams handling exceptions – this whole system rests on another crucial aspect of the philosophy – that of peer review. Performance reviews shouldn’t be done by immediate managers – that system is too easily gamed. Instead every individual must be rated by all members of the team that they are a part of, and also the people that report to them, and the people they report to. Only complete 360 degrees of feedback can ensure enough checks and balances for this democracy to succeed. This is true when silly things happen – someone is spending way too much on their expense account – to when serious things happen – someone wants to give themselves a fatter than deserved raise. People know when they’re trying to do something that isn’t necessarily fair, but when they also know that everyone will know about it and judge them for it, they will be less inclined to do it. There is no need of a draconian corporate police that throws a large policy document at people – the system handles itself in all such cases.

Part of the peer system, of course, is that the group hires their own team members. No HR department or some other team hires people and then sends them over to join a team. Instead, the team itself interviews and decides who to allow in. This is now common enough in several organizations, so I won’t go into it too much.

Another aspect of Semler’s ideas involve meetings – there are no compulsory meetings, and all meetings are open to any employee. It’s amazing – he says that if no one comes to a meeting, the agenda can’t be too interesting. And if that is the case, then whatever is being talked about or planned won’t ultimately be carried out very effectively. Again, when I first thought about this, I was concerned about the time when no one would come and listen to my brilliant idea about the next big thing. On questioning myself further, I realized that if I were to force people to listen to me and then insist that we go down a particular path because I said so, the result would certainly not be what I probably would hope for. However, if what I had to say appealed to people and they came and listened, then we could together take the merits of my idea, fix its weaknesses, evolve it with collective intelligence, and move forward together as a group of bought-in people. More importantly, if this were really the only way things got done in the organization, then no one would have to waste their time attending meetings about stupid things and, more importantly, no one would have to actually waste their time following along strategies that are bound to fail. (Any of you ever had to follow along as stupid ideas were being implemented?)

A different aspect of this meeting culture is – any meeting be open to anyone. And why not? I couldn’t think up of a single answer. My first reaction was – too many people in a meeting make it ineffective. I have personal experience of this – I have witnessed far too many meetings go on forever or be entirely ineffective because of too many people saying too many things. However, I don’t believe that the reason was that there were too many people in the room. It was often because of the quality of the people in that room – most were, quite frankly, less than able to pull their weight in the capacity that they had been hired for – but the team was not empowered enough to self-select the person out. (Any one know people like this?) If the fortunes of the company (and therefore compensation, job satisfaction, quality of the people that one had to work with, corporate culture) were truly in the hands of the people (indeed, they were directly responsible for them), then people would stop wasting time, and people would try to get the job done as quickly and as effectively as possible.

One other thing – even in organizations filled with super-smart and capable people – sometimes meetings run too long and without useful outcomes. I believe this is because of the way decisions are made in such places. I have seen, many times, that the person with the loudest opinion tries to win by using volume, or that two factions with equal volume argue forever. However, facts speak loudest. Organizations that use actual data to drive the decision making process don’t have to worry about opinions. Google is a prime example. Even things like the logo design is subject to real-usage data-analysis before one is finalized. There are no aesthetically driven shouting matches beyond a certain point. More importantly, data serves as the ultimate equalizer – it doesn’t matter if the CEO has a personal preference about something, the greenest engineer can win the argument in the face of data that proves his point. This makes for truly empowered employees.

See, it isn’t about the meetings – it is about the idea that no one leads by decree. If there is a proposed path that a group might go down, it needs to be democratically selected. It needs to prove itself to that group of empowered individuals that are looking out for themselves, and therefore the organization. It becomes impossible for dumb ideas to take hold – the brutal natural selection ensures no one even gives a damn, and so no one does anything about it. What a wonderful difference from so many organizations where dozens of people march towards the end of the cliff, and right off, because the ‘product manager’ had this great idea.

Such organizations that actually believe that people are their most important asset (cliche time!) and make it so through real empowerment are places where the Lean product development system isn’t some new-fangled idea from the east. It is often the most natural evolution of the collective thinking and creativity of the workforce. Conversely, and importantly, for lean processes to truly be their most effective – such empowerment is a necessary condition. The added benefit is that such places are joy to go to every morning.

For more, I highly recommend Maverick and The Seven Day Weekend. And all the lean development stuff.

Requirements management, user stories, mind-maps, and story-trees

For those who’ve been following along, I recently started a new project team that’s working on pretty much a green-field type of a project. I’ve been using mind-maps to kick things off, in various areas of the teams’ initial effort, and have now extended it as a way to build the functional backlogs that the teams will work on.

Essentially, what that means is this – imagine that we, along with an analyst (or a team of analysts) sit down to brainstorm an area of the application. We always start with the high-level ‘project’, lets call it Todos2Go. Just to retain context, lets also go ahead and list out the high-level functional areas of the application. These include modules we already discussed and elaborated, they certainly include new modules we want to include in our system. So in our example, Todos2Go would start with the basic todo-list creation module. It might also have a collaboration module, a search module with advanced search options, a calendar integration piece, an RSS generation module, an email interface module, a tagging and tag-browsing piece, and others. The analyst team doesn’t worry about trying to capture everything at any given point of time, knowing that they will be revisiting these requirements (the mind map) whenever they need to. So at this point, here’s what we have –

todos2go.png

Remember, we’re only capturing functionality from a business requirements perspective right now. Things like config-management, architecture, etc. are somewhat orthogonal areas. They’re captured in a different mind map. Also, we’re not concerned with scheduling right now – as in, when a particular piece might get built. Those decisions will be made based on business priorities, technical dependencies, and developer estimates. So, onward! Lets continue following the analyst team as they keep using the mind map to go further and further into details.

The analyst team picks one module that they’re going to focus on for the moment. They decide that since they’re just starting the project, they might as well pick the one they know best (from previous versions of the software, or from market research) – a basic task-management feature – the Todolists module. So they start to discuss the things the user might want to do, and/or how they would like to break the functionality down for development. Here’s something they might come up with –

Todos2Go-Todolists.png

The idea is simple. Keep breaking things down as you go down a path of functionality. In other words, go ahead and do plain ol’ functional decomposition. Again, remember to view things from a business requirements perspective. This means that ‘design database schema’ is probably not something that ought to appear on this mind map. The numbers next to the items are simply ID numbers, for tracking purposes, and will be of use later on. This view is what I’ve started to call Story Trees, thanks to a colleague – Roger Marlow.

Also, since most people use Excel to track projects (so do I, and I will continue to do so until Mingle launches), here’s how I capture this meeting artifact (it’s usually a whiteboard thing) in a spreadsheet –

todos2go-storylist-excel.png

The Parent ID column is simple. It is just the ID of the bubble in the mind map that is a ‘parent’ of a given bubble. This way, a simple script can be written to reconstruct the mind map if needed. The reconstructed version of the mind map can sport color-coding based on status, priority, or whatever else one might consider useful. More on this script (or my version of it) in another blog post.

So, I do hope this was useful. This is a simple meeting technique for project managers to use to facilitate requirement gathering workshops or business analysis sessions. Again, as I mentioned in one of my earlier posts, I do not use software/laptop/projector in my meetings because it is just not a good, interactive way to run a meeting. I use a whiteboard, flip-charts, and Post-Its. Later, I use software to capture it.

Coming soon – more benefits of Story Trees.

P.S. – Please pardon and ignore the horrid use of the generic term ‘user’ in my story list. We typically use personas, the focus of this post however is the story trees.

Using mind maps to start planning a project

As you might have read earlier, I recently started a new team that has been chartered with building a suite of new products from scratch. The business folks have been working in the domain for a long time and have built similar products before. Despite that, requirements for the new products have not all been nailed down yet, making this project somewhat of an ideal situation to start a new team. We’re using Agile (Scrum with XP practices) methods, and I’m infusing several ideas from Lean Software Development into the mix.

I used a new tool to kick-ff the high-level planning sessions this time. I’ve been using mind maps (FreeMind, Mindjet Manager, and NovaMind) for a while now: for various things – keeping track of to-dos, planning my personal projects, thinking about high-level goals, brainstorming and brain-dumping, and so on and so forth. I’ve always felt that such an intuitive tool might make for a great way to facilitate work sessions, especially when the work at hand is fairly high-level. An example might be – early stages of planning.

In my opinion, using the tool turned out to be quite a good idea. The first meeting lasted three hours (including coffee and a 10 minute break), and had about 12 participants with me facilitating using a whiteboard and sticky flip-charts. I ended up with several mind-maps including one for the overall project layout, one for training requirements, and another that detailed a particular module. The whole team was able to participate in building it as it was merely a form of a brainstorming exercise, with some extra structure.

Here’s a mock-up of one of these mind maps that got produced at this session –

projectX.png

One possible next step is to pick up the area which needs to be worked on first (actually this will probably be more than one – for example, the architecture-spike and config-management piece will need to be done in the beginning regardless of what else gets picked up) and continue to drill-down into its details. When sufficient granularity has been achieved, the leaf nodes of the mind map can be translated into backlog items (or phrase-level user-stories). Of course, this will mean that the brainstorming will need to be done with an eye towards this end goal – and all the guidelines around writing good user-story phrases will need to be kept in mind.

Overall, it was a good exercise – it was interactive and fun. More importantly, it was a visual way of representing the project and its potential scope – and was a great start at arriving at a shared mental-model of the system. Having all members of the team present (developers, QA, BA, and sponsors) helped a lot as well.

Note: I didn’t use the actual software in the meeting, instead, I chose to draw on the whiteboard and one of the participants (the sponsor!) volunteered to scribe. This let the session be more about the content than about formating and word-smithing. I later converted the notes to soft-copies using FreeMind.