Five years ago, when I visited Zappos, CEO Tony Hseih pointed out that as organizations grow they become less effective. Each person you add increases productivity by a little less than the previous employee. Eventually, a large organization takes man-years to do what a small one could do in man-weeks. We understand this intuitively. Hseih claimed this was universal to nearly all business – yet it didn’t have to be. He used the example of cities, where more people condensed in a small area actual improves quality of life by creating new small businesses, cultural and social events, things to do and so on. Living with enough people in a small enough area and you will have access to an incredible variety of resources, products and services. Hseih wanted to bring that idea to business. The idea was so radical that employees had to sign a non-disclosure agreement.
In a way, anyone can scale a project, throwing hundreds or thousands of people at it. The challenge of scaling is to do so without massive dilution; without running a project with 1,000 people in two years that could have been done by a hundred people in half the time.
The argument is that agile doesn't scale, that the initial agile methods – Scrum, XP, Crystal and DSDM – don’t work for large groups. It might be more accurate to say that, at the time, we didn’t know how to manage large groups. The creators of the Agile Manifesto gave up, choosing to define a set of methods that could make a small team very productive. After all, as Hseih put so well, large teams are not productive.
You could argue that the creators of the agile methods chose to punt.
Ken Schwaber, one of the creators of Scrum, once told me that we never knew how to scale software development in the first place. My own experience is that inside of every huge project with hundreds of people were a dozen or so sound craftspeople, who knew each other by name, who were doing the hard work of delivering the software. Outside those dozen were another dozen who were at least helpful – without them, things would take longer. Outside that group were the ones who at least didn't slow things down any, and provided a layer of protection from management so we could work. Beyond that layer were the people who did one of those two things, then the layer that did neither. Schwaber told me that in the ‘80s and ‘90s, when the crisis hit, the modest teams of heroes would come out and get the project done.
[Related: Comparing scaling agile frameworks]
Google, Facebook and Spotify all grew incredibly quickly and need to coordinate massive numbers of people. Spotify has been loud and public about building engineering culture, while Google and Facebook have quietly moved to tens of thousands of employees, some large percentage of them in software delivery. None of them seem to struggle with "going agile" or "doing agile." One reason why: They all started without any hierarchy (outside the founders), which is the real problem.
Big companies have layers and layers of people who have been promoted out of technology. Architects, analysts, managers and directors of a specific discipline. If we change to teams of teams, what do we do with the managers?
The managers are threatened. To them, agile looks like a lose-lose proposition. In an agile conversion, for a manager to go back to technical work, or even mere product management, is to lose face. To try to stick to line management probably means to lose their job. At most large companies, organizational inertia – the hierarchy itself – is self-perpetuating, preventing any significant change.
A little perspective on managing change
Just as the personal computer threatened the mainframe, streaming music threatened the CD business and Uber threatens the traditional taxi, a real agile adoption threatens whatever came before it. This isn't a new idea. Clayton Christensen explained it in his groundbreaking book, The Innovator’s Dilemma, as the reason that most innovation tends to come from outside established companies. Those that work in the company face strong incentives to add to the existing line of business, as success with the new idea will make the old structure obsolete. This will work just fine, according to Christensen, until someone from outside happens on the innovation. When that happens, the company may face a crisis that can force a change.
Agile methods, on the other hand, offer only to improve the speed and value of IT services. While that might work for a software company, established companies with existing offerings, like automobiles, financial services and insurance won't face a crisis for ignoring a change. They can simply over-pay for their IT, which will be a little less responsive. As Hseih pointed out, they probably will anyway.
Without management support at multiple layers, the best an agile initiative can do is to improve delivery within a few groups. But that itself is nothing to sneeze at. When Tom West was assigned a small team to improve Data General's processors while a much larger, ambitious team was going to develop the next generation, West's team ("Project Eagle") actually got it done, producing the Eclipse MV/8000. The story, documented in The Soul of a New Machine, earned Tracy Kidder a Pulitzer Prize.
Ultimately, though, it’s another example of a small team out performing a large team. Even after the success of Project Eagle, the members of the project mostly scattered shortly after it shipped. Tom West was transferred to Japan, a move he referred to as "effectively fired." After saving the company – not to mention the PR coup of the book itself – Data General couldn’t exactly fire him; he was promoted to research roles for technology that was never commercialized (and to sign deals and trade show work), retiring as Chief Technologist in 1998.
Project Eagle is a story about de-scaling – of doing with a dozen people what couldn’t be done with 500. The point is that ignoring the system and human forces at play in an agile transition is to ignore history and human nature. In order for a change to succeed, people need to see a personal win.
Moving forward
If "that won't scale" is a smoke screen, then one thing we can do is ignore it. Here are a few quick ways to improve the large organization without using the “S” word.
Join the CIO Australia group on LinkedIn. The group is open to CIOs, IT Directors, COOs, CTOs and senior IT managers.