The legend of hyper-performing teams

Introduction

Allegedly one core benefit of agile methods is an increase in productivity. That seems intuitive, as we know that improving teamwork pays off in terms of productivity at least over a longer period. Human interaction and collaboration is one focal point of agile methods after all. Agile methods such as Scrum for one incorporate wisdom that is much older than agile methods, such as the positive effects of co-location and thereby induced direct communication. But Scrum furthermore advocates cross-functional teams with direct customer interaction. It also allows the teams to self-organize. All of these points could be contributing factors to high-performing teams. More on how to create high performing teams can, for example, be found at http://p-a-m.org/2011/09/high-performing-teams/. This article now focuses on how well proven the productivity gains are based on studies of real cases.

How to measure productivity

But how can we measure productivity gains by agile methods in real life? Unfortunately productivity in software engineering still appears quite difficult to measure even in the 21^st century. One way would be to measure overall business output. But typically the overall output of software companies is determined by more factors than the output of its engineering. We would thus have difficulties isolating the impact of agile methods from other contributing factors. Other known approaches are synthetic benchmarks such as the function point analysis or – more simply – the lines of code (LOC) produced in a certain interval per FTE (Full Time Equivalent). Both have their shortcomings in measuring productivity. Particularly LOC as the older method does not sufficiently regard the complexity of the problem being solved: a hundred lines in programming a critical kernel function are hardly equivalent to hundred lines of standard web interfaces. Nor does it consider the quality of the implementation applied: good, state-of-the-art, object-oriented design might deliberately pursue lower LOC rates. Furthermore, many benchmarks are impeded by their own intention: some create a mirror of more or less comparable companies (or projects within one single organization), for example one using agile methods the other not. Unfortunately, the company or project that is reflected rather poorly in this mirror will be reluctant to have the outcome made public (an obstruction that probably goes for using any benchmarking).

Further complexity in productivity measurement is added by the circumstance that productivity is not an isolated factor. Even if we managed to compare projects where all other factors, such as product quality, are similar at a certain point in time, we would still face a possible trade-off: high short-term output could have been gained by piling up technical debt. In consequence, the outcome of a team should ideally be evaluated over the life cycle of a solution. A short-term solution might produce quicker results at the expense of maintainability; a more sustainable approach will consider the longer term with sufficient focus on product quality. But few researchers have the patience to conduct their studies over a stretch of 5-10 years. We certainly wouldn’t:)

Real-world studies

Yet there are studies to be found that take on the challenge. To start with, Scrum founder Jeff Sutherland tackled the question over years. He has delivered both productivity comparisons of companies as well as guidelines on how to achieve what he calls “hyper-performing teams”. (For instance see his presentation on “systematically achieving hyperproductivity”: http://jeffsutherland.com/PracticalRoadmapGoogle14Dec2009.pdf, or his response to the specific case of a company: http://scrum.jeffsutherland.com/2008/09/shock-therapy-bootstrapping.html). As far as the productivity comparison is concerned, again one must ask the question of how the data is measured. In the case of the presentation above, Jeff Sutherland uses velocity as a yardstick (he used other methods on other occasions). What does that measurement tell us? Assuming that the companies in this example follow the Scrum methodology, “velocity” tells us how the team rated the complexity of all of their own stories (requirements) implemented during one sprint. The yardstick is story points, an abstract unit, addressing relative and not absolute size, in a way that is not standardized – not even within the same organization. So we compare the subjective rating of two teams in an abstract currency. What does that tell us? It can certainly give us indications to formulate hypotheses of how productivity evolves with agile methods. But can it prove productivity gains of one team over time, or can it reliably measure productivity differences between two teams? We think not.

Other sources provided individual case studies such as by Eric D. Brown (http://ericbrown.com/agile-project-management-product-strategy-a-case-study.htm): “85% of the product features in half the time at less than half the cost.” Studies based on broader data have been conducted by universities and large companies. An overview of such studies has been provided David Rico et al. in “The Business Value of Agile Software Methods” (J.Ross Publishing, 2009). While the book delivers a good overview of studies, we found the conclusions derived to be… well… breathtaking. We learned that agile methods are 18 times as cost-effective as conventional methods. Quite impressive! But what does the figure 18 mean? Taking a closer look at the metrics revealed that Rico et al. focused on LOC per hour and number of defects. We can follow the qualitative arguments provided – that using agile methods leads to more code being produced and the code being of higher quality. But we are hesitant to believe that methods can be measured accordingly in such a one-dimensional way. Beside the fact that the old LOC measurement seems a bit outdated, we are uncertain how this metric is applied in this meta-study: it is unclear to us how the data across many studies with different methodologies had been compared. However, an interesting aspect becomes overt if we crosscheck the studies quoted by Rico et al: many of the studies quoted show significant improvements in productivity when applying agile methods, many of them around 80%. However the researchers measured it, they came to similar conclusions regarding productivity. So if we choose to live with the methodological questions raised above, this is a strong piece of evidence of productivity gains by applying agile methods.

Conclusions

All in all we see a reliable indication that productivity gains are to be expected by implementing agile methods, given they are properly applied in an enabling environment. That goes for increased output before a release due to high-performing teams, as well as for reduced maintenance effort after the release due to better software quality. We also acknowledge that determining the impact of organizational changes on productivity remains haphazard. Finding a quantitative indication in hard currency on what can be expected by introducing agile methods might have to wait for some time to come.