Perhaps were it not for the Manifesto for Agile Software Development, various agile and iterative techniques would have remained in relative obscurity. As Agile has grown in popularity so have they — none more so than Scrum. On the other hand, it’s no exaggeration to say that Scrum has long been the driving force behind the entire Agile movement itself. Such is its prominence that in the minds of many, Scrum is synonymous with Agile and some even believe them to be the same thing.
However, it would appear things are changing. The State of Scrum 2017-2018 report produced by the Scrum Alliance shows that while Agile practices are on the rise, only 16% of survey respondents in 2017 say they use Scrum alone — down from 32% in 2016, and 43% in 2015 — suggesting that a phasing out in favor of other frameworks is occurring. The question is, why?
In this three-part series we take take a speculative look at how and why Scrum came to be so widely adopted and what the future holds for it.
The trouble with requirements
The agile manifesto was written in early 2001, but long before that there was growing frustration with the dominant and heavily process-oriented waterfall methodology. The documentation overhead alone was enough to make any alternative to that appealing. But more than that was the realization that “big design up front” and a detailed plan — the norm in other engineering disciplines — did not work for large software projects aimed at solving business problems.
But why not?
The reasons are the same today as they were in the past; essentially boiling down to the pace and unpredictability of change.
Software development even now struggles to define itself as a true engineering discipline like civil or mechanical engineering. But it’s reasonable to think that aspiring to the practices and professionalism of these more established disciplines is a good idea. After all, one can find plenty of similarities between large construction projects and extensive software systems. However, they are not analogous in a key area: software requirements change at the pace that business requirements change and changes in the business environment, unlike the physical, can be very rapid indeed.
The first version of a large software product may take years to materialize with traditional approaches. The task facing customers and requirements gatherers thus entails some attempt at predicting the future, i.e: Knowing that the status quo is likely to change, specify what your future self will require.
This is neither fair nor realistic, especially in dynamic business environments that change rapidly. And that’s to say nothing of the underlying assumption that customers have the skill, time and precision of written language to describe all their requirements to a sufficient level of detail.
Requirements analysis, design and planning as activities in themselves are not to be thrown away cheaply, but when applied to software development, many problems have been observed. These include:
Customers and business analysts struggle to write requirements in a way that translates unambiguously to software features. Misspecification, underspecification and even overspecification are practically unavoidable.
There is a maintenance overhead associated with any document. This becomes non-trivial when changes must be made frequently. People tend not to keep them up to date.
Developers may not read large requirements documents properly, partly because they prefer writing software, but often because detailed specifications without inaccuracies and ambiguity are so uncommon.
Subject matter experts are often too engaged in the demands of daily work to have sufficient time to create a well-written specification document. They are not trained to do so either.
Developers will interpret ambiguous specifications or fill in the gaps when no specification exits. Inevitably, they will be wrong some of the time and build the wrong thing.
If a change process is too bureaucratic, people will stop bothering to push for change even when it’s the sensible thing to do.
Conformance to a rigid plan does not necessarily correlate with a quality product or satisfied customers - often the inverse is true as engineers are forced into taking shortcuts in order to hit arbitrary deadlines.
Requirements are not detached from the real-world concerns of cost and time to market. Customers may not have a good sense for what is easy to build and what is not. It is necessary to continuously assess the benefit-cost ratio and make adjustments accordingly to ensure that value is always achieved.
Customers put in the position of having to predict the future will understandably tend to over specify at the risk of leaving something essential out. This leads to a product that is bloated, complicated and less likely to be completed.
Primed for a revolution
Regardless of whether it’s software engineers, end users or management, nobody wants to dedicate years of time and genuine effort to a project only to realize the resulting product is not working as expected, if at all; not what was asked for anyway; delivered too late to be relevant anymore.
The problems and subsequent failed projects resulting from waterfall-like approaches were precisely what 17 software developers sought to address when they gathered in 2001. They came up with a judiciously phrased and succinct manifesto that states the following:
We are uncovering better ways of developing software by doing it and helping others do it. Through this work we have come to value:
Individuals and interactions over processes and tools
Working software over comprehensive documentation
Customer collaboration over contract negotiation
Responding to change over following a plan
That is, while there is value in the items on the right, we value the items on the left more. At the turn of the century, those involved with software development were more than ready for an Agile revolution. As it turns out, Scrum provided the vehicle for it.
