In traditional, so-called waterfall software development, the starting point for any project planning is a set of product requirements (a “PRD”). All of the requirements must be defined, reviewed and approved up-front before planning or development can begin. Additionally, the development team may produce a software functional specification (“SFS”) that takes each requirement in the PRD and elaborates it into more detail, adding technical requirements and constraints. Both of these documents are then used as the basis of planning the subsequent development activities required to deliver the content specified in the PRD. In order to derive a development plan and schedule, every task required to construct and test every feature must be estimated. Once development has been completed, there is a handoff of partially tested features to a test or “QA” organization, followed by several weeks or months of testing and bug-fixing. The length of this phase will be highly dependent on the quality of the code at the point of handoff. Finally, at some point, a decision to release the product to the market must be made, and this is where the usual tradeoffs between schedule, features and quality are fought over. The delivered software is usually accompanied by a set of ‘release notes’, listing all of the features that were actually delivered, plus all of the bugs that the team did not have time to fix.
Software development is referred to as an ‘empirical process’ by advocates of scrum. The output of almost every stage of software development is impossible to predict with precision, and each step produces defects that must be re-worked. This so-called ‘plan-driven’ paradigm has been demonstrated time and again not to be amenable to the level of precision expected from other forms of production such as manufacturing and construction. The output of a modern production line is usually specified to very high standards, such as ‘six sigma’ (less that 4 defects per million units). It is difficult to imagine any large software project coming remotely close to this standard. Scrum is a response to this problem.
We do not want to spend much time here listing the disadvantages of traditional waterfall development, however, we should note that the waterfall and it’s variants have delivered many successful software products over the past several decades, despite all of the known limitations and attendant budget overruns, quality problems, and stressed out development teams. Let us also note that there are many project activities beyond the boundaries of the scrum framework that need to occur for the release of a commercially successful software product. Product development teams in large companies operate within a broader ecosystem that includes multiple other functions which must work together to define, develop, market and support new products. Teams that want to adopt agile development practices must recognize that this cannot be accomplished in a vacuum, and must address how to achieve an effective integration with other inter-dependent functions.
Key Process Differences:
- Requirements are not defined in detail up-front. Requirements are continuously elaborated into technical details throughout the project. The goal is to defer making potentially irreversible decisions until the last possible moment.
- Release-level planning is based on relative sizing of requirements.
- Detailed task-level estimating is only performed at the iteration level.
- There are no hand-offs between developers and testers, testing runs concurrently with design and coding.
- Each iteration is completed to an agreed definition of ‘done’, which always includes the requirement that all bugs are fixed within the iteration. The output of an iteration is production-ready code.
- The team meets daily to synchronize on progress and to highlight any issues that are impeding progress.
- At the end of each iteration a retrospective review is carried out by the team to establish opportunities for improvement – this is how the team ‘inspects and adapts’ to continuously improve their processes.
Key Role Differences:
- Product Owner: The product owner owns and manages the product backlog. In addition to traditional product management responsibilities, product owners are an integral part of the team, and are involved in all aspects of the process from release and iteration planning, to daily standup meetings, acceptance of completed user stories, and participation in retrospectives and demos.
- Scrum Master: The scrum master is not the project manager, but rather fulfills the role of scrum process expert, guiding the team in all aspects of scrum
- Team: The team is integrated with all the functions and expertise necessary to deliver a fully shippable increment.
There is a common misunderstanding that scrum is an invitation to abandon process. An effective implementation requires a level of discipline that some may not fully appreciate, and which most teams will take time to adjust to.
The generic framework is quite simple and can be learned and put into practice very quickly by small, co-located teams. Minimal tooling is required for planning and tracking releases and iterations (scrumboards in the team standup room, or spreadsheets). A full-time product owner embedded within the team, and available for planning and attending daily stand-ups, means that requirements can be defined and translated into user stories without a lot formality, and that changes can be discussed and agreed quickly.
There is one universal framework but no single universal process that can be specified in detail. The situation for each team is unique in terms of product, technology and organization right down to the individuals on the team. Each team is tasked with continuously evaluating the results of their efforts and for identifying opportunities to improve.
In the following chapters we will be discussing approaches for scaling the generic framework to support the realities of large-scale software construction, and how to achieve this while preserving agility and the underlying lean principles behind scrum.