Author Archives: admin

1. Introduction

When I was first introduced to agile some 10 years ago, I was coaching teams in software engineering best practices like code inspections, formal requirements management and design reviews, all of which I had absorbed after many years in large SEI/CMM organizations, and several telecommunications equipment vendors who were either ISO 9001 or TL9000 certified.  The mantra at these organizations was “On-time, On-Scope and On-Quality”, and this quest was pursued vigorously using a vast range of practices (and people). Predictability, determinism and perfect control were the key objectives. Initiatives like Six Sigma, and “Zero Defects” (Infinite Sigma!) were the frequent outputs of annual process improvement rituals. ISO and TL auditors were brought in to verify that everyone in the organization was following defined procedures, and the resulting certificates were framed and proudly displayed in corporate lobbies for visitors and customers to be impressed and comforted. Despite all of this, it seemed to me that substantially little changed from year to year, and software continued to be delivered late, with missing features, and unacceptable quality.

I watched the emergence of Agile from a distance, and despite being skeptical of it’s claims, and even outright suspicious of the motives of it’s promoters, I was intrigued. I continued to observe it’s increasing popularity and growing momentum, but still something was holding me back and I could not buy in. I was attending a process improvement workshop at a company where I worked, and sat through a presentation on ‘agile defect tracking’ using Kanban. We were shown a Kanban board with defects recorded on post-its, and columns with the usual ‘To-Do’, ‘In Process’ and ‘Done’. When the presenter finished describing the mechanics of this agile technique, he got an enthusiastic round of applause. The demonstration was for a fairly small project which had to process about 50 software defects throughout the test phase. I asked a question about how the process could be scaled up for larger projects, with say, 1,000+ software bugs, and with the team distributed across multiple geographies and time-zones – a fairly typical situation in many large companies today. I did not get a convincing answer. This presentation reinforced my thoughts about what I was reading in the agile literature – successful application was based on small, co-located teams, doing fairly simple projects (like building shopping carts). Nonetheless the popularity of agile software development continued to grow, and soon teams within companies were seizing the initiative and adopting agile in a bottom-up revolution against the traditional software development establishment. My own company at the time employed a traditional phase-gate development model, where work was passed from one group of functional specialists to the next: product managers > architects > developers > testers, with each gate transition being sanctioned at a ‘Gate Review’ presided over by senior managers. It was the development team who had seized the initiative in this case, but all they had really done was introduce iterations into their development process, and at the same time started using the scrum terminology of user stories, sprints and burn-downs. This was a small step, but one that left the broader product development ecosystem completely intact, and, did little to improve organizational throughput or flexibility.

It is very well known that introducing change in any organization is a serious challenge, and that without the buy-in and active support from senior leadership, major change is virtually guaranteed to fail. In any software company, or any technology company whose products have significant software content, the people in positions of leadership, have invariably grown up with the waterfall and it’s variants, usually accompanied by a phase-gate framework for governance. These people are in leadership positions because they have a successful track record of delivering products to market. They know how to get projects executed, largely on-time, on-scope and with reasonable quality. They have seen it all and have the scars to prove it. They are business focused and do not have the time or patience to indulge the latest process fad. Above all, they know that turning a large organization inside out is a risk that could seriously hurt their company and their careers. Getting the development team bought into doing iterations is probably the easiest part of an agile transformation program in any large company.

Meanwhile, not long after the ‘Agile Defect Tracking’ demo, I had moved to a start-up company, one that had embraced agile practices end-to-end. This company had less than 100 employees, distributed across 3 locations, in Boston, Philadelphia and Beijing. The founders were strong supporters of agile practices, mandating it’s use as a company policy. But it was about much more than doing iterations. A continuous integration system was put in place at the outset, and coupled with a concurrent and highly automated testing process, releases of new functionality were delivered at a steady tempo to customers every 6 weeks. The customers loved the tremendous flexibility of being able to try things out in their lab, suggest changes, and have them provided 6 weeks later. Within a year, the company was acquired by a major communications equipment vendor. The start-up was soon thereafter required to adopt the acquiring company’s phase-gate model and other traditional development practices. However, they not only successfully retained all of their agile practices, but soon became a reference model for agile development throughout the company.

Who This Book Is For

Like any major change or improvement initiative, organizational leadership is a prerequisite for successful agile adoption. However, in order to set organizational direction and goals (or at least to support agile goals proposed by others), executives must invest the time to really understand Agile. There is much to learn, and the world is drowning in an ocean of seminars, literature and advice on agile methods. There are however a small set of basic principles that apply, and when understood, can help leaders provide guidance for implementation activities. Key topics for consideration include:

  • Why are short, time-boxed iterations, important?
  • What is velocity-based planning and estimating?
  • How do we leverage an agile product development framework to accelerate delivery of value and quality to customers?
  • How do we use agile to strengthen and deepen the customer relationship?

This book is intended to help senior managers and executives who are struggling with questions related to agile adoption. The aim of the book is to describe the underlying principles behind agile practices, to provide recommendations for making an orderly, incremental transition to an agile framework, and to show how to leverage a newly implemented agile framework for business advantage.

3.1 Introduction

Contrary to some common misperceptions, agile is a highly disciplined and rigorous framework for developing software. Like any other disciplined undertaking, agile requires solid planning. Agile planning is however fundamentally different from traditional waterfall planning. As previously discussed, in the waterfall model, a development plan and schedule involves decomposing each requirement for the release into a set of tasks and then making estimates for the effort required for every task. For even modestly scoped commercial software projects the number of tasks could range from hundreds to thousands, and it is clear to most that these plans quickly become of out of date and need a huge amount of effort to maintain. Agile planning on the other hand is designed to be adaptive. The principles behind this approach are: 1) plans should only be as detailed as the depth of information currently available, and 2) planning must be continuously adjusted as improved project knowledge becomes available. Agile teams should be prepared to make planning adjustments as information about requirements, design constraints and the team’s velocity improves.

2.1 Foundational Agile Practices

Agile software development requires 3 foundational sets of practices to be in place. Doing iterations alone does make you agile unless the output of your iterations is production-ready code. But how do you get production-ready code for a large complex software product out of a 2-week iteration?

We will spend a lot of time discussing Scrum in this book. Scrum has become the de-facto best framework for doing iterations. But scrum by itself does not produce deployable code as its output, unless backed up by 2 prerequisite sets of practices, namely Continuous Integration and Test Automation.

Take for example a communications product that resides at the core of a high-speed network. This product not only executes multiple stacks of routing protocols, and needs to be highly configurable and manageable, but also interacts with multiple external devices and products, against all of which it must be tested with to ensure interoperability and tolerance to many types of adverse events in the network. In order to do this, thousands of test cases must be executed, including performance and stress test cases, which traditionally may have taken months to complete. A very high degree of automation is required to get this done on virtually every change that is submitted.

But there is another prerequisite. Traditional construction has involved multiple developers working on private code branches. Changes from each developer are coded/unit tested on their own private branch and then checked into a trunk line from where the next build is produced. New builds must be ‘sanity tested’ for integrity, and then made available to the development team for verification of their changes before the system test team takes over. The process of getting a new build to system test may take several days, and many organizations working in this mode may typically make one new build per week. This is the ‘big batch’ approach to software construction which simply is not consistent with delivering production code in 2-week iterations. Changes from developers must be submitted continuously, and these changes must be of very high quality to ensure minimal re-work, and minimal accumulation of an open bug backlog. We will elaborate on the subject of Continuous Integration in a subsequent chapter.

Whereas scrum and be taught and implemented in a fairly short time frame, building an effective CI system, and automating large legacy test suites may require significant financial investment and time to get into place.

In summary, agile software development must comprise elements of:

  • Iterative development e.g. Scrum
  • Continuous Integration
  • Test Automation

We will be elaborating on each of these topics in subsequent chapters.

 

 

 

Managing Large Complex Backlogs

Topics

  • how to properly create user stories for complex systems
  • how to derive user stories from traditional customer or product requirements documents
  • how to ensure traceability between derived user stories and their parent requirements
  • how to measure progress in delivering the customer-facing product functionality

If our project is small enough we may start by creating a backlog populated with user stories. User stories are units of customer facing functionality that can be delivered within a single iteration. The scrum literature is full of examples like:

  • As a new user I can self-register
  • As a member I can login using the credentials supplied at registration
  • As a system administrator I can browse the list of registered users
  • As a system administrator I can search for registered users
  • As a system administrator I can disable a user’s account

A small release may have up to a few hundred of these types of user stories. The release backlog may be processed by one or more scrum teams who will implement the entire backlog over the course of several sprints.

At the next level of complexity the initial set of release requirements may not be defined with sufficient granularity that they can be individually implementable in a single iteration. Scrum refers to these requirements as ‘epics’. Epics are backlog items that must be broken down further before they can be delivered in a single iteration. If the epic list is larger than a few dozen items, this epic list could easily expand into several hundred items at the user story level. Hence it’s going to be important to impose some structure, otherwise we run the risk of losing sight of the forest in a vast expanse of trees.

Feature Categories

Grouping epics by feature category is one simple way of organizing a long list. Let’s say our product is a network router, typical feature categories might be:

  • Routing Protocols
  • QoS
  • Security
  • Management Features

All of the features (epics) to be delivered in the release fit within these categories. Each feature/epic is then decomposed further into units of functionality (user stories) that can be delivered in a single iteration:

A Structured Backlog

Note that this schema: Feature Category > Feature (Epic) > User Story is completely arbitrary. Furthermore, depending on the application, you may need more than 3 levels of granularity to specify and organize the customer’s requirements, and to get down to a level that can be delivered in a single iteration. There is no universal or standard way of doing this, and scrum does not prescribe how this should be done. Each organization or development team will need to develop the approach that best meets their needs. Let’s not forget the INVEST criteria for a good user story to help us determine if we have drilled down to the appropriate level of detail:

  • Independent
  • Negotiable
  • Valuable
  • Estimable
  • Small
  • Testable
Letter Meaning Description
I Independent The user story should be self-contained, in a way that there is no inherent dependency on another user story.
N Negotiable User stories, up until they are part of an iteration, can always be changed and rewritten.
V Valuable A user story must deliver value to the end user.
E Estimable You must always be able to estimate the size of a user story.
S Sized appropriately or Small User stories should not be so big as to become impossible to plan/task/prioritize with a certain level of certainty. A user story must be sized such that it can be fully delievered (to the definition of DONE) in a single iteration.
T Testable The user story or its related description must provide the necessary information to make test development possible.

 

 

 

2.2 Scrum Basics

In traditional, so-called waterfall software development, the starting point for any project planning is a set of product requirements (a “PRD”). All of the requirements must be defined, reviewed and approved up-front before planning or development can begin. Additionally, the development team may produce a software functional specification (“SFS”) that takes each requirement in the PRD and elaborates it into more detail, adding technical requirements and constraints. Both of these documents are then used as the basis of planning the subsequent development activities required to deliver the content specified in the PRD. In order to derive a development plan and schedule, every task required to construct and test every feature must be estimated. Once development has been completed, there is a handoff of partially tested features to a test or “QA” organization, followed by several weeks or months of testing and bug-fixing. The length of this phase will be highly dependent on the quality of the code at the point of handoff. Finally, at some point, a decision to release the product to the market must be made, and this is where the usual tradeoffs between schedule, features and quality are fought over. The delivered software is usually accompanied by a set of ‘release notes’, listing all of the features that were actually delivered, plus all of the bugs that the team did not have time to fix.

Software development is referred to as an ‘empirical process’ by advocates of scrum. The output of almost every stage of software development is impossible to predict with precision, and each step produces defects that must be re-worked. This so-called ‘plan-driven’ paradigm has been demonstrated time and again not to be amenable to the level of precision expected from other forms of production such as manufacturing and construction. The output of a modern production line is usually specified to very high standards, such as ‘six sigma’ (less that 4 defects per million units). It is difficult to imagine any large software project coming remotely close to this standard. Scrum is a response to this problem.

We do not want to spend much time here listing the disadvantages of traditional waterfall development, however, we should note that the waterfall and it’s variants have delivered many successful software products over the past several decades, despite all of the known limitations and attendant budget overruns, quality problems, and stressed out development teams. Let us also note that there are many project activities beyond the boundaries of the scrum framework that need to occur for the release of a commercially successful software product. Product development teams in large companies operate within a broader ecosystem that includes multiple other functions which must work together to define, develop, market and support new products. Teams that want to adopt agile development practices must recognize that this cannot be accomplished in a vacuum, and must address how to achieve an effective integration with other inter-dependent functions.

Key Process Differences:

  • Requirements are not defined in detail up-front. Requirements are continuously elaborated into technical details throughout the project. The goal is to defer making potentially irreversible decisions until the last possible moment.
  • Release-level planning is based on relative sizing of requirements.
  • Detailed task-level estimating is only performed at the iteration level.
  • There are no hand-offs between developers and testers, testing runs concurrently with design and coding.
  • Each iteration is completed to an agreed definition of ‘done’, which always includes the requirement that all bugs are fixed within the iteration. The output of an iteration is production-ready code.
  • The team meets daily to synchronize on progress and to highlight any issues that are impeding progress.
  • At the end of each iteration a retrospective review is carried out by the team to establish opportunities for improvement – this is how the team ‘inspects and adapts’ to continuously improve their processes.

Key Role Differences:

  • Product Owner: The product owner owns and manages the product backlog.  In addition to traditional product management responsibilities, product owners are an integral part of the team, and are involved in all aspects of the process from release and iteration planning, to daily standup meetings, acceptance of completed user stories, and participation in retrospectives and demos.
  • Scrum Master: The scrum master is not the project manager, but rather fulfills the role of scrum process expert, guiding the team in all aspects of scrum
  • Team: The team is integrated with all the functions and expertise necessary to deliver a fully shippable increment.

Elements of Scrum

There is a common misunderstanding that scrum is an invitation to abandon process. An effective implementation requires a level of discipline that some may not fully appreciate, and which most teams will take time to adjust to.

Generic Scrum

The generic framework is quite simple and can be learned and put into practice very quickly by small, co-located teams. Minimal tooling is required for planning and tracking releases and iterations (scrumboards in the team standup room, or spreadsheets). A full-time product owner embedded within the team, and available for planning and attending daily stand-ups, means that requirements can be defined and translated into user stories without a lot formality, and that changes can be discussed and agreed quickly.

A framework for Continuous Improvement

A framework for Continuous Improvement

There is one universal framework but no single universal process that can be specified in detail. The situation for each team is unique in terms of product, technology and organization right down to the individuals on the team. Each team is tasked with continuously evaluating the results of their efforts and for identifying opportunities to improve.

In the following chapters we will be discussing approaches for scaling the generic framework to support the realities of large-scale software construction, and how to achieve this while preserving agility and the underlying lean principles behind scrum.

 


2.3 Large Projects – Setting The Context

First, what do we mean by a ‘Large Project’? Rather than try to define this in terms of budget, staffing levels, code size, or complexity, I would prefer to characterize large projects as those carried out by typical large software companies. For example, any list of top software vendors would likely include:

  • Microsoft
  • IBM
  • Oracle
  • SAP
  • Computer Associates
  • Symantec
  • Nintendo

Additionally, many companies who do not sell software directly, develop very significant amounts of software for embedding or bundling with their core products or services. Notables would include:

  • Apple
  • Cisco Systems
  • Motorola
  • Google
  • Facebook
  • General Electric
  • Boeing

One big difference in how these companies run the front end of their development projects from the methods often cited in the Agile/Scrum literature is that these companies do not need to ‘discover requirements’. These companies operate a strategy that is either:

  • Roadmap-driven, (certainly true of technology leaders), or
  • Opportunity-driven (or ‘service-oriented’ – requirements derived from ‘RFP’s or customer contracts)

Nearly all of these companies will have mature product development methodologies and lifecycle models, and may even be subject to regulatory or quality management system standards. They also typically are organized around multiple independent functions – Engineering, Product Management, Manufacturing, Sales & Marketing and Customer Support – all of whom are involved in the planning of any new product release.

Finally, most will have development teams that are distributed across multiple sites, countries and timezones.

To summarize, large projects are frequently characterized by:

  • Roadmap- or RFP-driven lifecycles
  • Mature development processes
  • Organized with multiple independent functions
  • Regulatory/Compliance obligations
  • Distributed, multi-site development teams

This is the context within which many large companies embark upon the pursuit of an agile development framework.


Requirements Traceability

The marketing or product management team have produced something called a “PRD”, or product requirements document, the contents of which they have promised to a customer. Meanwhile the development team have constructed a product backlog, which has been been captured in an agile management tool like Rally or JIRA. You look in the agile tool for the current iteration and you see all sorts of records: Epics, User Stories, Sub-Stories, Tasks…and you wonder: which of the customer requirements are we delivering in this iteration?

Thus begins the customer-developer communication gap. Depending on the complexity of the system or application being developed, the requirements may need multiple levels of decomposition to get down to the user story level of granularity. For example: “As the pilot of the space shuttle, I wish to dock with the international space station” is not a user story. It is certainly not something that could be delivered in a 2-week iteration. On the other hand, a story like: “activate the ready-to-hook sensors”, which may be an actual 1-iteration user story, but may resemble nothing like the parent-level requirement to dock the shuttle, and without seeing the hierarchy, there is no way to assess its contribution to the overall requirement.

In a large project we may end up with thousands of user stories. Furthermore, the development team may add many so-called ‘infrastructure stories’ that have no direct relevance to the customer, and frequently complicate the reporting of PRD-level progress. Assuming we do not want our backlog to be an unstructured list with thousands of items, we might organize it around a generic requirements hierarchy based on something like:

  • Themes: Chapter headings in the PRD
  • Epics: Feature Requirements (customer-facing statements of need)
  • Parent Stories: Technical Requirements (feature technical requirements and constraints)
  • User/Component Stories: Implementable in a single iteration

Fig. 1 Generic Requirements Hierarchy

I’m calling the layer below the epics ‘Parent Stories’ (we could also use terms like ‘System Stories’ or ‘Master Stories’). This is the level of detail one would typically see in a traditional Software Functional Specification (SFS). I’m calling them parent stories because it may take one or more levels of further decomposition to get to the level of granularity that is appropriate for an iteration. From a planning perspective, Epics => Parent Stories belongs within the realm of release planning, while Parent Stories => User Stories is carried out in iteration planning.

The first step in developing a backlog that is traceable to a PRD is to get the PRD-level requirements enumerated, meaning each requirement is assigned a unique identifier. To do this a requirements management tool could be used to auto-generate the identifiers, or a spreadsheet could be used. Eventually we will end up with a set of requirements that is structured something like:

The next step is to get the PRD into the agile management tool. Some tools (e.g. Rally) support multiple levels of hierarchical decomposition which makes it mush easier to visualize the structure of the requirements. Other tools (e.g. JIRA) have limitations on the depth of any hierarchy. In both cases however, we need to do some configuration work in order to achieve 2 fundamental goals in support of traceability:

  1. Unambiguous identification of PRD items
  2. The ability of search and filter for PRD items

To support the above we need to add a couple of things to our agile tool:

  1. Use the “tag” attribute to identify those items which are from the PRD
  2. Addition of a user-defined field (most tools support this) to store the PRD reference id

Having done this our tool, when loaded with a mixture of PRD and non-PRD items will display something like:

Finally, reporting progress against completion of the PRD becomes straightforward. Most tools have custom views which can be constructed using specific fields or tags. Some also have query builders allowing you to create a query like:

select * where project = “XYZ” and sprint = “Sprint-3″ and tag contains “PRD” and status = “Accepted”


Agile Requirements Management

There are several reasons why some companies want to have a disciplined requirements management process:

  • The organization needs to comply with regulatory or quality management system standards, like ISO9001 or TL9000, and is subject to regular audits to verify compliance.
  • The organization builds software to contract and needs traceability mechanisms to ensure that every requirement has been delivered and validated.
  • The organization develops large scale software, and while embracing the principle of “working software over documentation”, is not confident that post-it notes are a reliable way of managing thousands of requirements.

Thus the organization needs a way of defining user requirements, tracing them through the development process, and managing changes. Documentation is certainly not the goal, but can be a necessary part of the process to achieve the goal of guaranteeing the needs of customers and stakeholders have been met. Just as doing iterations is not the same as being agile, there is no contradiction in being ISO9001-compliant and operating a highly agile development process – continuous integration, unit test automation with coverage measurement, and highly automated system and regression testing.

The process/tools we use to deliver an effective requirements management process must meet 2 key goals:

  1. Demonstrate Traceability
    • At minimum demonstrate upstream traceability between the customer-facing product requirements (“PRD”), or the customer contract, and the internal Epics and User Stories that the development team is building.
    • Ideally also show down-stream traceability between user stories and test cases (and software bugs) so that test coverage and delivered quality of the requirements can be measured and quantified
  2. Provide Structure
    • With many of the commercial agile tools it is very difficult to see the forest from the trees. When I look at the user story backlog for a product in a tool, I would like to see something that resembles the PRD, not an unstructured list of items that leaves me wondering if we have captured everything in the contract.

In Fig. 1 below we have (in a spreadsheet) the contents page from the PRD/Contract, with user stories created for each feature. This layout provides both traceability and structure.

Figure 1. Requirements Traceability and Structure

Agile Testing

The entire agile testing philosophy is based on the goal of having ship-ready code at the end of each iteration, where ‘ship-ready code’ means: 100% feature tested, 100% system/performance/stress tested, and zero open bugs.

This sounds like a very tall order, requiring a fundamentally different approach to the traditional practice of development hand-offs (typically of half-baked code), to the test organization. There is simply not enough time for this approach to work if we accept the above-stated objective. Here is the key point:

Agile iterations are not mini-waterfalls

Feature and system testing must happen concurrently with development, and for this to work, the development team must be making  available nothing but clean code to the testers.


Notice also how the agile approach significantly reduces project risk by avoiding the accumulation of unresolved defects – all defects are fixed within the iteration in which they are discovered.

Let’s break this down to see what is required for this to work in practice. Agile testing starts as soon as the first User Story is declared done (not at the end of the sprint!). But for this approach to have any chance of success, re-work must be minimized. By re-work we mean the traditional test and bug-fixing cycle, characteristic of waterfall development, that starts with the hand-off from development to the test organization. There are many definitions of ‘done’ for a user story, but at minimum this means:

  • Code clean compiles with all static analysis warnings removed
  • Code reviewed, with all review issues resolved
  • Story has been unit tested, and ideally units tests are automated
  • Test coverage based on unit testing meets some minimum threshold
  • Code and automated unit tests checked into build system, and system builds and passes all unit tests
  • Build passes all predefined build tests

Next, the test team verifies the user story based on its defined acceptance criteria. The majority of stories should be passing at this point. The manufacturing analogy is the production ‘yield’, and we should be striving for the highest possible yield, say > 90%. If the yield is low (and the corresponding re-work high), then we need to dig into the reasons for this, identify root causes, and apply corrective actions to drive the yield higher. Clearly, this will not happen overnight, and may require multiple iterations, if not releases, to get there. There are a couple of additional prerequisites that go along with getting to a high first-pass success rate:

  • A continuous integration environment with a high degree of automation of both the unit test and build sanity level
  • A high degree of system test automation
  • A continuous improvement mindset where the team routinely dissects test failures and institutes actions to push the bar higher for first-pass test success.

Each of the above points merits its own discussion which we will get to in future articles.