Category Archives: 3. Agile Requirements Management

Managing Large Complex Backlogs


  • how to properly create user stories for complex systems
  • how to derive user stories from traditional customer or product requirements documents
  • how to ensure traceability between derived user stories and their parent requirements
  • how to measure progress in delivering the customer-facing product functionality

If our project is small enough we may start by creating a backlog populated with user stories. User stories are units of customer facing functionality that can be delivered within a single iteration. The scrum literature is full of examples like:

  • As a new user I can self-register
  • As a member I can login using the credentials supplied at registration
  • As a system administrator I can browse the list of registered users
  • As a system administrator I can search for registered users
  • As a system administrator I can disable a user’s account

A small release may have up to a few hundred of these types of user stories. The release backlog may be processed by one or more scrum teams who will implement the entire backlog over the course of several sprints.

At the next level of complexity the initial set of release requirements may not be defined with sufficient granularity that they can be individually implementable in a single iteration. Scrum refers to these requirements as ‘epics’. Epics are backlog items that must be broken down further before they can be delivered in a single iteration. If the epic list is larger than a few dozen items, this epic list could easily expand into several hundred items at the user story level. Hence it’s going to be important to impose some structure, otherwise we run the risk of losing sight of the forest in a vast expanse of trees.

Feature Categories

Grouping epics by feature category is one simple way of organizing a long list. Let’s say our product is a network router, typical feature categories might be:

  • Routing Protocols
  • QoS
  • Security
  • Management Features

All of the features (epics) to be delivered in the release fit within these categories. Each feature/epic is then decomposed further into units of functionality (user stories) that can be delivered in a single iteration:

A Structured Backlog

Note that this schema: Feature Category > Feature (Epic) > User Story is completely arbitrary. Furthermore, depending on the application, you may need more than 3 levels of granularity to specify and organize the customer’s requirements, and to get down to a level that can be delivered in a single iteration. There is no universal or standard way of doing this, and scrum does not prescribe how this should be done. Each organization or development team will need to develop the approach that best meets their needs. Let’s not forget the INVEST criteria for a good user story to help us determine if we have drilled down to the appropriate level of detail:

  • Independent
  • Negotiable
  • Valuable
  • Estimable
  • Small
  • Testable
Letter Meaning Description
I Independent The user story should be self-contained, in a way that there is no inherent dependency on another user story.
N Negotiable User stories, up until they are part of an iteration, can always be changed and rewritten.
V Valuable A user story must deliver value to the end user.
E Estimable You must always be able to estimate the size of a user story.
S Sized appropriately or Small User stories should not be so big as to become impossible to plan/task/prioritize with a certain level of certainty. A user story must be sized such that it can be fully delievered (to the definition of DONE) in a single iteration.
T Testable The user story or its related description must provide the necessary information to make test development possible.




Print Friendly

Requirements Traceability

The marketing or product management team have produced something called a “PRD”, or product requirements document, the contents of which they have promised to a customer. Meanwhile the development team have constructed a product backlog, which has been been captured in an agile management tool like Rally or JIRA. You look in the agile tool for the current iteration and you see all sorts of records: Epics, User Stories, Sub-Stories, Tasks…and you wonder: which of the customer requirements are we delivering in this iteration?

Thus begins the customer-developer communication gap. Depending on the complexity of the system or application being developed, the requirements may need multiple levels of decomposition to get down to the user story level of granularity. For example: “As the pilot of the space shuttle, I wish to dock with the international space station” is not a user story. It is certainly not something that could be delivered in a 2-week iteration. On the other hand, a story like: “activate the ready-to-hook sensors”, which may be an actual 1-iteration user story, but may resemble nothing like the parent-level requirement to dock the shuttle, and without seeing the hierarchy, there is no way to assess its contribution to the overall requirement.

In a large project we may end up with thousands of user stories. Furthermore, the development team may add many so-called ‘infrastructure stories’ that have no direct relevance to the customer, and frequently complicate the reporting of PRD-level progress. Assuming we do not want our backlog to be an unstructured list with thousands of items, we might organize it around a generic requirements hierarchy based on something like:

  • Themes: Chapter headings in the PRD
  • Epics: Feature Requirements (customer-facing statements of need)
  • Parent Stories: Technical Requirements (feature technical requirements and constraints)
  • User/Component Stories: Implementable in a single iteration

Fig. 1 Generic Requirements Hierarchy

I’m calling the layer below the epics ‘Parent Stories’ (we could also use terms like ‘System Stories’ or ‘Master Stories’). This is the level of detail one would typically see in a traditional Software Functional Specification (SFS). I’m calling them parent stories because it may take one or more levels of further decomposition to get to the level of granularity that is appropriate for an iteration. From a planning perspective, Epics => Parent Stories belongs within the realm of release planning, while Parent Stories => User Stories is carried out in iteration planning.

The first step in developing a backlog that is traceable to a PRD is to get the PRD-level requirements enumerated, meaning each requirement is assigned a unique identifier. To do this a requirements management tool could be used to auto-generate the identifiers, or a spreadsheet could be used. Eventually we will end up with a set of requirements that is structured something like:

The next step is to get the PRD into the agile management tool. Some tools (e.g. Rally) support multiple levels of hierarchical decomposition which makes it mush easier to visualize the structure of the requirements. Other tools (e.g. JIRA) have limitations on the depth of any hierarchy. In both cases however, we need to do some configuration work in order to achieve 2 fundamental goals in support of traceability:

  1. Unambiguous identification of PRD items
  2. The ability of search and filter for PRD items

To support the above we need to add a couple of things to our agile tool:

  1. Use the “tag” attribute to identify those items which are from the PRD
  2. Addition of a user-defined field (most tools support this) to store the PRD reference id

Having done this our tool, when loaded with a mixture of PRD and non-PRD items will display something like:

Finally, reporting progress against completion of the PRD becomes straightforward. Most tools have custom views which can be constructed using specific fields or tags. Some also have query builders allowing you to create a query like:

select * where project = “XYZ” and sprint = “Sprint-3″ and tag contains “PRD” and status = “Accepted”

Print Friendly

Agile Requirements Management

There are several reasons why some companies want to have a disciplined requirements management process:

  • The organization needs to comply with regulatory or quality management system standards, like ISO9001 or TL9000, and is subject to regular audits to verify compliance.
  • The organization builds software to contract and needs traceability mechanisms to ensure that every requirement has been delivered and validated.
  • The organization develops large scale software, and while embracing the principle of “working software over documentation”, is not confident that post-it notes are a reliable way of managing thousands of requirements.

Thus the organization needs a way of defining user requirements, tracing them through the development process, and managing changes. Documentation is certainly not the goal, but can be a necessary part of the process to achieve the goal of guaranteeing the needs of customers and stakeholders have been met. Just as doing iterations is not the same as being agile, there is no contradiction in being ISO9001-compliant and operating a highly agile development process – continuous integration, unit test automation with coverage measurement, and highly automated system and regression testing.

The process/tools we use to deliver an effective requirements management process must meet 2 key goals:

  1. Demonstrate Traceability
    • At minimum demonstrate upstream traceability between the customer-facing product requirements (“PRD”), or the customer contract, and the internal Epics and User Stories that the development team is building.
    • Ideally also show down-stream traceability between user stories and test cases (and software bugs) so that test coverage and delivered quality of the requirements can be measured and quantified
  2. Provide Structure
    • With many of the commercial agile tools it is very difficult to see the forest from the trees. When I look at the user story backlog for a product in a tool, I would like to see something that resembles the PRD, not an unstructured list of items that leaves me wondering if we have captured everything in the contract.

In Fig. 1 below we have (in a spreadsheet) the contents page from the PRD/Contract, with user stories created for each feature. This layout provides both traceability and structure.

Figure 1. Requirements Traceability and Structure

Print Friendly