Category Archives: 2. Fundamentals of Agile and Scrum

Fundamentals of Agile and Scrum

2.1 Foundational Agile Practices

Agile software development requires 3 foundational sets of practices to be in place. Doing iterations alone does make you agile unless the output of your iterations is production-ready code. But how do you get production-ready code for a large complex software product out of a 2-week iteration?

We will spend a lot of time discussing Scrum in this book. Scrum has become the de-facto best framework for doing iterations. But scrum by itself does not produce deployable code as its output, unless backed up by 2 prerequisite sets of practices, namely Continuous Integration and Test Automation.

Take for example a communications product that resides at the core of a high-speed network. This product not only executes multiple stacks of routing protocols, and needs to be highly configurable and manageable, but also interacts with multiple external devices and products, against all of which it must be tested with to ensure interoperability and tolerance to many types of adverse events in the network. In order to do this, thousands of test cases must be executed, including performance and stress test cases, which traditionally may have taken months to complete. A very high degree of automation is required to get this done on virtually every change that is submitted.

But there is another prerequisite. Traditional construction has involved multiple developers working on private code branches. Changes from each developer are coded/unit tested on their own private branch and then checked into a trunk line from where the next build is produced. New builds must be ‘sanity tested’ for integrity, and then made available to the development team for verification of their changes before the system test team takes over. The process of getting a new build to system test may take several days, and many organizations working in this mode may typically make one new build per week. This is the ‘big batch’ approach to software construction which simply is not consistent with delivering production code in 2-week iterations. Changes from developers must be submitted continuously, and these changes must be of very high quality to ensure minimal re-work, and minimal accumulation of an open bug backlog. We will elaborate on the subject of Continuous Integration in a subsequent chapter.

Whereas scrum and be taught and implemented in a fairly short time frame, building an effective CI system, and automating large legacy test suites may require significant financial investment and time to get into place.

In summary, agile software development must comprise elements of:

  • Iterative development e.g. Scrum
  • Continuous Integration
  • Test Automation

We will be elaborating on each of these topics in subsequent chapters.




Print Friendly

2.2 Scrum Basics

In traditional, so-called waterfall software development, the starting point for any project planning is a set of product requirements (a “PRD”). All of the requirements must be defined, reviewed and approved up-front before planning or development can begin. Additionally, the development team may produce a software functional specification (“SFS”) that takes each requirement in the PRD and elaborates it into more detail, adding technical requirements and constraints. Both of these documents are then used as the basis of planning the subsequent development activities required to deliver the content specified in the PRD. In order to derive a development plan and schedule, every task required to construct and test every feature must be estimated. Once development has been completed, there is a handoff of partially tested features to a test or “QA” organization, followed by several weeks or months of testing and bug-fixing. The length of this phase will be highly dependent on the quality of the code at the point of handoff. Finally, at some point, a decision to release the product to the market must be made, and this is where the usual tradeoffs between schedule, features and quality are fought over. The delivered software is usually accompanied by a set of ‘release notes’, listing all of the features that were actually delivered, plus all of the bugs that the team did not have time to fix.

Software development is referred to as an ‘empirical process’ by advocates of scrum. The output of almost every stage of software development is impossible to predict with precision, and each step produces defects that must be re-worked. This so-called ‘plan-driven’ paradigm has been demonstrated time and again not to be amenable to the level of precision expected from other forms of production such as manufacturing and construction. The output of a modern production line is usually specified to very high standards, such as ‘six sigma’ (less that 4 defects per million units). It is difficult to imagine any large software project coming remotely close to this standard. Scrum is a response to this problem.

We do not want to spend much time here listing the disadvantages of traditional waterfall development, however, we should note that the waterfall and it’s variants have delivered many successful software products over the past several decades, despite all of the known limitations and attendant budget overruns, quality problems, and stressed out development teams. Let us also note that there are many project activities beyond the boundaries of the scrum framework that need to occur for the release of a commercially successful software product. Product development teams in large companies operate within a broader ecosystem that includes multiple other functions which must work together to define, develop, market and support new products. Teams that want to adopt agile development practices must recognize that this cannot be accomplished in a vacuum, and must address how to achieve an effective integration with other inter-dependent functions.

Key Process Differences:

  • Requirements are not defined in detail up-front. Requirements are continuously elaborated into technical details throughout the project. The goal is to defer making potentially irreversible decisions until the last possible moment.
  • Release-level planning is based on relative sizing of requirements.
  • Detailed task-level estimating is only performed at the iteration level.
  • There are no hand-offs between developers and testers, testing runs concurrently with design and coding.
  • Each iteration is completed to an agreed definition of ‘done’, which always includes the requirement that all bugs are fixed within the iteration. The output of an iteration is production-ready code.
  • The team meets daily to synchronize on progress and to highlight any issues that are impeding progress.
  • At the end of each iteration a retrospective review is carried out by the team to establish opportunities for improvement – this is how the team ‘inspects and adapts’ to continuously improve their processes.

Key Role Differences:

  • Product Owner: The product owner owns and manages the product backlog.  In addition to traditional product management responsibilities, product owners are an integral part of the team, and are involved in all aspects of the process from release and iteration planning, to daily standup meetings, acceptance of completed user stories, and participation in retrospectives and demos.
  • Scrum Master: The scrum master is not the project manager, but rather fulfills the role of scrum process expert, guiding the team in all aspects of scrum
  • Team: The team is integrated with all the functions and expertise necessary to deliver a fully shippable increment.

Elements of Scrum

There is a common misunderstanding that scrum is an invitation to abandon process. An effective implementation requires a level of discipline that some may not fully appreciate, and which most teams will take time to adjust to.

Generic Scrum

The generic framework is quite simple and can be learned and put into practice very quickly by small, co-located teams. Minimal tooling is required for planning and tracking releases and iterations (scrumboards in the team standup room, or spreadsheets). A full-time product owner embedded within the team, and available for planning and attending daily stand-ups, means that requirements can be defined and translated into user stories without a lot formality, and that changes can be discussed and agreed quickly.

A framework for Continuous Improvement

A framework for Continuous Improvement

There is one universal framework but no single universal process that can be specified in detail. The situation for each team is unique in terms of product, technology and organization right down to the individuals on the team. Each team is tasked with continuously evaluating the results of their efforts and for identifying opportunities to improve.

In the following chapters we will be discussing approaches for scaling the generic framework to support the realities of large-scale software construction, and how to achieve this while preserving agility and the underlying lean principles behind scrum.


Print Friendly

2.3 Large Projects – Setting The Context

First, what do we mean by a ‘Large Project’? Rather than try to define this in terms of budget, staffing levels, code size, or complexity, I would prefer to characterize large projects as those carried out by typical large software companies. For example, any list of top software vendors would likely include:

  • Microsoft
  • IBM
  • Oracle
  • SAP
  • Computer Associates
  • Symantec
  • Nintendo

Additionally, many companies who do not sell software directly, develop very significant amounts of software for embedding or bundling with their core products or services. Notables would include:

  • Apple
  • Cisco Systems
  • Motorola
  • Google
  • Facebook
  • General Electric
  • Boeing

One big difference in how these companies run the front end of their development projects from the methods often cited in the Agile/Scrum literature is that these companies do not need to ‘discover requirements’. These companies operate a strategy that is either:

  • Roadmap-driven, (certainly true of technology leaders), or
  • Opportunity-driven (or ‘service-oriented’ – requirements derived from ‘RFP’s or customer contracts)

Nearly all of these companies will have mature product development methodologies and lifecycle models, and may even be subject to regulatory or quality management system standards. They also typically are organized around multiple independent functions – Engineering, Product Management, Manufacturing, Sales & Marketing and Customer Support – all of whom are involved in the planning of any new product release.

Finally, most will have development teams that are distributed across multiple sites, countries and timezones.

To summarize, large projects are frequently characterized by:

  • Roadmap- or RFP-driven lifecycles
  • Mature development processes
  • Organized with multiple independent functions
  • Regulatory/Compliance obligations
  • Distributed, multi-site development teams

This is the context within which many large companies embark upon the pursuit of an agile development framework.

Print Friendly

Ideal Iteration Length – The Science

What is the underlying science behind iteration size selection?

For some background reading I refer readers to Donald Reinertsen’s book: The Principles of Product Development Flow. One of the book’s chapters is about batch size, and sets out a dozen ‘principles’ that promote the merits of smaller batch sizes. In the context of software development, Waterfall executes a software release in a single batch, whereas scrum breaks up the requirements for the release (the Release Backlog) into a set of smaller batches, and implements these in a sequence of Sprints. The question we have is: what is the optimum size for a sprint?

Much research has been done on determining the optimum batch size in the context of manufacturing logistics. The calculation of ‘optimum’ batch size – usually called Economic Batch Quantity (EBQ) – is based on the relationship with annual demand, machine set-up cost, and inventory carrying cost per item. Specifically:

EBQ = sqrt(2*Annual Demand*Setup Cost/Inventory Holding Cost per Unit)

where the inventory holding cost is the overhead cost associated with each item, and generally increases with the size of the batch, and the setup cost per item generally decreases as a function of batch size. This can be illustrated in the following chart:

Fig. 1: Optimum Batch Size

Optimum Batch Size

For manufacturing operations the impact of large batch sizes on cost and throughput may be fairly obvious: large batches increase work-in-process inventory levels, materials lead-times, and finished good stock levels. But more importantly, large batches drive more variability, and more variability means more re-work, more cost and delays.

We will look at exactly what this means for software, but before that, what we really care about is maximizing throughput, i.e. how do the maximize the number of User Stories (or Story-Points) delivered per sprint? The throughput chart is simply the inverse of the cost chart above, and will be an inverted U-Curve.

Let’s separate the production costs of User Stories into direct costs – those that contribute directly to the definition, design, coding and testing of each story, and indirect costs – those that represent overhead.

Direct Costs:

User Story Production Costs (Direct)

Indirect Costs:

User Story Production Costs (Indirect)

The fixed costs are associated with the fundamental development tasks that must be carried out to deliver the set of User Stories in the iteration: design, coding and testing. Building 100 user stories in a batch as opposed to building 1 at a time yields obvious economies of scale. (Think of test automation as an example). However, as the size of an iteration goes up, so too does the amount of indirect cost, examples of which include dealing with more complex branch merges, fixing broken builds, scrubbing long lists of bugs, plus all of the data collection and status reporting that goes with this. The simple fact of having to deal with an iteration of larger scope and complexity drives up the amount of overhead one encounters in a project. Thus the overall cost/story vs. iteration size graph will be quite similar to that in Figure 1.

The larger the batch, the fewer opportunities there are to inspect and adapt, and the more we are exposed to WIP and re-work. It is up to each team to find their own sweet spot for the size of an iteration. In scrum, we talk about getting to an ideal ‘velocity’, or the optimum number of Story Points per Sprint. Teams should pursue this objective by searching for and eliminating all sources of delay in their development process.

Print Friendly

Ideal Iteration Length – A Survey

Recently I put the question of the rationale for a max sprint length of 30 days to one of my LinkedIn groups. Here are the responses:

  • The idea is that anything over 30 days is too large to effectively break down and estimate properly and for everyone to keep that model in their head. It also keeps you focused on the quick evolution of the team to learn through regular small mistakes, instead of remembering what went wrong months ago.
  • The idea is to fail fast and more than 1 month is not fast enough
  • The shorter the sprint and the smaller the stories provides the ability to test early, deliver usable functionality incrementally in small batch sizes, with lower risk.
  • Delivering working software in every sprint over 30 days is called a project. If you keep in mind that the content of the sprint is frozen and that a scrum team should be around 7+-2 people you freeze 7-9 man months or more of work in advance without allowing the customer or product owner to have a say or see a working product in between. This is a huge investment with high risk of error. It also prevent you from reevaluating your way of working often and review your failures and learn from them continuously.
  • For failing faster, 30 days is too long to throw away (worst case).
  • At the beginning, the requirements for that Sprint have been froozen. So the question of Sprint length has also to do with the ability of the Product Owner and Scrum Master to keep these requirements untouched by external influences and to negotiate new requirements onto the Product Backlog for review on the next planning meeting. If you are getting alot of ‘this just can’t wait’, then shorter Sprints are better.
  • You’ll lose the pressure if you start a sprint longer than 30 days. I’ve experienced a 10 days sprint as the best. Try a discussion with the members of a possible fit after you’ve estimated all tasks. Is it realistic to “done” the entire sprint? It’s a challenge to start with a balanced sprint with natural pressure. 30 days sounds a little bit like a lazy PO.
  • I would also add to @Jurrian that for us the 10 days is the right amount. In addition, this helps with user exceptions for urgent changes or issues. It’s an easier sell if I tell them this sprint is set (or even the next one). At the most they typically only have to wait 3 weeks for us to get started on something for them.
  • Dina wisely commented that a shorter sprint duration reduces risk. I would add that shorter sprints force more frequent synchronization and convergence of the different workstreams occurring within a project. If there is any incompatibility between, say, modules of code it would be discovered sooner (and presumably be fixed sooner) thus reducing the risk of having a much bigger incompatibility problem that might arise from longer sprints. I believe this echoes Sachin’s comment that it’s better to fail fast because big problems discovered later than sooner is a recipe for blowing the schedule.
  • We have tried two week sprints and three weeks sprint and the feeling was that three weeks was the perfect length. When we did two week sprints it felt like we where always starting or ending and the teams stress level was too high.
  • Not adding anything really new, but this also comes down to how the human brain works. If you have a term paper due at the end of the semester, you tend to start working on it at the end of the semester. If you had an outline due three weeks into the semester, you’d start working on your Term Paper two weeks into the semester. Slicing into chunks (sprints) helps to prevent all the work slamming down at the end. This goes straight to the Agile Principle of “promote sustainable development.”
  • My Agile coach always recommends starting with one week sprints for new Scrum projects. When you are just starting, how you use Scrum is as much or more important than what is being built. With one week sprints you have a shorter time between learning cycles to adjust things like estimating, how many story points the team takes on and so on.
  • There are three reasons. Two are explicit in Scrum the other isn’t mentioned, but is one of the foundations on which Scrum is based. The explicit ones are feedback and enforced view of reality. The second is removing delays.
  • You want quick feedback. Longer than 30 days does not force you to get it. But feedback exists in many forms. There is the time from getting a request until delivering it. Time from starting a story until it is complete. This is where lessons are learned quickly. I personally don’t like to fail fast, I prefer to learn fast. One does not need to write code to discover the customer doesn’t want it, one needs to write a little where the customer is clear, show it to them, and enable the customer to learn a little bit more. But, if you are going to fail, do it quickly.
  • By enforced view of reality, I mean that things will show up to make it difficult to deliver value in 30 days. For example, customers may not want to talk to you that often, build processes may be inefficient, testing and coding may be too separated from each other. By having a time-box in which your work is intended to get completed, those things that work against you will become more visible and painful. These are impediments to your work actually, and instead of avoiding them by making the sprint length longer, you actually need to shorten the sprint to expose these even more. It is also harder to ignore them – if you haven’t tested your code by sprint’s end – it’ll be obvious and irrefutable. Both of these reasons actually tend to call for even shorter than 30 days sprints. Most successful Scrum teams I know of use 1-2 week sprints. In fact, I’d go so far as saying teams with 3-4 week sprints are showing a “smell” of not being engaged enough in solving their problems (not always true, but often true). The third reason, which in many ways is the biggest one, is removal of delays. Since about 2004, I have been claiming that Scrum is a weak implementation of Lean principles. I say “weak” because Scrum does not deal with optimizing the whole and it leaves out a lot of lean-education and management. But one of the key tenets of Lean is to remove delays so value can be delivered quickly. This is critical because delays between steps of work literally create more work ( see Real Tenets of Lean; Avoid Creating Waste by Eliminating Delays at a short video that is very well worth watching if I say so myself). The time-boxing of Scrum requires teams to complete their work quickly, it encourages swarming as well. This all reduces delays and therefore avoids incurring work that doesn’t need to happen.
  • The combination of feedback, awareness and removal of delays drives us to have shorter feedback loops until the overhead of the sprint overcomes the value achieved by them. For most teams this will be 1-2 weeks. Some teams that discover they don’t need the discipline of the time box will abandon it completely and move to Kanban.
    I might add that a simple value stream analysis will show most that “the shorter Sprint the better”. Scrum contains no technique or method for optimizing end-to-end, and it should not. The retrospective might uncover such a problem, but I generally advice to use Lean thinking to address end-to-end optimization explicitly.
  • On my projects we started off with 2 week sprints and I think this is a good duration to work with.
  • 30 days is also fairly typical ‘management reporting interval’. A Sprint longer than 30 days means that management may not get a ‘status update’ for two months.
  • With experienced teams and a well-defined product backlog, a 30 day sprint may be fine (not my preference). But when the teams are newly formed, new to Scrum or when the product backlog is very dynamic, it’s better, as someone pointed out, to fail earlier and adapt sooner.
  • A two-week sprint is my preference. Just long enough to develop some rhythm and velocity, but not so long that you risk going down the wrong road for a month.
  • 30-days is what is mentioned in early Scrum literature, notably Schwaber’s writings. Nowadays, Jeff Sutherland speaks about most teams worldwide practicing two-week Sprints, for all the reasons mentioned above (shorter feedback cycles, shorter learning cycles, shorter time to see visible results, and quicker visibility of pain-points that need to be addressed).
  • 30 days matched traditional development teams that were new to scrum, or where older technology was not nimble enough for rapid development for all the mentioned reasons especially quick review and feedback. Even with a 30 day sprint cycle, I have usually obtained feedback in shorter cycles before being fully accustomed to scrum. Maybe as technology and teams get more progressive we will see shorter sprint cycles.
  • All above answers are great. I will amend that by having frequent and not too far away reviews that show what was built (the increment) then you are being transparent and providing the visibility to the stakeholders, so everything that was mentioned (risk, done, value) are all observed and demonstrable. Also by repeating the cycle and having a time for inspect and adapt you can become agile. If you do them more than 30 days the people may not remember what happened and will not effectively adapt. BTW 30 days is too much for us, my experience that most teams use a 2 weeks (10 days) sprints.
  • Start with sprints as short as possible… one week or two week sprints. do not overpromise the productowners but in first place underpromise as you need to learn on what you can deliver on shippable products. If one week or two week sprints are not possible to deliver shipable products you can extend the dime of a sprint to for example 3 weeks or 4 weeks.
  • A 2 week sprint has effictevely just 9 days to build and test deliverables as you also need to reserve time for backlog grooming, sprint planning, sprint review and retrospectives. When you start your first sprint just underpromise. The first sprint needs to be succesfull to the productowner to get respect to each other.
  • Planning for 30 days is too long in my opinion. You really cannot promise deliverables and productowners need to wait too long.
Print Friendly