Agile Metric Estimation

posted in: Agile | 0

Estimation Dilemma

During the Egyptian revolution in 2011, thousands of people went to Tahrir Square in Cairo for mass demonstrations against the regime. The number of people in Tahrir Square was overwhelming enough but the variation of crowd estimation was even worse. Government news channels claimed only 8000 thousand in Tahrir Square, while others in support of the revolution recorded 1 million 125 times the figure reported!

Figure 1: Egyptian Revolution Tahrir Square

Politics aside, the fact is that crowd estimates can be quite variable. Estimates of the size of the crowd at the royal wedding of Prince William and Kate Middleton, for example, ranged from 500,000 to one million. At Obama’s inauguration ceremony, unofficial government estimates put the crowd at 1.8 million. Others gave estimates closer to one million.

Figure 2:William and Kate’s wedding day crowds

Estimates can be very deceptive and this plays out in our day-to-day working lives.  I recently discovered that one of the news channels used a metric estimation method to estimate how many people were in Tahrir Square. They calculated the size of the venue and the maximum number of people per meter that could be gathered and discovered that the maximum capacity of Tahrir Square was only 500,000 people.

The same estimation issues occur frequently in IT projects. It is not uncommon to have a variety of  estimates on  the same project when estimated by different people delivering the project based on factors such as;

  • Technical experience
  • Risk assessment skills
  • Clarity of the requirements from the client
  • Assumptions
  • Business Domain Knowledge

Cone of Uncertainty

Researchers have compiled the last 60 years’ worth of software projects to measure the accuracy of the project estimates and plotted it over a chart as shown in Figure 3.  The same concept can be applied to agile projects as shown in Figure 4

Figure 3: waterfall cone of uncertainty

Figure 4: Agile cone uncertainty


The chart shows that as time progresses, the estimate gets better. So, if the initial estimate of the project is one year, then your estimate is accurate by plus or minus four. This means the project can be finished in four years or as little as 3 months. However, as the project progresses, the estimate gets more accurate as a result of the development team become clearer about the product, business domain knowledge and more familiar with developing tools and technology used in the project. But our estimation will never be accurate 100% even during the last stages of the project.

The uncertainty around how long exactly the project could take to develop and be deployed into production causes a range of issues for the business such as:

  • Difficulty in predicting the project cost
  • Allocating resources
  • Planning training
  • Planning marketing events

How to Improve our Estimation

Split Big Projects into Smaller Projects

Studies show that agile projects have a much better success rate – almost 3 times as compared to waterfall projects. However, adopting an agile approach in your project doesn’t guarantee a high success rate in your project. As you can see in Figure 5 the success rate of large projects using the agile approach is only 18% compared to 58% success rate in smaller sized projects.

Figure 5: The resolution of all software projects from FY2011–2015 within the new CHAOS database, segmented by the agile process and waterfall method. The total number of software projects is over 10,000.

A large project will usually have a very wide cone of uncertainty, especially at the initial stage of the project. This makes the large project more likely to be challenged or even failed and delivered with much higher cost and time than the initial estimation.

One way of narrowing the uncertainty cone, therefore, is slicing larger projects into smaller ones, where each project is independent and more easily deployed to production, providing quicker business value to the organisation.

One Story Point Does Not Equal to One Day

Many people get confused about the significance of story points in agile projects. Unfortunately, many development teams have the concept that of 1 story point is equal to 1 day. This often means that the project managers also assume 1 story point is equal to 1 day which increases the cone of uncertainty instead of narrowing it down.

The purpose of using story points is to decouple time from effort and to use a different unit of measurement to measure the effort from time. Story points will help to measure the team velocity later on but will not measure how long (in time) it will take to finish a specific feature or user story.

The Power of Relativity

Humans naturally don’t make great estimators. We tend to be either optimists or pessimists and very rarely realists. For example, if you were asked to estimate how long it will take us to walk up all of the buildings as shown in Figure 6 using the stairs

Figure 6: how long will it take us to walk up all of the buildings?

Consider you’ve never climbed these buildings before or aren’t sure how physically fit we are or what types of obstacles you might need to negotiate in the stairwells. Unfortunately, our estimate will never be accurate regardless of which approach we take. . It will be subject to plus or minus 4x based on our cone of uncertainty.

On the other hand, let’s assume the first building will take 100 story points to climb then what’s the estimate to climb the rest of the buildings?

We can estimate the effort of climbing the other buildings as follows:

Building 1 100 Points
Building 2 300 Points
Building 3 250 Points
Building 4 750 Points

Note that all these estimated numbers are relative. Therefore, our estimates will always be relative and not absolute. So, if the first building took 1 hour to climb including rest time, then, how long it will take to climb the other buildings?

Using Fibonacci Sequences Will Make You Better at Estimating User Stories

The Fibonacci sequence (1,2,5,8,13,21,34, 55,….) is based on the golden ratio and it’s important the team stick to the exponential sequence growth during the estimation sessions. The shorter the time span, the more certainty. Longer tasks are more complex and have higher risk factors and time estimates are less precise. This will help the team to encapsulate the risk of large and complex tasks during estimation versus small tasks which carry less risk.

Stop Estimating and Use Metric

Let’s look at a practical example of how to use the agile metric approach to have better estimation, narrowing the cone of uncertainty and increasing the rate of successful projects.

Data to Collect

During each sprint, we will need to collect data to facilitate more accurate agile estimation

Product backlog size

Backlog sizes will change during each sprint as the team will complete work items as well as items will be added or removed including bugs. The product backlog items can be calculated as follows:

Backlog size = (Initial backlog size + item added to backlog + bugs) – (items removed from backlog + completed items)

Committed Sprint Backlog

At the beginning of each iteration, the team will be committed to completing selected items from the product backlog. It’s important to record the sprint size which the team is committed to do.

Completed backlog items

The actual backlog item sizes that are completed during the sprint. This will be an indication of how the team is performing during the iteration.

Cumulative team velocity

Measure team velocity during each iteration to record fluctuations as the project moves. Team velocity can be measured as:

The initial velocity sprint will not be accurate, but as the project progresses the velocity will become more accurate typically after the first 3 or 4 sprints

Items added to the backlog

An item that was added or requested by the business during the project

Item removed from backlog

Similar to an item added to the backlog, the business could request to remove a feature from the backlog or replace it with other features

Bugs added to the backlog

Regardless of the quality control you have in your project, there will always be a bug rate of which usually fluctuates during the project

Number of sprints to complete the project

To measure how many sprints are required to complete all backlog items and complete the project can be  measured as follows

Allocated Percentage for New Bugs & Rework

As the project progresses, the bug rate and effort of rework or redevelopment usually increase. The percentage will be different for each project based on many factors, for example:

  • Existence in coding quality controls like unit testing and integration testing
  • Length of the project
  • Technology and framework updates
  • Quality of solution architect design of the project

Based on statistics, the percentage of effort to develop new user stores versus the effort of rework and bug fixing could look  something like Figure 7

Figure 7: Allocation % of bugs and rework through the project

Weighted sprints to complete

The weight sprint to be completed is based on the allocation percentage of new bugs and rework.

Sprint Metrics Example

For the start of the project, let’s assume we have a sum of 100 story points for backlog items. These items are based on the initial project requirements and analysis of work that is required to build the project foundation. So our metrics could be something like the following

  Sprint 1 Sprint 2  Sprint 3  Sprint 4
Product backlog size 100 94 90 81
Committed Sprint Backlog 10 8 8 9
Completed backlog items 7 8 9 9
Team velocity 7 7.5 8 8.25
Item added to backlog 0 2 1 0
Item removed from backlog 0 0 1 0
Bugs added to backlog 1 2 3 2
Estimated Numbers of sprints to complete the project 13.4 12 10.5 8.9
Allocations % for new bugs, rework 1 0.95 0.90 0.90
Weighted sprints to Complete 13.4 12.6 11.6 9.9


As shown in the previous table, note the following:

  • Team velocity usually improves after 3rd or 4th The team starts becoming more familiar with the used technology with a better understanding of the business domain knowledge as project progresses.
  • Team velocity can be impacted if the team size keeps changing in every sprint. So, it’s important to have a stable team to get accurate results
  • There are no hard rules to allocate the percentage of new bugs and rework, and it will be different from team to team and from project to project.


Every Burndown Chart Has a Story to Tell​

The burndown chart is a great tool to help the team track progress since it shows progress on a daily basis, it helps scrum masters to predict if a team will be able to achieve the target. Also, it’s an indication of how the team is updating their tasks to ensure the accuracy of the data that will be collected during each sprint.

Ideal Chart

The graph below shows the ideal scenario where is actual and ideal lines are in sync and the team was able to finish all committed tasks at the end of the sprint.

Figure 8: Ideal Chart


Early Finish

Actual curve finishing below the ideal curve. If repeated, then you might need to consider increasing the team velocity and commitment to more work.

Figure 9:Early Finish


Unable to Finish All Committed Work

Actual curve finishing above the ideal curve. So, if repeated then you might need to consider decreasing the team velocity and commitment to less work.

Figure 10:Unable to Finish All Committed Work

Large peaks or valleys​

Large peaks in the actual curve could mean many things but the most common scenario is the team not updating their tasks regularly.

Figure 11:Large Peaks or Valleys

More Unexpected Work Discovered​

The jump in the actual burning curve could be because of more work discovered or more urgent bugs that are discovered during the sprint and must be resolved during the sprint. , if repeated then you might need to consider checking your quality control and increasing the allocation percentage of new bugs and rework.

Figure 12:More Unexpected Work Discovered

Stretched Curve

The team stretched towards the end to meet the commitment​

Figure 13:Stretched Curve

Inconsistent Curve

This could mean the team is not consistent through excessive meetings or not updating their tasks regularly.

Figure 14:Not Consistent Curve

Final Thoughts

The main idea of agile metrics is to narrow down the cone of uncertainty of the estimations and to notify the business as soon as possible for more accurate estimations. Importantly, metrics are just one part in building a team’s culture. They give quantitative insight into the team’s performance and provide measurable goals for the team.  This will help the team to be able to get unbiased estimations and to have more realistic estimations based on team effort.

Leave a Reply