Estimation Dilemma
During the Egyptian revolution in 2011, thousands of people went to Tahrir Square in Cairo for mass demonstrations against the regime. The number of people in Tahrir Square was overwhelming enough but the variation of crowd estimation was even worse. Government news channels claimed only 8000 thousand in Tahrir Square, while others in support of the revolution recorded 1 million 125 times the figure reported!
Figure 1: Egyptian Revolution Tahrir Square
Politics aside, the fact is that crowd estimates can be quite variable. Estimates of the size of the crowd at the royal wedding of Prince William and Kate Middleton, for example, ranged from 500,000 to one million. At Obama’s inauguration ceremony, unofficial government estimates put the crowd at 1.8 million. Others gave estimates closer to one million.
Figure 2:William and Kate’s wedding day crowds
Estimates can be very deceptive and this plays out in our day-to-day working lives. I recently discovered that one of the news channels used a metric estimation method to estimate how many people were in Tahrir Square. They calculated the size of the venue and the maximum number of people per meter that could be gathered and discovered that the maximum capacity of Tahrir Square was only 500,000 people.
The same estimation issues occur frequently in IT projects. It is not uncommon to have a variety of estimates on the same project when estimated by different people delivering the project based on factors such as;
- Technical experience
- Risk assessment skills
- Clarity of the requirements from the client
- Assumptions
- Business Domain Knowledge
Cone of Uncertainty
Researchers have compiled the last 60 years’ worth of software projects to measure the accuracy of the project estimates and plotted it over a chart as shown in Figure 3. The same concept can be applied to agile projects as shown in Figure 4
Figure 3: waterfall cone of uncertainty
Figure 4: Agile cone uncertainty
The chart shows that as time progresses, the estimate gets better. So, if the initial estimate of the project is one year, then your estimate is accurate by plus or minus four. This means the project can be finished in four years or as little as 3 months. However, as the project progresses, the estimate gets more accurate as a result of the development team become clearer about the product, business domain knowledge and more familiar with developing tools and technology used in the project. But our estimation will never be accurate 100% even during the last stages of the project.
The uncertainty around how long exactly the project could take to develop and be deployed into production causes a range of issues for the business such as:
- Difficulty in predicting the project cost
- Allocating resources
- Planning training
- Planning marketing events
How to Improve our Estimation
Split Big Projects into Smaller Projects
Studies show that agile projects have a much better success rate – almost 3 times as compared to waterfall projects. However, adopting an agile approach in your project doesn’t guarantee a high success rate in your project. As you can see in Figure 5 the success rate of large projects using the agile approach is only 18% compared to 58% success rate in smaller sized projects.
Figure 5: The resolution of all software projects from FY2011–2015 within the new CHAOS database, segmented by the agile process and waterfall method. The total number of software projects is over 10,000.
A large project will usually have a very wide cone of uncertainty, especially at the initial stage of the project. This makes the large project more likely to be challenged or even failed and delivered with much higher cost and time than the initial estimation.
One way of narrowing the uncertainty cone, therefore, is slicing larger projects into smaller ones, where each project is independent and more easily deployed to production, providing quicker business value to the organisation.
One Story Point Does Not Equal to One Day
Many people get confused about the significance of story points in agile projects. Unfortunately, many development teams have the concept that of 1 story point is equal to 1 day. This often means that the project managers also assume 1 story point is equal to 1 day which increases the cone of uncertainty instead of narrowing it down.
The purpose of using story points is to decouple time from effort and to use a different unit of measurement to measure the effort from time. Story points will help to measure the team velocity later on but will not measure how long (in time) it will take to finish a specific feature or user story.
The Power of Relativity
Humans naturally don’t make great estimators. We tend to be either optimists or pessimists and very rarely realists. For example, if you were asked to estimate how long it will take us to walk up all of the buildings as shown in Figure 6 using the stairs
Figure 6: how long will it take us to walk up all of the buildings?
Consider you’ve never climbed these buildings before or aren’t sure how physically fit we are or what types of obstacles you might need to negotiate in the stairwells. Unfortunately, our estimate will never be accurate regardless of which approach we take. . It will be subject to plus or minus 4x based on our cone of uncertainty.
On the other hand, let’s assume the first building will take 100 story points to climb then what’s the estimate to climb the rest of the buildings?
We can estimate the effort of climbing the other buildings as follows:
| Building 1 | 100 Points |
| Building 2 | 300 Points |
| Building 3 | 250 Points |
| Building 4 | 750 Points |
Note that all these estimated numbers are relative. Therefore, our estimates will always be relative and not absolute. So, if the first building took 1 hour to climb including rest time, then, how long it will take to climb the other buildings?
Using Fibonacci Sequences Will Make You Better at Estimating User Stories
The Fibonacci sequence (1,2,5,8,13,21,34, 55,….) is based on the golden ratio and it’s important the team stick to the exponential sequence growth during the estimation sessions. The shorter the time span, the more certainty. Longer tasks are more complex and have higher risk factors and time estimates are less precise. This will help the team to encapsulate the risk of large and complex tasks during estimation versus small tasks which carry less risk.
Stop Estimating and Use Metric
Let’s look at a practical example of how to use the agile metric approach to have better estimation, narrowing the cone of uncertainty and increasing the rate of successful projects.
Data to Collect
During each sprint, we will need to collect data to facilitate more accurate agile estimation
Product backlog size
Backlog sizes will change during each sprint as the team will complete work items as well as items will be added or removed including bugs. The product backlog items can be calculated as follows:
Backlog size = (Initial backlog size + item added to backlog + bugs) – (items removed from backlog + completed items)
Committed Sprint Backlog
At the beginning of each iteration, the team will be committed to completing selected items from the product backlog. It’s important to record the sprint size which the team is committed to do.
Completed backlog items
The actual backlog item sizes that are completed during the sprint. This will be an indication of how the team is performing during the iteration.
Cumulative team velocity
Measure team velocity during each iteration to record fluctuations as the project moves. Team velocity can be measured as:
The initial velocity sprint will not be accurate, but as the project progresses the velocity will become more accurate typically after the first 3 or 4 sprints
Items added to the backlog
An item that was added or requested by the business during the project
Item removed from backlog
Similar to an item added to the backlog, the business could request to remove a feature from the backlog or replace it with other features
Bugs added to the backlog
Regardless of the quality control you have in your project, there will always be a bug rate of which usually fluctuates during the project
Number of sprints to complete the project
To measure how many sprints are required to complete all backlog items and complete the project can be measured as follows
Allocated Percentage for New Bugs & Rework
As the project progresses, the bug rate and effort of rework or redevelopment usually increase. The percentage will be different for each project based on many factors, for example:
- Existence in coding quality controls like unit testing and integration testing
- Length of the project
- Technology and framework updates
- Quality of solution architect design of the project
Based on statistics, the percentage of effort to develop new user stores versus the effort of rework and bug fixing could look something like Figure 7
Figure 7: Allocation % of bugs and rework through the project
Weighted sprints to complete
The weight sprint to be completed is based on the allocation percentage of new bugs and rework.
Sprint Metrics Example
For the start of the project, let’s assume we have a sum of 100 story points for backlog items. These items are based on the initial project requirements and analysis of work that is required to build the project foundation. So our metrics could be something like the following
| Sprint 1 | Sprint 2 | Sprint 3 | Sprint 4 | |
| Product backlog size | 100 | 94 | 90 | 81 |
| Committed Sprint Backlog | 10 | 8 | 8 | 9 |
| Completed backlog items | 7 | 8 | 9 | 9 |
| Team velocity | 7 | 7.5 | 8 | 8.25 |
| Item added to backlog | 0 | 2 | 1 | 0 |
| Item removed from backlog | 0 | 0 | 1 | 0 |
| Bugs added to backlog | 1 | 2 | 3 | 2 |
| Estimated Numbers of sprints to complete the project | 13.4 | 12 | 10.5 | 8.9 |
| Allocations % for new bugs, rework | 1 | 0.95 | 0.90 | 0.90 |
| Weighted sprints to Complete | 13.4 | 12.6 | 11.6 | 9.9 |
As shown in the previous table, note the following:
- Team velocity usually improves after 3rd or 4th The team starts becoming more familiar with the used technology with a better understanding of the business domain knowledge as project progresses.
- Team velocity can be impacted if the team size keeps changing in every sprint. So, it’s important to have a stable team to get accurate results
- There are no hard rules to allocate the percentage of new bugs and rework, and it will be different from team to team and from project to project.
Every Burndown Chart Has a Story to Tell
The burndown chart is a great tool to help the team track progress since it shows progress on a daily basis, it helps scrum masters to predict if a team will be able to achieve the target. Also, it’s an indication of how the team is updating their tasks to ensure the accuracy of the data that will be collected during each sprint.
Ideal Chart
The graph below shows the ideal scenario where is actual and ideal lines are in sync and the team was able to finish all committed tasks at the end of the sprint.
Figure 8: Ideal Chart
Early Finish
Actual curve finishing below the ideal curve. If repeated, then you might need to consider increasing the team velocity and commitment to more work.
Figure 9:Early Finish
Unable to Finish All Committed Work
Actual curve finishing above the ideal curve. So, if repeated then you might need to consider decreasing the team velocity and commitment to less work.
Figure 10:Unable to Finish All Committed Work
Large peaks or valleys
Large peaks in the actual curve could mean many things but the most common scenario is the team not updating their tasks regularly.
Figure 11:Large Peaks or Valleys
More Unexpected Work Discovered
The jump in the actual burning curve could be because of more work discovered or more urgent bugs that are discovered during the sprint and must be resolved during the sprint. , if repeated then you might need to consider checking your quality control and increasing the allocation percentage of new bugs and rework.
Figure 12:More Unexpected Work Discovered
Stretched Curve
The team stretched towards the end to meet the commitment
Figure 13:Stretched Curve
Inconsistent Curve
This could mean the team is not consistent through excessive meetings or not updating their tasks regularly.
Figure 14:Not Consistent Curve
Final Thoughts
The main idea of agile metrics is to narrow down the cone of uncertainty of the estimations and to notify the business as soon as possible for more accurate estimations. Importantly, metrics are just one part in building a team’s culture. They give quantitative insight into the team’s performance and provide measurable goals for the team. This will help the team to be able to get unbiased estimations and to have more realistic estimations based on team effort.