Measuring ROI for Agile Projects

Every project has a goal and a set of acceptance criteria. In most instances, the goal is stated in financial terms of raising additional revenue or reducing costs. The business is required to present this goal, or group of goals, to gain project approval and for work to begin. Knowing this goal or number is the crucial variable to calculating ROI. 

Sometimes the difficulty isn't if the number exists but instead finding the number. For instance, "adding 50 new customers per month for the next 12 months" might be the project goal, but how much does each new customer generate in additional income? Hence, it might be tougher to find the ROI for these kinds of goals, but don't let this dash your hopes on calculating ROI. Let's be sure we have ROI defined before going further. 

ROI defined

In its simplest form, ROI is equal to Net income divided by the Cost of Investment, and the result is multiplied by 100 (ROI = Net Income / Cost of Investment * 100). 

Unfortunately, Net Income is a relatively complex calculation:

          Net Income = Gross Profit - Operating Expenses - Other Business Expenses - Taxes - Interest on Debt + Other Income

Let's simplify the Net Income variable as the Additional Revenue generated by our new feature(s). This modification may give those with Financial degrees a cold chill, but it allows us to simplify the calculation so that: (a) the calculation is still valid and (b) we can use the ROI to prioritize the projects in our backlog. Our new formula becomes:

          ROI = Additional Revenue / Cost of Investment * 100

Importance of ROI

Project approvals are typically given based on the additional income or revenue new features can generate. Below are examples of two very different projects:

Some may jump to approve and give Project #1 top priority due to the potentially large amount of additional revenue at stake. However, looking at the ROI, we can see that Project #2 should be given top priority. Hence, ROI helps us: (a) see what projects are profitable ventures and (b) help Product Management and Product Owners prioritize the projects in their backlogs. 

Calculating ROI

As stated above, all projects must undergo a project approval process to gain access to software development teams. How stringent that process is will vary from company to company, or even from department to department within a company. It will be more difficult to calculate ROI for organizations that allow nominal financial documentation for a project to gain approval. Missing financials or unclear financial benefits will make it more difficult to calculate ROI. Don't let this deter you if you are in one of these organizations. A few strategic questions will enable you to ferret out the Additional Revenue expected by the stakeholders. 

ROI algorithms

The ROI calculations will boil down to three algorithms, as described and defined in the below sections. 

The promise of additional revenue

The first calculation is for projects that promise additional revenue after introducing a new feature(s). In this scenario, "Additional Revenue" is the additional income expected after your new feature(s) are implemented. See Appendix, Example #1. 

The promise of cost savings

For projects which permit cost savings, the "Additional Revenue" variable in the ROI calculation is the total cost savings expected after your new feature(s) are implemented. See Appendix, Example #2.

The promise of both additional revenue and cost reductions

Larger projects that promise a combination of additional revenues and cost reductions will require a third calculation. In this instance, "Additional Revenue" is new revenue plus the total cost saved. See Appendix, Example #3.

Abstract numbers

You may find yourself in a situation where your stakeholder's estimate is based on something more abstract (e.g., new customers gained) instead of a hard additional revenue estimate. In this scenario, you should work with your stakeholders to find a way to calculate additional revenue from the abstract numbers (e.g., average net income for each newly added customer). 

Be cautious, though. Some abstractions can turn into lengthy calculations. Don't forget Agile Principle #10, "Simplicity – the art of maximizing the amount of work not done – is essential ." The time required to increase your calculation's accuracy might be better spent working on an additional feature that can generate other income. Tip: be careful about how much time you spend focusing on increasing the accuracy of your ROI calculation.

Total cost of ownership

Let's not forget about the total cost of ownership. This topic is too detailed and in-depth to address here, but it is still good to ask yourself, "Is it worth adding this feature if the costs required to maintain and support the application significantly rise after the new feature is added?" You will need the team's input to determine the answer to this question. Not addressing this question with the team before implementation might lead to some very disappointing financials you didn't expect.

Tracking ROI accuracy

So far, we have looked at ROI calculations before releasing a new feature. The variables used in those calculations are all estimated additional revenue and costs. However, should you also track the actual ROI? Today's technologies allow us to easily track customer experience as well as feature usage. Automating metric collection and calculating true ROI after a new feature is released is as simple as adding monitors or listeners to your services. These monitors track the information needed to calculate your feature's ROI (e.g., new customers, revenue generated by new customers, revenue generated by new features, time saved when using new features, etc.). In addition, analyzing and validating expected returns will help improve future ROI calculations. 

Agile transformation project ROI

We often hear about "failed agile transformations." But why did they fail? Was it that the teams made no improvement, or did the organization not track their progress in the transformation? Many Information Technology (IT) organizations undergo an "Agile Transformation" by adopting Agile mindsets and implementing Agile processes. You can measure an organization's or team's Agile Transformation ROI based on the below metrics. Each of these metrics can be fit into the ROI formula to identify individual performance against the specific metric or overall performance based on multiple metrics. 

Can ROI be calculated for agile transformations?

ROI can also be calculated for Agile Transformation projects as long as the:

1. The organization has identified a list of success factor metrics (see list below),
2. Metrics have been baselined, and
3. The organization is tracking and reviewing the metrics.

Agile transformation revenue generating success factors

• More value delivered per cycle (more value delivered for the same investment)
• Improved on-time delivery
• Enhanced team velocity / throughput
• Better team say/do ratio
• Decreased story churn
• Reduced team lead and cycle times
• Increased delivery team forecasting accuracy
• Improved application uptime
• Improved mean-time to recovery
• Healthier internal service satisfaction
• Elevated reputation (i.e., net promoter scores)
• Improved quality (i.e., reduced production bugs)

Vanity metrics

So far, we've looked at how to calculate ROI, different metrics to tie back to ROI, and how ROI can be used to prioritize your backlog of work. Unfortunately, all these metrics can lead management into the trap of using vanity metrics. Vanity metrics make you look good to others but do not help you understand your own performance in a way that informs future strategies. The metrics you or the business choose to monitor success must prove increased revenues, customer retention, and/or quality.

Eric Ries discusses vanity metrics in his book, The Lean Startup. One of the examples he gives of a vanity metric is when his team measured their product success by total enrolled customers. They purchased web advertisements linking to their product. They later declared victory when registered customers increased. Unfortunately, they found their assumption that revenue would increase as registered customers increased was incorrect. After a thorough investigation, they discovered the advertisements they purchased increased the number of registered customers, but those newly added customers were not buying goods. Hence, enrolled customers turned out to be a vanity metric for them. Thus, be sure you are measuring the right things. 

In summary

ROI is a relatively simple calculation that can help you identify what features should be worked on and prioritize your backlog of work. The figures to calculate ROI may not always be available. Be prepared to work closely with your Product Owner to clarify the numbers needed to perform ROI calculations. Although ROI is excellent as an estimate, don't forget to create automated monitors for each new feature to validate the actual ROI against estimated ROI – this is an important distinction. Future ROI calculations will become more accurate as you validate prior ROI estimations over time.

Appendix – example calculations

Example 1: Promise of additional revenue

For projects that promise additional income after introducing new feature(s), that additional income is placed into the "Additional Revenue" variable of the ROI formula. 

          Estimated Additional Review = $400,000

          Estimated Project Cost = $160,000

          ROI = Additional Revenue / Cost of Investment * 100

          ROI = $400,000 / $160,000 *100

          ROI = 250%

Example 2: Promise of cost savings

For projects that promise cost savings, the "Additional Revenue" variable in the ROI calculation is the total cost savings expected after your new feature(s) are implemented. 

          Estimated Additional Review = $100,000

          Estimated Project Cost = $80,000

          ROI = Additional Revenue / Cost of Investment * 100

          ROI = $100,000 / $80,000 *100

          ROI = 125%

Example 3: Promise of both additional revenue and cost reductions

For projects that promise a combination of additional income and cost reductions, the "Additional Revenue" variable in the ROI formula becomes the sum of all additional income and costs. 

          Estimated Additional Income = $50,000

          Estimated Cost Reductions Income = $15,000

          Estimated Project Cost = $60,000 

          ROI = Additional Revenue / Cost of Investment * 100

          ROI = ($50,000 + $15,000) / $60,000 *100

          ROI = 108%