Deployment Frequency
As the name already suggests, Deployment Frequency refers to the frequency of successful software releases to production. In other terms, it measures how often a company deploys code for a particular application.
The metric that uses the total number of deployments per day as a reference guide was developed on the basis of manufacturing concepts that measure and control the batch size of inventory that a company delivers.
Naturally, more successful companies tend to do smaller and much more frequent deliveries – or in the world of DevOps, more frequent but smaller deployments.
In general, one deployment per week is the standard, while a high-performing company releases up to seven deployments a day. Of course, the standard number of deployments differentiate by product.
For example, mobile applications which require customers to download the latest Update, usually make one or two releases per quarter at most, while a SaaS solution can deploy multiple times a day.
| Question it answers |
Elite performers |
High performers |
Medium performers |
Low performers |
| How often does your organization deploy code to production or release it to end-users? |
On-demand (multiple deployments per day) |
Between once per day and once per week |
Between once per week and once per month |
Between once per month and once every six months |
Source: 2019 Accelerate State of DevOps, Google
When DevOps teams realize that they fall into a low-performing category, they can install more automated processes when it comes to testing and validating new code as well as reduce the time span between error recovery time and delivery.
Lead Time for Changes
This metric measures the time that passes for committed code to reach production. While Deployment Frequency measures the cadence of new code being released, Lead Time for Changes measures the velocity of software delivery. It is used to get a better understanding of the DevOps team’s cycle time and to find out how an increase in requests is handled. The lower the lead time for changes, the more efficient a DevOps team is in deploying code.
In order to measure Lead Time for Changes, two pieces of data (or timestamps) are required: The exact time of the commit and the exact time of the deployment – in other words, the time from start to finish of a product The average time is then used as an indicator for overall performance.
| Question it answers |
Elite performers |
High performers |
Medium performers |
Low performers |
| How long does it take to go from code committed to code successfully running in production? |
Less than one day |
Between one day and one week |
Between one week and one month |
Between one month and six months |
Source: 2019 Accelerate State of DevOps, Google
If a high lead time for changes is detected, DevOps teams can install more automated deployment and review processes and divide products and features into much more compact and manageable units.
Mean Time to Recover
The Mean Time to Recover metric measures the time it takes for a service to bounce back from a failure. It doesn't matter how high-performing a DevOps team is, unplanned outages or incidents happen. And since failures can’t be avoided, it is really the time it takes to restore or recover a system or application that makes a difference.
If a company has a short recovery time, leadership usually feels more comfortable with reasonable experimenting and innovating. In return, this creates a competitive advantage and improves business revenue.
The metric is important as it encourages engineers to build more robust systems. It is usually calculated by tracking the average time between a bug report and the moment the bug fix is deployed.
| Question it answers |
Elite performers |
High performers |
Medium performers |
Low performers |
| How long does it take to restore service when a service incident or a defect that impacts users occur? |
Less than an hour |
Less than one day |
Less than one day |
Between one week and one month |
Source: 2019 Accelerate State of DevOps, Google
In order to improve their performance in regards to MTTR, DevOps teams have to practice continuous monitoring and prioritize recovery when a failure happens. It is also helpful to establish a go-to action plan for an immediate response to a failure.
Change Failure Rate
This metric captures the percentage of changes that were made to a code that then resulted in incidents, rollbacks, or any type of production failure. Thus, Change Failure Rate is a true measure of quality and stability while the previous metrics, Deployment Frequency and Lead Time for Changes don't indicate the quality of software but just the tempo of software delivery. According to the DORA report, high performers fall somewhere between 0-15%.
Change Failure Rate is calculated by counting the number of deployment failures and then dividing it by the total number of deployments. When tracked over time, this metric provides great insight as to how much time is spent on fixing errors and bugs vs. delivering new code. Needless to say, a DevOps team should always strive for the lowest average possible.
| Question it answers |
Elite performers |
High performers |
Medium performers |
Low performers |
| What percentage of changes to production or end-users results in degraded service? |
0-15% |
0-15% |
0-15% |
46-60% |
Source: 2019 Accelerate State of DevOps, Google
In order to improve a high average, teams should reduce deployment failures and time wasted due to delays.
Learn more
.svg)