You started DevOps practices in your organization because they improve application quality and time to market, according to multiple research reports, which result in better digital experiences and enhanced competitiveness
How do you figure out if your DevOps teams are actually delivering the desired results?
More than 80 percent of enterprises surveyed have adopted some level of DevOps practices, and “it has become the default approach for developing cloud-based applications”, according to RightScale’s 2017 State of the Cloud Report. DevOps implementation efforts are almost equally split between company-wide, selected business units or divisions, and specialized projects or teams. Regardless of the chosen approach, why do some organizations seem to be getting better results than others with their DevOps teams? Like many other processes, it really comes down to a combination of effective measurement and organizational maturity.
Don’t guess, measure
Deriving positive results from DevOps means getting reliable and real-time data on the results, not the efforts. If the metrics do not exist or do not align with the business objectives, the team is left to guess at the impact they are having, and process improvements will not necessarily translate into improved experiences. Key metrics fall into three categories: speed, quality, and experience.
The need for speed
DevOps is most often associated with speed; deploying applications and feature updates faster and with greater frequency, so these metrics are familiar to most people. Deployment frequency (how often you release new versions) and lead time (how long it takes from code commit to production) are the high-level metrics that should be visible to the team and updated frequently. Drilling down into each one will identify where the bottlenecks are and what processes need to be improved, automated, or eliminated. Mature DevOps organizations deploy updates multiple times a day and have lead times of less than an hour, according to the Puppet 2017 State of DevOps report.
Quality is not an accident
Speed without quality is of little value to the organization, so the next set of DevOps metrics are the failure rate (the percentage of releases that have problems) and the number of tickets how many issues a release has). Each organization or team needs to find its appropriate balance between speed and quality. The initial focus on quality numbers should be relative trends, not absolute values, to make sure that the teams are progressing in the right direction. Drilling down into failure causes should identify which steps of the process, such as code review or test coverage, need attention. An important detail from bug reports and trouble tickets is whether they are internal, or user reported. Mature DevOps teams have failure rates less than 10 percent, according to the 2017 Puppet Report, and have an increasing percentage of issues captured by monitoring tools before being reported by users.
It’s all experience
The metrics that underperforming DevOps organizations often miss are those related to a customer’s experience with the application, their usage (or not) of new features, and the resulting impact to the business. These measurements are outside the realm of traditional development and QA tools, and mean changing mindsets and adopting new tools. The percentage of users that are trying new features, specific demographic characteristics, and of course response times and transaction completion rates are all valuable insights into whether your developments are meeting customer’s needs. Finally, contributing to the bottom line is what the business executives care about, so it is essential to translate usage into relevant metrics such as revenue, productivity gains, cost savings, or customer satisfaction. Understanding, measuring, and responding to these outcomes are hallmarks of mature DevOps organizations.
Age doesn’t define maturity
Maturity is not a measure of the age of an organization, but of the completeness of its processes. Research by Enterprise Strategy Group (ESG) shows “a positive correlation between increasing maturity and the uptime of applications, time to resolution, and avoidance of negative impacts experienced.” At the highest level, mature processes are repeatable, clearly defined, actively managed, and continuously improved. According to ESG, the most mature DevOps groups not only measure the business impacts of their efforts, they enable self-service monitoring by the business managers and collaborate regularly with each business area about the policies, processes, and results. The result is a highly cooperative work environment, with IT viewed as a competitive differentiator by the line-of-business stakeholders.
Performance management is at the core
Companies of all shapes and sizes are embracing DevOps to better compete in the digital era. Such practices ultimately lead to higher revenue, user productivity, and customer satisfaction. To get the right metrics in a timely fashion, leading organizations have broad application performance management (APM) capabilities, covering every aspect of application delivery from development and deployment to user experience and business impact. With the right metrics, IT leaders can clearly demonstrate the value of DevOps to the business and continue to refine and improve their efforts. The results are accelerated application delivery, optimized performance, greater resolution of issues before they affect the business, and better understanding and connection between IT and the business.
About the Author
Gayle Levin is director of solutions marketing at Riverbed Technologies. Previously, she held product marketing and campaign roles at VMware, Oracle, and Splunk as well as several startups. Her interests lie in the impact of technology on the way we think and work today.
