For as long as we can remember, we have been ruled by Bill Gates’ words that ‘content is king’
And while in 1996 – and a long while thereafter – this might have been true, a lot has happened since then: new concepts have been given rise, the Internet of Things has boomed, digital applications now manage all aspects of our lives, everything has become digital, from our books to our banks, all underpinned by one constant; Data. So, with this in mind it’s fair to say, content is out, and a new king is in town.
Consultancy firm Accenture’s ‘Big Success with Big Data” study found that 79 per cent of enterprise execs say that companies who don’t embrace Big Data will lose market strength and may face extinction. By now, it is a well-known fact that data is king when it comes to being a successful organisation in the digital era. Benefits range from optimising business processes to understanding customers on a deeper level than ever before.
However, the bottom line of this is that many businesses are finding it difficult to operationalise their data pipelines. Findings from last year showed that only 17 per cent of respondents rated the performance of their big data stack as ‘optimal’. The reason for this struggle is the lack of the right skills, costs and the time it takes to actually derive usable insights from their data.
So how are we going to get through the door to data nirvana if the key lies just beyond our grasp? The answer is optimisation. But first, we must evaluate the principal issues that business leaders have with their current data stacks in order to fully understand why we as a DataOps community need optimisation so desperately.
As skills remain scarce, will the big data engine grind to a halt?
Our research revealed a broad range of pain points for those working across IT operations. However, a lack of skills kept coming back as a constant hindrance in the pursuit of data stack synergy, with 36 per cent of respondents listing it as a major pain point. Within this skills gap, the most pressing need is for data architects – an issue for almost half of organisations (45%).
As data continues to explode and flood our day to day lives with more and more information we need data architects to handle databases on a large scale and make it possible for data scientists and analysts to comb through this data deluge to pull out actionable insights to make life better for the stakeholders that need insights to make business decisions. They are crucial to achieving what business leaders want for their organisations, especially as they have their eyes set on improved data analysis, transformation and visualisation. Architects will be incredibly integral to enabling the enterprise to reach these goals.
Clouds are high in the sky and we are still stuck on the ground
One of the other principal issues which is likely holding businesses back in their venture for an optimised data stack is that so many organisations do not fully utilise public cloud platforms for their data driven applications. Many have intentions to: 82 per cent of respondents noted that they have a strategy to move existing big data applications to the cloud. The inference here is that a lot of them do not already have their applications sitting within the cloud and therefore face the challenges of scaling up and down at will – with all the preparation and maintenance of infrastructure that this entails.
The benefits of hosting in the cloud are well documented: More businesses are waking up to the possibilities that the cloud offers for these critical applications. The scalability of the cloud opens up the possibility for business infrastructure to encompass multiple servers and provide unprecedented levels of capacity. Hosting in the cloud also reduces the cost and improve the performance of applications. Migrating to the cloud will likely unlock a lot of potential from the data stack that businesses across the UK are yet to realise.
Dodge an early end to your organisation’s life with DataOps
At present data seems to be most profitable or effective when it’s used defensively. The top four reported use cases were:
- Cybersecurity intelligence (42%)
- Risk, regulatory, compliance reporting (41%)
- Predictive analytics for preventative maintenance (35%)
- Fraud detection and prevention (35%)
To move beyond the tried and tested into projects that promise a greater impact to the business DataOps and application performance management (APM) solutions are the solution to fine-tuning, caretaking, and supercharging the complex software and hardware collision that is the big data stack.
APM, although comparatively new to the data application stack, is a class of technology well-known to the DevOps teams, used to being tasked to manage the tools and technologies of varied project groups within the enterprise.
APM is one technology that can support both sides of the divide, and aide the enterprise in finding common ground. Whether it is missed SLAs, failed jobs or workflows, slow jobs or queries, or computing resources unwisely allocated and causing delays or end-user frustrations… Preventing or fixing these problems cannot be done by just monitoring the big data platform and trying to fix issues using logs and graphs. In a typical big data deployment, that approach could not scale. Metaphorically the traditional approach of monitoring and debugging would be like trying to unravel the intertwined wires from holiday lights. It just cannot scale. There are just too many potential problems across too many different systems for DevOps to troubleshoot issues through trial-and-error and meet SLAs.
This technology promises to bring new ways of using data to businesses, however, the DevOps team will likely be managing hybrid platforms for the foreseeable future as this is not an overnight transition. Leveraging the power of APMs and optimising processes within businesses will reveal the true possibilities of the big data stack, and more business leaders will begin to see this tech meeting its KPIs, aiding the reduction of costs and time management across the business.
As data steps up, we really need to get serious about solving the complex challenges that come as part of their fast, evolving data stacks. Today’s top priority is ensuring that data stacks are performing reliably and efficiently, and that DataOps teams have the right expertise and tools to deliver the next generation of applications, analytics, AI and Machine Learning.
About the Author
Kunal Agarwal is CEO of Unravel Data. Unravel provides full-stack visibility and AI-powered guidance to help you understand and optimize the performance of your data-driven applications.
