The data ditch: what is it and how to avoid it

An increasing number of businesses are investing in their data maturity to place them in prime position to make wiser and more expansive use of their data.

In doing so, many stumble across “the data ditch” – a metaphorical void of untapped potential, impacting businesses that have mastered data collection but cannot yet use this data to underpin all decision-making.

A recent survey by Calligo and Fivetran found that over half of SMEs and enterprises across Europe and North America are stuck in this data ditch today. The benefits of overcoming this stage are significant – a third of respondents that are already on the other side report a 33 percent uptick in productivity, a 68 percent increase in profitability, and 21 percent higher staff retention rates.

Yet the data ditch does not have to be a lengthy stopping point on organisations’ data maturity journeys. To explain what can be done to climb out of the ditch, we must first look at how businesses can spot when they are in it.

How do we spot the data ditch?

Data is at the heart of every business decision, whether it’s highlighting areas that need to be improved or showing what products are most popular amongst customers. It stands to reason, therefore, that when data is not being used to its full potential, the impact will strike all aspects of the business such as speed to market, customer satisfaction, employee retention, security and profitability.

The tell-tale signs that an organisation is in the data ditch also permeate all departments and aspects of business operations, and include:

1.       Nobody ‘owns’ data governance, data management is not aligned to corporate strategy

2.       The company has data ethics and privacy initiatives in place, but no standardised framework and lack of ‘privacy by design’ culture

3.       The IT architecture is centralised but planning remains project-based or reactive

4.       Collaboration tools exist but users still don’t have a unified view of insights

5.       Data analysts cannot yet take advantage of automation, AI and ML

It is important that business and technology leaders have meaningful conversations about these areas and use the tools available to them to identify inefficiencies in their operations so that they can enforce the changes needed to mitigate them.

How to climb out of the data ditch

It’s clear that although most companies have invested in technologies and processes to turn data into a competitive differentiator, they are still not getting enough bang for their buck. The good news is that they are in the final mile, where even small process improvements can catalyse data-driven innovation and increased profitability.

In order to become completely data-driven they need to focus more on the following key areas:

Data governance

Introducing data leaders who can oversee that data is structured, properly stored, treated and protected – so that everyone in the business can trust the data to make decisions – is the first step for businesses wanting to boost their data maturity. More mature, data-driven organisations have formalised, automated data management processes that are fully aligned to support overall business performance.

Data ethics

Next, it’s crucial that businesses have a detailed understanding of every source and type of data, workflows and purposes. Businesses that want to progress in this regard must routinely, first and foremost consider data ethics and privacy against all of their data processing initiatives.

IT security

The more expansive the data sources are, the more exposed businesses can be to security threats such as phishing attacks, data leaks and zero-day vulnerability exploits. Indeed, many businesses found remote working a challenge as employees had access to swathes of data in their own homes and on personal devices where it was harder to keep secure. Introducing security committees to ensure correct processes are implemented and continuously reviewed will help businesses stay vigilant and secure in the face of new threats.

IT architecture

Resilient businesses place IT architecture at the core of their organisational strategy to better plan their objectives. Planning and allocating funding on a project-by-project basis will stunt the growth of businesses and limit their adaptability to changes in the long run. Proactive planning and the provision of technology resources that support ambitious use of data – securely and efficiently – will set businesses up for consistent growth.

Data insights

A common mistake data-hungry businesses make is hiring data analysts to take advantage of AI and machine learning, only to find that underlying data processes make it impossible to build effective prediction engines on top of it. To make data insights available for the wider workforce, businesses need to democratise access to it by dismantling information silos and removing the manual data pipeline maintenance burden currently costing businesses over$500,000 a year. Automating the data integration process will enable data professionals to focus on value-added tasks and maximise the potential of AI and ML.

What’s next?

Businesses often view themselves as digitally mature because of their technology investments, but money alone won’t help them climb out of the data ditch – that’s a matter of careful data architecting, robust privacy and security processes and adequate accessibility in all corners of the organisation. Adopting a steadfast and secure strategy, that invites employees across departments to maximise the potential of data has never been more paramount – or easier done! The time to be proactive and take those first crucial actions to becoming data-driven is now.

About the Author/s

Alex James is Vice President of Global Customer Support at Fivetran. As a global leader his accomplishments span more than 15 years in strategy and execution, customer support and success management, product development, solution deployment and leading-edge technical sales

Tessa Jones is Vice President of Data Science Research and Development at Calligo. She uses nearly 20 years of experience in the industry to apply the scientific method to a range of interesting and challenging problems. She has a rich history of developing useful data science models that are meaningful to business users.

Featured image: ©Green Butterfly