When we think of entrepreneurs, we usually think of confident, assertive, decisive individuals.
They will give us the impression of a clear strategy or plan. We will use words like ‘vision’, ‘instinct’, and ‘conviction’ to describe them.
But in today’s economy, in which consumers expect products or services to be delivered almost instantly, the window that these individuals have to make decisions for the business is shrinking. The systems that business use often aren’t designed to deliver data within that small window of opportunity — forcing decision-makers to make assumptions and relying on previous experience.
For the startup or small-to-medium enterprise (SME), that puts an especially heavy burden on decision-makers. How can we expect entrepreneurs to make the right decision instinctively if they don’t have an accurate view of their business at the crucial moment?
Split decisions
To put that challenge into perspective, let’s run the numbers.
Confluent’s Quick Thinking report has found that 90% of business leaders in the UK are being forced to “make decisions in real-time.” Given that many simply don’t have the time or means to do so, more than three in five (61%) simply make ‘snap decisions, without reviewing the available data.’ A similar 58% rely on ‘gut feel’, specifically because “it’s too difficult to access insights in real-time.”
In other words – a majority of business owners are being forced to make decisions that are based on assumption and experience rather than evidence. Given the precarious position that many SMEs will endure in their infancy, this is a scary reality.
This isn’t to say that instinct should be ignored – in fact, it’s an important part of any business. We can’t discard previous experiences, and particularly past mistakes, when it comes to improving our business…
Consistency is key
…but we can’t rely on it exclusively, either.
The problem with experience is that it’s inherently retrospective. It’s the product of patterns that we recognise, in a context that we understand. The further we are from our experiences, the more likely things are to have changed in ways that we can’t pre-empt.
For the SME, there’s a smaller pool of people and experiences to draw from, too. SMEs are likely to rely on focused specialists for key business roles, which limits their ability to contribute to other areas of the business. Given that an SME also inevitably means a smaller workforce, our instincts can become less accurate over time, and there are more voices that can challenge the status quo in larger organisations.
If we can’t update our understanding of how things are from others, we need data – and if the numbers are correct, we need it fast enough to make decisions in real-time. When we’re unable to back our instinct with real-world evidence, we need to find that evidence!
The Quick Thinking report suggests that there’s a real appetite for this sort of data among business leaders. 85% think that they’d make better or more accurate choices if they were able to rely on real-time data, with an almost identical 84% actively agreeing that they’d like to make more data-driven decisions.
So – what’s stopping us? Given that we know a majority of business leaders want to access data in real-time but can’t, what are they wrestling instead?
Enter: data streaming
This is where we revisit the limitations of the businesses themselves, as many – especially SMEs – simply don’t have the technology to deliver this sort of information at the required speed.
One of many best practices in data integration is the concept of a “Shift Left”: that is to say, the aim to conduct data processing and governance as close to the source as possible. The theory is that by cleaning and processing the data earlier in the data lifecycle, all downstream consumers can share a single source of well-defined and well-formatted data.
Unfortunately, while many businesses will follow that mantra, they’re also likely to depend upon batch processing – the storing data in a lake or warehouse, and then dredging up huge chunks of that data to be analysed in more detail. It’s an approach typically used for tackling large volumes of data that aren’t time-sensitive, historically used heavily in industries like financial services.
For all its strengths, batch processing simply isn’t designed to deliver information rapidly. You need to process the data, then store it, then take a separate cut, and then wait for your hardware to crunch the numbers for you – which can take hours, or even days in some cases – before you’re presented with the information you need. It’s slow enough to undermine the benefits of “Shift Left”, even if your organisation is genuinely committed to best practice.
Time is money
Consequentially, we’re seeing many decision-makers at Confluent energetically driving the elimination of legacy systems in favour of data streaming platforms (DSPs) – literally streaming data across the business, rather than collating it in huge stores.
The DSP will connect data from every repository across organisation, pulling it all into a continuously moving whole accessible from a single source. Those data streams can have context built directly into them, allowing the platform to better recognise the business value of each individual point. It’s no surprise, then, that the Quick Thinking report expects a notable rise in the use of DSPs over 2025.
More than three quarters (77%) of business leaders surveyed are actively investing in data streaming solutions; a colossal 96% see real-time data streaming as “important” or “essential” to the decision-making process in the coming year.
Similarly, those that have invested ahead of the competition are seeing the returns that you might expect from adopting a disruptive technology at just the right time. Confluent’s Data Streaming Report 2024 found that more than two in five (41%) businesses are seeing a five-fold ROI on data streaming investments; a huge 84% report a 2-10x return.
This sort of return hammers home that data isn’t just an IT concern any more. Data is the understanding that underpins business success. The quicker you can access that insight, the more likely you are to make the right decision – and profit from it faster than your competitors.
About the Author
Jack Bingham is Director, UK&I Digital Native at Confluent. Confluent is pioneering a fundamentally new category of data infrastructure focused on data in motion. Our cloud-native offering is the foundational platform for data in motion — designed to be the intelligent connective tissue enabling real-time data, from multiple sources, to constantly stream across the organization. With Confluent, our customers can meet the new business imperative of delivering rich, digital customer experiences and real-time business operations. Our mission is to help every organization harness data in motion so they can compete and thrive in the modern world.
