Data is already the fuel that drives our real-time economy forward
But as businesses worldwide face unprecedented challenges in the coming year, from the supply chain crisis and US inflation to the green transition, harnessing the power of data will become even more crucial in enabling them to remain competitive. The web data collection industry is growing at a dizzying pace. In fact, recent research by Bright Data and Vanson Bourne revealed that over half (57%) of UK and US financial services, technology and IT organisations are already using data collection bots. And almost all (95%) plan to expand their automated functions, and with that bot usage, in the next two years.
Though the practice is only a decade old, web data collection is already ubiquitous in sectors like e-commerce, travel and fintech. In the future, we can expect it to become commonplace in a far broader range of areas, including healthcare, cybersecurity, law, adtech, social media, non-profits and legacy financial services firms.
Crucially, businesses are becoming more forthcoming about how they collect and use web data platforms. The practice is too often maligned and misunderstood due to the secrecy that surrounds it, so increased openness will have several key ramifications. Firstly, it will lead to greater scrutiny and standardisation of web data collection methods. And secondly, organisations will gradually realise that it’s better to allow “good data collection traffic” and have it visible rather than have it hide within the organic traffic. This will allow the organisation to filter out the data collection traffic from organic user statistics, monitor it, and even analyse and produce business-level insights, including knowing what competitors are looking at in your own site.
Let’s take a closer look at how this will play out in 2022 and subsequent years.
Greater scrutiny and regulation will build credibility
As web data collection becomes more widely accepted, practitioners should expect calls for greater transparency around their business practices to increase. Though this will no doubt help to ensure that high standards are upheld by all practitioners; self-policing alone will not be enough.
Given increased global scrutiny of data use and privacy, regulation around web data collection seems inevitable – and should be welcomed, not opposed, by practitioners. To manage the explosion in web data collection, governments should work alongside practitioners to create an extensive web data collection compliance framework for their jurisdiction. Given that data collection capabilities are becoming more complex and advanced, this certainly won’t be easy, as guidelines would need to be updated relatively frequently. However, to build long-term credibility and maturity in this space, regulation is a necessity.
Businesses will accept that they can’t prevent online data collection
From uncovering trends to conducting market research, there are countless reasons why businesses collect publicly available web data from their competitors. Though the competitors in question often also engage in data collection themselves, most will regularly block access attempts and make site changes to prevent their public data from being accessed, even though the information targeted is on public display.
All this could be about to change. While it may seem counterintuitive – after all, why would you want to give away information to your competitors – some businesses are beginning to realise that it’s in their best interests to allow their public data to be collected by responsible, well-defined, and compliant data practitioners. Firstly, preventing data collection is like a game of whack-a-mole: When you block one tactic, smart practitioners will simply find another. Secondly, accepting some forms of data collection will enable businesses to accurately distinguish between organic user traffic and collector traffic, giving them a clearer insight into what data is being collected and by whom.
Looking forward, as web data collection continues to boost business competitiveness, improve efficiency, and enhance strategic decision-making, it’s up to all data collection practitioners to prioritise responsible and compliant business practices. We’re already seeing significant progress in this area, and if current trends continue, we can expect online data collection to achieve widespread acceptance in 2022 and beyond.
About the Author
Ron Kol is CTO at Bright Data, a position he has held since 2019. He is a proven expert in developing and implementing data collection technology. As a result of his years of experience developing data-driven products, Kol has industry-leading expertise in multiple market verticals and has built technology to meet all their unique challenges, business goals, and growth targets. Bright Data serves more than 10,000 businesses globally, including Fortune 500 firms, major retail players as well as finance organisations, security companies, prominent travel sites, and more.
Featured image: ©gaihong
