Decoding tomorrow:
Insights in the age of AI
AI is set to be the biggest contributor to the evolution of our industry in the coming years. What we don’t fully know yet, is how exactly AI will shape the future of insights. From this comes a more prosaic question; what should we as an industry be doing now, to prepare for the future and maximise the reward?
No one ever predicts the future with complete accuracy, and we’re certainly not claiming to have all the answers. But, based on our experience of successfully launching an AI business, talking to many clients and closely following the AI conversation in our industry, we’ve some bold predictions to share.
Here’s the first: AI is going to change the insight industry more than anything that’s happened before. Let’s look at this in more detail…
Looking back: innovation in insight
The insight and market research industry has never been immune to innovation, and there have been several significant developments in the past few decades. Most data collection migrated online, and there have been significant developments linked to Behavioural Economics, semiotics, cultural insights, design thinking, neuroscience, and a host of other developments.
But, despite these developments, one could also argue that the backbone of the industry remains as it always was. Quantitative surveys and focus groups are still dominant, and whilst they might now be online in many cases, they’d still be recognisable to a time travelling market researcher from the 1950s. They have stood the test of time. But there are signs that we’re about to see fundamental changes in the years to come.
Looking ahead: AI in insight
Anyone who has worked in our industry will be accustomed to periodic enthusiasm around about the latest ‘big new thing’ promised to change the face of research. Most of us have seen these come and go with little to no lasting change. So we completely understand some of the well-placed scepticism that accompanies the hyperbole surrounding AI.
But if we look beyond the rhetoric, we’re extremely confident that AI will bring a genuine, ‘once in a generation’ disruption – and that disruption will be more rapid than anything we’ve seen before, with very significant change happening in the next five years. The exact nature and pace of that change is harder to predict, but we’ve compiled some early thoughts on what the landscape might look like in five years’ time.
For us, the most actionable and important question then becomes: “What should we be doing now to get ready for the future”. That’s the focus of this thought piece. So to answer that, let’s start by taking a quick look into the future at 2029.
Insight in 2029
First things first. We believe some things won’t change – if we as an industry are successful, then we will still be shaping strategy by surfacing consumer needs and preferences. We’ll be answering big strategic questions around product innovation, brand, marketing, and customer experience. So, in terms of what the industry is trying to achieve, there will be no drastic shifts.
But big changes are likely to happen in terms of how the industry is operating. If we were surveying the insight landscape in 2029, we expect to see:
Fewer surveys
Surveys are much shorter, engaging, and conversational
A much wider range of data sources, e.g.
synthetic data / intelligent personas / digital twins
hyper targeted data lakes based on search, social and storefront data
integrated datasets fusing traditional qual and quant with all of the above
Automated analysis, with increasingly powerful AI models trained to perform a wide range of tasks such as:
Market landscaping / surfacing needs and trends
Product innovation / development of new concepts
Diagnosing brand health
Foresight and predictive modelling
New ways of disseminating insights e.g.
insight bots answering big questions in real time, with high degrees of accuracy, reliability and engagement.
Automation of data visualization
Operational efficiency:
Less time spent on data collection, and more time spent on storytelling, interpretation and recommendations.
Back to the present moment
For many, speculation about the future is just that: speculation, with no real-world application. But for us, this speculation does serve a real purpose. It helps us understand the likelihood, scale, and shape of the change to come. And then: even if we don’t get the predictions 100% right, we can start to answer a more relevant and tangible question: what should we be doing now, to prepare for the changing landscape?
In the past 12 months we’ve learnt a lot from setting up a successful AI business, adopting new AI technologies in our processes, and talking to clients around the world about their AI strategies. Based on that, here’s our initial take on three big things we all could and should be doing now:
1. Transformative technology needs transformation
Lots of new technologies fail to make an impact at first. That’s largely because companies initially try to fit them into existing processes, ways of working, skillsets etc. A bit like trying to fit a combustion engine onto a horse to make it go faster.
To make the most of new technology, we need wider transformation. Using the example of the engine – we need to build a new vehicle, and then we need to build infrastructure (roads), skills and capabilities (driving licences, mechanics) and support (e.g. petrol stations).
In 2023, there was a lot of focus on the AI tools being developed. But to get the most out of AI, we believe we need to take a step back. And not see AI tools in isolation, or to focus on using AI to make small improvements to existing processes.
Instead, we need to see it as one piece of a much bigger transformation that allows us to reimagine insight. In taking this big picture approach we need to design a strategy for where we use it, how we use it and who it’s used by, and for. From that we would then be able to build out the infrastructure and capabilities to support it.
Implication #1: start to map out a comprehensive strategy; with organisational goals at the heart of it, and a clear view of where AI is an enabler / accelerator, and then mapping out the wider transformation that needs to take place for it to be a success.
2. Develop new skills and capabilities
AI is immensely powerful – but there are also plenty of examples of badly misfiring AI, and there’s a huge variety in value and quality across the supply landscape. To sort the wheat from the chaff, and to get the most out of AI, we believe the industry will increasingly need to focus on:
A working knowledge of AI: as the industry develops new models to tackle a wider range of tasks, we will increasingly need to understand how models are trained, how to mitigate bias (e.g. recency / cultural bias) and how to assess their accuracy as we start to base recommendations on outputs.
Learning how to navigate a completely new data world: the data sets associated with AI are new to most insight professionals. Capabilities will need to be built to enable interrogation of raw unstructured data lakes made of millions of data points. Assessing the validity of these, and understanding how they can add value, will be important. Big data and synthetic data will play a much bigger role, but for that to happen in a way that adds value, we will need to understand the benefits, watchouts, the questions to ask, and potential for delivering relevant business outcomes.
In addition to capabilities around AI and data, we expect to see an enhanced focused on skills that have always been important in the world of insight. With a shift in methods, and AI doing more of the heavy lifting in the data gathering and analysis, we expect to see a much greater emphasis on:
Communication skills: against the proliferation of data and insight sources, there will still be a need to distil insights into a clear narrative, to land recommendations, and to ultimately ensure insights are translated into action. These skills will continue to grow in importance as the process of gathering and analysing data increasingly becomes automated.
Continuous learning & and change management: This is a subject of great depth and complexity. In short, the pace of change in our industry is going to accelerate, and there will be a need for insight organisations to continuously adapt. As new tools and solutions come on the market, we will need to manage the change, and ensure our organisational structures and capabilities are keeping pace with the rate of AI adoption.
Implication #2: develop a clear view on the capabilities needed to harness the power of AI. Create capability frameworks, team KPIs and L&D programmes with AI in mind. Due to the fast-evolving nature of this area, reviews of the systems and processes will need to be done with a regular fast-paced cadence.
3. Ability to assess the supply landscape
It’s likely that we will see a proliferation of AI solutions and an increasingly fragmented supply landscape emerge in the next few years. Against this backdrop, insight organisations will need to develop new frameworks for assessing the quality and value of new solutions.
When online research panels became mainstream, ESOMAR developed a series of questions that all providers should answer. The list is still in operation today (at latest count, it covers 37 questions), and it provides the industry with a framework to evaluate online sample providers.
We believe insight organisations need something similar for AI – an ever-evolving set of questions, with detailed guidance on what to look for, when evaluating different AI solutions. Over the past year, Basis has been developing its own list – this is by no means the final set of questions, but it's our start-point.
Our current ‘ten questions we’re asking’ are:
What dataset is the AI trained on?
How recent and comprehensive is it – and how often is it updated?
Is this solution based on a syndicated data lake – are all clients accessing the same data?
How is this AI addressing GDPR / data protection concerns?
What safeguards are built in to prevent the AI from hallucinating?
If client data is fed into this – can other clients ultimately access / benefit from it?
What work has been done to validate this solution – and is that work in the public domain?
What human input is there into the process, to validate and elevate insights?
What underlying tech is this AI solution built on?
Is this AI solution using technology that is likely to become more affordable or cheaper to run in the future – will the provider pass on future savings, or am I tied into a fixed price?
But this list is ever evolving and is often augmented with specific questions that relate to a particular solution.
Implication #3: develop a list of questions to assess the quality and value of the AI solutions being adopted. Review these regularly to ensure the questions remain relevant as the landscape evolves.
Final thoughts
We think our industry is about to undergo a significant transformation. Some businesses will innovate and improve our industry immeasurably; others won’t. We also believe that the rate of this transformation will be fast, so preparation must begin now. Those who begin now give themselves a much greater chance of success and build the right momentum to keep up with the pace of evolution. Those who wait will almost certainly always remain behind the curve.
At Basis we’re already seeing the change and working with forward-thinking businesses to implement the right strategies as their AI partners. If you want to know more about how Basis can help you with using AI in insight please get in touch.