Five Years In: How The Data & AI Landscape Changed

Five years ago, I started this blog with the aim of sharing leadership lessons I learned on my data, analytics, and AI journey. Reflecting on the past 101 posts, what strikes me is how much the landscape has changed in certain areas, and how little it has in others. What is apparent is that the role of data leaders – and I use this term in the broadest sense of the word – is more important than ever. Let us consider a few of the changes, and some aspects that remain the same.

Perhaps most interesting is to note that getting from data to insight, considered the hardest part of analytics, has effectively been automated with the advent of generative AI. The ability to process, summarise, and take action based on large volumes of structured or unstructured data now seems to no longer require human effort. If Google Search put the world's knowledge at people's disposal, their work on transformers has given everyone their own personal analyst.

Another major change is the degree to which the average person is aware of the role of data and AI in their lives. While five years ago people might have quipped that "data is the new oil" few outside of the industry seemed to care much about data and analytics. Now, thanks to publicity around regulations such as the GDPR, the proliferation of consumer AI applications, the increased frequency of data breaches, and growing data literacy, awareness is significantly higher.

Thirdly, the cliche that every company will become a data company is certainly coming true. For one, the phasing out of third-party cookies means that every company needs to build up their first-party data platforms lest they risk losing targeting and personalisation capabilities. Similarly, the adoption of AI is driving a valorisation of data assets, given the impact that high quality data can have on model performance. For many organisations, data is their new economic moat.

That said, plus ça change! Not everything has changed to such a great extent. The topic of data governance is still of critical importance to the success of data programs, and yet remains about as popular as visits to the dental hygienist. There is some hope that the scaling of generative AI will improve matters, given how badly – and how publicly – these models can be wrong if trained with the wrong data or prompted with faulty parameters. Guardrails are not optional.

On a related note, the majority of organisations still struggle with realising the business impact of their data programs. Whether this can be attributed to a lack of data literacy, constantly shifting technology (and therefore skills) goalposts, or the transversal complexity inherent in such programs, what is clear is that the role of data leaders has not become any easier – even though a lot of the enabling technologies have become friendlier. I mean, remember Hadoop?

Perhaps most comfortingly, despite the proliferation of AI and the automation of so many stages of analytics, human judgement still matters. Arguably, in some scenarios, it matters more than ever. From the vast amount of AI generated misinformation to the myriad subtle errors that can sneak into generative AI outputs, the value of sound human judgment cannot be overstated. Yet how many of us are mindful of this fact, and reflect on this in our daily lives?

As AI solutions grow more powerful and more independently capable, we all have a duty to ensure the proper use of data, the correctness of models' outputs, and whether their usage in a particular context is appropriate. We humans need to maintain our agency in a world where an increasing number of decisions are taken for us. Looking ahead at the accelerating trajectory of data and AI, every person needs to become a data leader and consider these questions in earnest.

The irony is that whereas I started this blog with the aim of appealing to a small set of data professionals, in writing about these developments I realised we as data leaders collectively need to engage a much wider audience. At the end of the day, if we want to ensure that data and artificial intelligence play a meaningful and positive role in our society in future, it will take more than data specialists to make that happen. No matter how many blogs they write.

– Ryan

Cover image by DALL·E, based on my profile photo.