AI and real time data

Peter Pugh-Jones at Confluent explains why data streaming engineers are the missing link in AI success

 

No matter your industry, there’s a lot of noise to tune out when it comes to artificial intelligence. From banking to retail, to manufacturing, to healthcare and beyond, organisations are racing to take advantage of the capabilities of AI.

 

But while these initiatives made bold claims around efficiency, revenue, and the customer experience, many are still struggling to meet business expectations. Too often, organisations find themselves wrestling pilot projects, often frustrated by models or strategies that simply don’t deliver on their initial promise.

 

There’s a host of reasons why, and they’ll be different for every business. Research has found that a majority of UK businesses (52%) face a skills shortage in AI. The compliance landscape for AI is constantly evolving - as we see with the public debate over the UK’s AI Bill, for example - which makes it difficult for many businesses to keep up. And given the explosion of AI tools across the world of work, making sure that your new systems and technologies can integrate together is never a given.

 

However, these things are all themselves dependent on one of the most overlooked barriers to the effective use of AI: data.

 

 

Why real-time data is critical for AI

Data is the fuel that drives AI. To paraphrase a classic engineering term: “rubbish in, rubbish out.”

 

AI depends on access to the right data, in the right format, at the right time, and in sufficient quantity. Without this, even the most sophisticated algorithms are left deprived of the data they need to perform.

 

In today’s digital economy, that means real-time data. Whether powering fraud detection in financial services, personalisation engines in e-commerce, dynamic pricing in transport, or operational intelligence in manufacturing, the provision of real-time data is what allows AI to act and respond to the moments that can make or break a business. Static datasets alone are not enough.

 

For digital-native leaders, this is second nature. They design their systems with real-time data in mind. Consider companies like Uber, the entire ride-matching experience depends on continuously updated geospatial, traffic, and demand data. Without it, dynamic pricing and efficient routing would simply collapse.

 

Or take Expedia, whose customers rely on real-time travel data, such as flights and hotel availability, and pricing information to make booking decisions. Domino’s order tracking system updates in real time as pizzas are prepared and delivered, while Pinterest serves dynamic, personalised content based on live engagement trends.

 

These companies and many others like them run on real-time data streaming. It’s what enables their AI models to deliver accurate, actionable insights and adapt to changing conditions moment by moment; the services they provide could not be delivered without to-the-second insights.

 

But recent research from Confluent shows that many UK businesses are struggling to follow suit. Fewer than half (45%) of UK organisations have a central team advising on data streaming. An overwhelming 68% of UK data engineers — more than two-thirds — say that a shortage of AI-related skills is holding their organisations back. Even more telling, only 29% believe their business stakeholders can comfortably discuss data streaming, despite 93% agreeing that it will be critical to AI success.

 

For the businesses struggling with these issues, the numbers help to quantify a significant risk. Businesses that fail to establish a genuinely data-driven approach will increasingly see their AI projects fall short, while their competitors pull away ahead.

 

 

The role of the data streaming engineer

This problem has led to the emergence of a crucial role in the modern data stack: the data streaming engineer.

 

Data streaming engineers specialise in designing, building, and maintaining the pipelines that transport data across an organisation in real time. Their job is to ensure that this data arrives at its destination: clean, consistent, and with minimal latency. In other words, they’re responsible for ensuring that AI systems are left trying to reason on outdated or incomplete data, and the degradation of insights and efficiency that result from it.

 

In many ways, data streaming engineers are the underrated heroes of effective AI adoption. Just as we have seen the rise of data scientists and cloud architects over the past decade, and prompt engineers more recently, data streaming engineers are a response to a specific problem, which makes them an instrumental part of any serious data-driven organisation.

 

The alternative is to expect existing data engineers or software developers to handle these challenges as an add-on to their existing roles, despite the aforementioned skills gap. This often leads to burnout, the need to learn on the job, and compromising on existing engineering or software projects, none of which is ideal.

 

Real-time data infrastructure is complex and fast-evolving. It stands to reason that dedicated expertise is fundamental to keeping up. While technologies like Apache Kafka provide a powerful backbone for real-time data streaming, it’s the human touch that enables these tools to transform into robust, production-grade systems.

 

 

Investing in real-time data 

In all these sectors, the same principle applies: AI success depends on real-time data, and real-time data depends on the skills and infrastructure available to manage it effectively. This is why I truly believe that data streaming engineers will soon be as essential to enterprise success as any role in the data stack.

 

Organisations that recognise this early and invest accordingly will gain a significant competitive advantage — not only in AI, but across their digital strategies. To truly future-proof their data strategies, organisations need to embrace the real-time mindset from the top down.

 

Those that fail to adapt risk falling into the trap of overpromising on AI while underdelivering. As more AI-driven use cases become core to customer experience and operational excellence, businesses without real-time data capabilities will find themselves unable to compete.

 

Ultimately, the path forward is clear. AI is already reshaping industries and the way that almost every employee goes about their day-to-day. But to make AI work, businesses must ensure that their data foundations are fit for purpose.

 

That means embracing real-time data streaming, building teams with the right expertise and empowering them with the right tools. For businesses globally, the message is simple: invest now, or risk being left behind.

 

---

 

Peter Pugh-Jones is Director of strategic accounts at Confluent and KanawatTH

 

Main image courtesy of iStockPhoto.com