Building on GenAI’s foundations
Building on GenAI’s foundations
Current infrastructure developments and real case benefits
Where we are today, two years in to the generative AI innovation wave, is analogous to when the first smartphone was introduced and everyone was waiting for that first breakthrough app: which ultimately arrived in the guise of Uber. But in order for Uber to work, a broad infrastructure needed to be established, encompassing 4G networks, app stores, payment systems, GPS maps etc. It didn’t happen overnight.
Over the last 12 months, AI platforms and LLMs have raised approximately $25 billion of capital. OpenAI has further improved its LLM architecture with the introduction of its latest version, GPT-4o, released in May ’24, which enhances the chatbot’s multimodal capabilities, bringing faster performance and efficiency.
What we are now starting to see is the emergence of AI middleware, as the industry looks to build on last year’s foundation model architecture and more cloud providers push into this space.
As companies raise capital to develop AI middleware, the aim is to move generative AI from being a productivity assistant and co-pilot, to actually taking on entire parts of the workflow. This will allow companies to develop more complex use cases by combining, or orchestrating, different models.
This AI middleware layer – and the subsequent generative AI apps being built - presents an opportunity for software firms to change how they think about target market size. Take for example the US customer support market. There are 50 to 100 start-ups in Hg’s network who are looking to take a share not of the productivity uplift that is eminently achievable today, but rather a share of the entire employee cost base.
In other words, they are framing this not in terms of a $10 billion software spend market opportunity, but a TAM that is ten times bigger. That is what AI middleware and the emergent application layer is seeking to address. And it is likely to become a fiercely contested market.
of capital raised by AI platforms in the last 12 months
“We are starting to see a new layer emerge of AI middleware. The scaffolding that makes all of these models easier to use; everything from LLM operations, orchestration, hallucination monitoring."
An infrastructure triptych
Some organisations have moved quickly to build on today’s existing infrastructure, in ways that have already led to meaningful cost savings. VMware has developed a three-pronged approach to leverage generative AI technology by framing it around three strategic questions: 1) How to pivot to GPUs; 2) How to transform its internal operations and 3) How to make existing products better.
VMware serves large regulated entities, for whom 60 to 70% of their data remains on premises in their own data centres. As they didn’t want to wait for their data to be moved to a data lake, VMware started off with limited use cases. This led it to pivot to a GPU instead of a CPU offering and build a Private AI solution, which according to former CEO Raghu Raghuram is already off to a “fast start”.
Secondly, the firm sought to build internal enterprise language model-based infrastructures to help its various business functions. This has enabled VMware to reduce sales & marketing costs, R&D costs and improve its cost margins, which combined have led to a 10% increase in P&L; equivalent to $1 billion.
Thirdly, it built a co-pilot model to leverage 15 years of proprietary data (i.e. helpdesk requests) to sell as an adjunct to its existing management products.
This is a playbook that software CEOs can apply today, even before the longer-term revenue-driving benefits of AI middleware are taken into account.
Other organisations, such as Addepar, are seeing the meaningful benefits as a result of the work they’ve done, and the money they’ve spent, to normalise their data foundation layer.
“Starting with a data foundation that is normalised…you need that. Fortunately, a lot of the investment we’ve made over the last three or four year has made it so that every single data point is resident in the same data lake house. We partnered closely with Data Bricks on our underlying infrastructure.”
- Eric Poirier, Addepar
This focus on the data foundation layer has led to radical advances in productivity, helped further by Addepar choosing to use its own deep LLM, which was trained on its vast proprietary data lake.
The model is able to augment or replace some of the customer support tasks as a way to determine the best way to accomplish tasks with Addepar’s broad set of tools.
This approach to data management creates a human loop dynamic. And as AI middleware evolves, model orchestration will allow more complex, sophisticated use cases to be deployed in the enterprise space.
This is expected to lead to a third wave of AI, the discovery phase, where the human in the loop will guide the AI to perform core reasoning. That will be a huge transformational shift, especially in cognitive fields such as R&D.
As will be revealed in the next piece, this will lead to the rise of ‘agentic’ models and autonomous reasoning loops, enabling workflow tasks to not only be performed better but to drive new, proactive insights.
