Agents build things. Enterprises have to run them

I’ve never worked in the Kubernetes space, so KubeCon, its big event, is rarely top of mind, even when it’s in my backyard. Last week it attracted more than 10,000 attendees, yet a good friend still questioned its relevance in the age of AI. I’d mentioned KubeCon in a post and he texted back, “No one cares about the primitives anymore. Agents just build things!”

I found myself agreeing with him, but only for a moment. For those of us who live in AI, it’s easy to forget the reality of enterprise adoption. I’ve spent years warning against fashion-driven Kubernetes adoption, and I’m hardly nostalgic for developers spending their afternoons deciphering deployment configurations. If an agent can spare them that work, wonderful.

But there’s a rather large difference between no longer wanting to configure infrastructure and no longer needing infrastructure that works. For those who think Kubernetes’ obsolescence is just a prompt away, well, reality bites.

Indeed, many of the people embracing AI are precisely the people who understand why the primitives still matter. I posited this to Linux Foundation CEO Jim Zemlin, arguing, “AI changes how we create software faster than it eliminates the work of operating it.” “Yep,” he texted back, “Code is cheap [but] trust is expensive.” That’s a much better explanation of KubeCon’s relevance than nostalgia for YAML.

The AI crowd is already here

It’s not as if the Kubernetes crowd is allergic to AI. Far from it. As noted in a recent CNCF and SlashData report, 75% of cloud-native developers use AI coding tools, compared with 55% of their non-cloud-native peers. Of course, cloud-native developers aren’t synonymous with Kubernetes operators. Even so, it’s hard to find data to support the idea that the Kubernetes infrastructure community is stubbornly resisting AI. The two interests overlap.

Nor should we pretend enterprise agents are mostly imaginary. Zapier’s November 9 survey found that 71% of respondents’ organizations had agents in production. This may not be an entirely representative sample, given that it was 513 AI-experienced US managers at companies with at least 1,000 employees. And it may also not be the most interesting finding in the report. What drew my attention was that only 30% of leaders at organizations using agents said those agents completed more than 60% of assigned tasks without human intervention.

That’s cool. It’s real adoption. But it’s a long, long way from handing over the business for robots to run.

Indeed, both surveys challenge my friend’s cavalier “AI will do it all” response. Enterprises can enthusiastically use AI while insisting that their applications remain secure, recoverable, and affordable. This doesn’t make them slow. It shows maturity and a willingness to take responsibility beyond an impressive screen recording.

After all, an agent generating a service doesn’t settle which customer records it should access, how much capacity it deserves, or which other application gets priority when resources run short. Those decisions remain, even when software increasingly executes them.

We still have to pay for inference

Zemlin says the people at KubeCon were focused on security and getting more performance from less hardware and power. “That is not easy,” he adds.

Consider China Merchants Bank. In a September case study published by CNCF, the bank describes supporting AI training and inference across nearly 10,000 accelerator cards. Its problem wasn’t simply finding more hardware. Instead it was a matter of predictability and responsiveness: Training needs predictable capacity, and customer-facing inference needs to respond to changing demand. Keeping capacity available for peak traffic wastes money during quiet periods, while scaling too slowly leaves users waiting.

The bank combined Kubernetes with tools for scheduling, monitoring, scaling, and moving model data, with significant impact. It reported average accelerator compute utilization had risen from 35% to more than 60%, with inference cost per million tokens falling by over 60% for comparable models and services. Suddenly, not-so-cool Kubernetes became very cool indeed, because it helped to dramatically lower the cost of AI.

Apparently someone cared about the primitives.

That’s the part the “agents just build things” argument skips. An application can work perfectly well in a demonstration while consuming far more resources than it should. A deployment can succeed while leaving the next team to discover its permissions are too broad. Generating the configuration faster doesn’t establish that it’s the right configuration. Etc., etc.

As Isolvalent cofounder Thomas Graf told me in 2023, platform teams are increasingly expected to meet enterprise requirements “without disrupting and undoing the gains that have been made on agility and developer efficiency.” The job is to deliver the controls while preserving developers’ freedom

[…]
Content was trimmed to protect the source. Please visit the original article for the full text.

This article has been indexed from InfoWorld