When Even One of the World's Most Successful AI Companies Gets It Badly Wrong, Is It Any Wonder Their Customers Do Too?

Leadership Capital Group · The Board Intelligence Series

There is a particular kind of irony when the company building some of the world's most advanced AI models delivers, all by itself, a textbook example of how AI governance can go wrong. Not in the model's intelligence - but in the simple, human decision of when a customer needs to reach an actual person.

That is precisely what recently happened at Anthropic, the company behind Claude.

The case, briefly

A customer experienced three days of failed payment attempts on Anthropic's platform. The customer had done their homework thoroughly: the bank confirmed that no transactions were reaching it at all on their end, and - critically, an ordinary subscription payment on the same account went through without issue that same evening. That ruled out a general payment failure. The fault sat isolated somewhere in Anthropic's own sales flow.

The customer contacted support. Again and again. And explicitly requested escalation to a human three separate times.

What followed was nine consecutive replies from an AI support agent that, in substance, repeated the same generic checklist - billing address checks, 3D Secure, VPN interference, alternative payment methods - without ever engaging with the customer's central counter-evidence. Along the way, the agent contradicted itself on a core fact: the customer's organisation ID was first declared not to exist in the system, then described as "likely valid but not visible in all system views," and finally explained by stating the customer had no organisation at all. When the customer demanded to speak with a human, he was told that option is reserved exclusively for Enterprise customers.

Not a flaw in the model. A structural refusal.

Why this matters for your board

The tempting - but wrong - focus here would be to single out Anthropic as a technology company. What's genuinely interesting is something else: this is not an example of AI "hallucinating" or inventing incorrect facts. It is an example of an organisation that is, technically, among the most capable in the world at building AI - and that still failed on the most fundamental governance question of all: when does a human need to step in?

This is precisely the kind of failure we examine in depth in the Stakeholder Blindness workbook: human escalation was gated behind a payment tier, with no fallback for the customers experiencing the most critical failures. The design was optimised for customer-value segment - not for the severity of the situation. It is blindness toward an entire customer group, not malicious intent, but the consequence is the same: a customer in an acute, documented crisis, unable to reach anyone with the authority to resolve it.

Where the decision was actually made - or never was

Dig one layer deeper, and you find a question that the The Missing Decision workbook is built to raise: who actually decided that human support should be a matter of payment tier rather than a matter of severity? It is rarely a deliberate, malicious decision - far more often it is a decision that was never formally made by anyone holding the full picture. A product team optimises for cost. A support team optimises for volume. Neither owns the question: "What happens to that customer if everything goes wrong and they don't have access to the expensive plan?"

When no one owns that decision, it gets made by default - and default is rarely in the customer's interest.

What it costs to invest in AI without asking the question first

Finally, we land back at the core of the AI Investment Workbook: automation bias - the tendency to trust a confident system rather than asking whether it is actually solving the problem. The more convincing an AI agent sounds, the less likely we are to notice that it isn't answering what the customer is actually asking. Anthropic's own case illustrates how easy it is to invest heavily in AI capability without investing equally in the governance that determines whether that capability actually creates value for the customer standing in crisis at 11pm on a Sunday night.

The point for your board

The eight Board Intelligence workbooks are built to reinforce one another, because governance failures rarely have a single cause. This one case cuts across three different frameworks at once:

●      AI Investment Workbook - automation bias and a missing task-fit assessment of where AI genuinely belongs

●      Stakeholder Blindness - a design optimised for the wrong segment

●      The Missing Decision - a decision that was never formally owned by anyone

If one of the world's leading AI companies can fall into that trap - with all the technical capacity and all the resources that exist - how realistic is it that your organisation avoids the same trap without a deliberate governance framework?

That is the question we believe every board should ask itself before approving the next AI investment.

This article is part of Leadership Capital Group's Board Intelligence series.

Read more about the AI Investment Workbook, Stakeholder Blindness, and The Missing Decision at leadershipcapitalgroup.dk.