In a datacentre in Mahwah, New Jersey, an array of servers belonging to a single hedge fund is crunching through 10 million messages from the New York Stock Exchange per second. On the line is more than $10 billion of assets. All the execution authority sits with the machine, making trades faster than any person could ever supervise. The human is well and truly out of the loop. 

In the world of numbers that is the public markets, this type of agent operator has been possible for decades. In the new world of Large Language Models just emerging, the rest of us are all facing the same question: how can we trust AI to act without supervision too?

At Anthemis, this is a question we’ve been exploring closely. Our work with startups across banking, insurance, healthcare, supply chains and other industries has highlighted the importance of building AI systems that can be relied upon in environments where mistakes can carry meaningful consequences. This is why we see the next trillion dollars of value coming from High Assurance AI.

What do we mean when we say High Assurance AI? In short, High Assurance AI refers to systems granted execution authority in high-consequence environments. The last three years have been the “Copilot Era”, where AI acts as a whisperer in the ear of a human decision-maker. If the AI hallucinates or fails, the human is the safety buffer. The cost of failure is low (a bad email, a weird image).

High Assurance AI is the next step, where companies will be deploying large language models:

In high consequence environments, operating in a domain where error is has asymmetric downside risk, e.g.

• Financial: Payments, Trading, Insurance Claims, Fraud Prevention.
• Digital: Cybersecurity, Critical Infrastructure Code.
• Physical: Healthcare, Energy Grid, Manufacturing, Logistics.

With execution authority, the AI having the “keys” to the system: 

• Low Assurance: An AI that flags a suspicious transaction for a human to review.
• High Assurance: An AI that freezes the account and reverses the transaction instantly.

While engineering for correctness, the technology stack prioritising predictability over creativity:

• Use of formal verification, control theory, and deterministic guardrails.
• “Explainability” as a requirement.
• The system must fail safely (fail-secure/fail-safe) rather than fail erratically.

Right now, enterprise leaders in critical industries we are speaking to are seeing mixed results from their first AI transformation attempts. While some efficiencies have been shown in a few domains (software engineering being the meteoric example), many early attempts simply put a chat interface on top of a constrained proprietary knowledge-base and little else. Knowledge workers and decision-makers found that jumping out of their workflow to use a new tool was too inefficient to justify the switch, and as a result have become increasingly sceptical that today’s enterprise AI systems can satisfy the requirements of high-stakes environments

At Anthemis, we believe the recent stumbling of applied AI in these verticals came not from insufficient capabilities in the models, but from not enough responsibility being delegated to AI agents themselves. The promise of AI in the workplace is to remove administrative tasks from knowledge workers, to add semantic understanding into automated decision-making, and to operate at a scale that would be economically impossible with human oversight. In-house Google Search and meeting notetakers are nice, but we believe much more is possible with High Assurance AI.

So far, the blocker to high assurance has been that LLMs produce output variance and hallucinations, and under the current paradigm that doesn’t look like it’s going to go away. Large language models get much of their capabilities from their probabilistic nature. If you dial the variance down on LLMs to 0 and make them deterministic, they lose many of the ‘human-like’ intelligence features that make them such an exciting breakthrough. We believe the answer is therefore in the engineering outside of the model.

We’re keen to engage with startups that agree that High Assurance AI is an engineering problem that sits around the model, rather than in the model weights. That’s one of the reasons we co-founded CommonAI, and we see it as a focus area for Anthemis over the next five years. This isn’t about taking the next largest model from the big labs and giving it write access to the general ledger. This is about taking the raw intelligence of AI and giving it the harness, scaffolding and guardrails it needs to perform key operations in the economy at scale.

With High Assurance AI, we envision a future of far greater human capital efficiency in our most important verticals. We see:  

• Corporate treasurers no longer manually managing cash positions across dozens of accounts and currencies, with AI autonomously moving money between accounts, investing excess cash overnight, ensuring sufficient liquidity for payroll and payments.

• Healthcare diagnostic systems that don’t just suggest findings but autonomously order follow-up tests and route urgent cases to specialists in minutes.

• Insurance claims processed and paid autonomously in minutes, not weeks, with AI verifying damage, checking policy, calculating payout and transferring funds without human approval.

• Manufacturing lines detecting quality issues, adjusting processes and scheduling maintenance without stopping production for human review.

We’re taking a great interest in this area in the coming years at Anthemis. If you’re working on a startup that grants execution authority to AI in a high-consequence environment, we want to hear from you. Reach out to us on the contact page.