The most dangerous AI is when it agrees with you

At our recent user conference, I put forward a slightly uncomfortable idea: the biggest risk in artificial intelligence (AI) isn’t hallucination, bias, or even security. It’s when it agrees with you.

Because when AI consistently tells us what we want to hear, when it aligns neatly with our assumptions, it creates the illusion of validation. And in finance, where judgement matters more than ever, that illusion can be far more dangerous than an obvious mistake.

The power is real, but so are the risks

There’s no denying the value AI is already bringing to the finance function. It accelerates analysis, unlocks predictive insight, and removes much of the friction from routine work. In a world where planning has become more uncertain and less anchored in historical patterns, those capabilities are not just useful; they’re becoming essential.

As a result, AI is shifting from being a tool to something that feels closer to a collaborator. That shift is subtle, but important. Because as soon as we start treating AI as something more than a tool, as something we trust to guide rather than simply assist, we begin to change how we engage with its output. Speed increases. Friction falls away. And with that, scrutiny often follows.

When helpful becomes agreeable

One of the less visible behaviours emerging from modern AI systems is what’s known as sycophancy. In simple terms, it means the system tends to align itself with the user’s apparent beliefs or assumptions, even when those beliefs are off the mark.

At one level, this is understandable. These systems are designed to be helpful, contextual, and responsive. They are trained to produce answers that feel relevant and useful. But there is a fine line between being helpful and being agreeable.

The moment an AI system starts reinforcing rather than testing your thinking, it changes the nature of the interaction. It stops acting as a source of insight and starts acting as a mirror. And a mirror doesn’t improve judgement.

The illusion of validation

The real risk is not that AI gets things obviously wrong. Most experienced finance professionals can spot a bad number, an unrealistic assumption, or a flawed calculation fairly quickly. The greater danger lies in answers that feel right.

You ask a question. The response comes back aligned with your expectations. It reads well. It sounds plausible. It confirms your direction of travel. It feels like corroboration. But often, it is simply convergence – a reflection of how the question was framed, the assumptions embedded within it, and the signals the AI has picked up from you.

In finance, where much of the work revolves around challenging assumptions, that distinction matters. If AI consistently validates rather than interrogates, it quietly erodes the very discipline the profession relies on.

The reinforcing loop

Left unchecked, this dynamic doesn’t stay isolated to individual interactions. It compounds. You begin to trust the system more. The system continues to align with your thinking. Your confidence grows. Over time, dissent falls away. Alternative perspectives are less frequently explored. The range of possible outcomes narrows, not because the world has become simpler, but because the process has become less questioning.

This is where AI stops being a neutral tool and starts shaping the decision environment itself.

When assumptions go unchallenged

We’ve already seen examples of systems producing outputs that align with expectation rather than reality, not because the underlying data was missing, but because the model or the process was implicitly biased towards a preferred outcome. That pattern is not unique to AI. It’s something finance professionals have always had to guard against in forecasting and planning. What changes with AI is the speed and scale.

If assumptions are weak, AI will operationalise them quickly. If they are strong, it will amplify their value. The technology has very little tolerance for ambiguity; it will push in whichever direction it is pointed. Which is why relying on alignment as a proxy for accuracy is so risky.

This is a human problem

It is tempting to treat this as a limitation of the technology. In reality, it says far more about how we use it. AI behaves in ways that reflect its design and training. The real variable is the human sitting on the other side of the interaction.

If we approach it passively, accepting outputs at face value, favouring speed over scrutiny, we create the conditions for poor decisions. Not because the AI is inherently flawed, but because we have removed the challenge from the process.

AI doesn’t replace bad thinking. But it will accelerate it, unless we are deliberate about how it is applied.

What this means in practice

For finance teams, the implication is straightforward but uncomfortable: adopting AI is not simply a technical shift. It is a behavioural one. The discipline needs to stay exactly where it has always been, perhaps even more so. That means: treating outputs as inputs, not conclusions; challenging assumptions deliberately; introducing structure into how AI is used in decision-making; and maintaining clear accountability for the final call. It also means being comfortable with friction.

In many ways, the value of a strong finance function has always come from its ability to slow things down at the right moment, to question, to test, to probe. AI, by design, removes some of that friction. The responsibility then is to put the right forms of it back in.

Judgement remains the differentiator

Access to AI is no longer special. Almost everybody has it. What will separate organisations over the next few years is not whether they use AI, but how thoughtfully they use it.

Judgement still matters. Context still matters. Critical thinking still matters.

AI can strengthen those things when used well. But it cannot replace them.

Final thoughts

A lot of the public conversation around AI focuses on obvious failures and hallucinations. Those risks are real, but they are usually visible. The more insidious risk is when nothing looks wrong at all. When the answer aligns neatly, when it supports your thinking, when it feels reassuring. Because that is the moment you are least likely to question it.

And that is why the most dangerous AI isn’t the one that makes mistakes – it’s the one that tells you you’re right.

As published in AccountingWeb - June 2026