The Cost of Ignorance

Part 2 of Iterating in the Dark:
Organizational Blindness in AI Evaluations

← Part 1: A Structural Flaw in Judgment | Series Index

"Uncertainty is expensive. Ignoring uncertainty is negligent. Rewarding those who hide it is how organizations fail."
— again, just me

AI Evals are Estimations of Random Variables - let that sink in

Let's frame our evaluation problem as follows: we have M AI systems to evaluate. Each could go to production — or not, based on what we decide given our eval results (Let's assume for now we are deploying internally or at one customer - so let's not worry about variabilities across customers). If deployed, each system can generate positive value — or it can harm (generate negative value).

For example, consider an Incident Auto-Resolution agent that attempts to fix IT tickets automatically. Each execution might solve the problem completely (high positive value), give useful hint or workaround (still some positive value), do nothing useful (or say "i don't know" and do nothing - probably value around zero or slightly negative), or take a wrong action that makes things worse (high negative value).

Or, consider an Incident Summarization agent — each summary might accelerate triage, add nothing, or mislead the analyst (high negative value). The value generated varies, execution by execution.

For now, assume we have a magic function Vf that captures the value that one execution of an AI system generates. In reality this metric is hard to define and hard to measure, but let's assume we have it.

If we pick a random execution and apply Vf to it, we get a number — the value of that execution. The number we get depends on which execution we picked. Different executions yield different values.

If we could observe many executions and record Vf for each, we'd see a distribution of values emerge. For example, consider the Incident Auto-Resolution agent: Imagine we deploy it and observe 1,000 executions. For each, we record Vf — the value generated. We might see:

• 400 executions where the agent solved the problem correctly: Vf ranging from +5 to +50
• 200 executions where it gave a useful hint: Vf around +2 to +10
• 250 executions where it said "I don't know" and did nothing: Vf around 0 or −1
• 100 executions where it took a wrong action on a minor issue: Vf around −10 to −20
• 50 executions where it took a wrong action on a critical incident: Vf ranging from −50 to −200

Plot these 1,000 values as a histogram. That's the distribution of V for this agent.

We therefore conceptualize our understanding of value for this use case as a random variable V, described by a distribution of values, which in this case we obtain empirically. Each time the system runs, V takes a value — sometimes positive, sometimes negative, sometimes large, sometimes small. The Value random variable doesn't have one value. It has a shape — a distribution of outcomes, shaped by the mix of incidents it encounters, the actions it takes, and how those actions land in context.

If you take one thing from this series, let it be this: what you call "AI eval" in reality is "estimating the distribution of a random variable".

Taking decisions

When we evaluate an agent to choose whether to deploy for us (or across the board for a set of customers), ideally we want to know the distribution of values we expect it to generate, or at least have a ballpark idea. Doing so means deciding on what distribution shapes are acceptable.

Both measuring / estimating the distribution and deciding on what distributions are acceptable are hard problems.

So let's make some simplifying assumptions and decisions. Consider the following possible decision rules (E[V] is the expected value of V, take it as what we think the average value of the agent will be):

If this sounds too complex for you and you are ignoring it, you should not be in the business of taking decisions on AI systems, not even as an executive. You are taking decisions by making choices and assumptions you don't know you are making.

"Eval" implies "forming a belief about how the shape of value looks like"

Ideally you can run your agent at scale on the exact data it will see in production and get the distribution of V, which will in turn enable to you make informed decisions.

In practice, you have some measures obtained in some way by your team on some dataset. Based on this, you will form a belief about V. Or, more likely, you will form a belief about E[V], the expected value - or average value - of V, or, a belief about P(V > 0), the probability that V is positive, or whatever is your target metric. (Remember that figure with the metric of 89%? That implicitly states that the agent is expected to generate a positive value 89% of the time).

Your estimate can be wrong. Sometimes a little, sometimes a lot. The cost depends on how you are wrong and how wrong you are — not on why.

The cost of hiding uncertainty

If your estimate is off, you either deploy a bad system (harm in production) or miss a good one (opportunity cost). The cost scales with how wrong you are.

Take our Incident Auto-Resolution agent. Suppose the true distribution has E[V] = −5 — on average, it hurts slightly more than it helps. But your evaluation estimates Ê[V] = +8.

You deploy. Over the next quarter, the system runs 10,000 times. The expected total harm: 10,000 × (−5) = −50,000 value points.

You were off by 13 points. That error, multiplied by production volume, became a quarter's worth of accumulated harm.

But how can our estimates be wrong if we experiment? and are they wrong "by chance", occasionally, or are they structurally wrong? and what can we do about it? We'll come to this in a second. But for now let's focus on the cost of hiding uncertainty.

Consider a presentation that tells you that an agent is 85% correct. Then consider another presentation where somebody tells you: "I am fairly sure that the agent will be correct between 60 and 90% of the time - and that's all I can say."

The second report is more informative and in many cases more honest than the first. It exposes our believed uncertainty, and invites questions about the basis of our belief.

You rarely hear that. It makes the team look unprepared and executives are likely to object and complain - which by itself simply results in this kind of reports being rare and in an increase of reports that give a false sense of confidence. As a decision maker - and as an engineer or scientist - once you hear that conclusion with indication of uncertainty you may decide that that's "good enough" or you may decide that we need some more investigations. Or, that we can move to prod, but cautioning the customer that our belief is as stated.

Every time as a manager or executive you do not ask and reward the indication of uncertainty (or worse, you "punish" it by making negative considerations on the team), you are setting up your org up for failure. The AI industry should learn from airline safety and similar industry where a lot of effort is spent on making sure that report of issues (eg, pilot tiredness, even pilot errors) are communicated and not "retaliated" against.

What decisions look like without uncertainty

When someone reports a point estimate without uncertainty, you have exactly two options:

That's it. You act as if the estimate is the truth.

What decisions look like with uncertainty

When someone communicates their belief with a range, you have more options:

The third option — investigate further before deciding — doesn't exist when you only hear a point estimate.

Next: Part 2b: Uncertainty vs Variability — When you see a range, what does it mean?