Stop guessing. Start calculating.
The ADJUDON Confidence Engine does not trust the model's self-reported confidence. It independently evaluates every AI agent decision using three signals — base probability, decision entropy, and historical vector similarity — and produces a single Confidence Performance Index (CPI) score. Under 10ms. Every decision.
A model that is wrong is wrong with confidence.
Language models produce a probability distribution over possible outputs. The highest probability is often reported as "confidence." But this number is uncalibrated — a model that says 92% may be correct 60% of the time, or 99% of the time, depending on the domain, the prompt, and the input distribution. There is no universal relationship between a model's self-reported confidence and its actual accuracy.
For a compliance officer, this is not a theoretical problem. A regulator will ask: "How confident was the system in this decision?" If the answer is "the model said 92%," the follow-up is: "What does that number mean? How was it validated?" Self-reported confidence is not evidence. It is an internal model signal with no external calibration. The Confidence Engine provides that calibration.
Three independent signals. One score.
Each pillar evaluates a different dimension of decision quality. The composite CPI score is a weighted combination — not a black box, but an auditable, decomposable metric your compliance team can explain to a regulator.
Base Model Probability
40%The model's own certainty for the chosen output. Extracted from the output logprobs or the reported confidence score. This is the signal most tools rely on exclusively — and the one most often miscalibrated. We include it, but we do not trust it alone.
What it catches: Nothing on its own. A confident wrong answer scores high here. That is exactly why the other two pillars exist.
Decision Entropy
30%How clearly the chosen output outperforms the alternatives. If the model considered "approve" at 0.52 and "block" at 0.48, the decision is essentially a coin flip — regardless of what the base probability says. Low entropy means the model had no clear preference. High entropy means the top choice dominated.
What it catches: Decisions where the model picked an answer but had no strong reason to prefer it over alternatives. These are the silent failures — high base confidence, low actual conviction.
Historical Vector Similarity
30%How similar this decision is to past decisions with known outcomes. Every successful trace is stored as a vector embedding via OpenAI's embedding API. New traces are compared against the historical corpus using cosine similarity. A decision on a novel, unprecedented input scores low here — even if the model is confident.
What it catches: Novel situations the model has never encountered. A loan application from an industry the model was not trained on. A fraud pattern that does not match any historical precedent. These are the cases where "confident but wrong" is most dangerous.
From three signals to one decision
The CPI score is a weighted linear combination. No hidden layers. No secondary model. Fully decomposable.
Extract base model probability
Extract the base model probability from the trace payload (reported confidence or logprobs).
Calculate entropy
If alternatives were provided, calculate entropy across the probability distribution of all considered outputs. If no alternatives, entropy defaults to a conservative estimate (lower confidence).
Generate vector embedding
Generate a vector embedding of the trace (input + output) and compute cosine similarity against the historical corpus for this agent.
Compute CPI score
Compute CPI = (0.40 × base) + (0.30 × entropy_score) + (0.30 × similarity_score). Normalize to 0.00–1.00.
Return decomposed result
Return the composite CPI score and all three pillar breakdowns in the API response. The Policy Engine uses the CPI score for enforcement. The Audit Trail logs it with full decomposition.
The system learns from your decisions — without training.
Every approved trace is stored as a vector embedding in the agent's historical corpus. This corpus grows with every decision your agent makes. New traces are evaluated against the full history via cosine similarity — meaning the Confidence Engine becomes more accurate over time, without any model retraining, without any data leaving your environment for training purposes, and without any manual calibration.
How it works:
At ingest, the trace (input context + output decision) is sent to the embedding API. The resulting vector is compared against all stored vectors for this agent using cosine similarity. The similarity score (0.00–1.00) represents how "familiar" this decision is relative to the agent's history. A score near 1.0 means the agent has made very similar decisions many times before. A score near 0.0 means this is unprecedented territory.
Privacy:
Vector embeddings are stored per-agent, per-workspace. They are never shared across organizations. The embedding API call is the only external data flow — and it is covered by the Zero Training Policy and the sub-processor agreement with OpenAI (SCCs in place).
Under 10ms. Fail-open. No bottleneck.
The Confidence Engine evaluation pipeline — including embedding generation, vector similarity search, entropy calculation, and CPI scoring — completes in under 10ms end-to-end under typical load. This is faster than the round-trip to most LLM providers.
Fail-open architecture:
If the Confidence Engine is unavailable (network failure, infrastructure issue), the SDKs default to fail-open — your agent's decision passes through without scoring. Your application is never blocked by ADJUDON downtime. This is enforced by design in both the Node.js and Python SDKs.
Throughput:
The Confidence Engine supports sustained throughput for all pricing tiers. Per-agent rate limiting is applied to prevent abuse. Enterprise customers receive dedicated compute with no shared rate limits.
Latency targets:
Under typical production load.
Know how confident your AI actually is.
The Confidence Engine scores every decision independently. No trust in self-reported numbers. Start scoring in under 10 minutes.