Core Metrics: Correctness, Groundedness, Faithfulness, Format Validity
Prerequisites: Building Golden Datasets.
Key Takeaway: Evaluating AI applications requires quantitative, decomposable metrics. By mastering the RAG Triad (Context Relevance, Groundedness/Faithfulness, and Answer Relevance) alongside structural format validity, you can isolate exactly where your pipeline failed—whether the vector retriever pulled the wrong chunks or the model hallucinated unsupported facts.
Why Single-Score Evaluations Are Useless
When a product manager says "the AI's answers are only 70% good", this number is diagnostically useless. Did the model fail because:
- The vector search query returned irrelevant documentation?
- The prompt retrieval was perfect, but the model ignored the text and invented facts?
- The model answered accurately, but output broken JSON that crashed the frontend?
To systematically diagnose and fix AI bugs, production evaluation separates performance into distinct, orthogonal metrics.
graph TD
Query["User Query: 'How to cancel subscription?'"] --> Retrieval["1. Vector Retrieval Engine"]
Retrieval --> Context["Retrieved Context Chunks"]
Context --> Generation["2. LLM Synthesis Generation"]
Generation --> Answer["Generated Output Answer"]
subgraph "The RAG Triad of Metrics"
M1["Context Relevance: Are the retrieved chunks actually focused on the query?"]
M2["Groundedness (Faithfulness): Are all claims in the answer strictly supported by context?"]
M3["Answer Relevance: Does the generated answer directly solve the user's question?"]
end
Query -.->|Measure M1| Context
Context -.->|Measure M2| Answer
Query -.->|Measure M3| AnswerThe Four Foundational AI Evaluation Metrics
| Metric | Definition | How It Is Measured | Target Threshold |
|---|---|---|---|
| 1. Groundedness (Faithfulness) | The ratio of factual claims in the answer that can be directly verified against the provided context. | Claim extraction + Context entailment verification | >= 0.95 (Zero tolerance for hallucinations) |
| 2. Context Relevance | The ratio of retrieved sentences that are directly useful for answering the prompt. | Semantic relevance filtering against the query | >= 0.80 (Minimizes context window noise) |
| 3. Answer Relevance | How directly and completely the answer addresses the user's intent without wandering. | Semantic similarity / Reverse question generation | >= 0.85 |
| 4. Structural Format Validity | The percentage of model outputs that strictly conform to runtime schema constraints (e.g. Zod). | Strict schema parsing (zod.safeParse()) | 100.0% (Zero runtime parser exceptions) |
Deconstructing Groundedness (Faithfulness)
Hallucination is the fatal flaw of LLMs in enterprise applications. Groundedness measures whether the model is hallucinating:
$$\text{Groundedness} = \frac{\text{Number of claims supported by retrieved context}}{\text{Total number of factual claims in generated answer}}$$
The Evaluation Protocol:
- Sentence / Claim Extraction: The generated answer is split into atomic factual statements ($C_1, C_2, \dots, C_n$).
- Entailment Verification: For each claim $C_i$, an evaluator verifies whether $C_i$ logically follows from the context $K$.
- If the model says "Our refund policy covers 30 days and includes free shipping", but the context document only mentions "30 days" and says nothing about shipping, Groundedness drops to $0.50$.
Why Traditional NLP Metrics (BLEU and ROUGE) Fail for Generative AI
In classical machine translation, researchers used BLEU and ROUGE. These metrics count exact n-gram token overlap:
$$\text{BLEU} = \text{Geometric mean of n-gram precision} \times \text{Brevity Penalty}$$
Why BLEU and ROUGE fail for LLMs:
- Reference Dependency: They require an exact reference text.
- Synonym Blindness: If reference is "The flight was canceled due to bad weather", and model outputs "Severe storms caused the airplane cancellation", BLEU gives a near-zero score despite 100% semantic correctness.
- Negation Blindness: If reference is "You can cancel at any time", and model outputs "You cannot cancel at any time", ROUGE-1 gives an 80% pass score despite meaning the exact opposite.
Concrete TypeScript Implementation: Groundedness & Claim Entailment Scorer
// lib/evals/groundedness-scorer.ts
export interface GroundednessReport {
score: number; // 0.0 to 1.0
totalClaims: number;
supportedClaims: string[];
unsupportedClaims: string[];
isGrounded: boolean;
}
export class GroundednessEvaluator {
/**
* Evaluates if claims in an answer are strictly grounded in retrieved context
*/
public static async evaluateGroundedness(
contextText: string,
answerText: string,
judgeModelClient: any
): Promise<GroundednessReport> {
const prompt = `
You are an expert factual auditor. Analyze the following Generated Answer against the Verified Context.
Step 1: Extract all atomic factual assertions made in the Generated Answer.
Step 2: For each claim, determine if it is STRICTLY ENTAILED by the Verified Context (True) or UNVERIFIED/HALLUCINATED (False).
Verified Context:
"""${contextText}"""
Generated Answer:
"""${answerText}"""
Output ONLY valid JSON matching this structure:
{
"claims": [
{ "claim": "string", "supported": boolean, "citationQuote": "quote from context if true" }
]
}
`;
const rawResponse = await judgeModelClient.generate(prompt, { temperature: 0 });
const parsed = JSON.parse(rawResponse);
const claims: Array<{ claim: string; supported: boolean }> = parsed.claims || [];
if (claims.length === 0) {
return {
score: 1.0,
totalClaims: 0,
supportedClaims: [],
unsupportedClaims: [],
isGrounded: true,
};
}
const supported = claims.filter((c) => c.supported).map((c) => c.claim);
const unsupported = claims.filter((c) => !c.supported).map((c) => c.claim);
const score = supported.length / claims.length;
return {
score: Math.round(score * 100) / 100,
totalClaims: claims.length,
supportedClaims: supported,
unsupportedClaims: unsupported,
isGrounded: score >= 0.95,
};
}
}Common Pitfalls & Anti-Patterns
- Averaging Metrics into a Single Score: Never compute
(Groundedness + Relevance + Schema) / 3. If schema validity is 0% (runtime crash), having 100% groundedness does not matter. Keep metrics separate and enforce hard thresholds. - Ignoring Format Failures in Benchmarks: If an evaluation harness crashes on invalid JSON during automated runs, developers often catch the error and skip the row. This artificially inflates pass rates by ignoring hard failures.
- Evaluating Synthetic Ground Truth Without Human Review: If your reference answers were generated by a model without human verification, your evaluation metric is merely testing whether two models share the same hallucinations.
Senior Engineer Interview Angle
Staff Engineer Question: "Our documentation assistant recently answered: 'You can deploy to AWS using our CLI.' Our documentation mentions AWS and mentions our CLI, but never states that our CLI can deploy to AWS. How would our evaluation suite catch this subtlety?"
Exemplary Answer:
*"Keyword matching and standard vector similarity fail here because both 'AWS' and 'CLI' appear in the retrieved context with high cosine similarity.
To catch this, we use Atomic Entailment Groundedness scoring:
- The claim extractor isolates the specific proposition:
[Our CLI supports AWS deployment].- The evaluator executes a Natural Language Inference (NLI) entailment check:
Does Context logically entail Proposition?- Because the documentation does not contain an entailment relationship connecting the two entities, the assertion is marked as
UNSUPPORTED.- Groundedness drops below our 0.95 CI threshold, failing the automated regression gate before reaching production."*
Quick Self-Check
- What are the three components of the RAG Triad?
- Why do n-gram metrics like BLEU and ROUGE fail for LLM evaluations?
- What is the mathematical definition of Groundedness (Faithfulness)?