How to Evaluate LLMs, Metrics, Benchmarks & Python Code

You just deployed an LLM-powered feature. Users are complaining, some answers are wrong, some are great, and you can’t tell which is which. Why? Because you never set up proper evaluation.

I’ve seen this pattern dozens of times. Teams spend weeks on prompt engineering and fine-tuning, then ship without measuring whether any of it actually worked.

Evaluation isn’t optional. It’s the difference between “we think the model is good” and “we know the model scores 78% on our test set, up from 71% last week.”

In this guide you’ll learn:

• What the major LLM benchmarks (MMLU, HumanEval, MT-Bench) actually measure
• How to compute perplexity, BLEU, ROUGE, and BERTScore from scratch in Python
• How to build an LLM-as-judge pipeline for open-ended evaluation
• When to use human evaluation and how to measure annotator agreement
• How to evaluate RAG systems and avoid common evaluation mistakes
• Which evaluation frameworks (DeepEval, RAGAS) to use in production

Before we write any code, here’s how the evaluation landscape fits together.

There are three layers to LLM evaluation. The first is benchmarks, standardized tests like MMLU and HumanEval that let you compare models against each other. They answer: “How does this model rank overall?”

The second layer is metrics, quantitative scores you compute on your own data. BLEU, ROUGE, perplexity, pass@k. They answer: “How well does this model perform on my specific task?”

The third layer is judgment, human evaluation and LLM-as-judge. These catch what automated metrics miss: tone, helpfulness, safety. They answer: “Is this output actually good?”

Most teams need all three. Benchmarks for model selection. Metrics for tracking improvement. Judgment for catching edge cases.

We’ll build each layer from scratch. By the end, you’ll have a working evaluation pipeline you can adapt to any LLM project.

Prerequisites

• Python version: 3.9+
• Required libraries: numpy (1.24+), collections (stdlib)
• Optional libraries: openai (1.0+), rouge-score (0.1.2+), nltk (3.8+)
• Install:pip install numpy openai rouge-score nltk
• Time to complete: ~45 minutes
• Last reviewed: March 2026

What Is LLM Evaluation?

LLM evaluation is the process of measuring how well a language model performs on tasks you care about. It moves beyond “does the output look reasonable” to “can we quantify quality and track it over time.”

Think of it like grading student essays. You could eyeball a few and say “looks fine.” Or you could define rubrics, score consistently, and track improvement. LLM evaluation is the rubric approach, and I prefer it because it turns gut feelings into data.

There are three core questions every evaluation answers:

1. Correctness. Is the output factually accurate?
2. Relevance. Does it actually address what was asked?
3. Quality. Is it well-written, safe, and appropriate?

Different evaluation methods tackle different questions. Benchmarks handle correctness at scale. Metrics quantify specific quality dimensions. Human judgment catches everything else.

The evaluation landscape has exploded since 2023. Dozens of benchmarks, metrics, and frameworks compete for attention. I find the trick is knowing which ones matter for your use case, and ignoring the rest.

Key LLM Benchmarks Explained

Benchmarks are standardized tests. They present the model with predefined inputs, check the outputs against known answers, and produce a score.

You’ve probably seen benchmark numbers thrown around in model announcements. Here’s what the most important ones actually measure.

MMLU (Massive Multitask Language Understanding)

MMLU tests knowledge across 57 subjects, from abstract algebra to world religions. Each question is multiple-choice with four options.

The metric is simple: accuracy, what percentage of questions did the model get right. A score of 86% on MMLU means the model answered 86 out of 100 questions correctly across all 57 subjects. I like MMLU as a starting point, but it tells you less than you’d think.

Why it matters: MMLU is the most widely cited benchmark for general knowledge. But it has limits. Multiple-choice doesn’t test generation ability, and some questions have leaked into training data.

HumanEval (Code Generation)

HumanEval tests whether a model can write working Python code. It gives 164 programming problems and checks if the generated code passes unit tests.

Task: HumanEval/0
The model must complete the function
Then it runs against hidden unit tests

Sample test:
assert has_close_elements([1.0, 2.0, 3.9, 4.0], 0.3)
assert not has_close_elements([1.0, 2.0, 3.0], 0.25)

The metric is pass@k: generate k code samples, and check if at least one passes all tests. Pass@1 means one attempt. Pass@10 means ten attempts, did any of them work?

Other Important Benchmarks

Here’s a quick reference for benchmarks you’ll see in model comparisons.

The Limitations of Benchmarks

Benchmarks are useful but imperfect. Before you rely on them too heavily, know their failure modes.

Data contamination. If benchmark questions leaked into training data, scores are inflated. The model memorized answers instead of reasoning.

Narrow format. Most benchmarks use multiple-choice or short-answer formats. Real-world tasks involve long-form generation, multi-step reasoning, and ambiguous inputs.

Benchmark gaming. Models can be optimized specifically for benchmark performance without improving general capability. High MMLU scores don’t guarantee good chat behavior.

Decay. As models improve, benchmarks saturate. When every model scores 90%+, the benchmark stops being useful. MMLU-Pro was created because MMLU became too easy.

Evaluation Metrics You Can Compute in Python

Benchmarks compare models. Metrics measure specific quality dimensions on your own data. Here are the ones that matter most, and we’ll implement each one from scratch so you understand exactly what they compute.

I think building metrics yourself is the fastest way to develop intuition for what they actually capture, and more importantly, what they miss.

Perplexity, How Surprised Is the Model?

Perplexity measures how well a language model predicts text. Lower perplexity means the model finds the text less surprising, it assigns higher probability to the actual words.

The formula: perplexity = exp(-average log probability per token). A model with perplexity 10 is, roughly speaking, “choosing between 10 equally likely options” at each step.

Good model perplexity: 1.60
Bad model perplexity:  14.46

Lower = better (model is less 'surprised' by the text)

The good model has perplexity 1.60, it predicted the text with high confidence. The bad model’s perplexity of 14.46 means it was much more uncertain.

When to use perplexity: Comparing language models on the same text, measuring fine-tuning impact, detecting domain mismatch (perplexity spikes on unfamiliar text).

BLEU Score, Translation and Text Similarity

BLEU (Bilingual Evaluation Understudy) measures how much a generated text overlaps with a reference text. Originally designed for machine translation, it’s now used broadly for text generation tasks.

BLEU counts matching n-grams (sequences of n words) between the candidate and reference. A perfect match scores 1.0. No overlap scores 0.0.

Here’s BLEU computed from scratch. The core idea: count overlapping n-grams between candidate and reference, then combine them.

Two details make BLEU more nuanced. First, a brevity penalty punishes candidates shorter than the reference, you can’t game a high score by outputting only the words you’re confident about. Second, BLEU takes the geometric mean of n-gram precisions, so a zero at any n-gram level zeros out the whole score.

Watch how the geometric mean makes BLEU strict, even partial overlap isn’t enough if higher-order n-grams don’t match.

The exact match gets 1.0. Both the paraphrase and unrelated text score 0, they share too few higher-order n-grams with the reference.

You might be wondering: “Why does the paraphrase score zero?” Because while it shares some unigrams (“cat”, “on”, “a”, “mat”), it has zero matching bigrams or trigrams. The geometric mean kills it. This is BLEU’s biggest weakness, it’s purely lexical and can’t recognize synonyms or paraphrases.

ROUGE Score, Summarization Quality

ROUGE (Recall-Oriented Understudy for Gisting Evaluation) measures overlap from the reference side. While BLEU asks “how much of the candidate appears in the reference,” ROUGE asks “how much of the reference appears in the candidate.”

ROUGE-1 counts unigram overlap. ROUGE-L finds the longest common subsequence.

Summary 1 captures key terms (“president”, “climate”, “change”) and scores well. Summary 2 paraphrases everything, only “the” and “and” overlap with the reference, giving it a low 0.22 F1. ROUGE can’t recognize that “leader” means “president” or “weather” relates to “climate.”

BERTScore, Semantic Similarity Beyond Surface Overlap

BLEU and ROUGE both fail at the same thing: they can’t detect paraphrases. “The president spoke” and “the leader talked” share zero tokens, so they score zero, even though they mean the same thing.

BERTScore fixes this by comparing texts at the meaning level. Instead of matching exact words, it converts each token into a contextual embedding (using a pretrained model like BERT) and measures cosine similarity between the best-matching pairs.

In practice, you’d use the bert-score library: pip install bert-score. But understanding the cosine similarity core helps you interpret the scores, high BERTScore with low ROUGE means good paraphrasing, while low BERTScore with high ROUGE means surface copying without understanding.

Quick Comparison: Which Metric When?

Try It Yourself

Exercise 1: Compute ROUGE-1 for a summarization task.

Given a reference summary and three candidate summaries, compute the ROUGE-1 F1 score for each. Which candidate is the best summary?

Hints:
1. Overlap is the intersection of the two token sets. Precision = overlap/candidate_size. Recall = overlap/reference_size.
2. F1 = 2 * precision * recall / (precision + recall). Handle the case where precision + recall == 0.

LLM-as-Judge, Using AI to Evaluate AI

Here’s where things get interesting. Automated metrics have a fundamental problem: they can’t judge qualities like helpfulness, tone, or reasoning quality. These require understanding, not pattern matching.

The solution? Use a strong LLM to evaluate outputs from the model you’re testing. This is the “LLM-as-judge” pattern, and it’s become the standard approach for evaluating open-ended generation. I use it on nearly every project now.

The idea is simple. You give a judge model (typically GPT-4 or Claude) the prompt, the model’s response, and a scoring rubric. The judge returns a structured score.

Pairwise Comparison, A Better Judge Pattern

Single-score judging is noisy. The judge might give a 4 one time and a 3 the next for the same response. Pairwise comparison is more reliable: show two responses and ask “which is better?”

Running LLM-as-Judge at Scale

Here’s a complete evaluation function that calls the OpenAI API. In production, you’d run this on hundreds of test cases.

# NOTE: Requires OpenAI API key. Shown for reference.
# Set OPENAI_API_KEY environment variable before running.

"""
import openai
import json

def llm_judge(prompt, response, model="gpt-4o"):
    judge_msg = f'''Score this response (1-5) on:
    relevance, accuracy, clarity, completeness.
    Return JSON only.

    Prompt: {prompt}
    Response: {response}'''

    result = openai.chat.completions.create(
        model=model,
        messages=[{"role": "user", "content": judge_msg}],
        temperature=0,
    )
    return json.loads(result.choices[0].message.content)

# Example:
# scores = llm_judge("What is Python?", "Python is a programming language.")
# print(scores)
"""

print("LLM-as-Judge: use GPT-4o or Claude as evaluator")
print("Key parameters:")
print("  temperature=0  -> deterministic scoring")
print("  JSON output    -> structured, parseable results")
print("  Run twice with swapped positions for pairwise")

LLM-as-Judge: use GPT-4o or Claude as evaluator
Key parameters:
  temperature=0  -> deterministic scoring
  JSON output    -> structured, parseable results
  Run twice with swapped positions for pairwise

Try It Yourself

Exercise 2: Design a judge prompt for customer support quality.

Write a judge prompt that evaluates customer support responses on four criteria relevant to that domain. Include the scoring rubric.

Hints:
1. Good criteria for support: empathy/tone, problem resolution, actionability (did they give next steps?), accuracy of information provided.
2. Use a 1-5 scale and define what each score means. “1 = dismissive, 5 = warm and understanding” for empathy.

Score this customer support response (1-5) on each criterion.

**Customer Query:** My order hasn't arrived and it's been 2 weeks

**Agent Response:** I apologize for the delay. Let me check your order status.

**Criteria:**
1. Empathy (1-5): Does the agent acknowledge the customer's frustration?
   1=Dismissive, 3=Neutral, 5=Warm and understanding
2. Resolution (1-5): Does the response work toward sol

Human Evaluation, When Automated Metrics Aren’t Enough

Automated metrics and LLM judges are scalable. But they systematically miss certain failure modes. Human evaluation catches what machines can’t, and skipping it is a mistake I’ve watched teams regret.

When you need humans:
– Safety-critical applications (medical, legal, financial advice)
– Subjective quality (brand voice, creativity, humor)
– Edge cases that LLM judges handle poorly
– Calibrating your automated evaluation pipeline

Setting Up a Human Evaluation Protocol

A good human eval has five components.

=== Human Evaluation Protocol ===

1. Sample Selection
  Random sample of 50-200 outputs from production

2. Rubric Definition
  3-5 criteria with clear anchor descriptions

3. Annotator Training
  Show 10 examples with expected scores

4. Double Annotation
  2+ annotators per example, measure agreement

5. Agreement Metric
  Cohen's kappa > 0.6 means acceptable agreement

The most critical step is the rubric. Vague criteria like “quality” lead to inconsistent scores. Each criterion needs anchor examples.

But how do you measure whether your annotators actually agree? Raw agreement percentage is misleading, if 90% of examples are “Good,” two random annotators will agree most of the time by luck. Cohen’s Kappa corrects for this by subtracting the expected chance agreement from the observed agreement.

The annotators agree 80% of the time, but Cohen’s Kappa is only 0.58 (moderate). That’s because some agreement is due to chance. If you see kappa below 0.6, your rubric is probably too vague, tighten the criteria.

Building Your Evaluation Pipeline

Now let’s put everything together. This is the pipeline I’d recommend for any LLM-powered product, and it’s the structure I reach for whenever I start a new evaluation project.

The Three-Stage Pipeline

Building an Offline Evaluation Script

Here’s a complete evaluation harness you can adapt. It takes a test dataset, runs metrics, and produces a report.

The average ROUGE-1 F1 of 0.40 isn’t great, but that’s expected since the predictions paraphrase rather than copy the reference text. In practice, you’d combine this with an LLM judge for a fuller picture.

Try It Yourself

Exercise 3: Add length and vocabulary diversity checks to the evaluation pipeline.

Extend the evaluate_model_outputs function to also compute average response length (in words) and vocabulary diversity (unique words / total words).

Hints:
1. Word count: len(pred.split()). Vocabulary diversity: len(set(pred.lower().split())) / len(pred.lower().split()).
2. A diversity score near 1.0 means every word is unique (good). Near 0.5 means lots of repetition (may indicate degenerate output).

Evaluating RAG Systems

RAG (Retrieval-Augmented Generation) adds retrieval to the evaluation challenge. You’re no longer just evaluating the LLM, you’re evaluating the retriever, the context, AND the generation. This is where I see most teams struggle, because there are more moving parts to test.

Three metrics matter for RAG:

Faithfulness. Does the response only contain information from the retrieved context? An unfaithful response hallucinates facts not in the source.

Answer relevance. Does the response actually answer the question? A response can be faithful (everything it says is in the context) but irrelevant (it doesn’t address the query).

Context relevance. Did the retriever pull the right documents? Irrelevant context leads to irrelevant answers.

Common Evaluation Mistakes

I keep seeing the same mistakes across projects. Each one silently undermines your evaluation results, and you won’t realize it until production breaks.

Mistake 1: Evaluating on training data. If the model saw the test cases during training, your scores are inflated. Always use a held-out test set that was never part of training or fine-tuning.

Mistake 2: Using a single metric. ROUGE alone misses semantic similarity. Perplexity alone misses factual errors. Use 2-3 complementary metrics.

Mistake 3: Ignoring edge cases. Average scores hide failures. A model with 90% average accuracy might fail 100% on a critical subcategory. Break down scores by category.

Mistake 4: Not versioning your evaluation. When you change your test set or rubric, old scores become incomparable. Version everything: test data, prompts, scoring criteria, model checkpoints.

Here’s the full list with fixes, I keep this pinned on every project.

Choosing the Right Evaluation Strategy

Not sure where to start? Different tasks need different evaluation approaches. Here’s the decision framework I use.

Evaluation Frameworks Worth Knowing

Once you understand how metrics work under the hood, you’ll probably want a framework that handles the boilerplate. Here are the three I’d recommend looking at.

DeepEval is the most comprehensive open-source option. It gives you 14+ built-in metrics (hallucination, toxicity, answer relevance, faithfulness) and integrates with pytest for test-driven evaluation.

# DeepEval quickstart (pip install deepeval)
# from deepeval import evaluate
# from deepeval.metrics import AnswerRelevancyMetric
# from deepeval.test_case import LLMTestCase

# metric = AnswerRelevancyMetric(threshold=0.7)
# test_case = LLMTestCase(
#     input="What is Python?",
#     actual_output="Python is a programming language.",
# )
# metric.measure(test_case)
# print(f"Score: {metric.score}, Passed: {metric.is_successful()}")

print("DeepEval: pip install deepeval")
print("  14+ metrics out of the box")
print("  pytest integration for CI/CD")
print("  Best for: comprehensive evaluation pipelines")
#> DeepEval: pip install deepeval
#>   14+ metrics out of the box
#>   pytest integration for CI/CD
#>   Best for: comprehensive evaluation pipelines

RAGAS specializes in RAG evaluation. If you’re building retrieval-augmented systems, RAGAS gives you faithfulness, answer relevance, and context precision metrics designed specifically for that architecture.

Promptfoo takes a different approach, it’s a CLI tool for A/B testing prompts across multiple models. Think of it as unit testing for your prompts.

frameworks = {
    "DeepEval": {
        "focus": "General LLM evaluation",
        "metrics": "14+ (hallucination, toxicity, relevancy...)",
        "install": "pip install deepeval",
    },
    "RAGAS": {
        "focus": "RAG pipeline evaluation",
        "metrics": "Faithfulness, context precision, answer relevancy",
        "install": "pip install ragas",
    },
    "Promptfoo": {
        "focus": "Prompt A/B testing across models",
        "metrics": "Custom assertions + LLM graders",
        "install": "npx promptfoo@latest init",
    },
}

print(f"{'Framework':<12} {'Focus':<30} {'Install'}")
print("-" * 70)
for name, info in frameworks.items():
    print(f"{name:<12} {info['focus']:<30} {info['install']}")
#> Framework    Focus                          Install
#> ----------------------------------------------------------------------
#> DeepEval     General LLM evaluation         pip install deepeval
#> RAGAS        RAG pipeline evaluation        pip install ragas
#> Promptfoo    Prompt A/B testing across models npx promptfoo@latest init

Summary

LLM evaluation has three layers: benchmarks for model selection, metrics for quality measurement, and judgment (human or LLM) for nuanced assessment.

Here’s what to remember:

1. Benchmarks (MMLU, HumanEval, etc.) compare models but don’t predict performance on your task
2. Metrics (BLEU, ROUGE, perplexity, BERTScore) quantify specific quality dimensions, always use 2-3, not just one
3. LLM-as-judge scales evaluation for open-ended tasks, calibrate with human scores
4. Human evaluation remains the gold standard for safety-critical and subjective tasks
5. RAG evaluation requires testing retriever and generator separately
6. Frameworks (DeepEval, RAGAS) handle boilerplate once you understand the underlying metrics
7. Version everything, test sets, rubrics, and model checkpoints

# Complete code from: How to Evaluate LLMs
# Requires: pip install numpy
# Python 3.9+

import numpy as np
from collections import Counter

# --- Perplexity ---
def compute_perplexity(log_probs):
    return np.exp(-np.mean(log_probs))

# --- BLEU ---
def compute_bleu(reference, candidate, max_n=4):
    ref_tokens = reference.lower().split()
    cand_tokens = candidate.lower().split()
    bp = min(1.0, np.exp(1 - len(ref_tokens) / max(len(cand_tokens), 1)))
    scores = []
    for n in range(1, max_n + 1):
        ref_ng = Counter(tuple(ref_tokens[i:i+n]) for i in range(len(ref_tokens)-n+1))
        cand_ng = Counter(tuple(cand_tokens[i:i+n]) for i in range(len(cand_tokens)-n+1))
        matches = sum(min(c, ref_ng[ng]) for ng, c in cand_ng.items())
        total = max(sum(cand_ng.values()), 1)
        scores.append(matches / total)
    if any(s == 0 for s in scores):
        return 0.0
    return bp * np.exp(np.mean([np.log(s) for s in scores]))

# --- ROUGE-1 ---
def rouge_1(reference, candidate):
    ref_t = set(reference.lower().split())
    cand_t = set(candidate.lower().split())
    overlap = ref_t & cand_t
    if not overlap:
        return {"precision": 0, "recall": 0, "f1": 0}
    p = len(overlap) / len(cand_t)
    r = len(overlap) / len(ref_t)
    return {"precision": round(p, 4), "recall": round(r, 4),
            "f1": round(2*p*r/(p+r), 4)}

# --- Cosine Similarity (BERTScore core) ---
def cosine_similarity(vec_a, vec_b):
    dot = sum(a * b for a, b in zip(vec_a, vec_b))
    norm_a = sum(a ** 2 for a in vec_a) ** 0.5
    norm_b = sum(b ** 2 for b in vec_b) ** 0.5
    return dot / (norm_a * norm_b) if norm_a * norm_b > 0 else 0.0

# --- Cohen's Kappa ---
def cohens_kappa(a1, a2):
    n = len(a1)
    po = sum(x == y for x, y in zip(a1, a2)) / n
    labels = set(a1) | set(a2)
    pe = sum((a1.count(l)/n) * (a2.count(l)/n) for l in labels)
    return (po - pe) / (1 - pe) if pe != 1 else 1.0

# --- Evaluation Harness ---
def evaluate_model_outputs(test_cases, predictions):
    scores = []
    for tc, pred in zip(test_cases, predictions):
        r1 = rouge_1(tc["reference"], pred)
        scores.append(r1["f1"])
    return {"n": len(test_cases), "avg_rouge1": round(np.mean(scores), 4)}

print("All evaluation functions loaded. Script completed successfully.")

FAQ

Q: How many test cases do I need for reliable evaluation?

Minimum 50 for basic signals. 200+ for statistically significant comparisons between models. For category-level breakdowns, you need 20-30 per category.

Q: Can I use a weaker model as an LLM judge?

You can, but quality drops. GPT-4 and Claude 3.5 are the standard judges. Open models like Llama 3.1 70B work for simple rubrics but struggle with nuanced quality assessment.

Q: How often should I update my evaluation test set?

Quarterly at minimum. More often if your product scope changes. The test set should always reflect current user queries and edge cases.

Q: Is BLEU still relevant in 2026?

For translation and structured generation, yes. For open-ended generation, prefer ROUGE + BERTScore or an LLM judge. BLEU’s inability to handle paraphrasing makes it insufficient as a sole metric.

Q: What’s the fastest way to set up evaluation from scratch?

Start with 50 representative test cases from your actual use case. Compute ROUGE-1 and run an LLM-as-judge with a simple rubric. That gives you two complementary signals in under a day.

Q: Should I use an evaluation framework like DeepEval or build my own?

Both. Understand the metrics by building them from scratch first (as this guide teaches). Then use a framework like DeepEval or RAGAS for production pipelines, they handle scaling, CI/CD integration, and metric aggregation that you don’t want to maintain yourself.

Related Topics

1. Prompt Engineering, optimizing prompts before evaluating their impact
2. Fine-Tuning LLMs, when to fine-tune vs prompt and how to measure improvement
3. RAG (Retrieval-Augmented Generation), building and evaluating retrieval pipelines
4. LLM Deployment, serving models in production with monitoring
5. AI Safety and Alignment, evaluating model behavior for safety
6. DPO (Direct Preference Optimization), training models on human preferences
7. Vector Databases, the retrieval layer that feeds RAG systems

References

1. Hendrycks, D. et al. (2021). “Measuring Massive Multitask Language Understanding.” ICLR 2021. arXiv:2009.03300
2. Chen, M. et al. (2021). “Evaluating Large Language Models Trained on Code.” arXiv:2107.03374. (HumanEval)
3. Papineni, K. et al. (2002). “BLEU: a Method for Automatic Evaluation of Machine Translation.” ACL 2002.
4. Lin, C-Y. (2004). “ROUGE: A Package for Automatic Evaluation of Summaries.” ACL 2004 Workshop.
5. Zheng, L. et al. (2023). “Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena.” NeurIPS 2023. arXiv:2306.05685
6. Zhang, T. et al. (2020). “BERTScore: Evaluating Text Generation with BERT.” ICLR 2020. arXiv:1904.09675
7. Liang, P. et al. (2023). “Holistic Evaluation of Language Models.” TMLR 2023. (HELM)
8. Zellers, R. et al. (2019). “HellaSwag: Can a Machine Really Finish Your Sentence?” ACL 2019.
9. Lin, S. et al. (2022). “TruthfulQA: Measuring How Models Mimic Human Falsehoods.” ACL 2022.
10. Evidently AI. “30 LLM Evaluation Benchmarks and How They Work.” Link