lm-evaluation-harness - LLM Benchmarking Quick start

lm-evaluation-harness evaluates LLMs across 60+ academic benchmarks using standardized prompts and metrics.

Installation:

pip install lm-eval

Evaluate any HuggingFace model:

lm_eval --model hf \ --model_args pretrained=meta-llama/Llama-2-7b-hf \ --tasks mmlu,gsm8k,hellaswag \ --device cuda:0 \ --batch_size 8

View available tasks:

lm_eval --tasks list

Common workflows Workflow 1: Standard benchmark evaluation

Evaluate model on core benchmarks (MMLU, GSM8K, HumanEval).

Copy this checklist:

Benchmark Evaluation: - [ ] Step 1: Choose benchmark suite - [ ] Step 2: Configure model - [ ] Step 3: Run evaluation - [ ] Step 4: Analyze results

Step 1: Choose benchmark suite

Core reasoning benchmarks:

MMLU (Massive Multitask Language Understanding) - 57 subjects, multiple choice GSM8K - Grade school math word problems HellaSwag - Common sense reasoning TruthfulQA - Truthfulness and factuality ARC (AI2 Reasoning Challenge) - Science questions

Code benchmarks:

HumanEval - Python code generation (164 problems) MBPP (Mostly Basic Python Problems) - Python coding

Standard suite (recommended for model releases):

--tasks mmlu,gsm8k,hellaswag,truthfulqa,arc_challenge

Step 2: Configure model

HuggingFace model:

lm_eval --model hf \ --model_args pretrained=meta-llama/Llama-2-7b-hf,dtype=bfloat16 \ --tasks mmlu \ --device cuda:0 \ --batch_size auto # Auto-detect optimal batch size

Quantized model (4-bit/8-bit):

lm_eval --model hf \ --model_args pretrained=meta-llama/Llama-2-7b-hf,load_in_4bit=True \ --tasks mmlu \ --device cuda:0

Custom checkpoint:

lm_eval --model hf \ --model_args pretrained=/path/to/my-model,tokenizer=/path/to/tokenizer \ --tasks mmlu \ --device cuda:0

Step 3: Run evaluation

Full MMLU evaluation (57 subjects)

lm_eval --model hf \ --model_args pretrained=meta-llama/Llama-2-7b-hf \ --tasks mmlu \ --num_fewshot 5 \ # 5-shot evaluation (standard) --batch_size 8 \ --output_path results/ \ --log_samples # Save individual predictions

Multiple benchmarks at once

lm_eval --model hf \ --model_args pretrained=meta-llama/Llama-2-7b-hf \ --tasks mmlu,gsm8k,hellaswag,truthfulqa,arc_challenge \ --num_fewshot 5 \ --batch_size 8 \ --output_path results/llama2-7b-eval.json

Step 4: Analyze results

Results saved to results/llama2-7b-eval.json:

{ "results": { "mmlu": { "acc": 0.459, "acc_stderr": 0.004 }, "gsm8k": { "exact_match": 0.142, "exact_match_stderr": 0.006 }, "hellaswag": { "acc_norm": 0.765, "acc_norm_stderr": 0.004 } }, "config": { "model": "hf", "model_args": "pretrained=meta-llama/Llama-2-7b-hf", "num_fewshot": 5 } }

Workflow 2: Track training progress

Evaluate checkpoints during training.

Training Progress Tracking: - [ ] Step 1: Set up periodic evaluation - [ ] Step 2: Choose quick benchmarks - [ ] Step 3: Automate evaluation - [ ] Step 4: Plot learning curves

Step 1: Set up periodic evaluation

Evaluate every N training steps:

!/bin/bash

eval_checkpoint.sh

CHECKPOINT_DIR=$1 STEP=$2

lm_eval --model hf \ --model_args pretrained=$CHECKPOINT_DIR/checkpoint-$STEP \ --tasks gsm8k,hellaswag \ --num_fewshot 0 \ # 0-shot for speed --batch_size 16 \ --output_path results/step-$STEP.json

Step 2: Choose quick benchmarks

Fast benchmarks for frequent evaluation:

HellaSwag: ~10 minutes on 1 GPU GSM8K: ~5 minutes PIQA: ~2 minutes

Avoid for frequent eval (too slow):

MMLU: ~2 hours (57 subjects) HumanEval: Requires code execution

Step 3: Automate evaluation

Integrate with training script:

In training loop

if step % eval_interval == 0: model.save_pretrained(f"checkpoints/step-{step}")

# Run evaluation
os.system(f"./eval_checkpoint.sh checkpoints step-{step}")

Or use PyTorch Lightning callbacks:

from pytorch_lightning import Callback

class EvalHarnessCallback(Callback): def on_validation_epoch_end(self, trainer, pl_module): step = trainer.global_step checkpoint_path = f"checkpoints/step-{step}"

    # Save checkpoint
    trainer.save_checkpoint(checkpoint_path)

    # Run lm-eval
    os.system(f"lm_eval --model hf --model_args pretrained={checkpoint_path} ...")

Step 4: Plot learning curves

import json import matplotlib.pyplot as plt

Load all results

steps = [] mmlu_scores = []

for file in sorted(glob.glob("results/step-*.json")): with open(file) as f: data = json.load(f) step = int(file.split("-")[1].split(".")[0]) steps.append(step) mmlu_scores.append(data["results"]["mmlu"]["acc"])

Plot

plt.plot(steps, mmlu_scores) plt.xlabel("Training Step") plt.ylabel("MMLU Accuracy") plt.title("Training Progress") plt.savefig("training_curve.png")

Workflow 3: Compare multiple models

Benchmark suite for model comparison.

Model Comparison: - [ ] Step 1: Define model list - [ ] Step 2: Run evaluations - [ ] Step 3: Generate comparison table

Step 1: Define model list

models.txt

meta-llama/Llama-2-7b-hf meta-llama/Llama-2-13b-hf mistralai/Mistral-7B-v0.1 microsoft/phi-2

Step 2: Run evaluations

!/bin/bash

eval_all_models.sh

TASKS="mmlu,gsm8k,hellaswag,truthfulqa"

while read model; do echo "Evaluating $model"

# Extract model name for output file
model_name=$(echo $model | sed 's/\//-/g')

lm_eval --model hf \
  --model_args pretrained=$model,dtype=bfloat16 \
  --tasks $TASKS \
  --num_fewshot 5 \
  --batch_size auto \
  --output_path results/$model_name.json

done < models.txt

Step 3: Generate comparison table

import json import pandas as pd

models = [ "meta-llama-Llama-2-7b-hf", "meta-llama-Llama-2-13b-hf", "mistralai-Mistral-7B-v0.1", "microsoft-phi-2" ]

tasks = ["mmlu", "gsm8k", "hellaswag", "truthfulqa"]

results = [] for model in models: with open(f"results/{model}.json") as f: data = json.load(f) row = {"Model": model.replace("-", "/")} for task in tasks: # Get primary metric for each task metrics = data["results"][task] if "acc" in metrics: row[task.upper()] = f"{metrics['acc']:.3f}" elif "exact_match" in metrics: row[task.upper()] = f"{metrics['exact_match']:.3f}" results.append(row)

df = pd.DataFrame(results) print(df.to_markdown(index=False))

Output:

Workflow 4: Evaluate with vLLM (faster inference)

Use vLLM backend for 5-10x faster evaluation.

vLLM Evaluation: - [ ] Step 1: Install vLLM - [ ] Step 2: Configure vLLM backend - [ ] Step 3: Run evaluation

Step 1: Install vLLM

pip install vllm

Step 2: Configure vLLM backend

lm_eval --model vllm \ --model_args pretrained=meta-llama/Llama-2-7b-hf,tensor_parallel_size=1,dtype=auto,gpu_memory_utilization=0.8 \ --tasks mmlu \ --batch_size auto

Step 3: Run evaluation

vLLM is 5-10× faster than standard HuggingFace:

Standard HF: ~2 hours for MMLU on 7B model

lm_eval --model hf \ --model_args pretrained=meta-llama/Llama-2-7b-hf \ --tasks mmlu \ --batch_size 8

vLLM: ~15-20 minutes for MMLU on 7B model

lm_eval --model vllm \ --model_args pretrained=meta-llama/Llama-2-7b-hf,tensor_parallel_size=2 \ --tasks mmlu \ --batch_size auto

When to use vs alternatives

Use lm-evaluation-harness when:

Benchmarking models for academic papers Comparing model quality across standard tasks Tracking training progress Reporting standardized metrics (everyone uses same prompts) Need reproducible evaluation

Use alternatives instead:

HELM (Stanford): Broader evaluation (fairness, efficiency, calibration) AlpacaEval: Instruction-following evaluation with LLM judges MT-Bench: Conversational multi-turn evaluation Custom scripts: Domain-specific evaluation Common issues

Issue: Evaluation too slow

Use vLLM backend:

lm_eval --model vllm \ --model_args pretrained=model-name,tensor_parallel_size=2

Or reduce fewshot examples:

--num_fewshot 0 # Instead of 5

Or evaluate subset of MMLU:

--tasks mmlu_stem # Only STEM subjects

Issue: Out of memory

Reduce batch size:

--batch_size 1 # Or --batch_size auto

Use quantization:

--model_args pretrained=model-name,load_in_8bit=True

Enable CPU offloading:

--model_args pretrained=model-name,device_map=auto,offload_folder=offload

Issue: Different results than reported

Check fewshot count:

--num_fewshot 5 # Most papers use 5-shot

Check exact task name:

--tasks mmlu # Not mmlu_direct or mmlu_fewshot

Verify model and tokenizer match:

--model_args pretrained=model-name,tokenizer=same-model-name

Issue: HumanEval not executing code

Install execution dependencies:

pip install human-eval

Enable code execution:

lm_eval --model hf \ --model_args pretrained=model-name \ --tasks humaneval \ --allow_code_execution # Required for HumanEval

Advanced topics

Benchmark descriptions: See references/benchmark-guide.md for detailed description of all 60+ tasks, what they measure, and interpretation.

Custom tasks: See references/custom-tasks.md for creating domain-specific evaluation tasks.

API evaluation: See references/api-evaluation.md for evaluating OpenAI, Anthropic, and other API models.

Multi-GPU strategies: See references/distributed-eval.md for data parallel and tensor parallel evaluation.

Hardware requirements GPU: NVIDIA (CUDA 11.8+), works on CPU (very slow) VRAM: 7B model: 16GB (bf16) or 8GB (8-bit) 13B model: 28GB (bf16) or 14GB (8-bit) 70B model: Requires multi-GPU or quantization Time (7B model, single A100): HellaSwag: 10 minutes GSM8K: 5 minutes MMLU (full): 2 hours HumanEval: 20 minutes Resources GitHub: https://github.com/EleutherAI/lm-evaluation-harness Docs: https://github.com/EleutherAI/lm-evaluation-harness/tree/main/docs Task library: 60+ tasks including MMLU, GSM8K, HumanEval, TruthfulQA, HellaSwag, ARC, WinoGrande, etc. Leaderboard: https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard (uses this harness)