Evaluation and Alignment: The Seminal Papers book cover

Evaluation and Alignment: The Seminal Papers

Manning Publications

A practical guide to designing and implementing AI evaluation systems, grounded in seminal research papers.

Get Early Access — 50% Off

About the Book

Evaluation and Alignment is an engineering guide to the research behind modern evaluation and alignment. Each chapter starts from a seminal paper, reconstructs the problem it addressed, and carries the method into an implementation or system design.

The book covers application evaluation, model evaluation, and post-training. It shows what each method measures, the assumptions it makes, and where it fails.

What You’ll Learn

  • Evaluation fundamentals — Choose metrics that match the behavior and failure modes you need to measure.
  • Alignment techniques — Trace RLHF and constitutional AI from their original formulations to current practice.
  • Reading research — Separate a paper’s durable contribution from its experimental setting and implementation details.
  • Evaluation pipelines — Connect datasets, evaluators, regression tests, and release decisions in one system.

Table of Contents

  1. The Landscape of LLM Evaluation and Alignment: An Evolving Field
  2. The Dawn of Automatic Evaluation: BLEU and ROUGE
  3. Bridging the Semantic Gap with Learned Metrics: BERTScore and COMET
  4. LLM-as-a-Judge: The New Paradigm for Evaluation
  5. Detecting and Quantifying Hallucinations
  6. Evaluating Retrieval-Augmented Generation (RAG): The RAGAS Framework
  7. Reinforcement Learning from Human Preferences: The RLHF Foundation
  8. Constitutional AI: Alignment Through Principles
  9. Advanced Alignment and Safety: Red Teaming and Beyond
  10. Deliberative Alignment: Reasoning for Safety at Inference Time
  11. The Evolving Landscape of LLM Evaluation and Alignment

Table of contents is preliminary and subject to change during Early Access.

Why This Book?

The core methods are scattered across papers on metrics, model judges, hallucination, RAG, preference learning, constitutional AI, and red teaming. This book puts them in the order an engineer encounters the problems and explains how they fit into a production evaluation system.

Get Early Access at Manning

Papers

SkillsBench: Skills, Agent Harness, and Models hierarchy with resolution rate benchmarks
arXiv · Feb 2026
SkillsBench: Benchmarking How Well Agent Skills Work Across Diverse Tasks
Xiangyi Li, Wenbo Chen, Yimin Liu, Shenghan Zheng, Xiaokun Chen, ..., Steven Dillmann, Han-chung Lee

Agent Skills are structured packages of procedural knowledge that augment LLM agents at inference time. We present SkillsBench, a benchmark of 86 tasks across 11 domains paired with curated Skills and deterministic verifiers. Curated Skills raise average pass rate by 16.2 pp, but effects vary widely by domain, and self-generated Skills provide no benefit on average.