Evaluation and Alignment: The Seminal Papers book cover

Evaluation and Alignment: The Seminal Papers

Manning Publications

A practical guide to designing and implementing AI evaluation systems, grounded in seminal research papers.

Get Early Access — 50% Off

About the Book

Building AI systems is only half the battle — knowing whether they actually work is the other half. Evaluation and Alignment bridges the gap between foundational research and practical engineering, walking you through the seminal papers that shaped how we measure and align AI systems today.

Whether you’re building LLM-powered applications, fine-tuning models, or designing evaluation pipelines, this book gives you the conceptual and practical tools to do it rigorously.

What You’ll Learn

  • Evaluation fundamentals — From classical metrics to modern LLM-as-judge approaches, understand what “good” means for your system and how to measure it.
  • Alignment techniques — RLHF, constitutional AI, and the research lineage behind today’s alignment methods.
  • Reading research effectively — Each chapter is anchored to seminal papers, teaching you how to extract practical insights from academic work.
  • Building evaluation pipelines — Design end-to-end evaluation systems that catch regressions, measure progress, and build confidence in your deployments.

Table of Contents

  1. The Landscape of LLM Evaluation and Alignment: An Evolving Field
  2. The Dawn of Automatic Evaluation: BLEU and ROUGE
  3. Bridging the Semantic Gap with Learned Metrics: BERTScore and COMET
  4. LLM-as-a-Judge: The New Paradigm for Evaluation
  5. Detecting and Quantifying Hallucinations
  6. Evaluating Retrieval-Augmented Generation (RAG): The RAGAS Framework
  7. Reinforcement Learning from Human Preferences: The RLHF Foundation
  8. Constitutional AI: Alignment Through Principles
  9. Advanced Alignment and Safety: Red Teaming and Beyond
  10. Deliberative Alignment: Reasoning for Safety at Inference Time
  11. The Evolving Landscape of LLM Evaluation and Alignment

Table of contents is preliminary and subject to change during Early Access.

Why This Book?

The AI evaluation landscape is fragmented across hundreds of papers, blog posts, and tribal knowledge. This book distills the work that matters most into a single, coherent narrative — so you can spend less time reading papers and more time building systems that work.

Get Early Access at Manning

Papers

SkillsBench: Skills, Agent Harness, and Models hierarchy with resolution rate benchmarks
arXiv · Feb 2026
SkillsBench: Benchmarking How Well Agent Skills Work Across Diverse Tasks
Xiangyi Li, Wenbo Chen, Yimin Liu, Shenghan Zheng, Xiaokun Chen, ..., Steven Dillmann, Han-chung Lee

Agent Skills are structured packages of procedural knowledge that augment LLM agents at inference time. We present SkillsBench, a benchmark of 86 tasks across 11 domains paired with curated Skills and deterministic verifiers. Curated Skills raise average pass rate by 16.2 pp, but effects vary widely by domain, and self-generated Skills provide no benefit on average.