Easy ReviewAIF-C01
Fundamentals of GenAI13

Evaluating foundation models

Metrics, human review, LLM-as-a-judge and business KPIs for judging generative AI models and applications.

Generative AI output is open-ended, so no single score proves a model works. These sections cover the metrics, the judges (people or models) and the business measures that together show whether a model or application meets its goals.

Evaluation methods and metrics

Evaluating generative AI takes several lenses at once: automatic metrics against reference answers, a model acting as judge, and human reviewers for what only people can judge. All of them need a good evaluation dataset.

How it works

Method Measures Best for
ROUGE Word overlap, recall-focused Summarization
BLEU Word overlap, precision-focused Translation
BERTScore Semantic similarity of embeddings General text
LLM-as-a-judge Faithfulness, relevance, coherence, style Scale, fast iteration
Human evaluation Ratings and A/B preferences Subjective and critical cases
Benchmark datasets Standard tasks with known answers Comparing models
  1. Build a golden dataset from real logs, expert examples and synthetic edge cases.
  2. Run automatic metrics and an LLM judge on every change.
  3. Send the critical or subjective cases to human reviewers.

Exam cues

  • If the question says translation quality, think BLEU.
  • If the question says summarization quality, think ROUGE (or BERTScore for meaning).
  • If the question says scalable, automated quality scoring with explanations, think LLM-as-a-judge.
  • If the question says subjective quality or final sign-off, think human evaluation.

Remember

  • ROUGE = recall, summaries. BLEU = precision, translation.
  • Judge from a different model family; humans for critical decisions.

Trap: Automatic overlap metrics need a reference answer and reward matching words, not truth or tone. Pair them with faithfulness checks or human review.

From the AWS docs (10)
  • A generative AI evaluation strategy normally combines automated metric-based, model-based (LLM-as-a-judge) and human evaluation.
  • A high-quality, human-curated "golden" evaluation dataset of questions and answers, built from real-world logs, expert examples and synthetic edge cases, is the foundation of generative AI testing.
  • ROUGE (Recall-Oriented Understudy for Gisting Evaluation) measures n-gram overlap between generated and reference text, focusing on recall, and is typically used for summarization and question answering.
  • BLEU (Bilingual Evaluation Understudy) measures n-gram overlap focusing on precision, penalizes outputs that are too short, and is typically used for machine translation.
  • BERTScore compares embeddings of the generated text and the reference text, so it captures semantic similarity even when the wording differs.
  • Human evaluators rate outputs on a scale or compare two outputs (A/B testing); this suits subjective criteria but is slow, expensive and can be inconsistent.
  • LLM-as-a-judge uses one LLM to score the outputs of another; it is cheaper and faster than human review but can inherit the judge's biases, so humans should stay in the loop for critical decisions.
  • To avoid self-preference bias, use a judge model from a different model family than the model being evaluated.
  • Model-based metrics include faithfulness (groundedness), which checks that output is consistent with the source context, relevance to the query, coherence and style adherence.
  • Perplexity is an intrinsic language modeling metric based on how likely the model finds the test text; a lower perplexity means the model predicts the text better, but it does not always match human judgment.

Business and application metrics

A model can score well and still fail the business. Production generative AI is judged on what users and the company get from it: tasks completed, satisfaction, cost and revenue.

How it works

KPI group Examples
Engagement Active users, repeat usage, task completion rate
Satisfaction Net promoter score, user feedback
Efficiency Tokens, tool calls and cost per task or interaction
Business impact Productivity, cost savings, revenue growth
  1. Set the business KPIs during scoping, next to the technical metrics.
  2. Track them in production with user feedback.
  3. Revisit them as business goals change.

Exam cues

  • If the question says prove business value or ROI, think business KPIs such as revenue growth, cost savings or task completion rate.
  • If the question says is the agent economical, think cost per task and tokens per task.
  • If the question says user perception, think satisfaction scores and feedback.

Remember

  • Infrastructure metrics show uptime, not value.
  • Task completion rate, user satisfaction and cost per interaction align AI apps with business goals.

Trap: Model accuracy is a technical metric. A question about business objectives wants a business metric such as task completion rate or cost per interaction.

From the AWS docs (4)
  • Business and user-interaction health for a generative AI application tracks user engagement (active users, repeat usage, task completion rates), customer satisfaction (such as net promoter score) and business impact (productivity, cost savings, revenue growth).
  • Agent KPIs include operational ones (task completion rate, resolution time, error and escalation rate), quality ones (decision accuracy, hallucination rate, user satisfaction), efficiency ones (tokens, tool calls and cost per task) and business ones (outcome achievement, SLA compliance).
  • Infrastructure metrics such as CPU and invocation count show whether an agent runs, not whether it achieves business outcomes.
  • Collecting and monitoring user feedback on generated outputs helps identify bias and areas where the model needs adjustment.

Flashcards

tap to flip
Answer

·

All 13 cards as a list

Which metric fits a model that translates product descriptions from English to Spanish against reference translations?

BLEU, an n-gram precision metric made for machine translation.

Which metric compares a model's summaries with reference summaries by overlapping words, focusing on recall?

ROUGE (Recall-Oriented Understudy for Gisting Evaluation).

Two answers say the same thing in different words. Which metric still rates them as similar?

BERTScore, which compares embeddings for semantic similarity instead of exact word overlap.

A team must score thousands of chatbot answers for helpfulness every day, cheaply. Which evaluation approach scales best?

LLM-as-a-judge, with a judge model from a different family, plus human review for critical decisions.

When is human evaluation the best choice for a generative AI model?

For subjective qualities such as tone, style or brand voice, and for critical decisions, accepting that it is slow and costly.

What should a team build first to evaluate a new generative AI application consistently?

A curated golden dataset of prompts and expected answers that represents real usage, including edge cases.

Is a high ROUGE score proof that a summary is factually correct?

No. ROUGE counts word overlap with a reference; a summary can share words and still be wrong. Check faithfulness too.

Why should an LLM judge come from a different model family than the model it grades?

Models tend to prefer outputs like their own (self-preference bias).

A team compares two language models by perplexity on the same test text. Which result is better?

The lower perplexity: the model predicts the test text better. It is an intrinsic metric and may not match human judgment, so pair it with task metrics or human review.

A customer service AI assistant goes live. Which metrics show whether it delivers business value?

Task completion rate, customer satisfaction (such as NPS), cost per interaction and business impact such as cost savings.

An AI agent's dashboards show low CPU and no errors, yet customers complain. What is missing?

Outcome KPIs such as task completion rate and user satisfaction. Infrastructure metrics only show the agent is running.

Which efficiency metrics show whether an AI agent is economical?

Tokens per task, tool invocations per task and cost per task.

An e-commerce company adds a generative AI shopping assistant. Which business metric shows it increases sales?

Business impact metrics such as revenue growth, tracked against a baseline from before the assistant launched.

You finished every topic of AIF-C01!

Your badge turned gold. Next step: book the exam on AWS. Once you pass, add yourself to Who passed.

Share

Share that you finished this certification

I finished every topic of AWS Certified AI Practitioner (AIF-C01) on Easy Review. Come study too!