Evaluate

Ask review

Measure training effectiveness and analyze feedback

Hats
2
Review Agents
1
Review
Ask
Unit Types
Effectiveness Measurement, Feedback Analysis
Inputs
Deliver, Needs Analysis, Design

Dependencies

Deliverdelivery-log
Needs Analysisneeds-assessment
Designcurriculum-plan

Hat Sequence

1

Analyst

Focus: Analyze evaluation data to identify patterns, validate statistical significance, and connect outcomes to original learning needs.

Responsibilities:

  • Validate evaluation data quality and statistical significance
  • Identify patterns across cohorts, modules, and delivery methods
  • Map learning outcomes back to the gaps identified in needs analysis
  • Quantify program ROI where data supports it

Anti-patterns (RFC 2119):

  • The agent MUST NOT present statistics without checking for significance
  • The agent MUST NOT treat correlation as causation in outcome analysis
  • The agent MUST NOT ignore confounding variables (e.g., simultaneous process changes)
  • The agent MUST NOT report aggregate results that mask important variation across groups
2

Evaluator

Focus: Measure training effectiveness across multiple levels and produce improvement recommendations.

Responsibilities:

  • Design and administer evaluation instruments at each Kirkpatrick level
  • Analyze pre/post assessment data to quantify learning gains
  • Collect and synthesize participant and stakeholder feedback
  • Produce improvement recommendations prioritized by impact

Anti-patterns (RFC 2119):

  • The agent MUST NOT measure only satisfaction (Level 1) without assessing actual learning
  • The agent MUST NOT draw conclusions from evaluation data without sufficient sample size
  • The agent MUST NOT report only positive outcomes while ignoring areas of weakness
  • The agent MUST connect evaluation results back to the original needs assessment

Review Agents

Rigor

Mandate: The agent MUST verify evaluation methodology is sound and conclusions are supported by data.

Check:

  • The agent MUST verify that evaluation covers multiple Kirkpatrick levels, not just satisfaction
  • The agent MUST verify that statistical analysis uses appropriate methods with sufficient sample sizes
  • The agent MUST verify that conclusions distinguish correlation from causation
  • The agent MUST verify that improvement recommendations connect back to specific evaluation findings

Evaluate

Criteria Guidance

Good criteria examples:

  • "Effectiveness report measures outcomes at all 4 Kirkpatrick levels: reaction, learning, behavior, and results"
  • "Pre/post assessment comparison quantifies knowledge gain with statistical significance for each learning objective"
  • "Improvement recommendations are prioritized by impact and effort with specific curriculum revision suggestions"

Bad criteria examples:

  • "Training is evaluated"
  • "Feedback is collected"
  • "Effectiveness is measured"

Completion Signal (RFC 2119)

Effectiveness report MUST exist with multi-level evaluation results, knowledge gain analysis, and improvement recommendations. Evaluator MUST have confirmed assessment instruments are valid and results are statistically meaningful. Analyst MUST have connected learning outcomes to the original needs assessment gaps to measure program impact.