Evaluate

Ask review

Assess vendors and score against criteria

Hats
2
Review Agents
1
Review
Ask
Unit Types
Vendor Assessment, Scoring
Inputs
Requirements

Dependencies

Requirementsrfp-document

Hat Sequence

1

Evaluator

Focus: Score vendor responses against RFP criteria using the predefined evaluation methodology.

Responsibilities:

  • Apply the scoring methodology consistently across all vendor responses
  • Document scoring rationale for each criterion and vendor
  • Calculate total cost of ownership including hidden and ongoing costs
  • Produce a comparative ranking with clear differentiation

Anti-patterns (RFC 2119):

  • The agent MUST NOT chang scoring criteria mid-evaluation to favor a preferred vendor
  • The agent MUST NOT score based on vendor presentations rather than substantive responses
  • The agent MUST NOT ignore total cost of ownership in favor of upfront pricing
  • The agent MUST document the reasoning behind scores for auditability
2

Technical Reviewer

Focus: Validate vendor technical claims through proof-of-concept testing and reference verification.

Responsibilities:

  • Design and oversee proof-of-concept evaluations for shortlisted vendors
  • Verify vendor technical claims against actual product capabilities
  • Conduct reference checks with existing customers of each vendor
  • Assess architecture compatibility and integration complexity

Anti-patterns (RFC 2119):

  • The agent MUST NOT accept vendor demos as proof of capability without hands-on testing
  • The agent MUST contact vendor references or asking only the vendor-provided references
  • The agent MUST NOT evaluate technical capabilities in isolation without considering integration needs
  • The agent MUST NOT ignore scalability and performance under realistic loadd conditions

Review Agents

Objectivity

Mandate: The agent MUST verify vendor evaluation is objective and scoring is consistent across vendors.

Check:

  • The agent MUST verify that scoring methodology is applied consistently to all vendors
  • The agent MUST verify that technical claims are validated through proof-of-concept testing, not just vendor demos
  • The agent MUST verify that total cost of ownership includes all direct and indirect costs
  • The agent MUST verify that reference checks contacted actual customers, not just vendor-provided contacts

Evaluate

Criteria Guidance

Good criteria examples:

  • "Vendor scorecard rates each vendor against every RFP criterion using the pre-defined scoring methodology"
  • "Technical evaluation includes proof-of-concept results, reference checks, and architecture compatibility assessment"
  • "Total cost of ownership analysis covers licensing, implementation, integration, training, and ongoing maintenance"

Bad criteria examples:

  • "Vendors are evaluated"
  • "Scores are calculated"
  • "Best vendor is identified"

Completion Signal (RFC 2119)

Vendor scorecard MUST exist with all vendors rated against RFP criteria, technical evaluations complete, and total cost of ownership calculated. Evaluator MUST have confirmed scoring consistency across vendors. Technical-reviewer MUST have validated vendor claims against proof-of-concept results and reference feedback.