A Repeatable Prompt-Evaluation Checklist

A repeatable prompt-evaluation checklist fixes tasks and rubrics first, runs identical prompt packages on each provider, scores blind where possible, and archives results with model IDs and dates — so AI choices survive scrutiny and model updates. Use this checklist every time you compare OpenAI, Anthropic, and Google AI for adoption, routing, or regression testing.

Phase 1 — Define the decision

• [ ] Decision statement — What choice will this evaluation inform? (default model, coding assistant, research pipeline)
• [ ] Success threshold — Minimum rubric scores or maximum failure rates
• [ ] Disqualifiers — e.g., invented legal citations, JSON parse failures >10%
• [ ] Stakeholders — Who approves results?
• [ ] Re-test trigger — Provider release notes, quarterly calendar

Align thresholds with business AI model evaluation for team purchases.

Phase 2 — Select representative prompts

• [ ] 8–15 golden prompts from real work, not toy examples
• [ ] Coverage — Include easy, typical, and edge cases
• [ ] Task types — Match your use cases hub entries (coding, research, writing)
• [ ] Frozen versions — Store prompts in git or doc with version IDs
• [ ] No mid-run edits — Changing prompts invalidates cross-model comparison

See AI model comparison guide for fair-test rules.

Phase 3 — Build the prompt package

Each run includes:

• [ ] System/role instructions (if used)
• [ ] User message with explicit output format
• [ ] Attachments/context with documented token size
• [ ] Constraints (length, tone, citation rules)
• [ ] Negative instructions (what not to do)

Phase 4 — Match provider settings

• [ ] Comparable model tiers on OpenAI, Anthropic, Google AI
• [ ] Temperature and max tokens logged
• [ ] Tool configs identical or noted if unsupported
• [ ] Timestamp and model ID recorded per response

Run parallel API calls with BYOK on Smart AI Comparison.

Phase 5 — Rubric and scoring

• [ ] 5–7 weighted criteria (accuracy, format, completeness, clarity, safety)
• [ ] 1–5 scale defined with anchors
• [ ] Failure tags — hallucination, omission, over-refusal, unsafe
• [ ] Blind labels when feasible — bias guide
• [ ] Independent scores before group discussion

Add hallucination probes from AI hallucinations testing.

Phase 6 — Execute comparison

• [ ] Run full pack on all providers same day if possible
• [ ] Export raw outputs unchanged
• [ ] Note latency and approximate token usage — pricing guide
• [ ] Re-run ambiguous cases once at same settings

Use comparison pages for context: OpenAI vs Anthropic vs Google AI.

Phase 7 — Analyze and decide

• [ ] Compute medians and failure rates, not vibes
• [ ] Examine worst outputs, not only averages
• [ ] Document ties and portfolio splits (task A → model X)
• [ ] Capture dissenting reviewer notes

Phase 8 — Archive and communicate

• [ ] Store prompts, outputs, scores, model IDs, dates
• [ ] Write one-page decision summary with caveats
• [ ] Schedule next re-test
• [ ] Share templates with content/engineering teams

Phase 9 — Post-adoption monitoring

• [ ] Track production failures linked to prompt IDs
• [ ] Adjust prompts — re-run full pack on all providers after material changes
• [ ] Compare against benchmark claims only as secondary signal

Quick reference table

Anti-patterns that invalidate results

Stop the run if any of these appear:

• Prompt edits mid-batch to "help" a lagging model
• Reviewers discussing vendor reputations before scoring
• Missing model IDs in export logs
• Mixed temperature settings without documentation

Restart with frozen prompts rather than patching forward — consistency beats speed when procurement is on the line.

Handoff to production routing

When evaluation completes, translate scores into routing rules:

• Default model per workflow tag
• Escalation model for low-confidence paths (if you measure confidence)
• Kill switch owner and rollback steps

Store routing config next to prompt versions so engineers know which evaluation run authorized production behavior.

Versioning checklist templates

When you improve this checklist, bump its version ID in your wiki and note what changed (e.g., added multimodal assets). Teams reusing old checklists without versioning unknowingly compare scores across incompatible procedures.

Next steps

Copy this checklist into your team wiki, run your first pack on smartaicomparison.com with BYOK, and treat prompt evaluation as infrastructure — not a one-off demo before renewal season.