‹ 首页

comparative-evaluation

@owl-listener · 收录于 5 天前 · 上游提交 1 个月前

A/B testing, side-by-side comparison, and preference ranking for AI outputs.

场景模型评估效果对比

适合你，如果需要客观比较不同AI模型或提示词的效果

/ 下载安装
https://github.com/owl-listener/ai-design-skills/tree/main/claude-plugin/evaluation/skills/comparative-evaluation ↗
/ 通过 npx 安装 校验哈希
npx oh-my-skill add owl-listener/ai-design-skills/comparative-evaluation
/ 通过 bash 安装
curl -fsSL https://oh-my-skill.com/install.sh | bash -s -- owl-listener/ai-design-skills/comparative-evaluation
/ 已经装过?验证本机副本,不用重装
npx oh-my-skill verify owl-listener/ai-design-skills/comparative-evaluation
安装目标可用 --agent / --scope 或 --to 明确指定;省略时只会在唯一已存在的 agent 目录上自动选择,零命中或多命中会停止并提示。content_hash 缺失或不一致均拒装。
144GitHub stars
~505上下文体积 · 单文件
索引托管

怎么用

商店整理自技能原文 · 版本 f41b650 · 表述以原文为准
它做什么

装上后，Claude能帮助你进行AI输出的对比评估，包括A/B测试、并排比较、偏好排序等方法，并提供设计模板和评估协议。

什么时候触发

当你需要比较不同AI模型的输出质量，或进行A/B测试、偏好排序等对比评估时触发。

装好后可以这样说
触发并排评估
触发A/B测试设计模板
触发偏好排名分析
技能原文 SKILL.md作者撰写 · MIT · f41b650

Comparative Evaluation

Absolute quality scores are useful but limited. Comparative evaluation — putting outputs side by side and asking which is better — often reveals quality differences that rubrics miss.

Comparison Methods
  • A/B testing: Show different users different versions and compare outcomes
  • Side-by-side evaluation: Show evaluators two outputs for the same input and ask which is better
  • Preference ranking: Show evaluators multiple outputs and rank them from best to worst
  • Paired comparison: Compare every pair of options to build a complete ranking
  • Elo rating: Use tournament-style comparisons to develop continuous quality scores
Designing A/B Tests for AI

A/B testing AI is different from A/B testing UI:

  • Variance is high: The same prompt can produce different outputs, so you need more samples
  • Context matters: The same change might help for one task and hurt for another
  • Metrics lag: AI quality changes may take time to show up in user behavior
  • Interaction effects: A change to one part of the conversation affects all subsequent parts

Design A/B tests with:

  • Sufficient sample sizes to account for output variance
  • Segmentation by task type and user experience level
  • Multiple metrics (don't optimise for one at the expense of others)
  • Guardrails to catch severe quality regressions quickly
Side-by-Side Evaluation Design

For human evaluation of AI outputs:

  • Blind evaluation: Evaluators shouldn't know which version is which
  • Consistent inputs: Compare outputs generated from the same input
  • Structured criteria: Give evaluators specific dimensions to compare on, not just "which is better"
  • Multiple evaluators: Use at least 3 evaluators per comparison for reliability
  • Diverse inputs: Test across a representative sample of real user inputs
When to Use Comparative vs. Absolute Evaluation
  • Comparative: Best for choosing between alternatives, detecting subtle quality differences, and model selection
  • Absolute: Best for measuring against a standard, tracking progress over time, and certification
Design Artefacts
  • A/B test design templates
  • Side-by-side evaluation protocols
  • Evaluator instructions and rubrics
  • Sample size calculators for AI experiments
  • Comparison result analysis frameworks
按 MIT 许可原样转载,未经改动 · 在 GitHub 查看 →

评论

登录即可评论;带「已验证安装」的,是发布者名下有本店的安装或持有记录。
…