‹ 首页

longitudinal-measurement

@owl-listener · 收录于 5 天前 · 上游提交 1 个月前

Tracking AI product quality over time — drift, degradation, and improvement.

场景模型监控质量评估

适合你，如果你需要持续监控AI产品的性能漂移和退化

/ 下载安装
https://github.com/owl-listener/ai-design-skills/tree/main/claude-plugin/evaluation/skills/longitudinal-measurement ↗
/ 通过 npx 安装 校验哈希
npx oh-my-skill add owl-listener/ai-design-skills/longitudinal-measurement
/ 通过 bash 安装
curl -fsSL https://oh-my-skill.com/install.sh | bash -s -- owl-listener/ai-design-skills/longitudinal-measurement
/ 已经装过?验证本机副本,不用重装
npx oh-my-skill verify owl-listener/ai-design-skills/longitudinal-measurement
安装目标可用 --agent / --scope 或 --to 明确指定;省略时只会在唯一已存在的 agent 目录上自动选择,零命中或多命中会停止并提示。content_hash 缺失或不一致均拒装。
144GitHub stars
~468上下文体积 · 单文件
索引托管

怎么用

商店整理自技能原文 · 版本 f41b650 · 表述以原文为准
它做什么

安装后，Claude会定期评估AI产品的质量指标，监控模型更新、数据漂移等变化，并生成趋势报告和漂移预警。

什么时候触发

当用户要求进行质量跟踪、检测退化，或按设定时间表自动运行时触发。

装好后可以这样说
Claude将创建固定测试输入并安排定期评估。
Claude会调取仪表盘并分析最新数据。
Claude会列出漂移响应步骤。
技能原文 SKILL.md作者撰写 · MIT · f41b650

Longitudinal Measurement

AI products change over time — models get updated, usage patterns shift, and quality can drift without anyone noticing. Longitudinal measurement is how you track quality across time and catch degradation before users do.

What Changes Over Time
  • Model updates: New model versions may improve some capabilities and regress others
  • Prompt drift: System prompts accumulate edits that may interact in unexpected ways
  • Usage evolution: Users discover new use cases that weren't tested for
  • Data drift: The real-world inputs diverge from what was tested
  • Expectation drift: Users' expectations change as they become more experienced
What to Measure Longitudinally
  • Quality scores: Track rubric scores on a consistent test set over time
  • Task success rates: Monitor whether users are completing tasks at the same rate
  • Satisfaction signals: Track trends in explicit and implicit satisfaction
  • Error rates: Monitor failure frequency and type distribution
  • Latency: Response time changes can indicate degradation
  • Engagement patterns: Changes in usage frequency, depth, and breadth
Measurement Infrastructure
  • Golden test sets: A fixed set of inputs evaluated regularly to detect quality changes
  • Automated evaluation: Run golden test sets automatically on a schedule
  • Dashboards: Visualise trends and set alerts for significant changes
  • Regression detection: Statistical methods to distinguish real changes from noise
  • User cohort tracking: Follow specific user groups over time
Responding to Drift

When measurements show drift:

  1. Detect: Automated alerts flag significant changes
  2. Diagnose: Was it a model update, prompt change, data shift, or usage change?
  3. Assess: Is the drift harmful, neutral, or actually an improvement?
  4. Act: Adjust prompts, revert changes, update guardrails, or accept the new baseline
  5. Verify: Confirm the fix worked and set the new baseline
Design Artefacts
  • Longitudinal measurement plan
  • Golden test set specifications
  • Quality trend dashboards
  • Drift detection alert configurations
  • Response protocols for detected drift
按 MIT 许可原样转载,未经改动 · 在 GitHub 查看 →

评论

登录即可评论;带「已验证安装」的,是发布者名下有本店的安装或持有记录。
…