‹ 首页

failure-taxonomy

@owl-listener · 收录于 5 天前 · 上游提交 1 个月前

Classifying AI failures — hallucination, refusal, irrelevance, tone mismatch, latency.

场景模型评估质量检查

适合你，如果需要系统化识别和归类AI输出的错误类型

/ 下载安装
https://github.com/owl-listener/ai-design-skills/tree/main/claude-plugin/evaluation/skills/failure-taxonomy ↗
/ 通过 npx 安装 校验哈希
npx oh-my-skill add owl-listener/ai-design-skills/failure-taxonomy
/ 通过 bash 安装
curl -fsSL https://oh-my-skill.com/install.sh | bash -s -- owl-listener/ai-design-skills/failure-taxonomy
/ 已经装过?验证本机副本,不用重装
npx oh-my-skill verify owl-listener/ai-design-skills/failure-taxonomy
安装目标可用 --agent / --scope 或 --to 明确指定;省略时只会在唯一已存在的 agent 目录上自动选择,零命中或多命中会停止并提示。content_hash 缺失或不一致均拒装。
144GitHub stars
~534上下文体积 · 单文件
索引托管

怎么用

商店整理自技能原文 · 版本 f41b650 · 表述以原文为准
它做什么

装上后，Claude 学会识别 AI 输出的失败类型，包括幻觉、拒绝回答、语气不当、延迟等，并评估严重等级（致命/高/中/低），用于记录和优先处理。

什么时候触发

当你指出或询问 AI 输出的问题，例如“这个回答有错误”或“为什么回答得这么慢”。

装好后可以这样说
Claude 会归类并说明严重等级。
Claude 会指出是否幻觉及原因。
Claude 会识别语气不当等问题。
技能原文 SKILL.md作者撰写 · MIT · f41b650

Failure Taxonomy

Not all AI failures are the same. A hallucination is different from a refusal, which is different from a tone mismatch. A failure taxonomy classifies failure types so teams can track, prioritise, and address them systematically.

Failure Categories

Content Failures:

  • Hallucination: The AI presents false information as fact
  • Inaccuracy: The AI gets details wrong (dates, numbers, names)
  • Incompleteness: The AI misses important information
  • Irrelevance: The AI's response doesn't address the user's actual question
  • Contradiction: The AI contradicts itself within or across responses

Behavioral Failures:

  • Inappropriate refusal: The AI refuses a reasonable request
  • Missing refusal: The AI fulfils a request it should have declined
  • Tone mismatch: The AI's tone is wrong for the context
  • Persona break: The AI drops out of its defined persona
  • Over-generation: The AI produces far more than needed

Technical Failures:

  • Latency: Response takes too long
  • Truncation: Response is cut off
  • Format errors: Output is in the wrong format or structure
  • Tool failures: The AI attempts to use a tool and fails
  • Context loss: The AI loses track of conversation history

Safety Failures:

  • Harmful content: The AI generates content that could cause harm
  • Privacy violation: The AI reveals sensitive information
  • Bias manifestation: The AI's output shows bias against a group
  • Manipulation: The AI's output could be used to deceive or manipulate
Severity Levels
  • Critical: Causes harm or creates serious trust violation. Requires immediate fix.
  • High: Significantly degrades user experience or task success. Fix within days.
  • Medium: Noticeable quality issue that users can work around. Fix within weeks.
  • Low: Minor quality issue. Track and batch with other fixes.
Using the Taxonomy
  • Logging: Classify every detected failure by type and severity
  • Trending: Track failure type frequency over time
  • Prioritisation: Address highest-severity, highest-frequency failures first
  • Root cause analysis: Group failures by type to identify systemic causes
  • Prevention: Use failure patterns to inform guardrail design and prompt improvements
Design Artefacts
  • Failure taxonomy reference document
  • Failure logging templates
  • Severity classification rubric
  • Failure trend dashboards
  • Root cause analysis protocols
按 MIT 许可原样转载,未经改动 · 在 GitHub 查看 →

评论

登录即可评论;带「已验证安装」的,是发布者名下有本店的安装或持有记录。
…