‹ 首页

dspy-adapters-multimodal

@omidzamani · 收录于 1 周前

Use for DSPy adapter selection, JSONAdapter, XMLAdapter, ChatAdapter, native function calling, structured outputs, and multimodal inputs like dspy.Image or dspy.Audio.

场景多模态数据结构化输出

适合你，如果正在用DSPy构建需要图像或音频输入的AI管道

/ 下载安装
dspy-adapters-multimodal.skill双击,或拖进 Claude 桌面版 / Cowork,即完成安装↓ .skill↓ .zip
用别的 agent?下载 .zip 解压,把文件夹放进它的技能目录
Claude Code~/.claude/skills/(项目级 .claude/skills/)
Codex CLI~/.codex/skills/
Cursor自动读取上面两处目录
其他工具见其文档的「skills」目录;两个下载是同一份文件,只是名字不同
/ 通过 npx 安装 校验哈希
npx oh-my-skill add omidzamani/dspy-skills/dspy-adapters-multimodal
/ 通过 bash 安装
curl -fsSL https://oh-my-skill.com/install.sh | bash -s -- omidzamani/dspy-skills/dspy-adapters-multimodal
/ 已经装过?验证本机副本,不用重装
npx oh-my-skill verify omidzamani/dspy-skills/dspy-adapters-multimodal
安装目标可用 --agent / --scope 或 --to 明确指定;省略时只会在唯一已存在的 agent 目录上自动选择,零命中或多命中会停止并提示。content_hash 缺失或不一致均拒装。
113GitHub stars
~713最小装载
~713含声明引用
~774文本包总量
镜像托管

怎么用

技能原文 SKILL.md作者撰写 · MIT · f5db3b7

DSPy Adapters and Multimodal I/O

Goal

Choose an adapter deliberately and model image, audio, and file inputs with DSPy's typed primitives.

Adapter Selection

| Adapter | Use it for | |---------|------------| | dspy.ChatAdapter() | Default, human-readable field markers, broad model compatibility | | dspy.JSONAdapter() | Structured JSON output and native function calling where supported | | dspy.XMLAdapter() | XML-tagged fields when XML is easier for the target LM to follow | | dspy.TwoStepAdapter() | A separate extraction pass when parsing needs extra help |

Configure globally or for a limited scope:

import dspy

dspy.configure(
    lm=dspy.LM("openai/gpt-4o-mini"),
    adapter=dspy.JSONAdapter(),
)

with dspy.context(adapter=dspy.XMLAdapter()):
    result = dspy.Predict("question -> answer")(question="What is DSPy?")
Native Function Calling

JSONAdapter enables native function calling by default. ChatAdapter keeps text parsing by default. Override either behavior explicitly:

chat_native = dspy.ChatAdapter(use_native_function_calling=True)
json_manual = dspy.JSONAdapter(use_native_function_calling=False)

DSPy falls back to manual parsing when the configured LM does not support native function calling.

Image Inputs
class DescribeImage(dspy.Signature):
    image: dspy.Image = dspy.InputField()
    description: str = dspy.OutputField()

describe = dspy.Predict(DescribeImage)
result = describe(image=dspy.Image("./diagram.png"))

Pass a local path, HTTP URL, bytes, PIL image, or existing data URI directly to dspy.Image(...).

Audio and File Inputs
class SummarizeAudio(dspy.Signature):
    audio: dspy.Audio = dspy.InputField()
    summary: str = dspy.OutputField()

audio = dspy.Audio.from_file("./meeting.wav")
summary = dspy.Predict(SummarizeAudio)(audio=audio)
class SummarizeFile(dspy.Signature):
    file: dspy.File = dspy.InputField()
    summary: str = dspy.OutputField()

document = dspy.File.from_path("./research.pdf")
summary = dspy.Predict(SummarizeFile)(file=document)

Provider capabilities vary. Verify that the selected model accepts the media type before deployment.

Best Practices
  1. Start with ChatAdapter; switch only for a measured reason.
  2. Use typed signatures for structured output.
  3. Test adapter behavior against the exact production model.
  4. Avoid deprecated Image.from_file() and Image.from_url() helpers; call dspy.Image(...).
  5. Keep local file handling and uploaded file IDs within provider policy.
Related Skills
  • Design signatures: [dspy-signature-designer](../dspy-signature-designer/SKILL.md)
  • Build tool agents: [dspy-react-agent-builder](../dspy-react-agent-builder/SKILL.md)
Official Documentation
  • Adapters guide: https://dspy.ai/learn/programming/adapters/
  • Tools guide: https://dspy.ai/learn/programming/tools/
  • XMLAdapter API: https://dspy.ai/api/adapters/XMLAdapter/
  • Image API: https://dspy.ai/api/primitives/Image/
  • Audio API: https://dspy.ai/api/primitives/Audio/
按 MIT 许可原样转载,未经改动 · 在 GitHub 查看 →

评论

登录即可评论;带「已验证安装」的,是发布者名下有本店的安装或持有记录。
…