doc-gen-unicode-diagnostic

Document Generation with Diagnostic Workflow (Unicode-Safe)

When to Use

Use this skill when document generation tasks fail or return unclear errors, especially when:

• Generating documents in multiple formats (.docx, .pdf, .html)
• PDF generation fails with encoding/LaTeX errors
• shell_agent returns "unknown error" without diagnostics
• Documents contain special characters, symbols, or non-ASCII text
• You need systematic error diagnosis rather than blind retries

⚠️ Critical: Why This Skill Exists

Recent executions show 43% effectiveness because agents:

• Skip unicode sanitization before PDF conversion
• Don't try xelatex engine (better Unicode support)
• Don't capture stderr for proper diagnosis
• Exhaust iterations on repeated failures without systematic troubleshooting

This skill fixes those gaps with mandatory steps.

Pre-Flight Validation (NEW - Required Step 0)

Before starting document generation, verify your toolchain:

run_shell
command: which pandoc && pandoc --version | head -3

run_shell
command: which pdflatex xelatex wkhtmltopdf 2>/dev/null || echo "Some engines missing"

run_shell
command: python3 -c "import sys; print(sys.version)"

If pandoc is missing, install it:

run_shell
command: apt-get update && apt-get install -y pandoc

For PDF support, install LaTeX engines:

run_shell
command: apt-get install -y texlive-latex-recommended texlive-fonts-recommended texlive-xetex

Core Technique

Manually split the workflow into observable, diagnostic steps:

1. Pre-flight → Verify toolchain availability
2. Content creation → Use write_file for markdown source (visible content)
3. Unicode sanitization → MANDATORY for PDF: Create sanitized version
4. Format conversion with fallback → Try engines in order, capture stderr
5. Verification → Check outputs exist and validate content

Unicode & LaTeX Compatibility (MANDATORY for PDF)

PDF generation via LaTeX has limited Unicode support. Before PDF conversion, you MUST sanitize:

| Character | Issue | Safe Replacement | |-----------|-------|------------------| | — (em dash) | LaTeX incompatibility | -- | | – (en dash) | LaTeX incompatibility | - | | " " (curly quotes) | Encoding errors | " " (straight) | | ' ' (curly apostrophe) | Encoding errors | ' (straight) | | … (ellipsis) | May not render | ... | | → ← ↑ ↓ (arrows) | LaTeX incompatibility | -> <- ^ v | | ✓ ✗ (checkmarks) | May not render | [x] [ ] | | ★ ● (symbols) | May not render | * - | | © ® ™ | Require packages | (c) (r) (tm) | | Non-ASCII (é, ñ, ü) | Font-dependent | Keep for xelatex, sanitize for pdflatex |

DOCX and HTML handle Unicode natively - use original markdown for these formats.

Step-by-Step Workflow

Step 0: Pre-Flight Validation

run_shell
command: which pandoc || (apt-get update && apt-get install -y pandoc)

run_shell
command: which xelatex || echo "xelatex not available - will use fallback"

Step 1: Create Source Content with write_file

write_file
path: /tmp/document_source.md
content: |
  # Document Title
  
  ## Section 1
  Your content here with full Unicode support...
  
  ## Section 2
  Special chars: "quotes" — dashes … ellipsis ✓ checkmarks

Step 2: Create Sanitized Version for PDF (MANDATORY)

Option A: Manual sanitization with write_file

write_file
path: /tmp/document_source_sanitized.md
content: |
  # Document Title
  
  ## Section 1
  Your content here...
  
  ## Section 2
  Special chars: "quotes" -- dashes ... ellipsis [x] checkmarks

Option B: Automated sanitization script

First create the script:

write_file
path: /tmp/sanitize_for_pdf.sh
content: |
  #!/bin/bash
  INPUT="$1"
  OUTPUT="${2:-${1%.md}_sanitized.md}"
  sed -e 's/—/--/g' -e 's/–/-/g' \
      -e 's/"([^"]*)"/"\1"/g' -e "s/'([^']*)'/\`\1\`/g" \
      -e 's/…/.../g' -e 's/→/->/g' -e 's/←/<-/g' \
      -e 's/✓/[x]/g' -e 's/✗/[ ]/g' \
      -e 's/©/(c)/g' -e 's/®/(r)/g' -e 's/™/(tm)/g' \
      "$INPUT" > "$OUTPUT"
  echo "Sanitized: $INPUT -> $OUTPUT"

run_shell
command: chmod +x /tmp/sanitize_for_pdf.sh && /tmp/sanitize_for_pdf.sh /tmp/document_source.md /tmp/document_source_sanitized.md

Step 3: Convert to DOCX (from original, supports Unicode)

run_shell
command: pandoc /tmp/document_source.md -o output.docx 2>&1

Capture stderr with 2>&1 to see actual errors (not "unknown error").

Step 4: Convert to PDF with Engine Fallback Chain (CRITICAL)

Try engines in order: xelatex (best Unicode) → pdflatex → wkhtmltopdf

run_shell
command: pandoc /tmp/document_source_sanitized.md -o output.pdf --pdf-engine=xelatex 2>&1

If xelatex fails, try pdflatex:

run_shell
command: pandoc /tmp/document_source_sanitized.md -o output.pdf --pdf-engine=pdflatex 2>&1

If pdflatex fails, try wkhtmltopdf:

run_shell
command: pandoc /tmp/document_source_sanitized.md -o output.pdf --pdf-engine=wkhtmltopdf 2>&1

If ALL engines fail, diagnose with:

run_shell
command: file /tmp/document_source_sanitized.md && head -20 /tmp/document_source_sanitized.md

Step 5: Convert to HTML (from original)

run_shell
command: pandoc /tmp/document_source.md -o output.html 2>&1

Step 6: Verify All Outputs

run_shell
command: ls -lh output.* && file output.*

run_shell
command: [ -f output.pdf ] && echo "PDF created: $(wc -c < output.pdf) bytes" || echo "PDF MISSING"

Complete Example

# Generate Client Report in Multiple Formats

## Step 0: Pre-flight
run_shell
command: which pandoc xelatex || echo "Checking toolchain..."

## Step 1: Write markdown source
write_file
path: /tmp/client_report.md
content: |
  # Client Investment Report
  
  ## Executive Summary
  Portfolio performance shows strong returns — up 15% this quarter...
  
  ## Risk Analysis
  Key metrics: "Sharpe ratio" ✓ passed … continuing analysis

## Step 2: Sanitize for PDF (MANDATORY)
write_file
path: /tmp/client_report_sanitized.md
content: |
  # Client Investment Report
  
  ## Executive Summary
  Portfolio performance shows strong returns -- up 15% this quarter...
  
  ## Risk Analysis
  Key metrics: "Sharpe ratio" [x] passed ... continuing analysis

## Step 3: Convert to DOCX (original unicode OK)
run_shell
command: pandoc /tmp/client_report.md -o client_report.docx 2>&1

## Step 4: Convert to PDF (sanitized, xelatex first)
run_shell
command: pandoc /tmp/client_report_sanitized.md -o client_report.pdf --pdf-engine=xelatex 2>&1

## Step 5: Convert to HTML (original unicode OK)
run_shell
command: pandoc /tmp/client_report.md -o client_report.html 2>&1

## Step 6: Verify
run_shell
command: ls -lh client_report.* && file client_report.*

Error Diagnosis Decision Tree

When a conversion fails, capture stderr (2>&1) and diagnose:

If error contains "xelatex not found" or "LaTeX error":
  → Try next engine: --pdf-engine=pdflatex or --pdf-engine=wkhtmltopdf

If error contains "encoding" or "UTF-8":
  → Unicode not properly sanitized; re-check Step 2
  → Add -f markdown+utf8 to pandoc command

If error contains "template" or "class":
  → LaTeX template issue; try --pdf-engine=wkhtmltopdf

If error is "unknown error" (no stderr captured):
  → Re-run with 2>&1 to capture actual error message
  → Check if pandoc is installed: which pandoc

If wkhtmltopdf fails:
  → Install: apt-get install wkhtmltopdf
  → Or use Python alternative: reportlab or fpdf2

Alternative: Python PDF Generation (When pandoc Fails)

If all pandoc engines fail, use Python libraries directly:

Using fpdf2:

run_shell
command: python3 -c "
from fpdf import FPDF
pdf = FPDF()
pdf.add_page()
pdf.set_font('Arial', '', 12)
pdf.cell(0, 10, 'Document Title')
pdf.output('output.pdf')
"

Using reportlab:

run_shell
command: python3 -c "
from reportlab.lib.pagesizes import letter
from reportlab.pdfgen import canvas
c = canvas.Canvas('output.pdf', pagesize=letter)
c.drawString(100, 750, 'Document Title')
c.save()
"

Common pandoc Commands Reference

# Markdown to Word (Unicode-safe)
pandoc input.md -o output.docx

# Markdown to PDF with xelatex (BEST for Unicode)
pandoc input.md -o output.pdf --pdf-engine=xelatex

# Markdown to PDF with pdflatex (requires sanitization)
pandoc input.md -o output.pdf --pdf-engine=pdflatex

# Markdown to PDF with wkhtmltopdf (HTML-based, good fallback)
pandoc input.md -o output.pdf --pdf-engine=wkhtmltopdf

# Markdown to HTML
pandoc input.md -o output.html

# With metadata
pandoc input.md -o output.pdf --metadata title="Document Title"

# Force UTF-8 encoding
pandoc -f markdown+utf8 input.md -o output.pdf

Troubleshooting Quick Reference

| Symptom | Likely Cause | Solution | |---------|--------------|----------| | "xelatex not found" | Missing LaTeX engine | apt-get install texlive-xetex or try --pdf-engine=wkhtmltopdf | | "LaTeX error: encoding" | Unicode in source | Use sanitized markdown for PDF | | "unknown error" (pandoc) | stderr not captured | Re-run with 2>&1 to see real error | | PDF missing after conversion | All engines failed | Try Python (fpdf2/reportlab) as fallback | | DOCX has garbled text | Encoding issue | Add -f markdown+utf8 to pandoc command | | HTML renders but PDF fails | LaTeX-specific issue | wkhtmltopdf engine usually works |

Verification Checklist

Before marking task complete, verify:

• [ ] Pre-flight: pandoc installed and accessible
• [ ] Source markdown created with write_file
• [ ] Sanitized version created for PDF conversion
• [ ] DOCX generated from original (unicode preserved)
• [ ] PDF generated with xelatex (or fallback engine documented)
• [ ] All output files exist: ls -lh output.*
• [ ] File types verified: file output.*
• [ ] Content validated (spot-check with read_file if applicable)

When to Use shell_agent Instead

After successfully completing this manual workflow:

• For simple DOCX-only tasks (no PDF needed)
• When toolchain is verified working
• For repetitive tasks with known-good content

For documents with Unicode content requiring PDF, always use this manual workflow.

Related Skills

• spreadsheet-direct-python: For Excel/CSV generation with Python
• pdf-verification-cli: For verifying PDF page count and content after creation