document-gen-resilient-workflow

Resilient Document Generation Workflow

When to Use

Use this skill when document generation tasks encounter tool errors or failures, especially:

• run_shell commands return 'unknown error' without diagnostic details
• PDF generation fails with LaTeX encoding or missing engine errors
• You need fallback options when primary conversion methods fail
• Workspace or path issues cause file access problems
• Unicode/special characters cause rendering issues in PDF output
• shell_agent has failed multiple times on similar tasks

Core Technique

Split document generation into observable, verifiable steps with explicit error capture and progressive fallback:

1. Validate workspace → Confirm current directory and write permissions
2. Pre-flight checks → Verify required tools (pandoc, LaTeX engines) are available
3. Content creation → Use write_file for source Markdown with full visibility
4. Unicode sanitization → Preprocess for PDF conversion when needed
5. Progressive conversion → Try multiple engines/methods until one succeeds
6. Explicit error capture → Redirect stderr to diagnose failures
7. Verification → Confirm output files exist and are readable

⚠️ Critical: Tool Error Patterns

'unknown error' from run_shell: This typically indicates:

• Sandbox execution context issues (not the command itself)
• Missing stderr capture preventing diagnosis
• Transient file access problems

Solution: Use explicit error capture with 2>&1 and verify tool availability first:

run_shell
command: pandoc --version 2>&1 || echo "PANDOC NOT AVAILABLE"

Pre-Flight Checklist

Before starting conversion, verify your environment:

## Pre-Flight Checks

### Check 1: Verify working directory
run_shell
command: pwd && ls -la

### Check 2: Verify pandoc availability
run_shell
command: pandoc --version 2>&1 | head -3 || echo "PANDOC MISSING"

### Check 3: Check LaTeX engines
run_shell
command: which pdflatex xelatex wkhtmltopdf 2>&1 || echo "Some engines missing"

### Check 4: Check Python libraries (fallback option)
run_shell
command: python3 -c "import fpdf; print('fpdf OK')" 2>&1 || echo "fpdf missing"
run_shell
command: python3 -c "import docx; print('python-docx OK')" 2>&1 || echo "python-docx missing"

Complete Step-by-Step Workflow

Step 1: Validate Workspace

Confirm you're in the correct directory and can write files:

run_shell
command: pwd && echo "Test write" > /tmp/workspace_test_$$.txt && ls -la /tmp/workspace_test_$$.txt && rm /tmp/workspace_test_$$.txt

If this fails, you may have directory permission issues. Use absolute paths for all files.

Step 2: Create Source Content with write_file

Write document content as Markdown to a source file:

write_file
path: /tmp/document_source.md
content: |
  # Document Title
  
  ## Section 1
  Content here...

Step 3: Unicode Sanitization (PDF Only)

Only needed for PDF conversion via LaTeX engines. DOCX and HTML handle Unicode natively.

Create a sanitized version replacing LaTeX-problematic characters:

write_file
path: /tmp/document_source_sanitized.md
content: |
  # Document Title
  
  [Same content with: em-dash → --, curly quotes → straight, → arrows → text, etc.]

Common replacements (see full table below): | Original | Replace With | |----------|--------------| | — (em dash) | -- | | " " (curly quotes) | " " (straight) | | … (ellipsis) | ... | | → ← (arrows) | -> <- | | ✓ ✗ | [x] [ ] | | © ® ™ | (c) (r) (tm) |

Alternative: Use sanitization script (see Scripts section below).

Step 4: Progressive Format Conversion

Try multiple methods in order of preference. Capture stderr explicitly for diagnosis.

Method A: DOCX Conversion (Most Reliable)

run_shell
command: pandoc /tmp/document_source.md -o output.docx 2>&1 && echo "SUCCESS" || echo "FAILED: $?"

If this fails, try with explicit UTF-8 flag:

run_shell
command: pandoc /tmp/document_source.md -f markdown+utf8 -o output.docx 2>&1

Method B: PDF Conversion (Progressive Fallback)

Try engines in this order until one succeeds:

run_shell
command: pandoc /tmp/document_source_sanitized.md -o output.pdf 2>&1 | tee /tmp/pdf_error.log && echo "PDF SUCCESS"

If Method B fails, check error log and try next engine:

run_shell
command: pandoc /tmp/document_source_sanitized.md --pdf-engine=xelatex -o output.pdf 2>&1 | tee /tmp/pdf_error.log && echo "PDF SUCCESS"

If xelatex fails:

run_shell
command: pandoc /tmp/document_source_sanitized.md --pdf-engine=wkhtmltopdf -o output.pdf 2>&1 | tee /tmp/pdf_error.log && echo "PDF SUCCESS"

Method C: Python Library Fallback (When pandoc Unavailable)

If all pandoc methods fail, use Python libraries:

Option C1: FPDF2 for PDF

execute_code_sandbox
language: python
code: |
  from fpdf import FPDF
  
  pdf = FPDF()
  pdf.add_page()
  pdf.set_font('Arial', 'B', 16)
  pdf.cell(40, 10, 'Document Title')
  pdf.ln(20)
  pdf.set_font('Arial', '', 12)
  pdf.multi_cell(0, 10, 'Your content here...')
  pdf.output('output.pdf')
  print('PDF created successfully')

Option C2: python-docx for DOCX

execute_code_sandbox
language: python
code: |
  from docx import Document
  
  doc = Document()
  doc.add_heading('Document Title', 0)
  doc.add_paragraph('Your content here...')
  doc.save('output.docx')
  print('DOCX created successfully')

Step 5: Verify Output Files

Don't just check existence—verify files are valid:

run_shell
command: ls -lh output.* 2>&1

run_shell
command: file output.docx output.pdf 2>&1

For DOCX, verify content:

run_shell
command: unzip -p output.docx word/document.xml 2>&1 | head -50 || echo "Cannot read DOCX"

For PDF, check page count:

run_shell
command: pdfinfo output.pdf 2>&1 | grep Pages || echo "pdfinfo not available"

Alternative verification via Python:

execute_code_sandbox
code: |
  import os
  for f in ['output.docx', 'output.pdf', 'output.html']:
    if os.path.exists(f):
      size = os.path.getsize(f)
      print(f'{f}: {size} bytes')
    else:
      print(f'{f}: NOT FOUND')

Complete Example: Generating a Report

# Generate Quarterly Report (DOCX + PDF)

## Step 1: Pre-flight checks
run_shell
command: pandoc --version 2>&1 | head -1

## Step 2: Write Markdown source
write_file
path: /tmp/quarterly_report.md
content: |
  # Quarterly Business Report
  
  ## Executive Summary
  Q4 performance exceeded expectations...
  
  ## Key Metrics
  - Revenue: $1.2M
  - Growth: 15%
  
  ## Recommendations
  Continue current strategy...

## Step 3: Create sanitized version for PDF
write_file
path: /tmp/quarterly_report_sanitized.md
content: |
  # Quarterly Business Report
  
  ## Executive Summary
  Q4 performance exceeded expectations...
  
  [Rest of content with special chars replaced]

## Step 4: Generate DOCX
run_shell
command: pandoc /tmp/quarterly_report.md -o quarterly_report.docx 2>&1 && echo "DOCX OK"

## Step 5: Generate PDF (try xelatex for Unicode safety)
run_shell
command: pandoc /tmp/quarterly_report_sanitized.md --pdf-engine=xelatex -o quarterly_report.pdf 2>&1 && echo "PDF OK"

## Step 6: Verify outputs
run_shell
command: ls -lh quarterly_report.* && file quarterly_report.*

Full Unicode Replacement Table

| Character | Unicode | Issue | Safe Replacement | |-----------|---------|-------|------------------| | — | U+2014 | LaTeX incompatible | -- | | – | U+2013 | LaTeX incompatible | - | | " " | U+201C/U+201D | Encoding errors | " (straight) | | ' ' | U+2018/U+2019 | Encoding errors | ' (straight) | | … | U+2026 | May not render | ... | | → | U+2192 | LaTeX incompatible | -> or to | | ← | U+2190 | LaTeX incompatible | <- | | ↑ ↓ | U+2191/U+2193 | LaTeX incompatible | ^ v | | • | U+2022 | Font-dependent | - or * | | ✓ | U+2713 | May not render | [x] | | ✗ | U+2717 | May not render | [ ] | | ★ ● | U+2605/U+25CF | May not render | * - | | © | U+00A9 | May require packages | (c) | | ® | U+00AE | May require packages | (r) | | ™ | U+2122 | May require packages | (tm) | | é ñ ü | Various | Font-dependent | Use xelatex or replace |

Reusable Sanitization Script

Save this script for repeated use:

#!/bin/bash
# sanitize_for_pdf.sh - Replace problematic unicode chars for LaTeX/PDF
set -e
if [ -z "$1" ]; then
  echo "Usage: $0 <input.md> [output.md]"
  exit 1
fi
INPUT="$1"
OUTPUT="${2:-${1%.md}_sanitized.md}"

sed -e 's/—/--/g' \
    -e 's/–/-/g' \
    -e 's/"\([^"]*\)"/"\1"/g' \
    -e "s/'\([^']*\)'/\'\1\'/g" \
    -e 's/…/.../g' \
    -e 's/→/->/g' \
    -e 's/←/<-/g' \
    -e 's/↑/^/g' \
    -e 's/↓/v/g' \
    -e 's/•/-/g' \
    -e 's/✓/[x]/g' \
    -e 's/✗/[ ]/g' \
    -e 's/★/*/g' \
    -e 's/©/(c)/g' \
    -e 's/®/(r)/g' \
    -e 's/™/(tm)/g' \
    "$INPUT" > "$OUTPUT"

echo "Sanitized: $INPUT -> $OUTPUT"

Usage:

run_shell
command: chmod +x sanitize_for_pdf.sh && ./sanitize_for_pdf.sh /tmp/document_source.md /tmp/document_source_sanitized.md

Error Diagnosis Guide

Error: "unknown error" from run_shell

Diagnosis:

run_shell
command: pandoc --version 2>&1
run_shell
command: echo "test" | pandoc -f markdown -t plain 2>&1

If pandoc works in isolation, the issue is likely:

• File path problems (use absolute paths)
• Sandbox context issues (retry or use shell_agent)
• Transient errors (retry the same command)

Error: "LaTeX not found" or "pdflatex: command not found"

Solutions in order:

1. Try xelatex: --pdf-engine=xelatex
2. Try wkhtmltopdf: --pdf-engine=wkhtmltopdf
3. Install LaTeX: apt-get install texlive-latex-recommended texlive-fonts-recommended
4. Use Python fpdf2 fallback (see Step 4, Method C)

Error: "Encoding error" or "Invalid UTF-8"

Solutions:

1. Add UTF-8 flag: pandoc -f markdown+utf8
2. Check file encoding: file -i source.md
3. Re-save with explicit UTF-8: ``` run_shell command: iconv -f UTF-8 -t UTF-8 source.md -o source_utf8.md ```
4. Use sanitization step

Error: PDF generated but empty or garbled

Diagnosis:

run_shell
command: pdfinfo output.pdf 2>&1

execute_code_sandbox
code: |
  import fitz  # PyMuPDF
  doc = fitz.open('output.pdf')
  print(f'Pages: {doc.page_count}')
  if doc.page_count > 0:
      page = doc[0]
      print(f'Text: {page.get_text()[:200]}')

Solutions:

1. Use xelatex engine with Unicode content
2. Check sanitization step (if using pdflatex)
3. Verify source markdown is valid

Error: DOCX file won't open or shows error

Diagnosis:

run_shell
command: unzip -l output.docx 2>&1 | head -10

Solutions:

1. Add --reference-doc with valid template
2. Ensure markdown syntax is valid
3. Try pandoc verbose mode: pandoc -v input.md -o output.docx 2>&1

Workspace Troubleshooting

Files appearing in wrong directory

Diagnosis:

run_shell
command: pwd && find /tmp -name "*.docx" -o -name "*.pdf" 2>/dev/null | head -10

Solutions:

1. Use absolute paths in all commands
2. Explicitly change to target directory: cd /path/to/workspace && ...
3. After generation, copy files to correct location: ``` run_shell command: cp /tmp/output.docx ./output.docx ```

Permission denied errors

Solutions:

1. Use /tmp directory for intermediate files
2. Copy final output to target location
3. Check disk space: df -h .

When to Use shell_agent vs Manual Workflow

| Situation | Recommended Approach | |-----------|---------------------| | Simple DOCX generation, no special chars | shell_agent (faster) | | PDF with Unicode/symbols | Manual workflow with sanitization | | Multiple 'unknown error' failures | Manual workflow with error capture | | Need to debug conversion issues | Manual workflow (visible steps) | | Time-critical, simple format | shell_agent | | Critical document, must succeed | Manual workflow with fallbacks |

Alternative: Hybrid Approach

For reliability with less manual effort:

## Hybrid Workflow

1. **First attempt**: `shell_agent` with clear task description

2. **On first error**: Switch to manual workflow:
   - `write_file` for source
   - `run_shell` with error capture (2>&1)
   - Verify with explicit checks

3. **On repeated errors**: Use Python library fallback (execute_code_sandbox)

This balances speed with reliability.

Related Skills

• workspace-path-troubleshooting: For diagnosing file location issues
• pdf-verification-cli: For validating generated PDFs
• write-file-fallback-report: For content-first document creation when web sources fail

Quick Reference Card

QUICK START: Generate Document in 3 Formats

1. Write source: write_file → /tmp/doc.md

2. Sanitize (PDF only): 
   Replace: — → --, "" → "", … → ...

3. Convert with error capture:
   DOCX: pandoc /tmp/doc.md -o out.docx 2>&1
   PDF:  pandoc /tmp/doc_sanitized.md --pdf-engine=xelatex -o out.pdf 2>&1
   HTML: pandoc /tmp/doc.md -o out.html 2>&1

4. Verify: ls -lh out.* && file out.*

5. If pandoc fails: Use execute_code_sandbox with fpdf2 or python-docx