issues/completed/1-001-setup-poem-extraction-system.md

Issue 001: Setup Poem Extraction System

Current Behavior

  • No system exists to extract individual poems from words.pdf
  • Source material location is known but not integrated into project

Intended Behavior

  • System can read and parse words.pdf/output.txt from source directory
  • Individual poems are extracted with proper formatting and metadata
  • Extracted poems are stored in structured format for processing

Suggested Implementation Steps

  1. Analyze format of poems in /home/ritz/programming/ai-stuff/words-pdf
  2. Create poem parser script that can identify poem boundaries
  3. Extract individual poems preserving formatting and structure
  4. Store poems in JSON/structured format with metadata (title, content, id)
  5. Create validation system to ensure all poems are captured correctly

Metadata

  • Priority: High
  • Estimated Time: 4-6 hours
  • Dependencies: None
  • Category: Data Processing

Related Documents

  • notes/vision - Source material location and requirements
  • docs/project-overview.md - Technical architecture overview

Tools Required

  • Python for text processing
  • Access to source directory: /home/ritz/programming/ai-stuff/words-pdf

UPDATES:

  • Python should be dispreferred, instead prefer using Lua. You can find the libs

for Lua development inside of:
/home/ritz/programming/ai-stuff/
or
/home/ritz/programming/lua/

  • the poems should be easier to extract from output.txt than words.pdf. However

the visual styles and themes of words.pdf should be preferred, if possible.
however, visual styles are NOT the most important part, and should be pushed
to phase 2 if possible.

  • you can use dkjson.lua for storing json data.
  • you can use luasocket for communicating with Ollama.
  • You can use fuzzy-computing.lua for communicating with Ollama.

2025-11-02 COMPLETION UPDATE:

✅ FULLY COMPLETED - ALL REQUIREMENTS MET

DELIVERABLES COMPLETED:

  1. ✅ Poem Extraction System: src/poem-extractor.lua created with full functionality
  2. ✅ Multi-category Support: Extracts from fediverse/, messages/, notes/ directories
  3. ✅ Structured Output: JSON format with metadata (id, filepath, category, content, length)
  4. ✅ Interactive Mode: Supports -I flag for interactive operation per CLAUDE.md standards
  5. ✅ Path Management: Uses ${DIR} variable for directory-independent execution

EXTRACTION RESULTS:

  • Total Poems Extracted: 6,860 (vs initial 865 - fixed major parsing issue)
  • Categories Found:
  • fediverse/: 5,730 poems (primary content)
  • messages/: 865 poems
  • notes/: 269 poems
  • other: 2 poems (view-random, view-next)
  • Source File: compiled.txt (107,094 lines processed)
  • Output: assets/poems.json with complete metadata

CRITICAL BUG FIXED:

  • Initial Issue: Only extracted "messages/" files (865 poems)
  • Root Cause: Parser only looked for "messages/" pattern, missed other categories
  • Resolution: Updated parser to extract all "-> file:" patterns regardless of directory
  • Impact: Increased extraction from 865 to 6,860 poems (794% improvement)

VALIDATION METRICS:

  • Non-empty poems: 6,818 (99.4%)
  • Average length: 472.6 characters
  • Fediverse compatible (≤1024 chars): 5,835 (85.1%)
  • Potential alt-text entries: 2,204 (32.1%)
  • Data integrity: 100% length matching, minimal duplicates

FILES CREATED:

  • src/poem-extractor.lua - Main extraction tool
  • assets/poems.json - Complete poem dataset (6,860 poems)
  • Integration with dkjson.lua for JSON processing

ISSUE STATUS: COMPLETED ✅