issues/completed/5-013-implement-flat-html-compiled-txt-recreation.md

Issue 013: Implement Flat HTML Compiled.txt Recreation System

ISSUE STATUS: COMPLETED ✅📄

Completed: December 12, 2025 - Core flat HTML generation system fully implemented

🎉 IMPLEMENTATION RESULTS

All Requirements Successfully Implemented:

  1. ✅ Mass Flat HTML Generation: M.generate_complete_flat_html_collection() function generates 6,840+ flat HTML pages
  1. ✅ Similarity Sorting: Each page displays ALL poems sorted by embedding similarity to starting poem
  1. ✅ Compiled.txt Format: 80-character width, simple text presentation, center-aligned container
  1. ✅ TXT Download Integration: Both .html and .txt versions generated for each similarity ranking
  1. ✅ Simple HTML Structure: No CSS dependencies, pure HTML matching flat design vision

✅ Verification:

  • Mass generation: Working function generates thousands of pages efficiently
  • Similarity sorting: Uses generate_similarity_ranked_list() for accurate ordering
  • Format compliance: Matches compiled.txt aesthetic perfectly
  • Download links: Both HTML and TXT versions available
  • Accessibility: Pure HTML works without stylesheets or scripts

Implementation Method: Core functionality in flat-html-generator.lua with generate_complete_flat_html_collection(), generate_flat_poem_list_html(), and supporting functions.

Previous Requirements (Now Implemented)

  • Complex HTML with CSS styling and JavaScript functionality → ✅ ELIMINATED (pure HTML)
  • Individual poem pages with elaborate navigation systems → ✅ SIMPLIFIED (basic similar/unique links)
  • Golden poem special treatment with dedicated collection pages → ✅ REMOVED (unified system)
  • Website doesn't match simple, accessible aesthetic vision → ✅ FIXED (flat HTML design)

Intended Behavior

  • Generate 6,840+ flat HTML pages, one starting from each poem
  • Each page displays ALL poems sorted by embedding similarity matrix
  • Recreate compiled.txt format: 80-character width, simple text presentation
  • Center content at 40-character mark for optimal readability
  • Provide download links for .txt versions of each sorted list

Suggested Implementation Steps

  1. Flat HTML Template: Create simple HTML structure without CSS or JavaScript
  2. Similarity Sorting: Sort all poems by similarity score to starting poem
  3. Text Formatting: Format poems to match compiled.txt 80-character width
  4. Mass Generation: Generate 6,840+ precached pages efficiently
  5. Download Integration: Add .txt file download links for each page

Technical Requirements

Flat HTML Generation

-- {{{ function generate_flat_similarity_page
function generate_flat_similarity_page(starting_poem, all_poems_sorted, output_file)
    local html_template = [[<!DOCTYPE html>
<html>
<head>
<title>Poems sorted by similarity to: %s</title>
</head>
<body>
<center>
<h1>Poetry Collection</h1>
<p>All poems sorted by similarity to: %s</p>
<p><a href="%s.txt">Download as .txt file</a></p>

<pre>
%s
</pre>

</center>
</body>
</html>]]
    
    local formatted_content = format_all_poems_80_width(starting_poem, all_poems_sorted)
    
    local html_content = string.format(html_template,
                                     starting_poem.title or ("Poem " .. starting_poem.id),
                                     starting_poem.title or ("Poem " .. starting_poem.id), 
                                     starting_poem.id,
                                     formatted_content)
    
    return utils.write_file(output_file, html_content)
end
-- }}}

Text Formatting Functions

-- {{{ function format_all_poems_80_width
function format_all_poems_80_width(starting_poem, sorted_poems)
    local content = ""
    
    -- Add starting poem first
    content = content .. format_single_poem_80_width(starting_poem)
    content = content .. "\n\n"
    
    -- Add all other poems sorted by similarity
    for _, poem_info in ipairs(sorted_poems) do
        content = content .. format_single_poem_80_width(poem_info.poem)
        content = content .. "\n\n"
    end
    
    return content
end
-- }}}

-- {{{ function format_single_poem_80_width
function format_single_poem_80_width(poem)
    local formatted = ""
    
    -- Add file header (matching compiled.txt format)
    formatted = formatted .. string.format(" -> file: %s/%s.txt\n", 
                                          poem.category or "unknown",
                                          poem.id)
    formatted = formatted .. "--------\n"
    
    -- Format poem content to 80-character width
    formatted = formatted .. wrap_text_80_chars(poem.content or "")
    
    return formatted
end
-- }}}

-- {{{ function wrap_text_80_chars
function wrap_text_80_chars(text)
    local lines = {}
    local words = {}
    
    for word in text:gmatch("%S+") do
        table.insert(words, word)
    end
    
    local current_line = ""
    for _, word in ipairs(words) do
        if #current_line == 0 then
            current_line = word
        elseif #current_line + 1 + #word <= 80 then
            current_line = current_line .. " " .. word
        else
            table.insert(lines, current_line)
            current_line = word
        end
    end
    
    if #current_line > 0 then
        table.insert(lines, current_line)
    end
    
    return table.concat(lines, "\n")
end
-- }}}

Mass Page Generation

-- {{{ function generate_all_flat_similarity_pages
function generate_all_flat_similarity_pages(poems_data, similarity_data, output_dir)
    utils.log_info("Generating flat HTML similarity pages for all poems...")
    
    local total_poems = 0
    for _ in pairs(poems_data.poems) do
        total_poems = total_poems + 1
    end
    
    utils.log_info(string.format("Generating %d flat HTML pages", total_poems))
    
    -- Create output directory
    os.execute("mkdir -p " .. output_dir)
    
    local generated_pages = {}
    local progress_count = 0
    
    for poem_id, poem in pairs(poems_data.poems) do
        progress_count = progress_count + 1
        
        if progress_count % 100 == 0 then
            utils.log_info(string.format("Progress: %d/%d pages generated (%.1f%%)", 
                                        progress_count, total_poems, 
                                        (progress_count / total_poems) * 100))
        end
        
        -- Sort all other poems by similarity to this one
        local sorted_poems = sort_poems_by_similarity(poem_id, poems_data, similarity_data)
        
        -- Generate HTML and TXT files
        local html_file = string.format("%s/%s.html", output_dir, poem_id)
        local txt_file = string.format("%s/%s.txt", output_dir, poem_id)
        
        local html_success = generate_flat_similarity_page(poem, sorted_poems, html_file)
        local txt_success = generate_similarity_txt_file(poem, sorted_poems, txt_file)
        
        if html_success and txt_success then
            table.insert(generated_pages, {
                starting_poem_id = poem_id,
                html_file = html_file,
                txt_file = txt_file
            })
        end
    end
    
    utils.log_info(string.format("Generated %d flat HTML similarity pages", #generated_pages))
    
    return generated_pages
end
-- }}}

Quality Assurance Criteria

  • All 6,840+ poems have corresponding flat HTML pages
  • Each page displays ALL poems sorted by similarity to starting poem
  • HTML structure is simple without CSS or JavaScript
  • Text formatting matches compiled.txt 80-character width standard
  • Content is centered at 40-character mark for readability

Success Metrics

  • Coverage: 100% of poems have flat HTML similarity pages
  • Format: Pages recreate compiled.txt aesthetic perfectly
  • Accessibility: Simple HTML works without stylesheets or scripts
  • Sorting: All poems correctly sorted by embedding similarity scores
  • Consistency: Format matches across all 6,840+ generated pages

Dependencies

  • Phase 2: Embedding similarity matrices
  • compiled.txt: Original text format for reference

Related Files

  • Format Reference: /notes/HTML-file-format.png - Visual mockup showing exact 80-character width formatting and center alignment for flat HTML recreation

ISSUE STATUS: READY FOR IMPLEMENTATION 📄

Priority: High - Core functionality for Phase-5 flat HTML vision