issues/completed/phase-8/8-018-fix-embedding-directory-case-inconsistency.md

Issue 8-018: Fix Embedding Directory Case Inconsistency

Current Behavior

Two different directory naming conventions exist for the EmbeddingGemma model:

  1. EmbeddingGemma_latest/ - Hardcoded in many scripts and utils.embeddings_dir() default
  2. embeddinggemma_latest/ - Generated by get_model_storage_path() when sanitizing "embeddinggemma:latest"

This causes:

  • Embeddings generated via similarity-engine.lua go to embeddinggemma_latest/
  • Other scripts looking in EmbeddingGemma_latest/ don't find the new embeddings
  • User confusion about which directory is canonical

Discovery Context

User ran embedding generation for 7,793 poems. Generation completed successfully but script crashed afterward due to unrelated timing_data bug (see below). New embeddings (72MB) saved to embeddinggemma_latest/, but most scripts still point to EmbeddingGemma_latest/ (6,641 embeddings from Nov 2).

Secondary Bug: timing_data undefined

src/similarity-engine.lua:578 references timing_data.average_generation_time but timing_data is never defined, causing post-generation crash.

Intended Behavior

  1. Single canonical directory name: embeddinggemma_latest (matches automatic sanitization)
  2. All scripts and utilities use this directory consistently
  3. timing_data bug fixed so embedding generation completes cleanly

Suggested Implementation Steps

Step 1: Update utils.lua default ✅ COMPLETED

  • [x] Change embeddings_dir() default from "EmbeddingGemma_latest" to "embeddinggemma_latest"

Step 2: Update all hardcoded references ✅ COMPLETED

Files requiring updates (from grep):

  • [x] libs/utils.lua - default value and help text
  • [x] scripts/generate-html-parallel - 4 references
  • [x] scripts/precompute-diversity-sequences - 3 references
  • [x] scripts/generate-corner-box-demo - 1 reference
  • [x] scripts/test-diversity-quick - 1 reference
  • [x] scripts/test-html-generation - 2 references
  • [x] src/flat-html-generator.lua - 3 references
  • [x] src/centroid-html-generator.lua - 3 references
  • [x] src/centroid-generator.lua - 1 reference (model_storage_name)
  • [x] src/main.lua - 6 references
  • [x] src/run-validation.lua - 2 references
  • [x] src/run-validation-with-reports.lua - 2 references
  • [x] src/regenerate-clean-site.lua - 2 references
  • [x] src/semantic-color-calculator.lua - 3 references
  • [x] src/html-generator/test-embedding-list-generator.lua - 1 reference
  • [x] src/html-generator/golden-collection-generator.lua - 1 reference
  • [x] src/html-generator/similarity-engine.lua - 1 reference
  • [x] src/test-mass-diversity-generator.lua - 4 references
  • [x] src/test-diversity-chaining.lua - 4 references
  • [x] src/test-validation-engine.lua - 2 references
  • [x] phase-demo.sh - 2 references
  • [x] demos/2-demo.sh - 2 references
  • [x] demos/3-demo.sh - 2 references
  • [x] demos/4-demo.lua - 1 reference
  • [x] demos/5-demo.lua - 2 references

Step 3: Fix timing_data bug ✅ COMPLETED

  • [x] Remove or fix lines 576-580 in src/similarity-engine.lua

Step 4: Directory cleanup ✅ COMPLETED

  • [x] Verify embeddinggemma_latest/ has complete data (7,793 embeddings)
  • [x] Remove old EmbeddingGemma_latest/ directory
  • [x] Remove mystery y/ directory if empty

Files Modified

  • libs/utils.lua
  • src/similarity-engine.lua (naming + timing_data fix)
  • Multiple scripts and source files (see Step 2)

Phase: 8 (Website Completion)

Priority: High (blocking further pipeline work)

Created: 2025-12-25

Status: Completed

Completed: 2025-12-25