How the similarity actually works.
This is an AI generated description of the math behind this project.
EMBEDDINGS
Every poem is run through a sentence-embedding model
(embeddinggemma-300m) that turns its text into a fixed-length list
of numbers -- a point in a high-dimensional space. Poems that mean
similar things land near each other. Vectors are stored at half
precision (FP16) to save space and computed at full precision (FP32).
COSINE SIMILARITY
To compare two poems we measure the ANGLE between their vectors, not
the distance. Two poems point the same way (angle ~ 0) when they are
alike; they point apart when they differ. 'similar' sorts by smallest
angle; 'different' sorts by largest -- maximum contrast, which is not
the same as 'unrelated noise'.
THE MOOD COLOR MAP
The colors of each poem are determined by comparing the centroid of the poem
to the embedding of each color, alongside some "theme" words that help
"red" not be next to "blue" just because they were colors. So, the color
of the word on the wordcloud page and the color of the "progress bars" of
each poem are supposed to reflect the theme. Whether they did or not is
often unpredictable, but... I don't care :)
THE WORD CLOUD ON THE MENU
The menu's word cloud sizes each word by how often it shows up, on a
LOGARITHMIC scale. Word counts follow Zipf's law -- a handful of words
dominate and a long tail barely appears -- so a plain linear size would
flatten almost everything to the smallest tier; the log stretches the
small counts back out. Frequency decides SIZE, and nothing else.
Left in size order the big words would all bunch at the front, so the
list is shuffled first with a Fisher-Yates pass: walk it from the last
word back to the first, and swap each word with a randomly chosen one at
or before it. A single sweep, and every possible ordering is equally
likely.
THE SHAPE OF THIS COLLECTION (counted at build time)
Poems per source (of 8510 total):
fediverse | ████████████████████████████████████████ 5977
messages | ██████████······························ 1558
notes | ███····································· 470
fediverse_boost | ███····································· 458
bluesky | ········································ 47
Poem length (characters):
0-99 | ████████████████████████████████████████ 2938
100-249 | ███████████████████████████████········· 2254
250-499 | █████████████████······················· 1268
500-999 | ██████████████·························· 1018
1000-1999 | ███████████····························· 839
2000+ | ███····································· 193
Poems per year:
2021 | █······································· 102
2022 | █······································· 90
2023 | █████··································· 546
2024 | ████████████████████████████████████████ 4834
2025 | ███████████████████····················· 2285
2026 | █████··································· 653
Back to Explore
|