🧪 The Lab

The Lab Notebook

Our motif index is built by language models running on our own computer, with Claude (Anthropic's AI model) checking their work and the person who runs the site spot-checking Claude. Every change to how the models file and find stories is tried here first, on the person's own decisions, with its hypothesis written down before it runs, and kept only if it helps. This is the notebook: each experiment an entry on the day its first result came in, newest first, named after a figure from folklore. The misses are here too: an idea that didn't work is a finding. The methods log tells what each kept change did to the site.

📅 November 2026

O2

the Kilkenny Cats

answered

❓ What share of American front-page news is people reacting to each other ("slams", "responds", "calls for") rather than things happening? A word pattern over two weeks of headlines, measured against a blind read of 400 headlines and 150 stories; outlets and sagas scored (the Kilkenny Cats)

How much of American front-page news is people answering each other ("slams", "responds", "calls for") rather than things happening? Under the site's own rule a reaction is a story of its own, so we asked how much of the news is that kind of story. Two weeks of US front pages (Nov 8 to 21, 46,714 articles): a word pattern over every headline, measured against a blind read of 400 headlines and 150 stories of three or more outlets by two Sonnet readers (a third on splits). Outlets and sagas were then ranked by the pattern.

The upshot reactions 8.7% of front-page headlines (R 0.18; politics 0.30, the rest 0.07), 4.7% of stories 3+ outlets share (R 0.06); the word pattern fails its bar (reaction F1 0.48), so a model labeller is proposed, its prompt waiting for the person; partisan sites on both sides lead, wires trail (tentative)
Keep reading ✏️

🎯 What we expected

Written before any headline was classified: reaction about 12% of headlines and R (reaction over event plus reaction) about 0.20, 0.35 in politics; stories less reactive (about 10%); the pattern's reaction precision at least 0.70; partisan sites on both sides highest, wires lowest, a U by lean; political sagas above 0.4. If the pattern's F1 fell under 0.70, a model labeller would be proposed, its prompt shown first.

🔬 What we found

Reactions are 8.7% of front-page headlines (R 0.18; 0.30 in politics, 0.07 outside it), and only 4.7% of the stories three or more outlets share (R 0.06): most jabs stay on one or two sites. Statements that answer no one are as common as reactions; features, explainers and columns are 43% of headlines. Readers agreed 90% (kappa 0.85). The word pattern failed its bar (reaction F1 0.48), so outlet ranks are tentative: partisan sites on both sides lead (Mediaite, Raw Story, The Gateway Pundit, Alternet, The Daily Beast, Fox News), wires and business desks trail, and by lean it is an even U (±2 both 0.085, centre 0.049). A model labeller is proposed, not run; its prompt waits for the person.

A10

the Seven Sleepers

answered

❓ Which genres of today's top motifs were already told in the American papers of 1896? 75 motifs, 15 per genre, each searched on a fixed budget and read whole (the Seven Sleepers)

Which of today's political tropes were already told in the American papers of 1896? We took the 15 motifs with the most claims in each of five genres (Plots, Archetypes, Theories, Beliefs, Arguments) and gave each the same search budget in 413,186 political articles of 1896: our pipeline's filings, the motif's own phrases, and one search in 1896 wording written from its note before anything ran. Five Claude readers read every candidate article whole and called each motif an echo (pattern and content), shape only (the move is old, the content new) or none found; a sixth blind read of 15 checked them.

The upshot 49 of 75 told in 1896 (65%), 60 with shape-only (80%); Beliefs 80%, Plots and Theories 73%, Archetypes 53%, Arguments 47% (most shape-only: old moves, new content); phrase hits agree with the read 53%; audit agrees 10 of 15
Keep reading ✏️

🎯 What we expected

Written before any search: about half the 75 have an echo (two thirds with shapes); Theories highest, then Plots, Archetypes, Beliefs, with Arguments lowest and mostly shape only; a phrase count orders the genres differently from the read and agrees with it per motif no better than two times in three; money and class motifs at least a third of the echoes, the corpus's lean.

🔬 What we found

Two in three were told in 1896 (49 of 75, 65%), four in five counting shape-only (80%). Beliefs 80%, Plots and Theories 73%, Archetypes 53%, Arguments 47%; Arguments hold the most shape-only verdicts (Robespierre for Hitler, a Catholic candidate's critics called bigots where a Muslim candidate's are called Islamophobes, gas monopolies for data centers). The election's own machinery is barely changed: stolen counts, spoilers, coerced voters, distrusted straw polls, 'availability', the stay-at-home vote, the candidate in his dotage, 'a silly fake story'. Phrase hits agree with the read only 53% of the time. Money and class were 18% of the echoes, not the third expected. The audit agreed on 10 of 15, all splits close calls both ways; no person checked these readings.

V18

the Cuckoo

answered

❓ The story that doesn't belong in its saga: DeepSeek's impostor and saga-check prompts asked in both orders, the production judge and similarity, on 271 sets, then over the live sagas (the Cuckoo)

The upshot no single arm passes (best: detect 72-74%, false flags 19-21%); three arms agreeing in both orders: 64% / 7% (post hoc); live: 10 of 574 parts evicted, 1 plainly right, 5 plainly wrong, 4 vague; not shipped

S30

the Golden Goose

in use on the site

❓ Can a rule that splits stories by outlet count kill katamari stories without hurting event detection?

We tested a new rule to stop stories from growing into giant lumps that swallow each day's news. The rule splits a story when half its new headlines come from outlets that already have a different article in it. We also tried asking a language model to confirm the split, and lowering the outlet threshold to three. The goal was to cut the nine katamari stories from October without breaking the event detection that groups headlines about the same real-world happening.

The upshot The outlet split at three outlets with a model check killed all nine katamaris and raised event F to 0.750; it went live November 19. 0.713 → 0.750
Keep reading ✏️

🎯 What we expected

We expected the outlet split to leave three or fewer katamaris, event F to stay within 0.02 of base, and asking the model to preserve more recall than the rule alone. We predicted the model would call reactions the same event and undo most splits, which turned out wrong.

🔬 What we found

The outlet split at three outlets, with the model asked, left zero katamaris. Event F rose from 0.713 to 0.750 across all cases, with precision up from 0.657 to 0.777 and recall down from 0.843 to 0.736. The model almost always called the second articles a new development, so asking undid few splits. The rule missed trickles of single articles but fired on big second-article bursts. The seam check's gate never offered onsets to the judge, scoring them too high.

V11

Tam Lin

tried, not kept

❓ Can a model rewrite prompts to cut wrong ties without losing right answers?

We tested whether a large language model could rewrite the prompts that decide which news story a satire piece or fact-check belongs to. The site uses these tie prompts to place jokes and checks on story cards. DeepSeek V4 Pro drafted two rewrites, and we ran 64 questions through qwen3.5 9B to see if either version made fewer wrong ties than the current wording while getting at least as many right.

The upshot Neither rewrite beat the current wording, and production's model was best, so the prompts stay.
Keep reading ✏️

🎯 What we expected

We expected the rewrites to tie with the current wording, since the current prompts already state their none rule plainly. We also expected qwen3:8b to make at least as many wrong ties as qwen3.5 with the same wording.

🔬 What we found

Two blind readers agreed on a key for 58 of 64 questions. The current wording got 41 right with 5 wrong ties. DeepSeek A got 37 right with 4 wrong ties, trading one fewer mistake for five more misses. DeepSeek B got 34 right with 6 wrong ties. The production model, qwen3:8b, outperformed everything with 46 right and only 2 wrong ties.

V10

Turandot

in use on the site

❓ Does this claim share the story shape of this motif?

The site uses a small model, Nimble, to decide whether a new claim fits a story shape we already track. It answers a single question for every filing. We wondered if a sharper question would help. DeepSeek, a larger model, drafted two rewrites. Both dropped the explicit mention of a must-have element and asked about the shared story shape instead. We tested them against the current wording on 16,175 filings already decided, kept or removed: 1,419 by the person, the rest by Claude's reviews.

The upshot DeepSeek's wording A: AUC 0.780 to 0.796 on 16,175 decided filings (+0.017, 95% +0.014 to +0.020); live Nov 19 0.780 → 0.796
Keep reading ✏️

🎯 What we expected

We expected both rewrites to lose. An earlier experiment showed that asking about a motif's must-have element was what made the judge work, and the drafts removed that wording. We thought at least one would score lower, with its confidence interval entirely below zero.

🔬 What we found

DeepSeek's wording A lifted agreement with the person's decisions from an AUC of 0.780 to 0.796, a gain of 0.017 with a 95% confidence interval of +0.014 to +0.020. It improved on the person's own rulings (0.905 to 0.915) and on Claude's automated clears (0.767 to 0.784). Wording B showed a smaller gain of 0.005. The rewrite won even though it stopped naming the must-have element that an earlier test found crucial.

V9

Tom Tit Tot

in use on the site

❓ Can a prompt rewrite make the site's labels clearer and more accurate?

We asked a language model to rewrite the site's own prompts—the instructions that turn headlines into story titles, saga names, and translations—then tested the new wordings blind against the current ones. The idea was simple: a prompt that's easier for a small model to follow might give readers clearer, more accurate labels. Two Claude readers judged every output without knowing which wording produced it, picking the best and worst and flagging anything factually wrong.

The upshot DeepSeek's A for titles and names, B for translations, went live Nov 19; the Court page stayed put.
Keep reading ✏️

🎯 What we expected

We expected the rewrites to edge out current on titles and names—the current prompts work, they're just crowded. For translations, we expected little difference: a strong model translating headlines shouldn't be where wording matters most. We were wrong about that.

🔬 What we found

For story titles, DeepSeek's wording A was picked best 29 times against 25 for current, with the same error count (4). For saga names, A won 23 to 18, with half the errors (2 against 4). Current's translation prompt was the surprise: it was flagged wrong on 15 of 120 reads, including turning yen into dollars and putting words in Xi Jinping's mouth. DeepSeek's B was picked best for translations nearly twice as often (58 against 33) with half the errors. On the Court page, no wording helped: case summaries were wrong about 40% of the time under both, mostly because the model didn't know legal acronyms or guessed locations from circuit court names.

V8

the Fox and the Crow

in use on the site

❓ Do the story checks answer the headlines, or the prompt's lean?

The site's story checks decide whether two headlines are part of the same running story, a clean break, or the same event. The prompts they use lean: the saga wording spends most of its words on reasons to join, and the seam check frames one card as the default. We wondered if the checks were answering the prompt's lean rather than the headlines. We wrote balanced drafts for each check, tested them on three models against 780 keyed question pairs, and added two rounds of feedback-driven rewrites.

The upshot Seam check to Nimble with wording A; saga to Gemma with wording B; returning story and arcs stay.
Keep reading ✏️

🎯 What we expected

We expected the wording to move the small model most, that a balanced prompt would let our local judge model do the job on at least two checks, and that Gemma would also improve. We thought feedback-driven rewrites would beat the first balanced drafts.

🔬 What we found

On the seam check, a balanced prompt let our small local model, qwen3.5, jump from 11 right cuts to 36, passing the bar. Nimble matched Gemma's right cuts with fewer wrong ones in a fraction of the time. On the saga check, only Gemma passed, with wording B finding 46 right joins against 1 wrong. The returning story and arc tagger saw no improvement worth switching. Feedback-driven rewrites in rounds two and three did not beat the first balanced drafts.

V7

the Princess and the Pea

answered

❓ Does the headline judge feel what people feel?

We wanted to know if our headline judge's mood scores match how people actually feel. The judge had only been checked against a small set of labels from another model and one person. So we tested it against 1,250 news headlines from 2007 that human annotators rated for valence and six emotions. We also compared two servers, Ollama and llama.cpp, to see if the switch to llama.cpp earlier that day had moved the judge closer to or further from people.

The upshot The judge feels what people feel, but the server matters and good news is still flattened.
Keep reading ✏️

🎯 What we expected

We expected the judge's mood to correlate with people's valence at 0.60 or better on both servers. We thought the server switch wouldn't matter much, and that the judge would see at least 70% of good news and 80% of bad news. We also expected the average emotion correlation to be at least 0.40.

🔬 What we found

Both servers passed the main bar, with mood correlating with people's valence at 0.695 for Ollama and 0.647 for llama.cpp. But the servers were not the same: Ollama was clearly closer to people. Both gave positive mood to only 48.3% of clearly good news, missing the 70% bar. For bad news, Ollama hit 78.8% and llama.cpp 64.4%, both below the 80% target. Two rewrites of the rubric later got much closer to people's valence, with correlations of 0.719 and 0.747, but they drifted from our own hand-checked key.

V5

the Little Red Hen

answered

❓ Can one model do every job in the hourly build?

The hourly run swaps between three models, and each swap unloads one model and loads another, costing time. We tested whether the judge model, qwen3.5 9B, could take over every job that Gemma 26B and qwen3:8B do in the build, so we could retire those models and batch the leftover Gemma work into one phase after the run. We ran 3,258 calls through production's own prompts and schemas, comparing answers on keyed pairs, cached replays, and a settled arc key.

The upshot Only arc placement passes; too many wrong joins and too few cuts elsewhere, so nothing switched.
Keep reading ✏️

🎯 What we expected

We expected the saga judge to fail on precision by a few pairs, the seam and return checks to pass, and arc placement to be the closest call. We thought the small jobs would pass, since qwen3.5 had beaten qwen3:8B on headline labels before.

🔬 What we found

Arc placement passed: qwen3.5 placed 90.1% of live sagas right against Gemma's 93.4%, with precision above 85%. The story checks all failed. On the saga judge, qwen3.5 made 8 wrong joins to Gemma's 3 and turned 28% of Gemma's noes into yeses. On the seam check it made only 19 right cuts to Gemma's 82. On returning stories it made 33 wrong joins to Gemma's 0. The small jobs missed their bars: satire picks agreed only 70.7%, and the Court page missed 95% agreement by one headline. Speed was good: qwen3.5 took about a quarter of Gemma's time per call.

V4

Atalanta

in use on the site

❓ How fast can the headline judge run on llama.cpp's own server without losing accuracy?

We moved the hourly headline judge from Ollama to llama.cpp's own server, where every setting can be chosen. We tested slot counts, cache sizes, and micro-batch sizes on 200 fixed headlines, looking for the fastest setup that still agreed with the old judge. Midway we moved the outlet's name to the end of the prompt, then removed it entirely: the person doubted the judge should see which outlet ran a headline at all. The final no-outlet prompt runs in production.

The upshot 16 slots, micro-batch 128, no outlet's name: 250 headlines a minute against Ollama's 126, about 2 times; live from the Nov 18 18:00 run
Keep reading ✏️

🎯 What we expected

We expected llama.cpp's server to judge at least 1.5 times faster than Ollama while agreeing on 98% of news calls and 95% of mood. We also guessed the peak slot count would land near 10 or 11, not a round number, and that moving the outlet name last would double speed.

🔬 What we found

The fastest setting (16 slots, micro-batch 128, no outlet name) judges 250 headlines a minute, about twice Ollama's 126. Removing the outlet name dropped news agreement with the old judge from 99.5% to 96.5%, but a hand-checked key showed the servers tie on mood. Against 1,250 human-rated headlines, llama.cpp's mood correlates 0.647 with reader valence against Ollama's 0.695, a gap of 0.05.

V3

Briareus

tried, not kept

❓ Can vLLM judge headlines faster than Ollama and give the same answers?

We tried to speed up the hourly news judge by switching from Ollama to vLLM, which can keep a GPU busy by feeding it many headlines at once. The judge is a 9-billion-parameter model that reads four headlines at a time in production; vLLM promised to judge dozens in parallel on the same card. If the answers stayed the same, we could cut the judging time without changing the scores.

The upshot Untested: vLLM couldn't fit the judge on the card, so the switch ships off.
Keep reading ✏️

🎯 What we expected

We expected vLLM to judge headlines at least three times faster than Ollama, and to agree with production on at least 95% of news calls and 80% of mood scores. If it matched on all four bars, we could switch without a rescore.

🔬 What we found

vLLM could not load the judge on our 10 GB card. The planned 4-bit quantization method was missing from vLLM 0.34. A smaller 4-bit copy of the model loaded but left no memory for the per-request state that the model's attention layers need. Six configurations failed for want of cache blocks; the last attempt, with 16 sequences and the whole card, needed 0.11 GB per request and had 0.02 GB left. The hypotheses went untested.

S29

Rapunzel

tried, not kept

❓ How alike must headlines be, and how many links must they share, to form one story?

We tested whether tightening two knobs on our story clustering could fix its tendency to over-join. Production links any two headlines with a likeness score of 0.80 or more into one story, but a single link can bridge unrelated events. We tried raising the likeness threshold and requiring shared links between headlines before they count, running all 685 October pools through 15 settings to see if a stronger rope would improve precision without losing too many real stories.

The upshot Production's 0.80 threshold with one link stands; the knobs are done, and story quality moves to post-clustering checks.
Keep reading ✏️

🎯 What we expected

We expected a setting with a higher threshold or more required links would beat production by at least 0.03 event F on held-out cases, with shared links outperforming a simple threshold raise. We also required the winner to keep at least 85% of production's coverage.

🔬 What we found

No setting beat production's event F of 0.694 on the tuning cases. The best rival, r78k3, tied at 0.693 but lost 0.026 on the five held-out cases, where trump-michigan-rally's recall halved. Raising the threshold or requiring more links did trade precision for recall as expected: precision rose from 0.64 to 0.96 while recall fell from 0.85 to 0.22. The best k=2 setting, r78k2, scored 0.769 on held-out cases but was not chosen by our rule.

V2

Rhadamanthus

in use on the site

❓ Which model and rubric together best match the hand-checked headlines?

The headline judge scores mood, spice and feelings for every headline on the site, about 9,000 calls a day. A blind hand check of 40 headlines found the old rubric flattened good news to neutral and counted plain reporting verbs as loaded. We rewrote the rubric, ran six models against 59 hand-checked headlines plus 120 unlabeled ones, then checked the winner on 60 more hand-labelled headlines with three versions of the rubric.

The upshot Qwen3.5 9B now judges every headline with the old mood wording and the new spice wording (rubric 78d6c6b2): on the 59 headlines the old judge was tested on, mood 42 agree against 34, spice 44 against 43.
Keep reading ✏️

🎯 What we expected

No prediction was written down first: this comparison grew out of the rubric fix the same afternoon, so read it as a comparison, not a test. The hope was that the fix would lift both mood and spice, and that a newer model might see good news the old one missed.

🔬 What we found

The new rubric fixed spice but not mood on the old model: Qwen3 8B stayed blind to good news. Qwen3.5 9B through OpenRouter hit 74% mood and 79% spice. Run here at 4-bit, against 119 hand-checked headlines (60 more labelled blind), the good-news sentence turned out to hurt it: mood 74 of 119 with it, 82 without, because the newer model already sees plain good news. The spice fix still helped: 91 against 83. The old mood wording with the new spice wording scored 83 and 87, and called 16% of a week's news good news, against 20% in the hand labels. Gemma 4 26B agreed most on mood but called 29% good news and won't fit on the card. Phi-4 called nearly everything neutral.

S25

Clotho

in use on the site

❓ Can a local model group an arc's stories into threads as well as two blind readers?

Once sagas sat inside arcs, an arc's threads were still production's sagas, built to answer "is this one running story?", which is too coarse for a thread: the midterms' main saga held 57 stories, every race, the leadership fights and the rallies together. Clotho asked whether the local model (Gemma 4 26B) could group an arc's stories into threads (one turn of the affair followed as it develops: a front, a negotiation, one race, one case) the way careful readers do.

The upshot Full-hindsight thread naming beat the old system on every set but fell short on precision, so it was shipped anyway on the owner's decision and now runs weekly.
Keep reading ✏️

🎯 What we expected

Written before any key was read: on four sets (the live Iran war, midterms and Russia-Ukraine arcs, and July 2024's election), the method that could run in production, naming threads as the stories come in, would beat production's sagas by 0.05 or more in pooled B-cubed F (how closely groupings match two blind readers' keys, 0 to 1), be no more than 0.03 worse on any set, and keep precision at 0.80 or more. Those three were the ship rule.

🔬 What we found

Naming threads with full hindsight beat production on every set (pooled F 0.73 against 0.59; July 0.66 against 0.43) but missed the precision bar (0.72 against 0.80), because some threads were named at a beat's grain rather than a turn's: "Pressure on Biden to Withdraw" held 22 stories a reader put in 4 threads. Naming as stories come in (seeing only the first half) reached 0.66 and fell below production on the Iran war (0.69 against 0.79). Asking the model whether two groups were the same turn split nearly everything (precision 0.94, recall 0.35), and a second round that asked the model to split big threads cut up the wrong ones, trusting each thread's name. The two readers (Claude, Anthropic's AI model, reading separately) agreed at 0.92. Nothing passed its ship rule, but on Nov 13 full-hindsight naming was shipped on the site owner's call, against the precision bar it missed: once a week each open arc's stories are named into threads afresh, with new stories joining between namings, and we are watching for beat-sized lumps.

S24

Melampus

tried, not kept

❓ Can coverage jumps and meaning shifts together mark where a long story develops?

Our fifth try at splitting a long-running story at its new developments, again with no model asked. Melampus looked for two signals in the same hours: a coverage jump (the number of front pages carrying the story at least doubling and growing by 5 or more, as when the Fort Hood livestream's cancellation took a story from 6 outlets to 35 in an hour), and a shift in meaning (the new headlines sitting unusually far from everything the story had said before, measured with a small embedding model).

The upshot A surge in coverage was common but the shift in meaning was usually unmeasurable, so the method barely split any stories and was abandoned.
Keep reading ✏️

🎯 What we expected

Written before any split was computed, with Chanticleer's bars unchanged: 15% to 45% of long stories split at least once, no more than 5% into five parts or more; 4 or more of 5 known developments caught; and 15 or more of 20 splits judged real in a blind read.

🔬 What we found

It split 4 of 207 long stories (2%) and caught 1 of the 5 known developments (the plan to make the Fort Hood execution public), with no stray splits, and there were too few splits for a blind read. Not shipped. The coverage jump was common (77 of 426 questions, 18%); the meaning shift is what failed. In 56 of those 77 the story's earlier hours held too few headlines to know how far it usually moves, and in the 21 where it could be measured the jump's hour sat at about the usual distance (median ratio 0.9, against the 1.5 needed). A handful of headlines an hour moves about as much with wording as with news. After five tries, splitting is set aside for good; the day-by-day turns on story pages and the latest turn on the front page's card are the answer.

S23

Chanticleer

tried, not kept

❓ Can we split a long story by watching outlets swap articles in the same hour?

A story that runs for days can hide many turns under one moving headline. The Nvidia saga lasted five days and 787 headlines, from the affair admitted to the CEO's resignation, all as one story. Asking a model to spot new developments (our Sinbad experiments) topped out at 13 of 20 right. Chanticleer tried a simpler idea: let the outlets themselves signal a turn. When many outlets already covering a story put up a fresh article in the same hour, using words the story hasn't seen, that's a development. In the Nvidia case, 25 outlets did this the hour after the resignation, against a usual hum of 5 to 10.

The upshot Precise but rare: 3 of 3 judged splits right, but only 2% then 4% of stories split. Not shipped.
Keep reading ✏️

🎯 What we expected

We expected 15% to 45% of long stories to split at least once, no more than 5% into five parts, 4 or more of 5 known developments caught, and 15 or more of 20 splits judged real in a blind read.

🔬 What we found

The rule was precise but almost never fired. In round 1, a spike meant at least 8 outlets swapping, 2.5 times the baseline. Only 5 of 207 long stories split (2%). It caught 3 of 5 known developments with no false splits, and all 3 splits among already-judged questions were correct, but there were too few to draw a blind sample. Round 2 scaled the floor to 20% of outlets on the story. 8 of 207 stories split (4%), the same 3 of 5 known cases. Outside the biggest stories, outlets rarely post a second article at all: 318 of 422 windows peaked at 0 to 2 swaps in their busiest hour. The signal is real but belongs to the largest stories.

S22

Ariadne

in use on the site

❓ Can a local model place each saga into the larger affair it belongs to?

Some news is bigger than one running story. The US–Iran war produced 100 stories since October 2, spread across eight separate sagas and a pile of ungrouped ones. We wanted a way to gather those threads under one roof, an arc, without forcing the saga judge to merge stories it had rightly kept apart. Ariadne adds a step after sagas are formed: a local model reads each saga’s name and a handful of its story titles, looks at a list of open arcs with short notes, and places the saga in one arc or leaves it out. An earlier test had done the same move for Senate races and passed cleanly.

The upshot The arc tagger met its targets in both test months and began grouping live stories on November 12.
Keep reading ✏️

🎯 What we expected

We expected at least 90% of sagas placed correctly, at least 90% precision on sagas put into an arc, and at least 80% recall on sagas that belong in one. We also predicted the tagger would find the Houthi front, the UK base plots, and Hegseth’s impeachment.

🔬 What we found

It passed in both months. On the live November 2026 sagas, 123 of 133 were placed correctly, in the right arc or correctly left out, for 92.5%. Of the 40 sagas the tagger put into an arc, 36 belonged there (90.0% precision, exactly on the bar). It found 36 of the 42 sagas the key said should be in an arc (85.7% recall), and it caught all six of the US–Iran war’s known fronts. July 2024 was cleaner: 98.6% placed right, 96.3% precision, 100% recall. The four live false placements were the kind of error we expected: a SpaceX spectrum deal filed under Russia–Ukraine because the note mentions Starlink, a Cuba story under US–Iran because of the phrase “after Iran.” The six misses were all midterm threads that named a candidate or a state but not the election itself. Both the key and the second reader were Claude, so the 12-of-12 agreement on disputes tells us the model is consistent, not that the key is ground truth.

S21

Vasilisa

tried, not kept

❓ Can a recheck step shed lookalike incidents from sagas without breaking up the races?

Production never takes a part out of a saga once it joins. A recheck step already existed in the code but nothing called it: it asks the saga judge about each part against its mates, nearest first, and puts out any part it calls a different story from all of them. We tried it on top of Heimdall's reworded board, which kept Senate races together but had lost some precision to lookalike incidents, hoping to win the precision back without losing the races.

The upshot The recheck removed some lookalikes but broke apart race stories again, so the overall accuracy on hand-made keys dropped. 0.930 → 0.955
Keep reading ✏️

🎯 What we expected

Written before the run: on a random sample of stories read blind, the rechecked board's precision would come within 0.01 of production's, with recall no lower; on two hand-made keys its F would stay within 0.01 of the board's and 0.08 or more above production's; and on production's own sagas the recheck would change almost nothing.

🔬 What we found

The recheck put out 46 parts from 25 of the board's 109 sagas. On the random sample precision rose from 0.930 to 0.955, about 70% of the way back, with recall kept (0.791 against production's 0.775). But that still left it 0.011 under production, just past the 0.01 bar. And it gave back the races: Alaska from 90% of its stories together to 60%, Texas from 100% to 82%, so F on the keys fell 0.030 and the gain over production (+0.069) missed its 0.08 bar. Asked pair by pair, the judge called a race's debate, ad or poll separate stories: it can't tell a lookalike from a race's quieter part. On production's own sagas it was harmless but shed almost nothing (11 parts).

S20

Kairos

tried, not kept

❓ What if time were a second dimension of the clustering, penalizing headlines further apart in the day?

Every hour, our site groups the day’s headlines by how alike they are in meaning. Time only sets the window’s edge, so two headlines 23 hours apart are judged on wording alone. A first measure looked promising: among pairs our clustering had wrongly lumped, the median gap was 15 hours, against 4 hours for pairs from the same event. We tried subtracting a small penalty per hour apart from the likeness score, then tested clustering only the live headlines first and attaching the past ones after.

The upshot Adding a time penalty hurt clustering quality, and processing only current headlines first hurt even more, because the existing links within a single run were already correct.
Keep reading ✏️

🎯 What we expected

Written before the run: subtracting a small penalty per hour apart from the likeness would raise cluster F by 0.03 or more on ten tuning cases, gain 0.02 or more on five held-out cases, and lift precision by 0.10 on five cases known to be lumped, without losing more than 15% of coverage. A second round tested clustering the live headlines first and attaching the dropped-off ones after, with a +0.02 bar.

🔬 What we found

Every penalty made things worse, and the steeper the penalty, the worse. The gentlest (0.002 per hour) lost 0.040 F on the tuning cases and 0.048 on the held-out ones, and was worse on three of the five held-out cases. Our own prediction was wrong in direction. The reason: within one run the clusters are already precise (0.94) and short on recall (0.48), so a penalty that can only cut links sells recall to buy precision there is little need for. Round 2 did worse still (−0.067 F; recall fell from 0.483 to 0.396): the headlines that have dropped off the front pages are much of what holds a story’s waves of coverage together. The lumping readers notice is made later, as stories carry on from run to run and sagas join, not in one run’s clustering.

F31

the Shadow

answered

❓ Can we subtract topic from a claim's embedding and count how many tellings the claims fall into?

A claim is a topic told through a motif. The Shadow asked whether we can subtract the topic out of a claim's embedding (its numerical fingerprint of meaning) so that what remains groups by how it's told, not what it's about. We tried several ways of removing topic from about 6,900 claims, then counted how many groups the claims fall into on their own, with no motifs in sight. We also read the leftover groups that matched no known motif to see if they held shapes the index had missed.

The upshot Removing topic from claim embeddings never helped by more than 0.7 point, the number of motifs keeps growing past 2,000, and no fixed slot count exists.
Keep reading ✏️

🎯 What we expected

We expected a small gain at best from removing topic, then losses as more directions came out. We also expected no slot count to hold still, an intrinsic dimension of 10 to 40, a Chao1 estimate of about 1,930 motifs, and at least five leftover groups reading as one telling, three of them new.

🔬 What we found

Topic doesn't come off cleanly. Removing the top directions of variation helped only at the smallest setting on one embedder (+0.3 point) and lost up to 8 points beyond it. Subtracting each claim's nearest headlines lost up to 11 points, because the news says most of what a claim says. The best method, five coarse topic centres taken out, gained 0.7 point, mostly in the least topical genres (Perennials +3.5, Values +1.7) and almost nothing in Plots. No slot count holds still: every clustering's count follows the resolution asked and keeps rising as claims are added. Locally the claims lie on a surface of about 12 dimensions. Chao1, which guesses unseen kinds from how many are seen once or twice, puts the index at about 2,013 motifs if every claim were in, a floor that has risen every week. Claude read the 30 leftover groups blind: 3 told one shape and none was a shape the index lacked. A second model reader agreed on only 18 of 30, reading shared subjects as shapes.

S18

Heimdall

tried, not kept

❓ Can showing a model every open case at once keep running stories together better than a shortlist?

Sagas, our groups of related stories, are built by filing each new story into a running "case." Earlier methods showed the local model only a shortlist of the 5 or 6 most alike open cases, because a longer prompt doesn't fit on our graphics card, and Senate races kept falling apart into several sagas. We suspected the shortlist was hiding the right case, so Heimdall showed a long-context cloud model every case open in the last 14 days at once: the whole board.

The upshot Showing the model all open cases kept Senate race stories together but added little overall grouping accuracy, and the improvement didn't hold in a new month.
Keep reading ✏️

🎯 What we expected

Written before the run: on two hand-made answer keys, the board would beat the shortlist by 0.02 or more in B-cubed F (a score of how closely our groupings match the key, from 0 to 1) and beat production's sagas by 0.05; each of three Senate races would keep 60% or more of its stories in one saga; and precision would fall no more than 0.03 on the keys, or 0.05 on a random sample of stories read blind. Later rounds kept the same bars.

🔬 What we found

Round 1 (October 2026, 1,240 stories, about 14 cents) kept the races together (91%, 94% and 100%, against production's 27%, 44% and 30%) and lifted F on the keys from 0.752 to 0.895. But not for the reason we gave: only 4 of the board's 259 joins (2%) went to a case the shortlist would have hidden; seeing everything simply made the model open cases of its own more readily. It also lumped, naming cases for a whole field (one "Ukraine-Russia war" saga of 63 stories), and precision on the random sample fell 0.076. Round 2 asked for cases named for one event and a stricter join: precision came most of the way back (0.930, production 0.965) and the races held (100%, 94%, 90%), but the gain on one key shrank to +0.025 against a 0.05 bar. Round 3 ran it on July 2024, a month nobody had tuned on: +0.005 F, with production's own lumps carried along, so October's gain was October's. Not kept.

S17

Urashima Taro

answered

❓ Do the best saga-joining methods hold up on a month nobody tuned on?

Every saga-joining test so far was tuned on October 2026, with answer keys built from cases that already looked wrong, a setup that favors methods fixing what we saw. To break that cycle, we took the two best methods to July 2024, a presidential summer from the site’s archive that nobody had tuned on. We fixed seventeen running stories in advance from memory (the Butler shooting, Biden’s withdrawal, the CrowdStrike outage, the Paris Olympics, and more), plus look-alike pairs that must not merge, then replayed the site’s story and saga machinery over the month and ran both methods on top.

The upshot Both methods failed on untuned July data because the site’s own sagas already lumped major stories together, and the methods only added joins on top.
Keep reading ✏️

🎯 What we expected

Fixed before any July story was made: a method that fails here after winning in October doesn’t go into use. To pass, each had to beat the replayed site by 0.05 or more in B-cubed F, hold most cases together more often than the site, merge no look-alike pair, and keep precision on a random key within 0.05 of the site’s.

🔬 What we found

The replay made 2,079 July stories, and the key was read blind by Claude. Neither method survived: Heimdall’s board gained +0.005 F and the Scapegoat’s method +0.018, both inside the noise, against the 0.05 needed. July’s trouble was lumping, not splitting. The replayed site built one saga of 265 stories carrying the convention, Butler, and Vance together, and five sagas merged look-alike pairs (Butler with the convention, Biden’s exit with Harris’s campaign, the immunity ruling with the documents dismissal, among others). Both methods only add joins on top of the site’s sagas, so they inherited the lump. Outside the key they gained a little recall (+0.04 and +0.05) at no real cost in precision.

S16

the Scapegoat

tried, not kept

❓ Can a topic slot carry off the field so the case slot stays narrow?

Aegeus gave each story one label, but the biggest labels named whole fields like "Russia-Ukraine war," lumping dozens of running stories together. The Scapegoat tried giving each story two slots: a topic for the broad field and a case for the narrow running story, with only the case allowed to join sagas. A check asked whether each new case label named one case or a whole field, and a veto stopped one saga from chaining two cases together. We also scored precision on a new random key of 60 stories, read blind, to catch wrong joins the hand-picked keys miss.

The upshot The two-slot design sharply improved race detection but failed to separate fields from cases, so it was not shipped. 0.746 → 0.903
Keep reading ✏️

🎯 What we expected

At least 34 of 40 shared case labels would name one case, with at most 4 fields. F on the non-election key would rise by 0.05 or more, and on the races key it would reach 0.80. Precision would hold within 0.03 on the keys, and on the random key it would fall no more than 0.05 while recall rose.

🔬 What we found

The topic slot didn't carry the fields off. In the full run, 18 of 40 shared case labels were still fields, after a once-changed wording had brought them down only from 31% to 24% in the pilots. The field check did the work, bluntly: it threw out labels covering 317 stories, including a real race, and let a field through. The races rose (F from 0.746 to 0.903, precision kept, biggest saga no bigger than production's), but nothing else did (0.757 to 0.761 on the non-election key). On the random key, precision fell 0.052, a hair past the 0.05 bar, while recall rose 0.078. Taken to July 2024, a month nobody tuned on, it gained just +0.018 F, inside the noise, though it was the fairest method outside the key there.

S15

the Ugly Duckling

tried, not kept

❓ Can a different grouping method cut the chains that lump separate events into one story?

The site's story grouping sometimes chains separate events into one story because a single headline can bridge them. We tried four tighter ways to form groups: average likeness across a group, splitting groups from the inside, Louvain community detection, and keeping headlines only while they fit a group's centre. Each was tuned to cover the same share of headlines as today's method and replayed over a month of hourly runs to see if any could cut the chains without breaking the right stories apart.

The upshot Tighter grouping raised precision on the worst cases but tore apart correct stories just as much, leaving overall performance unchanged.
Keep reading ✏️

🎯 What we expected

The best rival would lift precision on the five worst-lumped cases by 0.10 or more, beat today's method by 0.03 F on ten tuning cases and by 0.02 on five held-out ones, and mix two events into one story at least 10 points less often.

🔬 What we found

No grouping method beat today's on F. The best rival was today's method with one carry-over setting changed, at +0.007 F (0.739 to 0.746), inside the noise. Tighter methods did raise precision on the five lumped cases (from 0.394 to as much as 0.731) but cut right stories apart just as readily, losing more than they gained. Average linkage scored 0.651 F overall against today's 0.746. In a blind read, 11 of 40 stories held two or more events under both methods, mostly one affair with a later turn like an arrest or a ruling.

R2

Chiron

tried, not kept

❓ Can a cloud model judge claim-motif pairs like a person, to teach our small reranker?

The finder's small reranker learns only from pairs a person has checked, but most pairs the shortlist shows are never judged. We asked whether a large cloud model, DeepSeek V4 Flash, could judge those unseen pairs like a person, giving the reranker ten times more to learn from. We tested it first on 558 decisions the person had already made, before spending anything on the rest, within a 50-cent budget.

The upshot The cloud model failed the entry test, so it was not used to label any unseen pairs.
Keep reading ✏️

🎯 What we expected

The teacher had to reach AUC 0.80, 75% agreement, and kappa 0.40 on the person's explicit decisions, all three, or its labels would not be asked. It was also expected to say yes to at least 80% of the person's motifs and at most 20% of shown-but-not-filed ones.

🔬 What we found

The gate failed on all three counts. AUC was 0.729, below the 0.80 threshold and the taught reranker's own 0.796. Agreement was 63.6% and kappa 0.316. The teacher was strict where the person was generous: it said yes to only 55% of the machine's filings the person kept, though it agreed on most removals (yes to 12%) and on motifs the person filed themselves (yes to 96%). It rarely said yes to shown-but-not-filed motifs (4.5%). Its labels would have taught the reranker to drop the very filings people keep, so nothing more was sent. The check cost 5 cents. We found the reranker hadn't been retaught since October 8, though about 10,700 decisions have been made since.

K1

the Doppelganger

answered

❓ Would a second reader agree with Claude’s blind reads?

Every precision number in our experiments rests on answer keys and blind reads made by one AI model, Claude. Nobody had checked them, and one reader’s habits become the key’s: a lenient reader makes every method look precise, a strict one makes every method look lumpy. So we sent the same items to a second reader, the open model DeepSeek V4 Pro, blind to Claude’s verdicts, to find where the reads are contestable.

The upshot A second AI reader matched Claude closely on mechanical tasks but disagreed on a third of story joins, yet none of Claude’s original verdicts were overturned.
Keep reading ✏️

🎯 What we expected

The second reader would agree on at least 85% of story joins and call no more than 10% of Claude’s fair joins unfair; back at least 2 of Claude’s 4 unfair joins and 1 of its 2 wrong tags; reach a kappa of 0.60 or more on how headlines group into events in at least four of five cases; and agree on 80% of show stances and 85% of story pairs.

🔬 What we found

Where the reads are mechanical the second reader agreed closely: its groupings of headlines into events matched Claude’s with a B-cubed F (a 0-to-1 score of how closely two groupings agree) of 0.88 to 0.98, four of five cases passing both bars, and it agreed on 92% of story pairs once 4 pairs broken by a bug were set aside. On story joins it agreed only 70%, calling 26 of Claude’s 86 fair joins (30%) unfair, though it backed 3 of the 4 unfair ones. On show stances it agreed 82.5% but with a kappa of 0.39, below the 0.50 asked. Claude re-read all 41 disagreements: its verdict stood on 32, 5 could go either way, 4 were the broken pairs, and none turned out to be the second reader’s, so no verdict changed. The second reader judged at a finer grain than the site’s rules: it rejected plain turns of one running story where the site’s sagas are loose by design, and missed the rule that a correspondent stating a fact as news is reporting it. The run cost $0.07.

C3

Mimir

tried, not kept

❓ Can a model shown all 1,384 motifs find what a shortlist of 12 hides?

Our filing model only sees a shortlist of 12 motifs, but our own logs show that 12.8% of the motifs a person later confirmed already existed when the claim was filed and just weren't shown. A motif that isn't shown can't be picked. Some models can now read all 1,384 live motifs at once, so we tried sending the whole index in one prompt to see if a model shown everything could find what the shortlist hid.

The upshot Giving a model all 1,384 motifs instead of a curated 12 made it miss 28.8% more correct ones and pick 18.8% more wrong ones.
Keep reading ✏️

🎯 What we expected

We expected the whole-index model to find at least 30% of the hidden motifs, at least 5 points more right motifs than the shortlist arm, and no more than 5 points more wrong picks. We also checked that the middle of the long list wasn't read worse.

🔬 What we found

It didn't. On 100 past claims where the shortlist hid the right motif, the whole-index model found only 5 of the 106 hidden motifs (5%). On 218 hand-confirmed claims, it found 25% of the right motifs that existed at the time, 28.8 points fewer than the same model on a shortlist of 12, and its wrong picks rose 18.8 points to 58%. Place in the list wasn't the problem: motifs in the first, middle, and last thirds were found 24%, 23%, and 24% of the time. Shown everything, the model reached for broader, nearer-sounding motifs, while the shortlist does real work narrowing the choice to motifs close enough to tell apart.

C2

the Titans

answered

❓ How much of the filing pick's error is model size?

Our filing picker, which chooses which motifs a news claim retells, gets about a quarter of its picks wrong. Every model we had tried was small enough to run on our own 10 GB graphics card. We wanted to know how much of that error is model size. So we rented seven models in the cloud, from the same Gemma 4 26B we use up to models of about a trillion parameters, and asked them exactly the same 240 questions, word for word. This was research only: filing stays on our own machine.

The upshot Larger models found more correct motifs but were no more precise, and nearly half of what looked like errors were actually defensible picks.
Keep reading ✏️

🎯 What we expected

Written before any model was asked. A bigger model would count as better if it made at least 5 points fewer wrong picks while finding no more than 3 points less, or found at least 5 points more of the right motifs with no more wrong picks, either way with the gain holding up in a resampling test. We also expected the cloud copy of Gemma to answer like ours, and a blind read to find the bigger model's wrong picks at least 10 points more defensible than Gemma's.

🔬 What we found

On the 218 claims that still have hand-confirmed motifs, the cloud Gemma matched our own on the scores (70% found, 30% wrong, within a point), though it changed some individual picks. GLM 5.3 found 7.6 points more of the right motifs (78% against 70%; resampled range +3.5 to +11.8) with no more wrong picks (28%). Gemma 4 31B, the nearest that might run here on a bigger card, found 5.3 points more at about the same wrong-pick rate. No model cut wrong picks by 5 points. DeepSeek V4 Pro and Kimi K2.6 found the most (81%) but paid with 41% and 36% wrong. A blind read by Claude of 30 such wrong picks from GLM and 30 from Gemma judged 13 of each (43%) defensible, so the bigger model's doubtful picks are no better, and an answer key built from confirming what Gemma proposed overstates every model's errors, about twice over. Size helps finding, not precision. The whole run cost $0.72.

S13

Aegeus

tried, not kept

❓ Can a model label a running story at its birth, so later parts are joined by the label alone?

Eurycleia showed that naming a race joins its parts, but the list of races was written by hand, so it could only ever help elections. Aegeus tried the general version: a model writes a short label for each story at its birth (a case, race, trial, investigation, outbreak or protest) and stories that share a label are joined on top of the site's existing sagas. Because the first answer key was mostly races, we built a second one from 14 running stories of the same month that had nothing to do with elections.

The upshot The model's labels joined stories well, but it often used broad field names like "Russia-Ukraine war," which created a single saga of 99 stories. 0.757 → 0.837
Keep reading ✏️

🎯 What we expected

Written before any story was labelled: on the new key, B-cubed F (how closely two groupings agree, 0 to 1) rises by 0.05 or more; on the races key it reaches 0.80; precision (the share of joins that are right) drops no more than 0.03 on either key, no look-alike cases merge, and no saga grows past 33 stories; and blind reads find the labels at the right grain and the joins fair.

🔬 What we found

In the first run most labels were the right grain, but the biggest were whole fields: "Russia-Ukraine war" (60 stories) and "2022 midterm elections" (42, the year copied from an example in the instructions). F rose on both keys (0.757 to 0.837 on the new one, 0.746 to 0.842 on the races), but precision fell 0.163 and 0.235 and one saga grew to 99 stories. Changing the wording once to name fields as off limits made it worse: 777 stories labelled instead of 413, fields the biggest labels, a 316-story saga, and F down to 0.756 and 0.789. Labels weren't put into use. Afterwards, a cap on how far one label may spread looked good on the keys, but a blind read of 30 of its joins outside them found 15 wrong. The lesson: labels at the right grain join well, field labels are the whole problem, and wording can't keep them out; the Scapegoat came next.

S12

Eurycleia

tried, not kept

❓ Which 2026 race is this story mainly about, if any?

Three tests had found the same gap: a long-running story's parts don't join because they don't look alike. Mnemosyne showed the gain came from naming the race, not from filing as such. So we kept the site's sagas and its pairwise judge as they were and added one question per story, with no catch-all to compete: which single 2026 race, from a fixed list of every Senate and governor's race, is this story mainly about, if any? Stories naming the same race join that race's saga.

The upshot Adding race tags improved story grouping and passed its test, but the site owner rejected it for being useful only for one election. 0.746 → 0.820
Keep reading ✏️

🎯 What we expected

Written before any story was tagged: B-cubed F (how closely two groupings agree, 0 to 1) rises by 0.05 or more, counting a rally held for a race's candidate as part of that race; Texas, Michigan and Alaska each have 60% or more of their stories in one saga; precision (the share of joins that are right) drops no more than 0.03, with no look-alike cases merged; Claude's blind read finds at least 90% of tags right; and joins outside the answer key are no less fair than the current method's.

🔬 What we found

Gemma tagged 65 of 1,240 stories with a race, across 16 races. F rose from 0.746 to 0.820 with precision unchanged at 0.984; Alaska went from 30% to 100% in one saga and Texas from 27% to 64%, but Michigan stayed at 54%, because its missed stories named a candidate and not the race. Claude's blind read found 36 of 38 tags right (95%) and every join it read fair (25 made by the tags, 30 by the current method). By its own rule the test passed and race tags were due to go live, with candidates' names added to the list. They didn't: the site owner turned them down as useful for one election season and nothing else, and later tests (Aegeus, the Scapegoat, Heimdall) tried to find a general form.

S11

Mnemosyne

tried, not kept

❓ What if stories were filed into named sagas instead of judged in pairs?

The site's saga joiner works in pairs: show two stories, ask if they're the same running story. The Prodigal Son experiment found this misses most parts of a long race because a poll story and a scandal story don't look alike on their face. Newsrooms don't work that way. A wire service gives every update the same slug. A newspaper morgue files each clipping into a named envelope. So we tried filing: each new story is shown a shortlist of open sagas and asked, as one multiple-choice question, which it belongs to, or none, which starts a new saga that gets a name and a one-line description as it grows.

The upshot Filing stories into named sagas improved grouping but created a giant, imprecise catch-all saga and missed the precision target. 0.763 → 0.812
Keep reading ✏️

🎯 What we expected

Filing would beat the current method by 0.05 or more on B-cubed F, hold each of three Senate races 60% or more in one saga, lose no more than 0.03 in precision, and never merge look-alike cases. Pre-opened races would add 10 points more. A blind read would find no more than 10 points more unfair joins.

🔬 What we found

Over 1,240 stories, filing lifted the B-cubed F score from 0.763 to 0.812, a gain of 0.049, just short of the 0.05 bar. It held races together where the pairwise judge couldn't: Michigan went from 54% of its stories in one saga to 92%, Alaska from 30% to 90%. Texas only moved from 27% to 36%. But precision fell 0.061, twice what we allowed. The biggest saga swallowed 53 campaign stories under the name "2024 US election developments" (the wrong year), because its one-line description kept widening as it grew. Pre-opening a saga for every 2026 Senate and governor's race added only 5 points and scored worse overall (F 0.769). A blind read of 60 joins outside the key found 2 unfair ones in each arm.

S10

the Prodigal Son

tried, not kept

❓ Does a reworded saga judge let a running story's parts come home?

The site groups stories into sagas by asking a language model whether two stories are one running story. On election night the vote results and the midterms saga stayed apart partly because the judge's instructions said separate events involving the same person are different stories. We tested a reworded judge that names an election, race or campaign as one running story, with its rallies, polls, debates, vote and results as its parts.

The upshot A reworded judge linked 47% of true pairs but created 11 extra false links, so the original instructions remained.
Keep reading ✏️

🎯 What we expected

We expected the current judge to join under 60% of true pairs, the election wording to join at least 15 points more, and wrong joins to rise by at most 2, with none among 6 look-alike pairs.

🔬 What we found

We judged 281 pairs from 15 hand-checked running stories. The current judge joined 49 of 127 true pairs (39%), well under 60%, so the first expectation held. The election wording joined 60 (47%), only 8.7 points more, and its wrong joins rose from 4 to 15, all tying the Michigan Senate race (14) or the Texas race (1) to Trump's Michigan rally. Long races were where both wordings failed: Texas 0 of 15 pairs under either, Alaska 0 then 1. The judge stayed as it was.

F30

Morgiana

tried, not kept

❓ Do the tellers' own carrying words help the shortlist find the right motif?

We tested whether the shortlist could use carrying words—the tellers' own phrases that carry a motif, like "weaponized"—to find the right motif more often. Ali Baba had tried to find such phrases by statistics and got mostly topic words. By November, real marked phrases existed, so we matched a claim's text against each motif's carrying words, holding out claims from the same episode or source so a claim couldn't match its own twin.

The upshot Using the tellers' own phrases to guide the shortlist barely moved the needle, adding at most 0.6 points even in a perfect scenario.
Keep reading ✏️

🎯 What we expected

Top-12 recall on filings in motifs with carrying words should rise by 1 point or more, with the 95% interval above zero. Moves should gain 2 points. At least 8% of hard misses should come back. The marked phrases should beat Ali Baba's machine-found ones by 1 point.

🔬 What we found

The first run with 1,375 phrases over 604 motifs gained +0.03 points at top 12, with a 95% interval of -0.05 to +0.14. Only 5 of 408 hard misses came back, and 5 others fell out. A rerun the same day after Claude filled in carrying words for 8,233 of 9,113 filings—about ten times as many phrases—gave the same +0.03 points, interval -0.11 to +0.16. Moves gained +0.27, short of the 2 points needed. Ali Baba's machine phrases did no worse. A bug check found the signal fires but mostly on names and topics, and even a perfect combination could add at most 0.6 points.

S7

Osiris

answered

❓ Are a saga's stories its events?

We built the first answer key for the whole story pipeline: fifteen running stories, 4,200 headlines, each assigned to the event it reports. Two sagas looked wrong in early November (one braided an old feud into its newest part, another cut three clear events into parts that weren't those events), so we scored the site as it stands and tried fixes against it. Claude drafted the key without seeing the site's own grouping, and it was approved as drafted.

The upshot The site grouped stories into events with 0.75 precision and 0.85 recall, but none of the attempted fixes improved the overall F score of 0.70.
Keep reading ✏️

🎯 What we expected

We expected a replay of the month's hourly runs would match production closely enough to test changes (within 0.05 on 12 of 15 cases). We guessed production's weak side would be splitting one event across several stories, not lumping, and that Gemma sorting headlines straight into events would beat the clustering by 0.10 on at least 11 cases.

🔬 What we found

Production scored F 0.76 (precision 0.75, recall 0.85): it lumps more than it splits, the opposite of our guess. The lumping sits mostly in five running stories whose developments share the same names (the Ramaswamy ad's precision was 0.25). The replay failed its fidelity test (close to production on only 6 of 15 cases), so its tests counted only as replay against replay. The stay-put rule didn't help, the rule added that day barely acted but stayed, and Gemma's event sort tied the replay at F 0.70 while cutting 140 events into 183 stories. We later found the replay fed each hour only live headlines, missing 21% of the keyed ones.

📅 October 2026

S6

Odysseus

tried, not kept

❓ Can a reworded question and a day’s minimum split stories where a person sees a new development?

Sinbad’s first round split long stories much too eagerly, calling retellings, new details and reactions new developments. Odysseus kept Sinbad’s replay and changed two things: a reworded question that defines a development as something newly happened (a new act or decision, the next stage of an event) and lists what counts as the same news (retellings, new details, reactions, analysis, condition updates), “when in doubt, the same event”; and a full day of coverage before a story may split.

The upshot Rewording the question fixed old mistakes but failed on new stories, so the split feature was not turned on.
Keep reading ✏️

🎯 What we expected

Written before the run: 15% to 40% of long stories would split, with at most 5% in five parts or more; at least 4 of 5 known developments caught; on Sinbad’s 30 judged questions, the old wrong splits mostly gone and the real ones kept; and on a fresh blind read, 15 or more of 20 splits real, with 3 or fewer of 10 “same news” answers hiding a development.

🔬 What we found

The reach was right (51 of 207 long stories split, 25%, only 1.4% into five parts or more) and it was perfect on the cases it was written from: none of Sinbad’s 12 wrong splits split again, 6 of the 7 real ones still did, and no control did. On fresh questions, though, only 12 of 20 splits were real (60%, against 15 needed), and it caught 3 of 5 known developments, again missing the Fort Hood livestream’s cancellation. The misses were the very kinds the prompt names: retellings, a hurricane strengthening, a reaction. It learned Sinbad’s cases, not past them, which told us wording alone had reached its limit. The split stayed off.

F29

Hestia

answered

❓ Does a fuller motif index make filing easier, or do more motifs just crowd the old claims?

The person who runs the site suspected that earlier experiments had failed partly because the motif index was too thin: with too few motifs there is nowhere right to file a claim. We measured how the centres of motifs (the average of their claims' embeddings, numerical fingerprints of meaning) behaved as the index grew from 409 motifs on October 8 to 1,027 on October 30, using only saved versions of the index and cached embeddings, no new model calls.

The upshot A sparse index was the real problem: 43% of later claims had no motif on October 8, dropping to 5% by October 27, while adding motifs did not crowd old claims.
Keep reading ✏️

🎯 What we expected

Written before anything was counted: on the same claims, today's index should put their motifs among the 12 nearest centres 3 points more often than mid-October's; the share of incoming claims whose motif already existed should rise from 60% or under to 80% or over; a claim's distance from its centre should flag filings a reviewer later removed; and centres close together should mark linked motifs but not motifs of the same genre.

🔬 What we found

On the same 875 claims, today's index ranked their motifs in the top 12 only 1.5 points better than the October 11 index, with the interval crossing zero: new motifs didn't crowd old claims, but didn't help them much either. Coverage told the story: 43% of claims that came after October 8 had no motif yet on that day's index, against 5% after October 27. Incoming claims whose motif did exist were found in the top 12 more often against later indexes, rising from 60% to 78%. Distance from the centre flagged removed filings at an AUC of about 0.70, though plain likeness to a claim's own centre did better at 0.75 to 0.78. Centres knew kinship well (linked motifs scored 0.94 to 0.96) and genre not at all, at 0.52, no better than chance.

F28

the Judgement of Paris

tried, not kept

❓ Which model picks the motifs from the shortlist?

Three shortlist experiments in a row had moved nothing, so we turned to the next lever: the model that picks which motifs a claim fits from the twelve the shortlist offers. Gemma 4 26B had held that job since October 5. Three newer models had done well as a yes-or-no judge of one claim against one motif but had never been asked to pick among twelve. We set them against Gemma on 600 claims, scored against confirmed filings.

The upshot Gemma stays; the pick is where right motifs are lost.
Keep reading ✏️

🎯 What we expected

A newer model would replace Gemma if it found about as many of the right motifs with at least 3 points fewer wrong picks, or 3 points more with no more wrong, at no more than twice Gemma's time a claim, and without losing much on the filings humans had added by hand.

🔬 What we found

The first run on October 22 failed: the three newer models are decision models that answer yes or no about one pair, and they refused every request to pick. Gemma alone gave a baseline: of the right motifs among the twelve it picked 53.6%, and 33.7% of its picks were wrong, at 3.4 seconds a claim. A second asking on October 27 had each decision model score the twelve one at a time and keep its top picks past a threshold. On the 502 claims all four answered, each found 8.7 to 9.1 points fewer than Gemma's 54.7% and made 10.4 to 11.0 points more wrong picks. At a plain 0.5 cut-off the best, Clef, found 6.4 points more, but with 19.6 points more wrong and about ten times Gemma's time.

O1

the Wolf in Sheep's Clothing

answered

❓ Do some outlets run a barbell: mostly straight news hiding a small, separate tail of propaganda?

We wanted to know whether some outlets run a “barbell”: a great bulk of plain, wire-style news that lends them credibility, beside a small, separate tail of propaganda. An earlier word-list probe couldn’t tell dramatic reporting from ideological framing, nor a true two-part outlet from one that simply runs a little hot all over. So Gemma read 27,013 front-page headlines from 76 outlets over a month, never told which outlet wrote each one, and scored how far each headline’s own wording pushes a line, from 0 (straight, as a wire would write it) to 4 (propaganda or fringe). A statistical model then asked, outlet by outlet, whether its hottest headlines form a separate group beside its straight ones, or are just the hot end of one bell slid toward heat.

The upshot No outlet hid a small propaganda tail inside mostly straight news; most leaned consistently, and none showed a true barbell pattern.
Keep reading ✏️

🎯 What we expected

Written before the model read a headline: the two wire services, AP and Reuters, would show no tail (a check that the reading isn’t flagging straight copy); One America News Network would be a barbell even on its news reports alone; Newsmax’s tail would turn out to be its opinion columns; the New York Times, The Hill and the openly partisan sites would not be barbells; and at least one outlet somewhere would be. Before the full run, Gemma also had to agree well enough with 120 headlines Claude (Anthropic’s AI model) had labelled blind beforehand.

🔬 What we found

Gemma cleared the agreement bar (a weighted kappa of 0.70 on slant, where 0 is chance agreement and 1 perfect; within one level of the hand label 96% of the time). No outlet was a barbell, on everything it ran or on its reports alone. Of the 76: 44 read as heat (the typical headline leans, by this model’s reading), 14 as one shifted mode, 8 as heat with an extra tail (hot all over, with more at the far end, such as The Gateway Pundit and The Daily Beast), and 10 can’t tell (a straight centre and a tail too small to tell from none). The controls stayed clean. One America News Network had just 3 of 254 headlines at the hot end (1.2%), too few to say either way; all five of Newsmax’s hottest headlines were opinion pieces, and on its reports alone it read as one shifted mode. The question got its answer: at this resolution, front pages show no barbells, though every verdict rests on one model’s reading of each headline, and the Sinclair-owned stations that prompted the idea weren’t yet followed by the site.

F27

Ali Baba

answered

❓ Can the exact words tellers use, found by weighted log-odds, improve the shortlist?

We tried giving the shortlist the exact words tellers use. For each motif we found the phrases its claims use more than other claims do, using a standard word-statistics method, with names left out. A claim’s score became how many of its motif’s phrases it uses and their combined weight. The idea was that embeddings miss stock phrases and get irony backwards, but the tellers’ own words might catch what travels across stories.

The upshot Using the tellers' own phrases barely changed the shortlist's accuracy, and the phrases were just the topic in single words.
Keep reading ✏️

🎯 What we expected

We expected top-12 recall to rise by 1 point or more, with moves gaining 2 points, at least 8% of hard misses returning, and the tellers’ text beating the claim sentence alone by half a point.

🔬 What we found

Every test failed. Top-12 recall moved from 87.5% to 87.6%, a gain of 0.1 points with an interval from -0.3. Moves fell 0.3 points. Only 8 of 159 hard misses came back, half the 8% we needed. The phrases were mostly the topic again in single words like “economy, prices, tariffs.” Stock phrases a teller would recognise were rare, and some were web boilerplate. With two to forty claims a motif there is too little text for the method, which was built for corpora thousands of times bigger.

F26

Tiresias

answered

❓ What point is the teller making, in one plain sentence?

After Indra’s Net, we read 159 hard misses and saw the shortlist was good at finding what a claim is about but bad at catching the move a teller makes. Arguments, Values and Exhortations were missed about twice as often as Plots and Theories. We asked Gemma to write one plain sentence for each claim: the point the teller means the listener to take away, not why they said it. We tried it two ways (from the claim alone, and from the claim with its sources) and added each as a signal to the shortlist.

The upshot Adding a plain sentence of the teller's point barely improved the shortlist, recovering only 9 of 159 hard misses.
Keep reading ✏️

🎯 What we expected

The point written from sources would raise top-12 recall by 1 point or more, with the interval above zero, and by 3 points for moves without costing stories more than half a point. At least 10% of hard misses would come back, and sources would beat the bare claim by half a point.

🔬 What we found

Every hypothesis failed. With sources, top-12 recall rose from 87.5% to 87.8% (+0.3 points, the 95% interval touching zero); the bare point added nothing (+0.0). Moves gained 0.5 points with sources, short of the 3 we needed, and lost 0.3 without. Only 9 of 159 hard misses came back (6%), three of them plain news; the rest climbed from around 45th to the 25–40 band, still outside the filing model’s view. The point carried almost nothing the shortlist’s other readings of the claim didn’t already have.

F25

Indra's Net

answered

❓ Can 20 varied versions of each motif help the shortlist catch a claim's shape instead of its cast?

The shortlist that picks which motifs to show the filing model can mistake a claim's cast for its plot. A fox tale draws another fox tale, not a story with the same shape. We had Gemma write 20 one-sentence versions of each motif from its name and note alone, varying the setting, cast and voice. A claim is then compared with all of them, not just the motif's own claims, to see if the wider net catches the shape.

The upshot Generating varied motif versions barely improved recall on filings (from 87.5% to 87.8%) and helped folktales only a little (from 79.9% to 81.8%). 87.5% → 87.8%
Keep reading ✏️

🎯 What we expected

The versions should raise top-12 recall on confirmed filings by 2 points or more, with the 95% interval above zero. Motifs of five claims or fewer should gain 4 points while large motifs lose no more than 1. At least 10% of hard misses should come back. On folktales, versions should add 5 points.

🔬 What we found

The versions did carry each motif's shape: for 66% of filings the claim was closer to its own motif's versions than to 90% of other motifs'. But the shortlist already knew that. Top-12 recall moved from 87.5% to 87.8% (+0.3, interval +0.0 to +0.7). Small motifs gained only 0.3 points, not 4. Of 159 hard misses, 12 came back (8%) while 7 new ones fell out. On folktales the versions added 1.9 points (79.9% to 81.8%) and alone did worse (66.7%). A close reading of the misses found most were a claim's third, fourth or fifth motif, or the rhetorical move a teller makes rather than the topic.

F24

Baron Munchausen

answered

❓ Does a report-or-telling signal and discounting bulk passes improve the confidence model's calls?

The confidence model scores how likely a filing is to be kept, and claims above its sure line get only a light check. On Oct 18, 12 of 82 sure claims needed attention, many of them plain news reports filed in an opinion motif just because the subject matched. We tested two fixes: a signal for whether a claim is a report or a telling, and counting for less the filings once passed in bulk (230 at a click), which might have taught the model to trust itself too much.

The upshot Adding report signals did not shift any of the 12 problem claims, and a blind gate later caught 10 of 41 claims needing attention while keeping 81% of clean ones.
Keep reading ✏️

🎯 What we expected

Either report signal should raise AUC by 0.005 or more, and counting bulk passes at 0.3 should lift the sure line's precision to 95% or more. To ship, a variant had to move at least 5 of the 12 problem claims out of the sure list while keeping at least 85% of the 70 clean ones in.

🔬 What we found

Nothing moved. Across eight variants the accuracy score stayed within 0.002 of today's 0.870, not one of the 12 problem claims left the sure list, and counting bulk passes for less changed precision on deliberate decisions only from 94.8% to at most 95.2%. The signals weren't blind (Nimble rated the problem claims far more report-like than the clean ones), but the model learns from past decisions, and in those, reports had been kept nearly as often as tellings (68% against 74 to 75%): a signal can't teach a standard the labels don't hold. A follow-up tested a gate (keep anything Nimble calls a report out of the sure list) blind on Oct 19 and 20: it caught 10 of 41 claims that needed a human (24%, 40% asked) and kept 81% of clean ones in (85% asked), because most problem claims were tellings in the wrong motif, not reports. Neither went into production; the lesson was to take reports out of their motifs outright, so the model can learn from those decisions.

F22

Psyche

tried, not kept

❓ Does a claim's motifs fall on different posts, and can we split it there?

Social media posts that say roughly the same thing get grouped together, but a group often stitches several distinct tellings into one claim, which then gets filed under many motifs. We built a claim organizer, a second pass where Gemma reads a claim's posts alongside its motifs and says which posts tell which. If the motifs land on different posts, it drafts a split into child claims, one per telling, as a proposal for a person to approve or reword.

The upshot The organizer split about a quarter of claims regardless of filing, and its split proposals were never shown to a person for approval.
Keep reading ✏️

🎯 What we expected

Shown the human-chosen motifs of claims filed under three or more, the organizer should find them on different posts at least half the time. On the filing model's own picks it should split those many-motif claims at least 20 points more often than one-motif claims. Children should land in fewer motifs than their parent without losing the right ones.

🔬 What we found

The organizer split about a quarter of claims no matter what they were: 28% of claims filed under three or more motifs and 25% of one-motif claims, a gap of 2.2 points, well under the 10 needed to say it followed a human reading. Shown the human-chosen motifs it split only 35% of the many-motif claims, mostly reading the motifs as told by the same posts. Where it did split, on 28 claims, the children looked good on paper: each landed in 0.62 fewer motifs than its parent, and together they reached more of the right motifs, 3.54 against 2.61, though the scoring model had been trained on these same claims, which flatters them. Splits were best predicted by the post group itself, many authors and varied wording, which points upstream to how claims are written for groups. On 203 recent claims it drafted 45 splits, 22%, but they went to the lab's copy of the data and never reached a person, so whether people would approve them was never measured.

F20

the Blind Men and the Elephant

tried, not kept

❓ Does a claim writer shown the most varied posts cover more of a group?

When Gemma writes a claim for a group of Bluesky posts, it sees only ten posts picked at random, so in large groups it misses most of the story. We tried picking the ten posts most unlike each other instead, hoping that if a group held several sides, they would all appear in the claim. Before the run, we added a third method: the same varied ten, but Gemma could write one to four claims, each in one side's own voice.

The upshot Showing the writer the most different posts did not improve coverage, but letting it write multiple claims covered 79% of posts.
Keep reading ✏️

🎯 What we expected

We expected the varied claim to be about at least 8 points more of the group's posts, reach more of the person's motifs, and be filed in more motifs and run longer. We also thought a varied single claim might pass the half-about check less often, while the split method would cover the most with the least stitching, writing 1.5 to 2.5 claims a group.

🔬 What we found

Picking the most varied posts changed nothing. On 51 groups, the varied and random claims were each about 65% of the group's posts, and both reached the same 2.10 of the person's motifs. One sentence holds one telling, whichever ten posts it is written from. The split method covered 79% of posts, a gain of 14.4 points, but it wrote 3.6 claims a group, close to its limit of four whether a group was mixed or not. It still wrote claims for groups that shared none, and more than doubled the motifs a person would have to check, from about 6 to 13.8 a group. A model-server outage lost 1,085 of 6,832 answers, but a recount on the 42 untouched groups told the same story.

F9

Isis

answered

❓ Can single posts filed on their own recover the motifs found under a group's one claim?

We reversed the usual order: instead of writing one claim for a group of similar posts and filing that, we had Gemma write a claim for each post on its own and filed them one by one. The motifs then did the grouping. The Chimera had found that group claims often stitch several different tellings together, and we wanted to see if single posts were solid enough to file and could recover the motifs the person found under the group's one claim.

The upshot Filing posts one by one recovered most of the person's motifs, and nearly all posts found a home, but the motifs layered within single posts far more often than they spread across them.
Keep reading ✏️

🎯 What we expected

We expected posts from a group filed in three or more motifs to go to more distinct motifs than those from a one-motif group, about 3 against 2. We guessed a quarter to a third of posts would claim nothing. We expected the posts to reach at least 60% of the group claim's motifs, with at least half landing in one of them, and at least a third of the motifs reached to gather posts from two or more groups. An amendment added a bet that stitched groups would outnumber layered ones.

🔬 What we found

Across 159 groups and 2,000 posts, posts from groups the person filed in three or more motifs went to 3.83 distinct motifs, against 2.81 for one-motif groups, the gap we expected. The posts reached 79.5% of the group claim's motifs, and 74.5% of filed posts landed in one of them. 109 of 278 motifs gathered posts from two or more groups. Only 0.7% of posts claimed nothing, far below our guess of a quarter to a third. The amendment's bet failed: where a group's posts reached two or more of the person's motifs, a single post carried several of them 75 times, against only 3 where the motifs were spread across different posts. The many motifs look more like layers of one telling than stitched-together tellings. The fit sorts posts only weakly: raising the line from 0.3 to 0.7 left 22.6 points more posts unfiled for a 4-point gain in how many landed in the right motifs.

F7

Procrustes

answered

❓ Can the machine tell when nothing fits and propose instead of forcing?

The filing model often forces a claim into the nearest motif when the right one doesn't exist yet, and a person has to undo it. We tested whether the confidence model, which scores how likely a filing is to stick, can spot these misfits ahead of time. If every motif on a claim's shortlist gets a low score, the machine could send that claim to the new-motif proposals instead of forcing a bad fit.

The upshot A low confidence score flags claims that don't fit any existing motif, catching half of them while misrouting only 9% of good fits.
Keep reading ✏️

🎯 What we expected

We expected the best shortlist fit to separate nothing-fit from fits at AUC about 0.75, with a line catching half of nothing-fit claims sending at most 10% of fits to the proposals. We thought wrong-motif claims would score higher than nothing-fit ones, and the margin over the second-best fit would add little.

🔬 What we found

On held-out data with 953 fits, 108 nothing-fit claims, and 80 wrong-motif claims, the best shortlist fit spotted nothing-fit claims at AUC 0.84, better than the 0.75 we expected. A line at 0.695 caught half of them while sending 9.2% of fitting claims to the proposals, just under the 10% cap. Wrong-motif claims looked nearly identical to nothing-fit ones, with median best fits of 0.73 against 0.70, so the line can't tell them apart. About half the claims caught under 0.695, 88 of 180, did fit a motif. The gap between best and second fit was near chance at AUC 0.56. Plain claims with no story stood far apart at AUC 0.98, but only 23 were scored.

A3

Echo

answered

❓ Viral Texts: do reprints of one piece file alike?

A filing method should be consistent: the same text, filed twice, should land on the same motifs. Nineteenth-century newspapers offer a natural test, because popular pieces were reprinted from paper to paper, each copy with its own typesetting, cuts, editors’ changes and OCR errors. We took 60 political pieces from the Viral Texts project’s reprint clusters (American newspapers, 1836 to 1861), four copies of each from different papers, and filed every copy on its own.

The upshot When the same text is reprinted with errors and edits, our filing method still groups the copies together, matching a shared motif 59% of the time.
Keep reading ✏️

🎯 What we expected

Written before anything was filed. Copies of one piece should share their motifs far more than copies of different pieces (by Jaccard overlap, the share of motifs two sets have in common: about 0.35 within against under 0.05 between); at least half the pieces should have one motif common to every copy; differences should come mostly from different claims being pulled from each copy rather than one claim filed two ways; and our Golem method, building motifs from scratch, should keep a piece’s copies together at least as often as our index.

🔬 What we found

231 of the 240 copies got filings. Copies of one piece overlapped 0.32 on average against 0.03 for copies of different pieces, about ten times chance, and in 59% of pieces one motif was common to every copy that got filings: both main hypotheses held. Why copies differ when they do (different claims, or the same claim filed differently) wasn’t measured, so that question is still open. The Golem, building 23 motifs from scratch out of 600 of the claims, kept a piece’s copies under one shared motif more often than our index did (44% against 35% of pieces), but its broader motifs also make that likelier by chance (6% against 1%). The answer: our filing holds steady across noisy copies of the same text.

A2

Rip Van Winkle

answered

❓ Does our filing hold up on 1896 newspapers, and do the controls turn up?

We ran our full pipeline on 520 political articles from American newspapers of 1896, 130 years off and through OCR noise, to see if it holds up on text it was never built for. Control topics came first: the age's famous money-conspiracy talk, the Crime of '73, the Cross of Gold, and the Rothschilds and British gold. A method should find what we know is there before we trust what else it finds.

The upshot Nearly all claims found a home, but the system read known conspiracy talk as class conflict and only 60% of checked filings looked correct.
Keep reading ✏️

🎯 What we expected

Most claims would find a motif, 70% or more, but fewer would pass as sure than today's do, under 40% against about 69%. Over half the control claims would land in our money and banker-conspiracy motifs. 80% or more of a spot-check of sure filings would be right. New motifs would take 15% to 30% of claims. Building motifs from scratch on the same claims would lump them into a few broad ones that overlap our index's filing only loosely.

🔬 What we found

From 2,194 kept claims, 96% found a motif in today's index and 40% passed as sure, right at the line we drew. The controls failed as worded: the silver and gold talk went mostly to Class conflict, 46 claims for the Crime of '73 alone, not to our conspiracy motifs. The filing heard the many against the few, not a hidden hand. New motifs held only 5% of filings, 24 motifs holding 122 claims, though one is a shape we predicted: Diluting the standard, the argument that the currency is being debased. A spot-check of 126 sure filings found 60% a fit, against 80% hoped for. The misses were near neighbours and plain reports filed to broad motifs, not OCR damage. The Golem made 23 motifs from 600 claims, overlapping our filing loosely: B-cubed precision 0.10, against 0.06 to 0.07 by chance. A later correction found the claims counted as told in several papers were mostly one topic, the money question, not true reprints, so the finding that those found surer homes does not show what it first seemed to.

S5

Sinbad

answered

❓ Can a model split a long-running news story into its developments by reading the headlines?

A long-running news story takes in each new development under a title that moves with it: the Fort Hood execution story ran for ten days, ending with the livestream's cancellation, as one story. Sinbad tested a split built into the pipeline but switched off: whenever a long story is due a new title, Gemma is asked whether its newest headlines report a new development or the same news, and a new development starts a new part of the saga. It was replayed over the 207 stories that ran 36 hours or more between Oct 2 and 15.

The upshot Three rounds of prompt changes never met the original bars: the best round caught 13 of 20 real splits but still mistook reactions and new details for developments, so the feature stayed off.
Keep reading ✏️

🎯 What we expected

Written before the run: 20% to 45% of long stories would split, with no more than 5% in five parts or more; at least 4 of 5 known developments in two long stories would begin a part, with at most 2 stray splits; of 20 proposed splits read blind, 15 or more would be real; and of 10 "same news" answers, 3 or fewer would hide a development. Later rounds kept these bars.

🔬 What we found

Round 1 split far too much: 54% of long stories, 10.6% into five parts or more. It caught 4 of the 5 known developments but missed the Fort Hood cancellation, the very case that started it, and in a blind read (by Claude, Anthropic's AI model, Oct 30) only 7 of 19 splits were real, the rest retellings, new details and reactions; none of 10 controls was wrong. Round 2 (Nov 11-12) reworded what counts as a development and required a day of coverage: the reach came right (29%) and all 12 old errors were fixed, but only 12 of 20 new splits were real and 2 of 5 known developments were caught. Round 3 added judged examples and a rule that the newest headlines carry a new word: 13 of 20 real, the misses all reactions and new details, a new speaker read as a new event. It stays off, with no fourth round of wording; the next tries used other evidence (Chanticleer, Melampus).

F23

Ptah

answered

❓ Does writing the scope note first give better names and fewer abstract labels?

We replayed 141 past naming decisions through Gemma 4 26B, once with the current order (name first, then scope note) and once with the note written first. The idea came from a question: if the model describes the story before naming it, would its names and notes land closer to the ones a human settled on, and would it reach for fewer abstract labels?

The upshot Writing the note first did not improve name quality, as the small likeness gain was offset by a 7-point rise in abstract labels.
Keep reading ✏️

🎯 What we expected

We expected note-first names to be closer in meaning to the human's (likeness up by 0.02 or more, interval above zero), near matches to rise by 3 points or more, and abstract labels to fall by at least 5 points. A named risk: if abstract labels rose by 5 points or more, the name had become the note squashed into a label.

🔬 What we found

Writing the note first barely moved anything. Name likeness to the human's rose 0.008, with the 95% interval straddling zero (-0.008 to +0.025). Notes came no closer (-0.002). Near matches did rise from 3% to 7% (+3.4 points), but the risk we named before the run came true: read blind by Gemma against the naming rules, note-first names were abstract labels 41% of the time, up from 34%. Asking the filing model for a note before its name did no better (+0.003 likeness) and gave more abstract names by a word-ending count (13% to 20%). The bigger gap was one of style: the human's own names were read as plain topic labels 44% of the time, where Gemma reaches for story names.

F21

Charybdis

answered

❓ Does Straight news work as a sink for claims that fit no motif?

When a claim arrives, the filing model picks the best motif from a shortlist. When nothing truly fits, it often grabs the nearest wrong one anyway. We had kept a catch-all motif called Straight news for plain reports that tell no story, and we wanted to know whether it earns its place as a sink: offered beside the wrong motifs, would the model drop nothing-fit claims into it instead of misfiling them?

The upshot Adding a Straight news option did not reduce misfiling, as wrong picks barely changed, and it only helped with plain reports.
Keep reading ✏️

🎯 What we expected

Written before the run: offered Straight news, the share of nothing-fit claims put into a wrong motif should fall by at least 10 points, while claims that do fit a motif lose almost nothing (at most 5% swallowed, at most 3 points fewer picks of their own motif).

🔬 What we found

It didn’t work as a sink. On 107 claims where nothing in the index fit, Gemma put 92% into a wrong motif without Straight news on offer; with it, wrong picks moved by +0.9 points (95% interval –2.8 to +4.7), where a fall of 10 was needed. It did take plain reports (picked alone for 48% of 23), but only 2% of claims that called for a brand-new motif. The cost was small (1% of well-fitting claims swallowed, their own picks unchanged), but there was little gain to pay for. Straight news almost never reached the shortlist on its own (0 to 4%).

F6

Hermagoras

answered

❓ Can a telling's rhetoric find its motifs better than its topic can?

We wanted to see if a story's rhetoric could group it with its kin better than its topic could. Hermagoras asked Gemma to label each claim with four things: its stasis (what is in dispute), its driver (what moves the story), its time shape, and its speech act. The idea was that two tellings of the same motif would share these rhetorical bones even when their subjects differ, giving us a way to spot motifs that word likeness misses.

The upshot A story's rhetoric grouped same-motif claims together, and claims later removed from a motif stood out as different.
Keep reading ✏️

🎯 What we expected

We thought two claims of one motif would share a stasis and a speech act more often than two claims of different motifs, by 10 points or more. The driver and time shape would vary more within a motif. A claim carved out of a motif would differ from its motif's commonest label more often than claims in general do.

🔬 What we found

It worked. Two claims of one motif shared a speech act 56% of the time, against 38% for random pairs, an 18-point gap. Stasis agreed 44% against 31%, a 13-point gap. The driver gap was smaller at 7 points, as we guessed, but the time shape gap was 12 points, which we didn't expect. Claims later carved out of a motif were the odd one out on all four labels: their speech act differed from the motif's commonest label 40% of the time, against 32% for claims in general.

F4

Aesop

answered

❓ Can a proverb find a story's motifs better than its own words?

The shortlist finds a claim's motifs by comparing its own words and generic restatements that still keep the topic. We asked whether stripping the topic away entirely would work better. Aesop asked a language model for the proverb, idiom or fable a telling is an instance of, folklore's own way of abstracting a situation. We tested whether that topic-free version finds a claim's motifs better, alone or added to the plain claim, on 2,078 confirmed filings.

The upshot Using a proverb instead of the original claim dropped the success rate from 71% to 20%, and adding it to the claim hurt performance by 17 points.
Keep reading ✏️

🎯 What we expected

The proverb alone would find the right motif less often than the plain claim, since it drops specifics, but the plain claim plus the proverb would beat the plain claim by 2 points or more in the top 12. It should also pick out the carved-out claim more often than chance (over 20%).

🔬 What we found

The proverb alone put the right motif in the top 12 for 20% of filings against 71% for the plain claim, as expected. But summed with the plain claim it cost 17 points (54%) instead of gaining 2. It did a little better than chance at spotting the claim a reviewer later carved out of a motif (21% against chance 15% and the plain claim's 17%). The test used a simple stand-in for the real shortlist, and an equal-weight sum lets a noisy view outvote a good one. None of the abstractions earned a place.

F19

Theseus

answered

❓ Can a claim rewritten as its course of action alone find its motif better?

A 2026 shared task on telling stories apart by structure, not topic, found that the best systems rewrote a story as its course of action alone, then embedded that with an ordinary model. We tried the recipe on our own claims: Gemma distilled each into what happens and how it turns out, using kinds of actors and no names, places, dates or subject words. An embedding model turned that into numbers, which we compared against our other abstractions (Aesop, the claim as a proverb, and Loki, the claim with subjects swapped for kinds).

The upshot Rewriting a claim as just its course of action hurt performance, dropping accuracy by 5 points when combined with the original text.
Keep reading ✏️

🎯 What we expected

Alone, Theseus would find the right motif less often than the plain claim but more often than Loki. Summed with the plain claim, it would gain at least 1 point at top 12, at least as much as Aesop, and most on Plots. On the seams, it would spot the carved claim more often than the plain space.

🔬 What we found

On 2,078 confirmed filings, Theseus alone found the right motif in the top 12 for 38% of claims, against 71% for the plain claim and 41% for Loki. Summed with the plain claim, it reached 66% at top 12 (a loss of 5 points, not a gain). On the Gordian Knot’s seams, it picked the carved claim as the odd one out 15% of the time, the same as chance and below the plain claim’s 17%. A claim is a sentence or two with little course of action to distil, and our existing shape rewrite already covers most of what this adds.

F17

Castor and Pollux

in use on the site

❓ Do the learned weights put the right motif among the 12 candidates more often than the plain weights?

The site's checker suggests other motifs when a claim is filed weakly, but those suggestions were still picked by an older ranking that used six basic likeness measures. Filing itself had already switched to a learned ranking that uses every finder and holds the right motif in its top 12 more often. We made the suggestions share that same learned ranking, so the checker and the filing read from one formula.

The upshot The person's hand-added motif in the top 12 rose from 62.9% to 68.4%, a gain of 5.5 points. 62.9% → 68.4%
Keep reading ✏️

🎯 What we expected

We expected the learned ranking to hold the person's hand-added motif in its top 12 at least 8 points more often than the plain one, that 4 or more of the 12 candidates would change per claim, and that more weak claims would get a suggestion.

🔬 What we found

The learned ranking put the person's hand-added motif in its top 12 candidates 68.4% of the time, up from 62.9% for the plain ranking, a gain of 5.5 points with a bootstrap interval of +3.0 to +9.0. That is clear of chance but short of the 8 points we expected. At top 8 the gain was 3.9 points, at top 40 it was 4.3 points. On average, 4.7 of a claim's 12 candidates changed between the two rankings.

F16

the Magic Mirror

in use on the site

❓ Is the taught reranker better than the untaught one, on old and new decisions?

We taught a small reranker model the decisions of the person who runs the site, hoping it would better judge whether a claim fits a motif. An earlier test, Pygmalion, saw a small gain, but a later check, Goliath, found the taught version doing slightly worse in use. The weekly teaching always installed the taught model without checking. We changed the process so each teaching now also scores the untaught reranker on the same decisions and keeps the taught one only if it wins, then measured both.

The upshot The taught reranker scored lower on the teaching's own decisions and tied on later ones, so the process now keeps whichever model wins.
Keep reading ✏️

🎯 What we expected

We expected the taught reranker to beat the untaught one by 0.02 to 0.03 of AUC on the teaching's own decisions, but to do 0.01 to 0.03 worse on claims decided after the teaching. We also thought the new check inside a teaching would keep the taught one anyway, making it the wrong test.

🔬 What we found

Neither hypothesis held. On 739 decisions from the Oct 8 teaching, the untaught reranker scored 0.776 against the taught one's 0.758, a difference of -0.018. On 838 decisions about claims decided after the teaching, they were level at 0.738 against 0.739. The teaching is not hurting, but nothing shows it helping. The bootstrap intervals, about plus or minus 0.04, are wider than any difference we have seen. On a few hundred claims, two rerankers cannot be told apart at the second decimal.

F15

Thumbelina

answered

❓ How does held-out recall vary with motif size, and how much does the alike floor remove?

The shortlist’s headline score, 86.5% of the right motifs in its top 12, tested on claims it hadn’t learned from, is one number over the whole index. It might be carried by big motifs while small ones, with little to go on, are served badly. An ‘alike floor’ also hides any motif whose plain likeness to the claim is under 0.45 before the learned weights see it. Thumbelina broke the score down by motif size and counted what the floor removes. While designing it we spotted a likely leak: holding a claim out of a one-claim motif leaves that motif empty, which no other motif ever is, so the test can recognize the answer.

The upshot A test leak gave one-claim motifs a perfect score, and fixing it dropped the honest top-12 recall from 86.7% to 72.7% while the alike floor removed only 0.7% of filings. 86.7% → 72.7%
Keep reading ✏️

🎯 What we expected

Written before the run: recall at 12 would rise with size, from roughly 60% for two-claim motifs to above 95% for motifs of 11 or more; one-claim motifs would score near-perfectly because of the leak; with the leak removed the overall figure would fall 8 to 15 points while motifs of two or more claims stayed within a point; and 2-5% of the person’s confirmed filings would fall under the floor.

🔬 What we found

The leak was real: all 251 filings into one-claim motifs were ‘found’ in the top 12. Without it the honest whole-index figure is 72.7%, not 86.7% (on 1,761 confirmed filings), while motifs of two or more claims score 84.8%, as well as before. Recall rises in a step rather than a slope: 68% for two-claim motifs, about 85% from three to ten claims, 93% from eleven. The floor catches 5.3% of filings, but outside one-claim motifs the weights would have put only 11 of them in the top 12 (0.7%), so the floor stays. On Oct 12 both of the shortlist’s trainings were changed to leave the leak out and report the honest figure (the score on the dashboard stepped down by design), and two-claim motifs are where a finder that reads names and notes would pay most.

F14

Tyche

answered

❓ How much do shortlist recall figures move if the claims scored had been a different draw?

Several shortlist improvements from early October were kept or dropped on gains of one to three points, from single runs on a few hundred claims, with no idea how much such a figure moves by chance. Tyche added bootstrap intervals to our test bench, re-drawing the scored claims at random 200 times to see how far a figure wanders, and re-read four decisions: Lethe (signals from the day's news), Matryoshka (a claim's layers), the Pleiades (not kept) and Babel (a pair of newer embedding models, which turn text into numbers that can be compared).

The upshot Babel's gain of 2.9 points held clearly above chance, while Matryoshka's earlier gain shrank to 0.2 points once Babel was included, and the Pleiades' gain vanished under repeated train-test splits.
Keep reading ✏️

🎯 What we expected

Written before the run: one method's 95% interval would be about ±3 points at top 12 on Oct 8's 428 claims; read as a paired difference, the gains of the Pleiades, Lethe and Matryoshka would include zero, while Babel's +2.9 would not; and on Oct 10's larger set, taking each kept signal back out would cost about what it had gained.

🔬 What we found

A single method's interval was about ±2.5 points on Oct 8 (82.3%, 79.6 to 84.5) and ±1.7 on Oct 10. Paired, the intervals were narrower than we guessed: all four first-run gains cleared zero, Babel's by far (+2.9, +1.7 to +4.2), the Pleiades' only just (+0.6, +0.2 to +1.5). But when we re-dealt which claims trained and which tested five other ways, the Pleiades' gain ranged from -0.2 to +0.5, inside that noise, so it was rightly not kept, while Matryoshka's held every time (+1.2 to +1.9). The bigger finding: with Babel's embedders in, Matryoshka now adds only 0.2 points (inside the noise) and Lethe 0.6 at top 12 and nothing at top 8, so the new embedders seem to carry much of what those signals added. They stay (they're cheap and leak nothing), but since then every test result prints both kinds of interval, and the finders should be re-weighed together now and then.

F11

the Danaids

in use on the site

❓ How much does the confidence model's leak flatter its held-out AUC and sure line?

The site's confidence model scores every filing to set a 'sure line' above which filings pass without a check. A review found one of its 29 features, how many motifs a claim sits in, was counted from the live index after removals. The model was partly learning the answer from the answer. We ran an experiment called the Danaids to measure how much that leak flattered the model's held-out performance.

The upshot A data leak made the model look slightly better than it was, and the fix on October 14 confirmed the real performance drop was small.
Keep reading ✏️

🎯 What we expected

We expected the honest count would drop the held-out AUC by 0.005 to 0.02, not more, and dropping the feature would cost about the same. The sure line would move at most one step (0.05) with its precision staying at 95% or above. A fall of more than0.03 would have meant re-reading every model comparison since Oct8.

🔬 What we found

The leak was real and small. On the 1,032 decisions made after the index's first saved version, the AUC was 0.873 as in production and 0.861 counted honestly, a fall of 0.012, inside the expected band. Dropping the feature altogether cost 0.019. The sure line stayed at 0.85, its precision at 95.5% or more, with about 2 points fewer kept filings passing as sure. The honest count still carries some signal: a claim that fits no other motif is a weaker filing (2% of kept filings' claims against 11% of removed ones'). We fixed it in production on Oct 14 with a simpler stand-in for the honest count, since reading old versions of the index was too slow for a daily retrain.

F10

Pandora

in use on the site

❓ Do the person’s biggest hand-built motifs hold distinct tellings worth a motif of their own?

The person who runs the site had built 36 motifs by hand, each holding 10 or more claims. An earlier test showed that a reader could see distinct tellings inside the machine’s own broad motifs, but the hand-built ones are already fine-grained, with notes and must-have elements. We asked Gemma to open those 36 and name the tellings inside, to see whether they hold shapes worth a motif of their own.

The upshot Gemma broke every motif into smaller pieces, but almost no one has checked whether those pieces are actually meaningful.
Keep reading ✏️

🎯 What we expected

We expected Gemma to keep about a third of the motifs whole and split the rest into 2 to 4 tellings, with broad-category motifs splitting more often. At least 40% of the tellings would be judged worth a motif. After Gemma split everything, we added a control: random groups of claims, named the same way, which Gemma’s tellings had to beat by 20 points.

🔬 What we found

Gemma split all 36 motifs into 227 tellings, keeping none whole. The first two hypotheses failed: Gemma splits when asked to split. 91 tellings (40%) held a single claim. Only 4 tellings (2%) repeated the name of a motif already resting on theirs. Just 5 verdicts were recorded, all from one motif and all yes, and none of the 24 controls or the blind set of 20 has been judged. The question of whether Gemma finds real seams, or any well-named handful would do, remains open.

F3

Ma'at

in use on the site

❓ Is the machine getting better, measured the same way every week?

The daily kept rate swings with what the person checks, and each model’s own test moves with its training data. Neither tells you if the machine is actually getting better. So we built a fixed yardstick: 200 past decisions on machine filings, drawn once and frozen. Each week a fresh copy of the confidence model trains on every other decision and scores those 200. We added a second, torn set of about 100 filings the model was unsure of at decision time, since improvement should show there first.

The upshot After four weekly tests, the fixed yardstick showed a slight downward drift instead of the expected rise.
Keep reading ✏️

🎯 What we expected

We expected the main set’s AUC to land between 0.88 and 0.91, with 93 to 97% right when sure and 65 to 70% of kept filings passing as sure, then rise about 0.01 a week. The torn set we expected much lower, 0.65 to 0.75. Weekly changes under 0.03 on 200 filings were known to be noise.

🔬 What we found

The first weighing on October 12 gave an AUC of 0.920 on the main set, with 97% right when sure and 67% of kept filings passing as sure, close to what we expected. The torn set scored 0.818, well above the 0.65 to 0.75 we predicted. Through November 1 the main set’s AUC drifted from 0.920 to 0.901, and the torn set’s from 0.818 to 0.787, even as training decisions grew roughly fourfold. No sign of the hoped-for rise, though each weekly step is within the noise. Filings past the sure line stayed 97 to 100% right, but as the model’s own sure line rose from 0.85 to 0.90, the share of kept filings passing as sure fell to about half on the main set and about a quarter on the torn set.

U1

the Golem

answered

❓ What motifs does our filing make when it starts from an empty index?

We wanted to see what our filing machinery would do without a person's index. Normally a curator approves every motif, name, and merge. The Golem experiment started from an empty index and filed 600 claims on its own, matching each to a motif it had already made or inventing a new one. We later ran the same test on 600 claims from 1896 newspapers and 600 from the Viral Texts reprint collection.

The upshot It lumps: 63 motifs for 600 claims where the hand-built index has 465.
Keep reading ✏️

🎯 What we expected

Nothing was written down before the first run. Our guess was that an empty index would grow motifs about as fine as the hand-built ones, since filing names a new motif whenever nothing fits. The later archive runs did register expectations: that it would lump and overlap our index's filing only loosely.

🔬 What we found

It lumps. The 600 claims, which the hand-built index spreads over 465 motifs, got 63. The biggest were broad archetypes: Wolf in sheep's clothing took 75 claims. B-cubed precision was 0.19 and recall 0.53. The first motifs it makes are broad, and each later claim finds one that fits, so the index never grows fine. On the 1896 newspapers and the reprints it lumped harder still, making 23 motifs for 600 claims each. A blind test of whether its motifs are interesting was drawn but never rated.

S4

Atropos

answered

❓ Can word matching find where one news event ends and another begins inside a story?

Stories are built by embeddings, which group headlines by meaning but can merge two events on the same subject into one. We tried using word matching to find the seams inside those stories. For each story, we cut the headlines where the shared words were weakest and gave each cut a score from 0 to 1, where a low score means the two sides use very different words. We then asked a language model whether the cut had actually split two news events.

The upshot Cuts scoring under 0.6 were right 18 of 18 times, but the line was not tested independently.
Keep reading ✏️

🎯 What we expected

A first look at 35 stories ranked real seams first, with scores from 0.44 to 0.56. We expected low-scoring cuts to be two events and high-scoring ones to be one event split only by wording, with the dividing line near 0.6.

🔬 What we found

We made 345 cuts across 169 stories and got a consistent judgment on 260 of them. Every cut scoring under 0.6 was a real seam between two events (18 out of 18). Between 0.6 and 0.7, 69% were correct. Above 0.8, almost none were. The 0.6 line was first seen in an earlier look at 35 stories, so this run did not test it independently. A dry run before launch showed the score alone would split 9 stories, but 3 of those were wrong, cutting a single event in two. We now only cut when the score is under 0.6 and the judge, asked two ways, agrees it is two events.

S3

Rumpelstiltskin

answered

❓ What makes headlines one story: TF-IDF, embeddings or hybrids

Stories were made by word matching (TF-IDF, which scores headlines by the rare words they share) until Oct 3, then by embeddings (mxbai, which compares meaning) from Oct 4, after a test that compared embedders only: word matching was never measured. Embeddings catch paraphrase but can blur two events on one subject together. Rumpelstiltskin compared word matching, embeddings and blends of the two on one day's headlines.

The upshot Embeddings reached 0.88 precision while word matching fell to 0.25, and the hybrids showed no real gain over embeddings alone.
Keep reading ✏️

🎯 What we expected

Not written beforehand; reconstructed afterwards from the question as asked. We expected word matching to be precise but miss paraphrase, embeddings to lump events on one subject, and a hybrid (embeddings checked by shared words, or a blend) to beat both on precision at little cost in recall.

🔬 What we found

One Sunday's 4,989 headlines, with pairs judged by Gemma under two wordings. Word matching was clearly worse and collapsed as coverage grew: precision 0.64 with about 400 headlines in stories, 0.25 with about 500. Embeddings held at 0.88 at both. The hybrids gated or cut by shared words sampled 0.85 to 0.95, but a later check found two of them made exactly the embeddings' stories yet scored 0.92 to 0.95, so differences that size are sampling noise: the hybrids weren't shown to be better. Recall couldn't be measured (only 3 same-event pairs turned up in its pool). The solid finding confirmed the move to embeddings; nothing changed in production.

S1

Janus

answered

❓ Can a model tell a new event from an update to the same event?

The site groups headlines into stories hour by hour. A new event can get pulled into an old story if the words look alike, even when the meaning is different. We wanted to know if a language model could tell a new event from a simple update to the same event, using only a story's earlier headlines and a day's new articles. We tested two models, Gemma 26B and the cheaper Nimble 9B, to see which one could make this call reliably enough to split stories automatically.

The upshot Gemma agreed with the key 78% of the time, Nimble 60%.
Keep reading ✏️

🎯 What we expected

We expected a direct yes-or-no question to work where likeness fails, and Nimble, which had matched Gemma at judging filings, to do about as well here.

🔬 What we found

We tested 89 day-to-day turns across 50 stories. Claude labelled 53 new events and 36 updates as the answer key. Gemma agreed with the key 78% of the time, catching 79% of new events but wrongly calling 25% of updates new. Nimble agreed only 60% of the time, with an AUC of 0.715. A quarter of updates called new is too many wrong yeses to split a story on Gemma's word alone, since each mistake breaks one event in two. The key was drafted from Gemma's own answers and corrected by a model, so the agreement may be flattered.

F8

the Chimera

answered

❓ Do the posts behind one claim actually contain several distinct stories?

We noticed that claims written from people's posts landed in three or more motifs 32% of the time, while claims from shows, fact-checkers, and our focus group did so only 12 to 17% of the time. We worried our method of grouping similar posts and writing one claim for the whole group was stitching several different stories together. To test this, we had Gemma read the original posts behind a claim and list every distinct claim it could find.

The upshot Yes, nearly all groups contain about four distinct claims, but this stitching doesn't explain why some claims attract more motifs.
Keep reading ✏️

🎯 What we expected

We expected claims filed in three or more motifs would split into several distinct claims far more often than claims filed in just one. We also thought the multiple motifs would land on different sub-claims, revealing stitched tellings, rather than all pointing to a single, layered story.

🔬 What we found

Nearly every group contained multiple claims. For claims filed in three or more motifs, 92% split into an average of 4.4 sub-claims. The control group, claims filed in just one motif, split 95% of the time into an average of 3.9 sub-claims. When we asked Gemma to match the original motifs to the new sub-claims, the motifs landed on different sub-claims in 46 of 59 cases, confirming the groups are stitched together. However, since the control group split just as often, this stitching does not explain why some claims attract many motifs.

F2

Cerberus

answered

❓ Was the motif the person added already in the claim’s shortlist?

Every hour the site reads the front pages and files the stories it finds into motifs, the recurring story shapes folklorists catalogue. From October 8 to 11, the person who runs the site added a claim to a second motif 312 times: the machine had filed it in one and missed another. A miss happens in one of two places. Either the shortlist of candidate motifs never showed the right one, or it was shown and the filing model didn’t take it. Cerberus asked which.

The upshot When the machine missed a second motif, the shortlist had failed to include it about half the time, and the filing model had overlooked it the other half.
Keep reading ✏️

🎯 What we expected

Reconstructed after the run. The plan named both outcomes: if the added motif was usually in the shortlist, asking for a second motif would be cheap; if not, the shortlist was where second motifs got lost. We expected “usually,” with the added motif ranking a little below the one the machine chose.

🔬 What we found

The motif the person added was in the top 8, 12 and 40 for 42%, 50% and 72% of cases. The motif the machine had chosen reached those ranks 48%, 56% and 86% of the time. So about half the misses were the filing model’s (shown, not taken) and half the shortlist’s (never shown). A quarter of the added motifs sat outside even the top 40. The shortlist measured used a plain six-measure weighting, not the learned one production uses, so the absolute numbers understate what the real shortlist would show. The comparison between added and chosen is the useful part.

F1

the Gordian Knot

answered

❓ Can a word-based cut spot a motif holding two tellings?

When the filing machine can't find the right motif for a claim, it puts it in the nearest one. Later, a person carves a new motif out by hand. We had a word-based cut from an earlier experiment, Atropos, that splits a set of texts in two where they share the fewest words. It worked on news stories, finding two events tangled inside one. We tried the same cut on motifs, hoping it could spot a motif holding two tellings and the claim that doesn't belong, so the machine could propose splits before anyone has to make them.

The upshot A word-based cut failed to reliably find motifs needing to be split, performing worse than random guessing on carved claims.
Keep reading ✏️

🎯 What we expected

We expected a carved claim to be its motif's least alike claim well above chance, and motifs the person carved from to have low cut scores where motifs never carved from don't.

🔬 What we found

Words can't see it. Over 224 carvings since Oct 8, the carved claim was its motif's odd one out 11% of 123 times, below the 14% chance would give. A correction that evening found most were copies, and on the 35 claims that really left their motif the cut picked them 9% of the time, against 17% chance. Cut scores told the 54 motifs carved from apart from the 35 never carved at an AUC of 0.67, but with too many false alarms: 15 of the 35 never-carved motifs fell under the 0.6 line, against 32 of the 54 carved. A story's two events differ in their words, but a motif's two tellings share the topic's words and differ in shape. Nothing went into production.

E13

Anansi

answered

❓ Does a three-sentence retelling of what speakers said find a claim's motifs better?

Our claims are one sentence, and the shortlist searches with that sentence alone. For claims from talk shows, podcasts, and the Focus Group we keep the speakers' own words, so there is more to search with. Anansi asked whether a three-sentence retelling of what the speakers said, written by Gemma, finds a claim's motifs better, alone or alongside the claim. On folktales, three sentences instead of one had helped.

The upshot Adding a three-sentence retelling to the claim lifted the correct motif's top-12 placement from 68% to 73% of filings. 68% → 73%
Keep reading ✏️

🎯 What we expected

Reconstructed after the run: on folktales, three sentences raised the right type's place in the top 8 from 34% to 47%, so we expected a fuller telling to help on our claims too, most when searched together with the claim.

🔬 What we found

On 644 filings of 388 claims, the claim and its telling together put the right motif in the top 12 for 73% of filings against 68% for the claim alone (top 8: 69% against 63%). The telling alone did worse (65%): it kept the speakers' words but lost the claim's point. The gain showed only under the shortlist's learned weights, not plain likeness, and only at the top of the list (about the same at top 40), though the top is what counts, since the filing model sees about 12. It was measured against an older, simpler shortlist (the full one already reached 86.5% in its top 12), so the gain may shrink there, and no interval was computed. Nothing went into production; the next step is to try the telling as a signal in the full shortlist.

E12

Jack

answered

❓ Do our own seed motifs start buried in the shortlist?

We often plant a 'seed' motif, a name and note with no claims, so the filing model can find claims for it. But the shortlist scores motifs partly by how alike their existing claims are to a new one, and a sister test on folktales suggested empty motifs rank low. If seeds start buried, the filing model never sees them and they stay empty. Jack checked whether that happens on our own index.

The upshot Empty motifs get an accidental score boost from a default value, so seeds actually rank higher than they would with claims.
Keep reading ✏️

🎯 What we expected

Reconstructed after the run, from the folktale write-up: we expected a motif emptied of its claims to rank well below where it ranks with them, plain likeness to its name and note to rank it better than the shortlist's learned weights, and a fix (scoring an empty motif's missing claims by its name instead) to win back most of the loss.

🔬 What we found

It failed, the other way round. Taking 1,860 confirmed filings and emptying each one's motif as if it were a seed, the right motif landed in the top 12 for 93% of claims, against 70% with its claims in place. Plain likeness managed only 51%, and a planned fix 33%. The cause is a quirk of the weights: a motif with no claims gets a default value on one measure that the weights never saw in training, and it lifts the motif's score. So seeds aren't buried, and nothing was changed for them. The flip side, flagged for watching: every empty motif in the index gets the same unearned lift on every claim, so it may take shortlist places it hasn't earned.

P2b

Nobody

answered

❓ Does banning a leading "The" make model-written motif names read more like the site's own?

Under the site's new naming rules, 64 of 101 draft motif names from our model began with "The" (The Villainous Turn, The Great Unraveling), a pattern the site's own names rarely follow. We added one line to the model's instructions: no leading "The" unless it's part of the idiom. We wanted to see if that would make the names read more like the ones a person writes.

The upshot The site owner decided to keep the leading "The" as house style, so the instruction to remove it was left out of the model's rules.
Keep reading ✏️

🎯 What we expected

We expected the line would simply remove the article without otherwise changing the names, and that the names would come a little closer in meaning to the site's own.

🔬 What we found

The article nearly vanished: names starting with "The" fell from 63% to 4%. But closeness to the site's own names didn't rise (0.546 to 0.540), and abstract labels crept up from 10% to 13%. Some names changed wholesale rather than losing a word (The Villainous Turn became Moral panic). Measures couldn't settle a matter of style, so the site owner did: the leading "The" is house style, and should be kept for Archetypes. Told that in a second run, the model began 80% of all its names with "The", Archetypes and the rest alike (The punchable face). Naming the word primed it. The line stays in the written rules for people and out of the model's instructions.

P2

Adam

in use on the site

❓ Does a new rule for the judge change its scores, and do new naming rules produce better motif names?

Before the site’s filing rules went live on October 10, we tested two changes that touch the machine. One was a new instruction for the judge that scores whether a claim fits a motif: the motif is ours to name, and the teller doesn’t have to name it, as long as the shape is in what they said. The other was a new way to ask a model to name a motif, using plain language, no abstract labels, one to five words. Until then, every machine-made name had been renamed or thrown out by hand, so we wanted to know if the new rules would get closer to something usable from the start.

The upshot The new naming rules went live on October 10 because they produced shorter, less abstract names the site owner preferred, despite a drop in an old similarity score.
Keep reading ✏️

🎯 What we expected

We expected the judge already read filings that way, so the added rule would change little. We expected the new naming rules would produce shorter, far less abstract names, closer to the ones the site actually uses.

🔬 What we found

The judge barely noticed the new rule. We measured how well its scores separated kept filings from dropped ones on 1,524 decisions. Without the rule, the score was 0.827; with it, 0.825, where 0.5 is chance. The rule was left out. For naming, we had Gemma rename 100 motifs under the new rules. Abstract labels, a rough count of words like “of” and endings like “-tion”, fell from 51% to 10%. Name length dropped from 4.9 words to 3.2, close to the site’s own 2.8. But closeness in meaning to the site’s names fell from 0.587 to 0.546. That yardstick was built from older, plainer names made before the rules, so it couldn’t fairly judge the new ones. Many new names started with “The” (The Villainous Turn), alongside good ones like Eat the rich and Collateral damage. The site owner read them and judged them far better.

M1

the Changeling

in use on the site

❓ Can Gemma 26B answer the match question as well as Qwen3 30B?

Every time the filing model spots a new motif, a second model checks whether an existing motif already tells the same story, so we don't create a duplicate. That match question was the last job still running on Qwen3 30B, and each time it ran, the 30B had to swap places with Gemma 26B on the graphics card mid-filing. The Changeling asked if Gemma could take over the job, letting us remove one model from the pipeline entirely.

The upshot Gemma took over; Qwen3 30B is retired.
Keep reading ✏️

🎯 What we expected

We expected Gemma to find about as many of the past merges as Qwen and wrongly join no more of the pairs ruled 'not the same'. An earlier, smaller test on 31 merges and 45 not-same verdicts had been too close to call.

🔬 What we found

On 77 past rulings from the curation log, Gemma found 20 of the 33 merges to Qwen's 18, wrongly joined the same 4 of 44 'not the same' pairs, and answered faster at 12.9 seconds a call against 17.6. Two merges is within the noise, so the decision rested on 'no worse, and one model fewer' rather than on Gemma being better. Since October 10 Gemma answers the match question and Qwen3 30B is retired.

D5

Punch and Judy

tried, not kept

❓ Do a claim's roles and threaded domains find its motif and genre?

Folklorists describe a story by its actants: the stock roles, who does what to whom. We had Gemma read 914 claims for their roles, whether the threat comes from inside the teller's community or outside, and how many domains of life the claim threads together. We wanted to know if these signals could help the shortlist find a claim's motif, and whether threading many domains marks a Theory apart from a Plot.

The upshot Strong alone, but the embeddings already carry who does what to whom.
Keep reading ✏️

🎯 What we expected

We expected that matching who-does-what-to-whom would link a claim to its motif's other claims across topics, adding to the learned shortlist. We also thought Theories would thread more domains than Plots, enough to help the genre drafter.

🔬 What we found

The roles alone were a strong signal for matching a claim to its motif, scoring an AUC of 0.899. But plain word likeness already scored 0.933, and the two were closely related. Adding the roles to the learned shortlist left it unchanged: the right motif was in the top 12 for 86.7% of claims, before and after, in every genre. The threat label nudged that down to 86.5%. Theories and Plots threaded almost the same number of domains, 2.99 against 2.90, an AUC of 0.519 where 0.5 is chance.

D3

the Hydra

tried, not kept

❓ Does a judge’s agreement with itself across varied prompts tell us when to trust it?

An AI judge decides whether a news story fits a motif, a recurring story shape. Research suggests a judge that agrees with itself across varied prompts is more trustworthy. We asked our judge five times per filing, each time showing a different random sample of the motif’s claims, to see if the average answer ranked stories better and if disagreement flagged the ones a person would remove. That matters for the sure line: filings the machine passes without a human check.

The upshot Asking the judge five times did not improve the model, so the original single-check method remains unchanged.
Keep reading ✏️

🎯 What we expected

We expected the average of several asks would sort kept from removed filings a little better than one. We thought the asks would disagree more on filings a person removed, making the spread a useful signal. We also guessed that requiring every ask to say yes would make the sure line safer at little cost.

🔬 What we found

On 918 decided filings, the average of five barely beat a single ask on its own (AUC 0.819 vs 0.814; AUC measures how well a score ranks kept stories above removed ones, where 0.5 is chance). Inside the full confidence model it added nothing: 0.883 with the average, 0.885 as it was. The asks disagreed more on kept filings (0.086) than on removed ones (0.052), the wrong direction for a trust signal. The current model passes 69% of kept filings without a person at 95% precision; with the average that fell to 61%, and requiring all five asks to say yes cut it to 41–42%.

D2

Bluebeard

unproven either way

❓ Does showing the judge a person’s past removals from a motif improve its decisions?

The judge that feeds the site’s confidence model sees every claim filed under a motif but never the ones a person later took out, the nearest thing to a line drawn around what belongs. Prompted by work on few-shot examples and on showing a model its own past corrections, we gave the judge up to three past removals from the same motif, chosen by meaning, and asked whether seeing those counter-examples would help it decide better.

The upshot Showing past removals nudged the judge’s accuracy from 0.796 to 0.809, but the shift was too small to rule out chance. 0.796 → 0.809
Keep reading ✏️

🎯 What we expected

We expected that showing removals as counter-examples would raise the judge’s AUC, especially in the later half of decisions where more removals were available, and that adding a drafted reason for each removal would help further.

🔬 What we found

On 624 decided filings whose motif had at least one earlier removal, the judge’s AUC rose from 0.796 to 0.809 when shown the removals. Adding a short reason drafted by another model for each removal erased the gain, dropping AUC to 0.795. The improvement of 0.012 has a bootstrap range of –0.003 to +0.028, which includes zero. The later half of decisions gained slightly more (0.793 to 0.810) but stayed within the same noise.

D1

Cinderella

answered

❓ Does the judge read the must-have element?

Each motif has a must-have element, a sentence that says what a telling needs to count. Since October 9 the judge, a small model called Nimble 9B, sees that element and answers yes or no. A paper warned that a model can score well on a coding task while ignoring the codebook, so we checked whether our judge really reads the element or just goes by the motif's name, note and claims.

The upshot The judge's accuracy dropped sharply when given a wrong element, confirming it genuinely reads the must-have sentence.
Keep reading ✏️

🎯 What we expected

If the judge reads the element, a wrong one would cost AUC and cut its yes on kept filings far more than on removed ones. A wrong element from the same genre would cost less than any other motif's.

🔬 What we found

We tested 1,465 decided filings across 375 motifs. With its own element the judge scored AUC 0.821. With another motif's element it dropped to 0.641, and with another of the same genre to 0.654. A wrong element cut its yes on kept filings by 0.36 to 0.38 but on removed ones by only 0.09 to 0.11, and flipped the answer by more than 0.1 on 63% of filings. The judge reads the element, and reads more than just its genre. One catch: a swapped element clashes with the note shown beside it, so the judge may be reacting to the mismatch.

A1

Scheherazade

answered

❓ Can our pipeline file folktales into a folklorists' codebook it never saw?

We had only ever tested our filing machinery against our own index, so we ran it unchanged on a folklorists' one: 1,518 folktales, each labelled with its ATU type (the standard catalogue of 2,247 tale types). Each type became a motif and each tale a claim, summed up in one plain sentence or in three, with the folklorists' labels as the answer key. It was the one test we could run against an answer key nobody at the site had written.

The upshot On 1,518 folktales, the right tale type made our shortlist about 9 times in 10 and was picked 4 times in 5.
Keep reading ✏️

🎯 What we expected

No expectation was written down beforehand; this one is reconstructed after the run. We expected the machinery to shortlist and pick the right type about as well as it does on our own claims, three-sentence summaries to help a little, and the shortlist's learned weights (trained on motifs that all have claims) perhaps to behave oddly where most types have no tales at all.

🔬 What we found

Among the 182 types that have tales, the shortlist put the right type in its top 8 for 88% of tales, Gemma 4 26B picked it 79% of the time (88% when it was shown), and the judge model reached an AUC of 0.93 to 0.95 (how well a score ranks right answers above wrong ones; 0.5 is chance), better than the 0.83 or so it gets on our own filings, while almost never saying yes to a wrong type. Three sentences helped a little (80% picked), and that idea went on to Anansi. Over all 2,247 types, though, the learned shortlist had the right type in its top 8 only 34% of the time, against 64% for plain likeness. We first read that as burying motifs with no claims yet, but a follow-up on our own index (Jack) found the opposite, empty motifs getting an unearned lift, so the likelier story (not tested) is that the 2,065 empty types crowded out the right ones. Reading the failures showed the shortlist matching a tale's cast rather than its plot (one fox tale filed as another fox tale), which led to Indra's Net; nothing from this run itself went into production.

W1

the Brothers Grimm

unproven either way

❓ Can rules from a person's corrections make models write claims more like them?

The site writes a one-line claim for every story from its source material. When a claim misses the mark, the person running the site corrects it by hand. After 80 corrections, we noticed the fixes fell into a few repeating patterns. We wondered if we could teach the models those patterns. The Brothers Grimm experiment tested whether rules drawn from the corrections, and examples of those rules, would make two different models write claims closer to the person's own wording.

The upshot Gemma wrote closer than Qwen3; a short-sentence rule helped most, but the rules remain unproven.
Keep reading ✏️

🎯 What we expected

We expected Gemma to write closer than Qwen3 from every source. We thought rules would help, and examples would help more. We also guessed both models wrote too long, so a rule for a single short sentence would bring them closer to the person's style.

🔬 What we found

We tested 74 corrected claims. Gemma 4 26B started closer to the person's wording than Qwen3 8B, with a cosine likeness score of 0.758 versus 0.726. Qwen3 sometimes got stuck in loops and invented facts. Adding rules alone barely helped; a blind judge preferred Gemma's rules-only claims over its usual ones just 34% of the time. Adding examples raised that to 47%. A new rule to write one short plain sentence worked best, lifting Gemma's likeness to 0.784 and its word count to 17.7, very close to the person's 16.8. The judge preferred this short version 54% of the time.

R1

Goliath

tried, not kept

❓ Does a bigger, untaught reranker judge better and earn its card space?

The confidence model estimates how likely a filing is to be kept when checked by hand. One of its signals is a reranker, a small model that reads a claim and a motif together and scores how well they fit. We had been using Qwen3-Reranker 0.6B, taught weekly on our own decisions. Goliath asked whether its big brother, the 4B, would judge better, and whether it was worth the room it takes on our 10 GB graphics card.

The upshot The 4B judged best alone but added almost nothing inside the confidence model, so we kept the 0.6B.
Keep reading ✏️

🎯 What we expected

Not written down beforehand; reconstructed: the 4B would judge best alone, but add little inside the confidence model, since the other signals already carry most of what it adds.

🔬 What we found

On 976 filings checked by hand, the 4B, untaught, scored AUC 0.793 alone (how well it ranks kept filings above removed ones; 0.5 is chance) against 0.749 for the untaught 0.6B and 0.737 for our taught one. Inside the confidence model it added only 0.003 (0.895 against 0.892), within a run's noise, and it needs about 8 GB of a card the filing model needs too. Not worth it: we kept the 0.6B. One surprise: the taught 0.6B judged worse than the untaught one on this larger set, a question passed on to its weekly teaching.

P1

Puck

in use on the site

❓ Can Nimble judge link proposals as well as Gemma?

The site's motif index links motifs as "related" or "rests on," and a machine proposes these links for a person to approve or reject. A small model orders the proposals by their chance of approval, weighing plain facts like how alike two motifs are with a language model's judgement. Gemma 26B had been that judge, scoring each proposal 0 to 10. Nimble 9B, a small decision model from Bespoke Labs, had just matched Gemma at judging claim filings, so we asked if it could judge link proposals too, answering a simple yes or no.

The upshot Nimble alone did worse, but inside the combined model it averaged 0.772 to Gemma's 0.755.
Keep reading ✏️

🎯 What we expected

Reconstructed after the run: Nimble would rank approved proposals above rejected ones at least as well as Gemma's score, both alone and inside the combined model.

🔬 What we found

We tested both on 301 decided proposals, 150 approved, using AUC: how often an approved proposal ranks above a rejected one, where 0.5 is chance. Alone, Nimble scored 0.657 to Gemma's 0.681, so that half failed. Inside the combined model, Nimble scored 0.783 against Gemma's 0.754 and 0.751 with no judge. Over 20 data splits, Nimble averaged 0.772 to Gemma's 0.755, ahead in all 20. Using both judges together scored lower, 0.768, likely from overfitting on little data, though we didn't test why. Nimble went into use on Oct 9, at about 1.6 seconds a proposal.

N4

Tom Thumb

tried, not kept

❓ Does the 4-bit Nimble that fits the card whole judge as well and run faster?

We run a jury model that scores how well a news story fits a motif. The judge is a language model called Nimble 9B. The usual 8-bit version is 9.5 GB and spills 2 GB onto the CPU on our 10 GB card. We tried a 4-bit build that fits entirely in GPU memory, hoping it would judge as well and run faster.

The upshot judged worse: AUC 0.791 alone against Gemma 26B's 0.814 (the 8-bit 0.824 on its own set); 0.14 s a filing against 0.22 s; the 8-bit stays
Keep reading ✏️

🎯 What we expected

The 4-bit build loses little in judging quality and runs noticeably faster, since nothing waits on the CPU.

🔬 What we found

The 4-bit build judged worse. On 976 filings it scored 0.791 AUC alone, below Gemma 26B's 0.814. The 8-bit Nimble scored 0.824 on its own set of 817. In the confidence model the 4-bit reached 0.887 against Gemma's 0.892. It was faster at 0.14 seconds a filing versus 0.22, but the GPU was no busier: the time left between requests is the server's own overhead per request, not the part of the model run on the CPU.

J2

Solomon

in use on the site

❓ Does the judge do better when shown what the hand rulings actually turn on?

The judge, a model that decides whether a news story fits a recurring pattern called a motif, was only seeing a one-line summary of the story. But the hand rulings on the hardest cases often turned on details it never saw. We tried giving the judge more: the tellers' own words, a must-have element that any story must contain to fit the motif (like something going wrong and being deflected for Passing the buck), or a question framed in the motif's genre.

The upshot The must-have element took the judge from near chance to 0.79 on the hard cases. 0.54 → 0.79
Keep reading ✏️

🎯 What we expected

We expected each addition would help most on the 96 hard cases where the judge was near chance. The must-have element would help most of all, since it states the line the hand rulings draw. All three together would be best.

🔬 What we found

We measure success with AUC, where 0.5 is chance and 1 is perfect. On 96 hard cases ruled by hand, the must-have element lifted the judge from 0.54 to 0.79. The tellers' words reached 0.72 and the genre question 0.68. Giving the judge all three at once scored 0.70, worse than the element alone. Across all 897 filings the gains were tiny, within a run's noise, so the case rests on those 96 hard cases. The element went into production the same day.

G2

Noah

tried, not kept

❓ Would a description of what a group's members share help the model say which motifs belong?

The motif index sorts news stories into recurring shapes called motifs. Motifs can also belong to groups, a separate axis from genre, and a model named Gemma proposes which motifs fit a group from its name and its members. Groups could carry a written description, but none had one. We wanted to know whether a description of what a group's members share would help the model say which motifs belong.

The upshot Adding a written group description did not improve the model's overall accuracy, so we are not using them.
Keep reading ✏️

🎯 What we expected

A description drafted from half of a group's members would help the model admit the other half and keep the nearest outsiders out.

🔬 What we found

We tested 32 groups with four or more members, asking about 342 members and 342 nearby outsiders. Balanced accuracy, the average of members admitted and outsiders kept out, barely moved, from 69.6% to 70.2%. The description traded members for outsiders: it admitted fewer members (62.3% against 64.6%) and kept out more outsiders (78.1% against 74.6%). The drafted descriptions read as abstract sociology, and two halves of the same group sometimes described different things. We are not using them.

G1

Mother Goose

in use on the site

❓ Does telling Gemma what each genre is make it agree with us more?

Every motif in our index gets a genre, a layer of story like Plot, Theory, or Belief. A language model called Gemma drafts the genre for each new motif, and a person checks it. But Gemma only ever saw five example motifs per genre, and it often confused close pairs. Mother Goose gave Gemma written descriptions of what each genre is, plus tests for telling close ones apart, to see if that helped it match the person's calls more often.

The upshot Written descriptions lifted agreement from 53% to 64-67%, but refinements hurt Theories.
Keep reading ✏️

🎯 What we expected

We expected the written descriptions to raise the share of correct calls, especially for the genres the examples confused most, like Theories and Beliefs. We also thought each refinement of the descriptions would help further.

🔬 What we found

On about 453 hand-labeled motifs left out of the examples, Gemma got 52.8% right with examples alone. Adding descriptions and tests lifted that to 64.0%. A test for Argument versus Theory pushed it to 66.0%, with Theories jumping from 38% to 72%. That version went live. But later refinements backfired: they widened Plots and dragged Theories down to 54%, then 44%, even when the overall score held near 66.9%. The answer key also shifted between runs as genres were set and moved.

F18

Cordelia

in use on the site

❓ Does the Brothers Grimm's winning wording, deployed in every claim prompt, shorten claims and cut rewording?

Claims are the one-sentence statements our model writes of what people are telling. The person who runs the site rewords the ones that come out wrong. An earlier experiment, the Brothers Grimm, found a wording that came closest to theirs: rules drawn from their corrections, a rule asking for one short plain sentence, and up to 8 of their own corrections as examples. That winner was never deployed until a review noticed, so Cordelia put it into every claim prompt on Oct 12 and measured it forward.

The upshot Posts shortened by five words; rewording fell by more than a third; fact-checks didn't shorten.
Keep reading ✏️

🎯 What we expected

Claims from posts would fall under 18 words on average and fact-checks under 17, with the already short shows and Focus Group staying within two words. The share of claims the person rewords would fall by a third or more. Nothing would be lost on the way.

🔬 What we found

Claims from posts got shorter, from 20.7 words to 15.9. Shows stayed about the same, 14.3 to 13.4. Fact-checks did not shorten, 19.9 to 20.8. The Focus Group has written no claims since. Measured over only the claims the person actually judged, 3 of 300 new claims were reworded (1.0%) against 17 of 484 before (3.5%), a fall of more than a third in every source. One thing to watch: show claims dropped because their quote didn't support them rose from 6.4% to 8.0%, and quotes not found from 0.5% to 1.2%.

E11

Babel

in use on the site

❓ Which embedding models put more of the right motifs in the shortlist?

The shortlist works like a search engine: the claim is the query, the motifs are the documents, and an embedding model, which turns text into numbers so likeness can be measured, decides what looks alike. Until Oct 9 we used two, mxbai-embed-large and nomic-embed-text. Babel tried every embedding model we could reasonably run on our own machine, thirteen in all, to see which put more of the right motifs in the shortlist, alone and added to every signal we already had.

The upshot Adding two models together lifted the share of searches with the right motif in the top 12 from 82.3% to 85.2%. 82.3% → 85.2%
Keep reading ✏️

🎯 What we expected

Reconstructed after the run: we expected larger, newer retrieval models to beat mxbai on their own, but, added on top of every existing signal, most to add little since they read the same topic; one or two might add a point or more, and the best pair more than either.

🔬 What we found

Both parts held. Alone, the best (EmbeddingGemma 300m and Qwen3-Embedding 8B, each putting the right motif in the top 12 about 71% of the time) beat mxbai's 65.4% by about five points. Added to everything else, only Qwen3-Embedding 4B and 8B and Snowflake Arctic 2 moved the shortlist by a point or more; Qwen3-Embedding 8B with Arctic 2 together took the top 12 from 82.3% to 85.2% and cut the hard misses (right motifs outside even the top 40) from 76 to 52. The pair went into production on Oct 9, and a later re-check with confidence intervals (a gain of 2.9 points, somewhere between 1.7 and 4.2) made it the one decision of those early days that clears every reading of the noise. On Oct 27 a rebuilt version of Arctic 2 scored exactly the same as ours, so ours stayed.

E8

Narcissus

tried, not kept

❓ Does showing a motif’s nearest claims help the pick?

The filing model picks which motifs a claim belongs to from a shortlist. It sees each motif’s name, its note, and three of its claims, usually the latest three. An earlier experiment, Hansel and Gretel, changed two things at once: it put the note first and showed the three claims nearest the new one. That run missed more. We wanted to isolate the nearest-claims half to see if it helped on its own.

The upshot Swapping the latest claims for the nearest ones barely moved the model’s accuracy or its error rate.
Keep reading ✏️

🎯 What we expected

We expected that showing the claims most like the new one would help the model find a little more of the right motifs and pick fewer wrong ones.

🔬 What we found

We saw no real difference. Showing the latest three claims, the model found 161 of 239 correct motifs (67%) and 54 of its 215 picks (25%) were wrong. With the nearest three, it found 158 (66%) and 50 of 208 picks (24%) were wrong. The prompt stayed as it was. This also suggests the note-first framing caused Hansel and Gretel’s extra caution.

E6

Ariadne

in use on the site

❓ Does the order of checking filings change how fast the model learns?

We tested whether the order a person checks filings changes how fast the confidence model learns. The model ranks filings by how likely they are right, and it learns from the ones a human corrects. We replayed 753 past decisions in three orders: least certain first, at random, and likeliest first, to see which order taught the model fastest.

The upshot Least certain first teaches the model fastest.
Keep reading ✏️

🎯 What we expected

We expected least certain first to teach the model fastest, and likeliest first slowest, since filings the model is already sure of teach it least.

🔬 What we found

Least certain first taught the model fastest. After 150 decisions it reached an AUC of 0.884, a level random order took about 550 decisions to approach (0.882), and likeliest first lagged throughout (0.839 at 150). All three orders met by 550 decisions, so the gain is in speed, which matters while decisions are few. The caveat is it is a replay, and someone facing only torn filings may decide them differently and more slowly.

E5

Arachne

unproven either way

❓ Does spreading scores along a motif's links find more of the right motifs?

The index groups news into motifs, and the person who curates it links them: 'rests on' for logical dependence, 'related' for looser ties. A claim that fits one motif might also fit its neighbours. We tried spreading a motif's score to its neighbours through those links, hoping to catch the right motif when the direct match missed. Then we checked whether any gain was real or just the links echoing claims already filed together.

The upshot Promising in the lab, but the gain vanished under stricter tests and a small forward sample.
Keep reading ✏️

🎯 What we expected

We expected spreading to recover hard misses and improve the top 12, but we suspected part of the gain was leakage: until October 7 links were suggested from shared claims, so a link might encode the very claim being tested. Using only older links should shrink the gain.

🔬 What we found

On held-out claims, spreading through rests-on and related links lifted the right motif into the top 12 for 84.7% of claims, up from 82.3%, and cut hard misses from 76 to 64. But when we kept only links made before each claim arrived, the gain shrank to 82.5%. That strict test was weak because most links were made after the claims. A forward test on 30 new claims showed no gain at top 12: 58.7% with or without the spread.

E4

the Three Wise Men

tried, not kept

❓ Does having Gemma answer as three different readers improve its judgment?

Before a human checks a filing, a confidence model estimates how likely it is to be correct. One of its inputs is Gemma's own yes-or-no answer on whether the claim fits a story motif. We tried having Gemma answer as three different readers: a folklorist cataloguing story types, a rhetorician, and a reader from the claim's other side. We wanted to see if any persona judged better, or if their disagreement was useful.

The upshot No persona beat the plain question; the gain was too small to keep.
Keep reading ✏️

🎯 What we expected

We expected the folklorist to judge better than the plain question, and the three together to add a small boost to the confidence model.

🔬 What we found

It didn't work. Measured by AUC, where 0.5 is chance and 1 is perfect, the plain question scored 0.820 alone. The folklorist scored 0.794, the rhetorician 0.796, and the other side 0.762. Adding all three to the confidence model moved it from 0.882 to 0.886, and the share of filings it could pass unchecked at 95% precision rose from 66% to 67%. The plain question already asks the folklorist's question, so a persona adds nothing.

C1

Goldilocks

tried, not kept

❓ Can small models pick motifs and bump only their doubtful picks to Gemma?

The site’s filing model picks which of 12 candidate motifs a news story retells, but the model we use, Gemma 4 26B, is too big for our 10 GB graphics card, so part of every answer runs on the slower main processor. We tried letting smaller models that fit the card do the picking and handing only their doubtful picks up to Gemma, hoping to save time without losing accuracy.

The upshot Letting Gemma handle all picks on its own gave the lowest error rate, and routing only disagreements made the process slower.
Keep reading ✏️

🎯 What we expected

We expected the small models to find about as many right motifs as Gemma but make more wrong picks, and that our confidence model would flag those wrong picks, so bumping low scorers up to Gemma would recover Gemma’s accuracy at well under its cost.

🔬 What we found

On 240 hand-confirmed claims, Gemma 4 26B found 64% of the right motifs with 24% of its picks wrong, at 3.5 seconds a claim. The smaller models found about as much (Gemma 4 12B got 67%, Ministral 3 14B 65%, Qwen3.5 9B 58%) but 32% to 35% of their picks were wrong. Bumping the lowest-scoring fifth of picks up to Gemma only brought wrong picks down to 28% to 30%. Bumping when two small models disagreed matched Gemma (64% found, 24% wrong) but sent 57% of claims up and took 5.6 seconds a claim, slower than Gemma alone.

J1

the Norns

in use on the site

❓ Can a smaller model judge as well as Gemma 26B, and does a jury judge better?

The confidence model that estimates whether a claim filing will be kept needs a yes-or-no judge. The judge was Gemma 4 26B, which took one to two seconds per filing and didn’t fit our 10 GB graphics card. We sat a jury of other models (including “decision models” built to answer yes or no in a single pass) to see if one could judge as well and faster, and whether several together would do better than one.

The upshot A smaller model, Nimble 9B, matched the larger judge's accuracy and was faster, so it took over the job.
Keep reading ✏️

🎯 What we expected

First sitting, reconstructed: decision models would judge nearly as well as Gemma 26B and much faster, and a jury would add a little over its best member. Later sittings: a new juror replaces the judge only if it scores 0.005 better inside the confidence model at no more than twice the time, or the same at twice the speed or more.

🔬 What we found

Scores are AUC, where 0.5 is chance and 1 is perfect. On 817 filings checked by hand, Nimble 9B scored 0.824 alone against Gemma’s 0.818, and 0.895 against 0.894 inside the confidence model, at 0.22 seconds a filing. A jury of eight added only 0.002 to 0.005 over it. Nimble became the judge on Oct 9. A second sitting on 1,954 decisions found pplx-decider 27B added 0.010 over Nimble inside the confidence model but was seven times slower, so Nimble stayed. A third sitting tried three newer models; a small one, d3, scored 0.875 against Nimble’s 0.872 at under a quarter of the time, but Nimble remains the judge.

E9

Pygmalion

in use on the site

❓ If the reranker learns a person's taste, does it judge new claims better?

The confidence model that decides whether a filing is good enough to pass without review uses several signals. One is a reranker, a small model that reads a claim and a motif together and scores how well they fit. Off the shelf, it knows nothing of how this index files. Pygmalion trained the top 6 of its 28 layers on our own past keep-or-remove decisions, to see whether it would judge new claims better and whether the confidence model would gain.

The upshot The taught reranker improved on held-out claims, but the gain vanished on a larger set a day later.
Keep reading ✏️

🎯 What we expected

We expected the taught reranker to sort held-out decisions better than the untaught one, but the confidence model to gain less, since much of what it learns is already carried by its other signals.

🔬 What we found

Both parts held. On claims it never saw in training, the taught reranker scored an AUC of 0.796 against 0.765 untaught (AUC: how well a score ranks good filings above bad ones, where 0.5 is chance). The confidence model barely moved, 0.887 to 0.890; the share of good filings it could pass unchecked while staying 95% right rose from 66% to 76%, but that jump rests on one coarse step and is fragile. A day later, on 976 decisions, the taught reranker judged alone at 0.737 against the untaught 0.749, so the gain did not hold on the larger set.

E7

Matryoshka

in use on the site

❓ Can a claim described layer by layer in generic words find its motif by structure alone?

Our hardest misses were claims a person reads for their structure while every finder reads only their topic. Matryoshka had Gemma describe each claim layer by layer in plain generic words: a character type, a plot, a theory or belief, an argument, a value at stake, or 'none' where a layer was absent. Each layer was then matched against the motifs, with the layer that goes with a motif's genre (a plot against Plots, a character against Archetypes) as a signal of its own.

The upshot Describing a claim's story layers in generic words helped find its motif, but stronger embedding models later made the layers unnecessary.
Keep reading ✏️

🎯 What we expected

Reconstructed after the run: we expected it to beat Morpheus (imagined instances, tested the same day), since reading a claim as the stack of stories it tells is how a folklorist reads it, and to carry no leak, since the layers read only the claim.

🔬 What we found

The day's best leak-free gain: the right motif reached the top 12 for 83.7% of claims against 82.3%, the top 8 for 79.7% against 77.7%, and hard misses (right motifs outside even the top 40) fell from 76 to 67 (17 brought back, 8 newly lost). Its two signals went into production, the layers asked once per new claim and kept. A re-check with confidence intervals confirmed the gain on that day's shortlist (+1.4 at top 12, somewhere between +0.2 and +2.7). But once Babel's stronger embedding models joined, taking the layers out cost only 0.2, inside the noise: Qwen3-Embedding 8B and Arctic 2 now find most of what the layers found. Later experiments that drop the topic altogether start from its limit, that its layers keep the topic in generic words.

E3

the Pleiades

tried, not kept

❓ Do motifs that are often filed together help find a claim’s second and later motifs?

A claim can be told in several motifs at once, and some motifs are often filed together: a claim in *Pious hypocrite* is often in *Double standard* too. The Pleiades asked whether those habits help find a claim’s second and later motifs. Given the shortlist’s three likeliest motifs for a claim, we scored every other motif by how often it has been filed alongside them, then added that score to the weighting. The idea was simple: if two motifs travel together in the person’s filing, knowing one might point to the other.

The upshot Adding co-filing habits nudged the top-12 hit rate from 82.3% to 82.9%, but the tiny boost disappeared across five other test splits.
Keep reading ✏️

🎯 What we expected

Reconstructed after the run: we expected co-filing counts to lift a claim’s later motifs into the shortlist by a point or so in the top 12, without hurting its first. It was only ever meant to suggest filings, never to link motifs: being told together isn’t being related.

🔬 What we found

The right motif reached the top 12 for 82.9% of claims, against 82.3% without the bundles. Hard misses (right motifs outside even the top 40) fell from 76 to 73. That is a gain of 0.6 points. But when we re-checked with confidence intervals and dealt the claims into test groups five other ways, the gain ranged from -0.2 to +0.5. The first run was its luckiest split. The gain sits inside the noise, so we did not keep the signal. The question of where second motifs get lost passed to Cerberus.

E2

Morpheus

tried, not kept

❓ If we imagine a motif on new subjects, can a claim on yet another subject find it?

A motif with only one or two claims tends to match only claims on the same topic, but motifs are shapes that travel from subject to subject. Morpheus had Gemma dream up eight imagined instances of each motif, each on a different subject, written from the motif's name and note alone, to see whether a new claim on yet another subject would then find it. It aimed at the hardest misses, where a human reads a story's structure and our finders read its topic.

The upshot Imagined instances added almost nothing over the original notes.
Keep reading ✏️

🎯 What we expected

Reconstructed after the run: we expected the imagined instances to bring back some hard misses (right motifs that fell outside even the top 40 of the shortlist), most for motifs with few claims, with no leak since they were written without seeing any claims.

🔬 What we found

Almost nothing. The right motif reached the top 12 for 82.5% of claims against 82.3% without, and hard misses went from 76 to 74 (6 brought back, 4 newly lost). The imagined instances added little over the notes they were written from. Not kept; its sister test that day, Matryoshka (reading a claim layer by layer), did better and went into production.

E1b

Chronos

unproven either way

❓ Does removing the news with headlines from a claim’s own time find more motifs?

We tested whether removing the news from old claims works better when you use headlines from each claim’s own week instead of a single snapshot of today’s front pages. The idea was that older claims carry the topics of their moment, and today’s news can’t cancel those out. We ran the same shortlist stage with both sets of headlines and compared how many of the person’s motifs the system recovered.

The upshot No difference, but the headline history was too thin to test the idea properly.
Keep reading ✏️

🎯 What we expected

Directions drawn from headlines across the claims’ whole span would cancel the topics of each claim’s own time, letting the older claims’ framing come through and recovering more motifs than today’s snapshot.

🔬 What we found

The numbers barely moved. With 40 directions, the snapshot gave 83.7% of motifs in the top 12 and 69 hard misses; the span gave 83.6% and 68. With 80 directions, the snapshot hit 83.7% and 72 hard misses against the span’s 83.2% and 67. The baseline with no news removal was 82.3% and 76 hard misses. The headline table held only 840 headlines by first-seen date, drawn from seven days, so the “span” was mostly this week anyway.

E1

Lethe

in use on the site

❓ Does subtracting the day's news from claims and notes find more of the right motifs?

Our shortlist ranks every motif by likeness to a new claim, but it often misses stories told by their shape rather than their topic. A party's comeback read as a phoenix rising from the ashes would fall outside the top 40 because every finder matched on what the claim was about. Lethe tried washing the day's news out: we found the main directions headlines vary along, which are mostly topic like a country or a storm, and subtracted them from claims and motif notes before comparing.

The upshot Kept as cheap and leak-free, but the gain is small and newer models now carry most of it.
Keep reading ✏️

🎯 What we expected

We expected removing topic directions to bring back some hard misses and add about a point to how often the right motif lands in the top 12. We thought taking out too many directions would strip meaning too, so the gain would peak and then fall. We also expected any gain to be free of leakage since headlines know nothing of our filings.

🔬 What we found

With 40 directions removed, the right motif reached the top 12 for 83.6% of claims, up from 82.3%, and hard misses dropped from 76 to 68. There was no peak: 5, 10, 20, or 40 directions all gave about the same point. It helped find motifs but not judge a filing once found; the confidence model scored 0.882 with the new signals and 0.883 without. A later re-check with confidence intervals put the gain at about one point, probably real but small, and found that newer embedding models now carry most of it. We also discovered the 6,000 latest headlines were one snapshot of that day's front pages, so the news washed out was today's rather than each claim's own.

E0.6

Athena

in use on the site

❓ With every new kind of evidence, how well does the fit sort the person's decisions?

The confidence model gives every filing a score: the chance a person will keep it after checking. That score orders the queue and decides which filings pass as sure. On the morning of October 8 it weighed only likenesses, scored AUC 0.82, and had stopped improving at about 150 decisions. That day we added new kinds of evidence: new finders, a reranker, and Gemma's yes or no. Athena put them all in to see if the fit got better.

The upshot AUC 0.885; new signals helped, but more checking alone did not.
Keep reading ✏️

🎯 What we expected

We expected the new evidence to lift AUC well above 0.82 and pass many more kept filings at 95% precision. We thought boosted trees would beat a plain logistic regression, and that hand-added filings used as extra examples would help.

🔬 What we found

On 749 hand decisions, the logistic regression with every signal scored AUC 0.885. At a fit of 85%, it passed 76% of kept filings with 95.1% precision. Boosted trees did no better, and hand-added filings used as extra examples didn't help (AUC 0.883). Both learners levelled off by about half the decisions: more checking alone moved the fit little, while new kinds of evidence moved it. The fit went live that midday and is still the one the site uses, retrained daily. The 76% fell to 66% that afternoon as new decisions came in.

E0.5

the Bremen Town Musicians

in use on the site

❓ Do other finders catch motifs the likenesses miss, alone or together?

Before the filing model can assign motifs to a claim, a shortlist narrows the index to about a dozen candidates; whatever it misses is lost forever. On October 8 the shortlist used six likeness scores from one embedding model, weighted together. We wondered whether other finders: the motif's note alone, keywords, a bare rewrite of the claim's shape, votes from its nearest filed claims, a second embedding model, or the person's motif groups, could catch motifs the likenesses overlooked.

The upshot Weighting all finders together raised the top-12 recall from 79.6% to 82.5%, and a daily retrained version reached 83.9%.
Keep reading ✏️

🎯 What we expected

Not written down beforehand; reconstructed: no finder alone would beat the shortlist, which was already a blend, but together they would find more, provided they were weighted on hand-confirmed filings; mixed without weights, the weak finders would drown the strong.

🔬 What we found

Scored by the share of a claim's hand-confirmed motifs that appeared in the top 12, on 891 confirmed filings: the day's shortlist hit 79.6%, the best single new finder (the note alone) 62.9%, all finders fused without weights 72.3%, and all of them weighted 82.5%. Both guesses held: weighted, the band added about three points; unweighted, it cost seven. The weighted blend went into production the same day as the learned shortlist, retrained daily and used only while it tests better (its own test then put 83.9% of motifs in the top12, against 79.8% for the plain weights). Each new claim costs two more questions of the filing model. A caveat: the answer key was found partly through the old shortlist, which favours finders that agree with it.

E0.4

Hansel and Gretel

tried, not kept

❓ Does showing motifs by their note, with their nearest claims, improve filing?

We tried showing the filing model each motif by its note instead of its name, and with the three claims closest to the one being filed rather than the latest three. The idea was that the note describes what a motif covers, and nearby claims give the model better examples. We ran 120 confirmed claims through both prompts on the same model and shortlists.

The upshot More cautious: found 57% of the person's motifs against 66%, wrong picks 21% against 26%
Keep reading ✏️

🎯 What we expected

Leading with the note would steer the model toward what a motif covers rather than a playful name, so it would find more of the person's motifs. Showing the nearest claims would give it the closest instances, so wrong picks would fall too.

🔬 What we found

The note-first prompt cut wrong picks from 26% to 21%, but it also missed more of the person's motifs, finding 57% against the current prompt's 66%. The model grew cautious: it saved five points of wrong picks but lost nine points of correct ones. Because we changed two things at once, we can't say which caused the drop. A follow-up test of nearest claims alone made no difference, pointing to the note-first framing as the likelier cause.

E0.3

the Emperor's New Clothes

in use on the site

❓ Does comparing a claim with the motif's note alone put more of the person's motifs in the shortlist?

We wanted to know if a motif's playful name was throwing off the shortlist, the step that picks the best story shapes for a news claim. The shortlist already compared a claim to six likenesses of each motif. We added a seventh: the claim against the motif's note alone, without the name. The idea was that the note is a cleaner description of what the motif covers, so it might help the person's own motifs rank higher.

The upshot Top 8 rose from 73.9% to 75.0%, top 12 unchanged at 79.3%; the note was already read, glued to the name. 73.9% → 75.0%
Keep reading ✏️

🎯 What we expected

We expected the playful names to pull the name-and-note embedding away from what the motif really covers. The note on its own would be a cleaner target, so we thought it would add a few points at both the top 8 and top 12.

🔬 What we found

The note alone made almost no difference. With just the shortlist's usual facts, 73.9% of the person's motifs landed in the top 8 and 79.3% in the top 12. Adding the note-alone likeness nudged the top 8 to 75.0%, but the top 12 stayed exactly the same at 79.3%. A single point at top 8 is within what a few claims can move, so it is not a real gain. The note was already being read, glued to the name, and the names did less harm than we feared.

E0.2

Sisyphus

in use on the site

❓ Can an untrained, off-the-shelf reranker help tell kept filings from removed ones?

The confidence model decides whether a filing stays or goes. It already uses likeness scores from an embedding model and a yes-or-no vote from a large language model we call the Oracle. We wondered whether a cheap, off-the-shelf reranker could add a third signal. A reranker reads the claim and the motif note together instead of comparing two separate summaries, and Qwen3-Reranker-0.6B is small enough to run on the same CPU as the filing model without slowing things down.

The upshot AUC reached 0.887 with the reranker, likenesses, and the Oracle's vote.
Keep reading ✏️

🎯 What we expected

We expected the reranker alone to sort worse than the Oracle, a 26-billion-parameter model, but clearly above chance. We also thought it would add a small boost when used alongside the likeness scores and the Oracle's vote, as a third kind of evidence.

🔬 What we found

The reranker alone scored an AUC of 0.766, where 0.5 is chance. That is weaker than the Oracle alone, but it still carried useful information. When we combined it with the likeness scores, AUC rose to 0.852. Adding the Oracle's yes-or-no vote pushed it to 0.887. For a loose comparison, the Oracle's own run that morning reached 0.879 with likenesses and its own vote on a different set of pairs. The reranker went into the confidence model and is in production. Because it was never trained on the person's decisions, teaching it that taste became the next experiment.

E0.1

the Oracle

tried, not kept

❓ Does Gemma's own yes or no sort filings as well as the likenesses?

The confidence model estimates how likely a filing is to be kept when checked by hand. On the morning of Oct 8 it weighed only likenesses, how close a claim and a motif sit by an embedding model's measure. It knew how alike things look, not whether a claim tells a motif, and it had stopped improving. The Oracle asked a model to read the pair: Gemma 4 26B's own yes or no, with the probability it gave.

The upshot Gemma matched the old method's sorting ability but was too confident in its wrong answers, so it was swapped for a faster model the next day.
Keep reading ✏️

🎯 What we expected

Not written down beforehand; reconstructed: alone, Gemma's yes would sort kept from removed filings about as well as the likenesses together; it would be wrong about different filings, so the two together would do better; and its probability would be poorly calibrated.

🔬 What we found

On 728 filings checked by hand, Gemma alone scored AUC 0.818, level with the likenesses' 0.817, and together they reached 0.879. The share of kept filings that could be passed without asking, at 95% precision, more than doubled, from 29% to 64%. But it was overconfident: its yes was right about 91% of the time at every probability from 0.5 to 0.99, so it says yes just as surely when wrong. It went into the confidence model the same day as one weighed signal, and the next day was replaced by Nimble 9B, which judged as well in a fifth of the time.

📅 Loose pages (no date kept)

?

S27

Horatius

in use on the site

❓ Cutting single-edge graph articulation bridges, SNN filtering, and temporal gating on Osiris and Nov 17

Evaluated CPU-only graph and temporal clustering refinements—interior single-edge articulation bridge cutting (bridge_cut), shared-nearest-neighbor (SNN) edge filtering, dual-threshold temporal gating, hard time-gating, and average-linkage—across both event grain (case_id:part) and saga grain (case_id) on osiris-events (4,071 cached mxbai headlines) and nov17-cases (1,235 cached mxbai headlines).

The upshot cutting interior single-edge articulation bridges (min_side>=4, cross_sim<0.72) lifts osiris-events EVENT F 0.376 to 0.427 (+0.051) and SAGA F 0.781 to 0.856 (+0.075), lifts nov17-cases EVENT F 0.341 to 0.348, and holds nov17-cases SAGA steady (0.729 to 0.725); shipped in app/analysis/clustering.py (cut_bridges) and app/analysis/joining.py 0.376 → 0.427
Keep reading ✏️

🎯 What we expected

Single-link 0.80 occasionally welds distinct events or sagas via a single weak bridge headline. Cutting interior articulation bridges (nx.bridges) where both sides have >= 4 nodes and cross-side mean cosine similarity < 0.72 will sever accidental merges without shattering genuine multi-day stories.

🔬 What we found

bridge_cut(ms=4, cross<0.72) improves or holds steady across every grain: osiris-events EVENT F rises 0.376 to 0.427 (+0.051), osiris-events SAGA F rises 0.781 to 0.856 (+0.075), nov17-cases EVENT F rises 0.341 to 0.348 (+0.007), and nov17-cases SAGA F holds at 0.725 vs 0.729. Time-gating (<=12h) and average-linkage lift EVENT F (0.733, 0.612 on osiris-events) but over-split multi-day sagas (0.424, 0.451). Shipped cut_bridges into app/analysis/clustering.py and app/analysis/joining.py.

?

F5

Loki

answered

❓ Can a story’s shape, retold about unrelated subjects, find its motif better than its words can?

When a new claim arrives, the machine draws up a shortlist of motifs it might belong to by comparing words. Words carry the topic, so a claim about a mayor looks like other claims about mayors, even when it tells a different story. Loki tried washing the topic out: Gemma retold each claim’s story about a small-town school board, a professional football club, and a medieval royal court. We averaged the three retellings, hoping only the story’s shape would survive and that shape would match motifs better than the plain words could.

The upshot Stripping a claim’s topic to keep only its story shape made motif matching worse, and combining it with the original words hurt accuracy by 6 percentage points.
Keep reading ✏️

🎯 What we expected

Written before anything was scored: Loki alone would find a claim’s motif less often than the plain claim and less often than Aesop. Added to the plain claim it would gain 1 point or less. We also hoped it would spot claims carved out of a motif as the odd one out more often than chance.

🔬 What we found

We held out each of 2,078 confirmed filings and asked how often the right motif landed in the top 12. Loki alone managed 41%, against 71% for the plain claim. It did beat Aesop (20%), the reverse of what we expected. Added to the plain claim by summing the two scores, it cost 6 points (65% against 71%) instead of helping. It also picked out carved claims less often than chance (11%, against 15%). An equal-weight sum let a noisy way of looking outvote a good one.

?

S26

Barnacles

answered

❓ Pruning stray barnacles and cutting multi-day story blobs on Nov 17 and Osiris

Evaluated two CPU-only graph post-processing steps—degree-1 centroid pruning (barnacle stripping) and Louvain community cuts—on the hand-sorted nov17-cases multi-day benchmark (1,965 headlines across Jensen Huang/Nvidia and Trump–Xi–Putin; 1,239 cached mxbai vectors) and the 24-hour osiris-events benchmark (4,071 cached mxbai vectors).

The upshot barnacle pruning (<0.68, zero core words) lifts nov17-cases Event F 0.342 to 0.359 while holding osiris-events steady (0.781 to 0.780); blind Louvain splits Nvidia's 13 developments (+0.155 F on nov17-cases) but over-splits 24h stories (0.781 to 0.673 on osiris-events) 0.342 → 0.359
Keep reading ✏️

🎯 What we expected

Pruning degree-1 peripheral headlines with low centroid similarity (< 0.68) and zero core-word overlap will strip stray barnacles without hurting 24-hour recall, while Louvain community cuts on the >= 0.80 graph will separate multi-day developments within Katamari story blobs.

🔬 What we found

On nov17-cases (1,239 cached mxbai headlines), baseline single-link 0.80 scores Event F=0.342 (P=0.209, R=0.946). Degree-1 barnacle pruning (<0.68, zero core words) improves Event F to 0.359 (P=0.222, R=0.930). Louvain (res=1.0) lifts Event F to 0.495 (P=0.507, R=0.484), and Louvain + prune scores F=0.495 (P=0.509, R=0.482). On osiris-events (4,071 cached headlines in 24h pools), barnacle pruning is safe (F=0.780 vs 0.781 baseline, P=0.724 vs 0.717), whereas blind Louvain over-splits normal 100+ headline stories (F=0.673, P=0.989, R=0.510). Barnacle pruning is safe to deploy; breaking multi-day Katamari blobs requires time-windowed development splitting rather than static Louvain.

?

U1b

the Sorcerer's Apprentice

answered

❓ Can a second pass split one broad motif into its distinct inner tellings?

We had an automated system, the Golem, that groups news stories into broad motifs. Left to run on its own, it made a few large, messy piles. The Sorcerer's Apprentice experiment asked whether a second pass could clean them up. We took all the claims inside one of the Golem's big motifs and fed them to a language model, Gemma 4 26B, with instructions to split them into the distinct, smaller tellings inside, each with its own who-did-what-to-whom.

The upshot A second pass with a language model split 63 messy story groups into 225 much smaller, cleaner ones, nearly tripling how often grouped claims truly belonged together. 0.19 → 0.54
Keep reading ✏️

🎯 What we expected

We expected that breaking the broad motifs apart would make the groupings much more precise, and that we would lose some related claims along the way, causing recall to fall somewhat.

🔬 What we found

Gemma split all 26 of the Golem's motifs that had six or more claims, turning 63 motifs into 225, with a median size of just 2 claims. The groupings became much more precise. B-cubed precision, which measures how often claims grouped together belong together, nearly tripled from 0.19 to 0.54. Recall, the share of related claims that were actually grouped, fell from 0.53 to 0.44. A caveat: precision naturally rises as groups get smaller, and we didn't run a random split of the same sizes to compare, so part of the gain may be from size alone.

?

V17

the Lost Sheep

answered

❓ A new development joins its big saga: the core check against four variants (the Lost Sheep)

The upshot key 140 (101 joins, 39 not); right/wrong joins: production 54/1, nearest-2 63/1, nearest-3 59/1, no core for young 69/2, newest parts 52/2; CORE_NEAREST=2 proposed, not shipped