For a long time, we knew our scripture detection was good. We didn't know how good. We didn't know if last month's tweak made it better or worse. We didn't know precisely where it failed, only that it sometimes did, and that when it did, it was in front of a real congregation, mid-sermon, with no second take.
That's an uncomfortable place to build from. So before we touched a single line of ranking code, we built something that could tell us the truth: an eval.
The problem with vibes
Pewbeam listens to a sermon and tries to put the right verse on screen at the right moment. Underneath, three detectors vote: a regex that catches an explicit "John 3:16," an n-gram matcher that catches a near-verbatim quote, and a semantic search that tries to catch everything else: paraphrase, allusion, "you know, the one about the prodigal son." That third one, transcription_semantic, does the most work and takes the most risk. It's also the one nobody could put a number on.
We had a 200-case test set sitting in the repo, hand-labeled months earlier, and a scoring script to go with it. Both were dead. The script talked to a Python server we'd since deleted in a native-Rust rewrite. Good instincts, no way to act on them. So step one wasn't "improve the model." It was: build a thing that can tell us if we improved the model.
What we built
A small Rust binary, feature-gated so it never ships in the app, that calls the exact same detection code the live product runs. No HTTP hop, no mock, no drift between what we measure and what a church actually experiences. It runs our labeled cases through the real semantic, shingle, and direct-reference detectors and reports hit@1/3/5, false-positive rate, and latency, broken down by category: direct quotes, paraphrases, partial quotes, context-dependent references, multi-verse passages, noisy transcription, and deliberate negatives.
The first number it gave us was humbling.
Direct quotes: nearly perfect. Paraphrase, someone describing a verse in their own words, the way real sermons actually sound: 46%. Context-dependent references, worse.
Then we found something more interesting than the number: why. We traced actual failing cases through the ranking code and found that the correct verse was usually already there, in the candidate pool, just outranked. It was buried under a scoring formula that rewarded exact word overlap, which is precisely the wrong instinct for a paraphrase that, by definition, doesn't share words with the source text.
One failing case made the problem obvious: "steal, kill, destroy," an almost word-for-word match for John 10:10, got outranked by Matthew 4:11 for sharing one incidental word: "devil."
We found a real bug hiding in plain sight: a tokenizer that didn't strip punctuation, silently failing on every query with a comma in it, which is to say, on ordinary speech. We found our n-gram matcher confidently returning the wrong verse a quarter of the time on paraphrased input, not because it was bad at paraphrase, but because it was never built for paraphrase and had no way to say, "I don't know."
Once you can see a failure mode clearly, fixing it stops being guesswork. We rebalanced how much weight lexical overlap gets versus raw semantic signal, taught the n-gram matcher to require real density of evidence instead of scoring a lucky coincidence, and added a lightweight reranking pass. We deliberately scoped it to the candidate queue only, never the live broadcast decision, because getting it wrong live is a cost we're not willing to trade for a few points of recall.
We ran it, and ran it again, because it turns out our own semantic model gives slightly different scores from run to run due to floating-point noise in the inference engine, and we didn't want to celebrate a change that was really just normal variation. Pulling in eight independent runs from earlier in the day, we could finally tell signal from noise with real confidence.
Paraphrase detection: 46% → 77%. Context-dependent references: 37% → 68%. False positives on the n-gram matcher's weakest categories: cut roughly in half. And the part we checked most carefully, by hand, line by line: none of it touched the auto-publish decision that puts something directly on a screen in front of a church. That stays exactly as conservative as it was.
Why this matters more than the numbers
The numbers are good. The habit is the real win. We now have a standing answer to "did we make it better?" Not a feeling, not an anecdote from one demo, but a number we can compare run over run, with the noise accounted for. Every future change to detection, a new fusion weight, a fine-tuned model, a different rerank strategy, gets judged against this before it ever reaches a live service.
That changes what kind of work is possible. It's the difference between tuning by intuition and measuring the effect of each change properly.
Where this goes
Right now, the eval runs on 200 cases we wrote by hand. That's good enough to find real bugs, but not yet good enough to claim we understand every way a sermon can be misheard. Next, we're grounding it in reality: labeling real sermon transcripts, generating noise patterns from documented transcription errors instead of guessed typos, and mining real operator behavior, the moments someone in a live service swaps out a verse we got wrong, as ground truth we didn't have to write ourselves.
Sermon audio and transcript content never leave the device, full stop. The anonymous usage analytics that make signals like operator overrides mineable at all contain no sermon content, just detection metadata like verse references and scores. They're on by default and can be turned off in Settings.
Longer term, we want this to stop being our eval and start being the eval: a benchmark we publish, with a fixed test set and a scoring spec, so anyone building scripture detection can measure themselves against it, and so our own numbers can be tested publicly. Getting the right words of scripture in front of the right congregation at the right moment isn't a metric we optimize once. It's a standard we intend to keep raising, in the open, for as long as we're building this.
We used to ship changes and hope. Now we ship changes and know. That reflects a much bigger shift in how we build.
Check out Pewbeam, we have free 60 minutes for you: pewbeam.com
