We test our AI in public

How Earshot classifies mentions, and how we measure that it works

Every mention Earshot collects passes through an AI classifier before it reaches your dashboard. This page explains how that classifier is built, why we trust its output enough to alert you on it, and exactly how we measure its accuracy. It answers the question a careful buyer should ask: is this one generic call to a language model, or an engineered, measured system? It is the second, and here is what that means in practice.

Throughout, "the model" means Claude, the language model family from Anthropic. Earshot uses two of them: a small fast model for filtering and a large model for the judgment calls.

What the classifier decides

For every mention, the classifier returns one structured record with these fields:

• relevant: is this genuinely about your brand? A coffee chain named Acme should not see cartoon anvils, dictionary uses of the word, or a plumbing company that happens to share the name. Mentions of a competitor count only when they also say something about your brand, such as a switching story.
• sentiment: the writer's stance toward your brand: positive, negative, or neutral. Stance, not topic. A calm, factual complaint is negative. Sarcasm and backhanded praise are negative. A warning that protects your customers from a scam is neutral toward you.
• topic: a short phrase naming what the mention is about, such as "refund delay" or "store service", so mentions cluster into themes you can act on.
• severity: a 0 to 3 scale. 0 is noise, 1 is a mild grumble, 2 is a serious complaint or service failure, and 3 is reserved for crisis: safety or legal risk, regulator involvement, a data breach, a scam wave, or a clearly viral negative.
• language: the dominant language of the text. This matters because real mentions are not always written in one language at a time.
• needsAttention: should a human look at this? True for genuine complaints, safety and legal risks, scams and impersonation, churn threats, and crisis stories. False for praise, questions, and mild grumbles.
• confidence: the model's own calibrated certainty in the whole record. Low confidence is not hidden; it triggers a re-check.

The output is constrained to a fixed schema. The model cannot return a malformed record, an invented field, or a severity of 7. Anything that fails validation is rejected and retried later rather than guessed at.

How classification works: two stages plus a verify pass

Stage 1: the relevance gate. A fast, inexpensive model reads each mention and answers one question only: is this genuinely about the monitored brand? Homonyms, namesakes, and lookalike businesses are filtered out here. This is also what keeps costs sane: irrelevant mentions never reach the expensive model.

Stage 2: full classification. Mentions that pass the gate go to a more capable model that returns the full record above. Its instructions carry three things beyond the field definitions:

• Your project's context. Your brand terms, your competitor terms, and your industry are part of the prompt, so relevance and attribution are judged against your actual setup, not a generic idea of a brand.
• Language and slang lexicons. A general English lexicon teaches internet slang and sarcasm cues, the patterns that flip a positive-looking sentence into a complaint. Projects monitoring South African audiences also get a South African pack covering code-switching between English, Afrikaans, isiZulu, isiXhosa, Sesotho, Sepedi, Setswana, and tsotsitaal, the urban township slang. A mention that reads as calm English can carry a serious complaint in the embedded vernacular, and the classifier is taught to read it.
• Worked hard cases. The prompt includes a curated set of examples chosen because naive classifiers get them wrong: sarcastic praise, code-switched complaints, homonyms, competitor switching stories, scam warnings, and mixed-sentiment mentions. The model learns the decision boundary from boundary cases.

Stage 3: the verify pass. Any mention flagged as severity 2 or higher, marked as needing attention, or classified with low confidence is not trusted on a single pass. It is re-examined individually by the large model, which confirms or corrects every field. When the two passes disagree, the careful second look wins. This catches the expensive mistakes, a missed crisis or a false alarm, while leaving the bulk of easy, confident mentions to a single efficient pass.

How accuracy is measured

Engineering without measurement is just assertion. Earshot ships its evaluation harness in the same repository as the classifier, and the numbers on this page's companion report come from running it.

The gold set. Sixty mentions, each labelled by hand against the full schema. It is a synthetic-but-realistic benchmark: written to mirror real public mentions, grounded in real failure modes, and de-identified by design, so no real person's post is quoted. It deliberately over-samples the hard cases: sarcasm, mixed sentiment, competitor confusion, irrelevant homonyms, scams, crisis mentions, and a substantial South African code-switched slice, because those are the mentions where a classifier earns or loses its keep.

What the harness reports. It runs the real production pipeline, both stages and the verify pass, over the gold set and compares every field to the human label:

• accuracy per field, separately, so a weak field cannot hide behind a strong one: relevance, sentiment, severity (exact and within one band), language (including on the code-switched slice), and needs-attention;
• sentiment macro-F1, a score that weights each sentiment class equally so the rare classes count as much as the common ones;
• a confusion matrix showing exactly which sentiment gets mistaken for which;
• every single miss, printed with the text, the expected record, and what the classifier said instead.

That last item matters most. The harness does not just print a score; it shows its homework. Every mistake is inspectable, and every inspected mistake is a candidate for a new worked example in the prompt, which is how the classifier improves.

Where the numbers live. We do not print accuracy numbers on this page on purpose. They change whenever the model, the prompt, or the gold set changes, and a number frozen into marketing copy goes stale silently. The harness writes its results, dated and versioned with the model names, to a companion report, and that report is the only source of published accuracy figures. If you see an Earshot accuracy number anywhere, it came from a harness run.

Honest limitations

• The model is not fine-tuned. The domain knowledge lives in the prompt, in plain text anyone can read and audit, rather than frozen into model weights. We accept a somewhat lower ceiling in exchange for transparency and fast iteration.
• The gold set is sixty items. Large enough to catch real regressions and report honest per-field accuracy, small enough that the rarest categories carry wide uncertainty. It is a starting benchmark that grows from real classified mentions, not a finished one.
• The benchmark is synthetic. It mirrors real mention patterns but cannot capture every quirk of live data. As real mentions flow through, misses and edge cases are labelled and folded into the gold set.
• Some judgments are genuinely close. Two careful people can disagree on whether a complaint is a 1 or a 2, or whether a mixed mention is negative or neutral. The harness measures against one consistent labelling, and the misses list is where those close calls get debated and the rules tightened.
• Topics are free text. Topic phrasing is guided, not locked to a fixed list, so the harness reports topics for inspection rather than scoring them as right or wrong.

What this buys you

When Earshot alerts you at 2am about a severity 3 mention, that alert has passed a relevance gate, a full classification against your brand's own context, and an independent second look. When our methodology page cites an accuracy figure, you can trace it to a dated harness run over a labelled benchmark, with every miss on the record. That is the standard we think brand monitoring should be held to, and it is the one we hold ourselves to.