We test our AI in public
Reading a mention correctly is the hard part, and the part most tools ask you to take on faith. So here is the whole machine: a two-model cascade with a verify pass, a lexicon that reads code-switching, and a test set we score in the open. Every number comes from that test, never from a copywriter.
Latest eval run 2026-07-08
Gold set n=63
Relevance
98.4%
95% CI [92%, 100%] on 62 of 63
Real brand mentions kept, namesakes and lookalikes filtered out
Sentiment
94.9%
95% CI [86%, 98%] on 56 of 59
Stance toward the brand: positive, neutral or negative
Severity, within one band
100%
95% CI [94%, 100%] on 59 of 59
The 0 to 3 urgency scale, within one band of the human label
Code-switched language
100%
95% CI [82%, 100%] on 17 of 17
The slice that mixes South African languages mid-sentence
Gold set n=63 (63 scored, 0 dropped; 59 expected-relevant; code-switched slice n=17). Models under test: relevance gate claude-haiku-4-5, classification + verify claude-opus-4-8. These figures are written to this page by the eval harness and change only when it reruns. Each 95% CI is the range the true rate plausibly sits in given the sample size, so a small slice like the code-switched one carries a visibly wider band. It is derived from the same counts, not a separate measurement.
One mention, up to three reads, then measured
Reading a mention correctly is harder than collecting it. So Earshot does not make one call to a language model and hope. It runs a cascade: a cheap model filters the noise, a stronger model does the judgment, and a third read double-checks the mentions that matter most.
Every mention
News · reviews · social · forums
The raw stream we collect for your brand, before anything has been read or judged.
Relevance gate
claude-haiku-4-5A fast, low-cost first pass that decides one thing only: is this genuinely about you, or a namesake? Homonyms and lookalikes stop here.
Classification
claude-opus-4-8A stronger model reads what survived the gate and returns the full record: stance, topic, urgency, language, and whether a human should see it.
Verify pass
claude-opus-4-8Anything urgent or uncertain gets read a second time, on its own. When the two reads disagree, the careful second look wins.
Published metric
Versioned gold-set eval
The exact same pipeline is run over a hand-labelled test set, scored field by field, and the score is published on this page.
Why a cascade · Cost
Most of what gets collected is noise or namesakes. Filtering it with the cheap model first means the expensive model only ever reads mentions that are really about you.
Why a cascade · Latency
The gate clears the easy calls in bulk, so the slower, more careful model is reserved for the smaller set of mentions that actually need judgment.
Why a cascade · Accuracy where it counts
A missed crisis or a false alarm is the costly mistake. The verify pass spends extra effort on exactly those high-stakes items and leaves the easy, confident ones alone.
Collecting mentions is easy, reading them is the edge
The edge is judgment. A sentence that reads like calm English can carry a serious complaint in the vernacular underneath, so the stronger model is briefed for exactly that: your brand context, a lexicon, and worked examples.
A lexicon that reads code-switching
Code-switching is mixing languages inside one sentence, the way South African mentions actually read. Earshot ships two cue packs in plain text: 48 English slang and sarcasm cues, and 41 South African cues spanning Afrikaans, isiZulu, isiXhosa, Sesotho, Sepedi, Setswana and tsotsitaal, the urban township slang. They teach the model the words that flip a positive-looking sentence into a complaint.
Gold-set mention g004
“Bafo, Acme Coffee is giving me the runaround nge-order yami. Three weeks now, no delivery, the call centre just says 'we are escalating'. Ngeke, this is too much for a coffee machine.”
Cues the pack catches
ngeke never, no way (isiZulu; refusal or outrage)
giving me the runaround being deflected repeatedly (South African English complaint idiom)
Read as Negative · severity 2 · needs attention · dominant language English. This is g004 in the receipts below, read correctly in the latest run.
Output that cannot be malformed
Every call returns one fixed record, not free text: relevant, sentiment, topic, severity, language, needs-attention, and a confidence. The shape is pinned by a schema and re-checked by a validator, so a made-up field or a severity of 7 is rejected and retried, never shown to you.
Worked hard cases in the prompt
Twelve examples chosen because naive classifiers get them wrong ride in every call: sarcastic praise, code-switched complaints, cartoon namesakes, competitor-switching stories, scam warnings, mixed sentiment. Few-shot means the model learns the decision boundary from boundary cases, not from the easy middle.
Hand labels, blind runs, published misses
A gold set is a batch of mentions a person has already labelled by hand, the answer key. We run the classifier over it without letting it see those answers, then score every field against them.
- 01
A person labels the gold set
Real-style mentions, labelled by hand: about the brand or not, what the stance is, how urgent it is, what language it speaks.
- 02
The classifier runs blind
The exact pipeline that scores your mentions runs over the whole set. It never sees the answers.
- 03
Every miss goes on the record
The harness prints each mistake next to the expected answer. Misses become new test cases, and the dented score stays published.
How the labels were made
Today one labeller wrote the answer key, working to a fixed rule set so the calls are consistent rather than a matter of mood. A mention is relevant when it is genuinely about the brand, not a namesake or a competitor. Sentiment is the writer’s stance toward the brand, so sarcasm counts as negative and a scam warning that protects customers counts as neutral. Severity runs 0 to 3 by urgency: 0 is noise, 1 is a mild grumble, 2 is a real complaint or failure, 3 is a crisis such as a safety, legal, regulator or scam-wave case. Language is the primary language the mention is written in, and a code-switched item keeps its matrix language.
A single labeller is the honest limit of the current set. A second rater and a measured agreement score are planned, not yet done, so we do not quote one. When that lands it goes on this page like everything else.
No answers leak into the prompt
The classifier prompt carries a dozen worked examples that teach the hard cases. None of them is a gold-set mention. A test in the suite checks this on every run and fails the build if any gold text turns up among the worked examples, so the test set stays a genuine test and the accuracy above is judgment, not memory.
Built from the mentions classifiers get wrong
63 real-style mentions, written to mirror how public posts actually behave and weighted toward the hard ones: sarcasm that reads like praise, complaints that switch between English, Afrikaans, isiZulu and tsotsitaal mid-sentence, lookalike businesses that share your name, and scam warnings that protect your customers rather than attack you.
Easy mentions prove nothing. The set over-samples the boundary cases because that is where a classifier earns or loses its keep.
Honest limitations
- The model is not fine-tuned. The domain knowledge lives in the prompt, in plain text anyone can read and audit, rather than frozen into model weights. We accept a somewhat lower ceiling in exchange for transparency and fast iteration.
- The gold set is sixty-three items. Large enough to catch real regressions and report honest per-field accuracy, small enough that the rarest categories carry wide uncertainty. It is a starting benchmark that grows from real classified mentions, not a finished one.
- The benchmark is synthetic. It mirrors real mention patterns but cannot capture every quirk of live data. As real mentions flow through, misses and edge cases are labelled and folded into the gold set.
- Some judgments are genuinely close. Two careful people can disagree on whether a complaint is a 1 or a 2, or whether a mixed mention is negative or neutral. The harness measures against one consistent labelling, and the misses list is where those close calls get debated and the rules tightened.
- Topics are free text. Topic phrasing is guided, not locked to a fixed list, so the harness reports topics for inspection rather than scoring them as right or wrong.
Count the misses yourself
Here is the whole scorecard from the run dated 2026-07-08: every one of the 63 scored mentions, the label a person gave it, and the verdict the classifier returned. The misses are below in full, first. Every percentage on this page is just one of these counts.
Relevance
62 / 63
correct
95% CI [92%, 100%]
Sentiment
56 / 59
correct
95% CI [86%, 98%]
Severity, exact
57 / 59
correct
95% CI [88%, 99%]
Severity, within one
59 / 59
correct
95% CI [94%, 100%]
Language
58 / 59
correct
95% CI [91%, 100%]
Code-switched language
17 / 17
correct
95% CI [82%, 100%]
Needs-attention
57 / 59
correct
95% CI [88%, 99%]
Rows fully correct
57 / 63
correct
95% CI [81%, 96%]
The needs-attention alarm, judged as an alarm
For an alerting tool the number that matters is not overall accuracy, it is whether the alarm catches the urgent mentions and does not cry wolf. Recall is the share of genuinely urgent mentions the alarm caught. Precision is the share of the mentions it flagged that were genuinely urgent. Both are counted from the same scorecard below.
Recall, urgent caught
100%
Of the 22 mentions a person flagged urgent, Earshot caught 22. It missed 0.
95% CI [85%, 100%]
Precision, flags that held
92%
Of the 24 mentions Earshot flagged urgent, 22 genuinely were. 2 were routine, called urgent to be safe.
95% CI [74%, 98%]
The 6 misses, in full
Shown in full, and up top. On the urgent alarm specifically, none under-flagged, and 2 over-flagged a routine mention as urgent, the safer direction (g014, g027). The rest are sentiment or severity calls that left the alarm untouched. Each miss becomes a new worked example in the next prompt release.
“Quick question, how long before Acme Beans rewards points expire? Trying to understand the programme before I sign up.”
“Nobody explained that Acme Coffee's 'free month' auto-renews at full price. The signup screen rushed me through. I understand it now but it was not upfront, and that leaves a bad taste.”
“I understand prices go up with inflation, but a clear email from Acme Coffee explaining the increase would help. I only noticed when the debit was higher. Communication could be better.”
“I hit gold tier on Acme Beans rewards last month and was told I qualify for the birthday voucher, but nothing arrived. Emailed twice, no reply. It was a big reason I stayed with Acme Coffee.”
“URGENT: due to the Acme Coffee system upgrade you must reverify your account here to avoid losing your rewards points. Click the link and confirm your ID and card details now. Limited time only.”
“Acme Coffee has donated equipment to twelve township coffee startups as part of its incubator programme, the company announced at a ceremony in Khayelitsha. The programme also offers mentorship from senior roasters.”
Open the full scorecard, all 63 mentions
| Mention | Relevant | Sentiment | Severity | Language | Attention |
|---|---|---|---|---|---|
g001 · reviews Ordered an espresso machine from Acme Coffee three weeks ago. Still no delivery and still no refund. Every time I call they say the request is under review. I paid R4,000 for silence. | yes | negative | 2 | en | yes |
g002 · reviews · code-switched Sengikhathele. Bengithenga ku-Acme Coffee iminyaka eyisi-9 kodwa manje bayala ukungibuyisela imali, bathi isikhokhiwe. Asikholwa lokhu, sinabo ubufakazi bokukhokha. Shame on you AcmeCoffee. | yes | negative | 2 | zu | yes |
g003 · news Complaint escalated to the Consumer Goods and Services Ombud. Complainant alleges Acme Coffee continued charging a 12-month subscription after cancellation and refuses to reverse the debits. The company has been asked to respond within the prescribed period. | yes | negative | 3 | en | yes |
g004 · facebook · code-switched Bafo, Acme Coffee is giving me the runaround nge-order yami. Three weeks now, no delivery, the call centre just says 'we are escalating'. Ngeke, this is too much for a coffee machine. | yes | negative | 2 | en | yes |
g005 · reviews · code-switched Ndidiniwe yi-Acme Coffee. Umntu wabo wathi i-subscription ayinayo i-cancellation fee, kodwa ngoku bafuna i-R600 ukuyirhoxisa. Sasingazi ukuba kukho le fee, akukho apho yayibhalwe khona. | yes | negative | 2 | xh | yes |
g006 · google reviews · code-switched Acme Coffee lebenkeleng la Maseru le re emisitse hora e le nngwe bakeng sa dicappuccino tse pedi feela. Re ne re hloka thuso ka pele empa ho ne ho sa tsamaye letho ho fihlela re belaela ka lentswe le phahameng. | yes | negative | 1 | st | no |
g007 · x Reporting @acmecoffee to my bank. Subscription debit went off twice this month and support has ignored three emails. Cancelling today and never coming back. | yes | negative | 2 | en | yes |
g008 · x Order 'in preparation' for 45 minutes while my whole office waited. App says delivered, nothing arrived, no driver, no call. How is this acceptable, Acme Coffee? | yes | negative | 2 | en | yes |
g009 · facebook · code-switched Acme Coffee ba nkentse hampe, the bill was R200 more than the menu price. Ke kopa thuso, who do I escalate to? They keep saying prices changed mara we have the old receipt right here. | yes | negative | 2 | st | yes |
g010 · reviews Acme Coffee catering failed us completely. Booked and paid for my sister's baby shower, the order arrived two hours late and stone cold. The catering service is the whole reason we chose them. | yes | negative | 2 | en | yes |
g011 · reviews Ek het Acme Coffee se boonmasjien teruggestuur omdat dit stukkend aangekom het, en hulle weier om my geld terug te betaal. Agt weke se gestryery al. Ons gaan dit nie net los nie. | yes | negative | 2 | af | yes |
g012 · google reviews Store confirmed my refund was approved but two weeks later there is still no money in my account. The phone goes unanswered and the manager is 'in a meeting' every time. Exhausting. | yes | negative | 2 | en | yes |
g013 · facebook Quick question, how long before Acme Beans rewards points expire? Trying to understand the programme before I sign up. | yes → no | neutral | 0 | en → und | no |
g014 · reviews Nobody explained that Acme Coffee's 'free month' auto-renews at full price. The signup screen rushed me through. I understand it now but it was not upfront, and that leaves a bad taste. | yes | negative | 1 → 2 | en | no → yes |
g015 · x · code-switched Le subscription ya Acme Coffee e bitsa bokae ka kgwedi? Ke batla ho tseba pele ke saena le bona. Anyone who knows please ngiazise. | yes | neutral | 0 | st | no |
g016 · google reviews · code-switched The Acme Coffee Soweto staff were so patient when I came in to swap a bag of beans I bought by mistake. Sorted in ten minutes with a smile. Ngiyabonga, great service. | yes | positive | 0 | en | no |
g017 · google reviews Queued at the Sandton Acme Coffee for 25 minutes this morning with only two baristas working. The coffee was fine but please add more staff at peak times. | yes | negative | 1 | en | no |
g018 · google reviews Uitstekende diens by die Acme Coffee in Bloemfontein. Die barista het my bestelling perfek gemaak en die nuwe bone geduldig verduidelik. Baie dankie, ons kom beslis terug. | yes | positive | 0 | af | no |
g019 · reviews Barista at the Pretoria Acme Coffee was rude and dismissive when I asked about decaf options. Made me feel like a nuisance for asking. Do better with how you treat walk-ins. | yes | negative | 1 | en | no |
g020 · facebook · code-switched Acme Coffee increased the subscription price again this year and I am on a pension. Every year it goes up but the beans are the same. Eish, these increases are getting hard to afford. | yes | negative | 1 | en | no |
g021 · x My Acme Coffee-intekening het met meer as 10 persent gestyg hierdie jaar. Niemand het my vooraf laat weet nie. Hoekom so 'n groot sprong sonder kennisgewing? | yes | negative | 1 | af | no |
g022 · reviews I understand prices go up with inflation, but a clear email from Acme Coffee explaining the increase would help. I only noticed when the debit was higher. Communication could be better. | yes | neutral → negative | 1 | en | no |
g023 · facebook Acme Coffee took the subscription debit twice off my account this month, two days apart. Double debit on a tight month. Please refund the extra amount immediately. | yes | negative | 2 | en | yes |
g024 · reviews · code-switched Acme Coffee e ntse e nkgula chelete le ha ke khansetse subscription likhoeli tse tharo tse fetileng. Ke ntse ke lefa for coffee I no longer get. Please stop the deduction and pay me back. | yes | negative | 2 | st | yes |
g025 · x How do I set up an Acme Coffee office subscription for a team of 12? Do I go through the app or is there a business team? Just need the process, thanks. | yes | neutral | 0 | en | no |
g026 · facebook · code-switched Got my Acme Beans rewards voucher this month and I am so happy. After a year of buying it feels good to get something back for loyalty. Ngiyabonga Acme Coffee. | yes | positive | 0 | en | no |
g027 · reviews I hit gold tier on Acme Beans rewards last month and was told I qualify for the birthday voucher, but nothing arrived. Emailed twice, no reply. It was a big reason I stayed with Acme Coffee. | yes | negative | 1 → 2 | en | no → yes |
g028 · x · code-switched Le Acme Beans rewards ya bona e sebetsa jwang? Motswala wa ka o re o fumana i-free coffee after 10 orders. Is this true kapa ke leshano feela? | yes | neutral | 0 | st | no |
g029 · google reviews · code-switched I-Acme Coffee indithumelele umatshini we-espresso ukusuka eRhawutini ukuya eLusikisiki ngaphandle kwengxaki, kwaye umnxeba wokuseta wawuluncedo kakhulu. Enkosi kakhulu, yiyo le nto sibathembayo. | yes | positive | 0 | xh | no |
g030 · reviews The replacement-part voucher from Acme Coffee took forever to be issued and the approved courier list is very limited. We wanted a weekend delivery slot but were told it is not covered. More flexibility please. | yes | negative | 1 | en | no |
g031 · reviews When I phoned Acme Coffee in a panic because my machine died the day before I was hosting a family lunch, the consultant stayed on the line, talked me through a fix, and arranged a loaner. Small thing but it mattered. | yes | positive | 0 | en | no |
g032 · facebook · code-switched Phoned Acme Coffee support about my missing order and the agent was cold and just kept reading a script. Bekungasekho lutho lobubele. Sometimes you need a human, not a robot. Please train your line better. | yes | negative | 1 | en | no |
g033 · news Acme Coffee confirms it is investigating a payment system outage that disrupted its app this week. The company says it took systems offline as a precaution while services are restored. No comment yet on whether any card data was affected. | yes | neutral | 2 | en | yes |
g034 · x Been trying to order on the Acme Coffee app for three days and it just spins. The store says systems are down. I need to change my delivery address and I cannot do anything. Sort it out please. | yes | negative | 2 | en | yes |
g035 · facebook URGENT: due to the Acme Coffee system upgrade you must reverify your account here to avoid losing your rewards points. Click the link and confirm your ID and card details now. Limited time only. | yes | negative → neutral | 3 | en | yes |
g036 · x · code-switched Heads up to all Acme Coffee customers: after the outage there are fake SMSes going around asking for card details. Acme will never ask for your PIN. Ningakhohliswa, do not click those links. | yes | neutral | 2 | en | yes |
g037 · instagram Loved seeing the Acme Coffee youth barista academy graduation this year. Fifty young people trained and placed in jobs across three provinces. This is what giving back looks like. | yes | positive | 0 | en | no |
g038 · news Acme Coffee has donated equipment to twelve township coffee startups as part of its incubator programme, the company announced at a ceremony in Khayelitsha. The programme also offers mentorship from senior roasters. | yes | positive → neutral | 0 | en | no |
g039 · x Applied for the Acme Coffee incubator but the portal kept timing out and I never got a confirmation my entry was received. Frustrating, the deadline is close. Did anyone else struggle? | yes | negative | 1 | en | no |
g040 · facebook · code-switched Ndiyayithanda i-Acme Coffee incubator. Abantu abatsha bakuthi bafumana amathuba okuqala amashishini ekofu kwiilali zethu. Halala, oku kukuphuhlisa uluntu ngenene. | yes | positive | 0 | xh | no |
g041 · google reviews The Acme Beans rewards have actually saved me money on my morning coffee and even on groceries at partner stores. Did not expect that from a coffee chain. Pleasantly surprised. | yes | positive | 0 | en | no |
g042 · facebook · code-switched Sebenzisani i-Acme Beans rewards bantu, ama-voucher ayasebenza ngempela. Ngithenge ikhofi ngama-points inyanga yonke. Worth it for something we already buy every day. Highly recommend. | yes | positive | 0 | zu | no |
g043 · reviews My family has bought from Acme Coffee for three generations, back when it was one roastery in Durban. When my machine died they replaced it under warranty with zero fuss. Loyal for life. | yes | positive | 0 | en | no |
g044 · google reviews · code-switched Lilemo tse 20 le Acme Coffee mme ha ho mohla ke bileng le bothata. Ke ba tshepa ka pelo yohle. Ha order ya ka e ile ya lahleha hang, ba e lokisitse ka lona letsatsi leo. Nke ke ka ya kae kapa kae. | yes | positive | 0 | st | no |
g045 · x Honestly Acme Coffee has been solid for my family for years. People only post when they are angry but I want to say thank you, the subscription has never missed a delivery. Credit where it is due. | yes | positive | 0 | en | no |
g046 · x Wow Acme Coffee, only three weeks to refund a cancelled order, truly record breaking speed. Absolutely worth the four phone calls and two 'escalations'. 10/10 would wait again. | yes | negative | 2 | en | yes |
g047 · x The new Acme Coffee app update is 'amazing'. Love logging in three separate times to order one flat white. Great job guys, really nailed it. | yes | negative | 1 | en | no |
g048 · reviews Acme Coffee's beans are genuinely the best I have had, but the checkout crashes every single time I try to pay with Apple Pay. Two updates now and still broken. Fix it and you are perfect. | yes | negative | 1 | en | no |
g049 · x Switched from Acme Coffee to Bravo Beans last month. Half the price, twice the quality, and their app actually works. Should have done it years ago. | yes | negative | 2 | en | yes |
g050 · x Bravo Beans just opened a second location in Austin and the queue was out the door all morning. Best cold brew in town, no contest. | no | not scored | not scored | not scored | not scored |
g051 · reddit Rewatching old Looney Tunes and the Acme anvil catalogue still makes me laugh every time. Wile E. Coyote never learns, and I hope he never does. | no | not scored | not scored | not scored | not scored |
g052 · news Reaching the acme of his career, the striker signed a five-year deal last night and will captain the side from August. Fans queued overnight for shirts. | no | not scored | not scored | not scored | not scored |
g053 · google reviews Acme Plumbing quoted me $900 to replace a single valve and then charged a callout fee on top. Absolute rip-off, avoid these guys. | no | not scored | not scored | not scored | not scored |
g054 · news The specialty coffee market remains crowded, with chains like Acme Coffee and Java Junction both expanding subscription offerings, according to an industry review released on Tuesday. | yes | neutral | 0 | en | no |
g055 · facebook My daughter's hot chocolate from Acme Coffee had a shard of glass in it. The manager shrugged and offered a free muffin. Doctor visit done, reporting to trading standards, and I will post this everywhere until someone senior responds. | yes | negative | 3 | en | yes |
g056 · x This Acme Coffee thread is wild, 4k retweets in an hour. They charged a food bank triple the quoted price for an event and refused to refund the difference. Read the whole thing before you order again. | yes | negative | 3 | en | yes |
g057 · google reviews Popped into the Acme Coffee on Shoreditch High Street and the barista remembered my order from last week. Lovely little chain, deserves the hype. | yes | positive | 0 | en | no |
g058 · instagram Queue was long at Acme Coffee this morning, not gonna lie, but the flat white was worth every minute. The new Rwandan roast is superb, easily their best this year. | yes | positive | 0 | en | no |
g059 · x Ngl I came here to complain about my late Acme Coffee order but support refunded me before I even finished typing the DM. Okay, that is how you do customer service. | yes | positive | 0 | en | no |
g060 · reddit Does the Acme Coffee subscription ship to Ireland? Their site only lists UK postcodes but I have seen Irish unboxing videos. Anyone here actually receiving it in Dublin? | yes | neutral | 0 | en | no |
g061 · app store Rated 1/5 stars. The app charged me twice for the monthly subscription and three weeks later I still have no refund. Every ticket gets closed with a canned reply. Do not link a card to this. | yes | negative | 2 | en | yes |
g062 · app store Rated 5/5 stars. Ordering my Acme Coffee is one tap, rewards apply automatically, and the pickup timing is always spot on. Easily the smoothest coffee app I have used. Keep it up. | yes | positive | 0 | en | no |
g063 · google play Rated 3/5 stars. The Acme Coffee beans are genuinely excellent and delivery is quick, but the app logs me out every couple of days and I have to re-enter my card each time. Fix the login and this is five stars. | yes | negative | 1 | en | no |
Severity is scored on the strict 0 to 3 scale here; the headline “within one band” figure counts a call that lands one band off as close. Language is scored by primary language, so a mention gated out as not about the brand shows its language unset. Both are the harness’s real rules, shown as they run.
The honest fence around the numbers
Every accuracy page should be as clear about its limits as its results. Here is what Earshot does not claim, on purpose.
We only read what is already public
Earshot reads mentions that are already out in the open: reviews, news, social, forums. It does not read private inboxes, direct messages, or anything behind a login you have not connected yourself.
The numbers are our tested accuracy, not a guarantee on your brand
Everything on this page is measured on our gold set of 63 mentions. Your brand, your languages, and your mix of channels will differ. Treat these as how the classifier scores on a hard, fixed test, not a promise about your account.
The model is not fine-tuned on your data
The domain knowledge lives in a prompt you could read, not frozen into model weights. We prefer that for transparency and fast fixes, and we accept a slightly lower ceiling in exchange.
We watch the record behind AI answers
AI assistants learn about your brand from your public reviews and mentions, which is what Earshot reads. We do not yet query ChatGPT or other assistants live and show you their answer, and we will not claim to until that feature ships.
The benchmark is a starting point, and we say so
The gold set is synthetic-but-realistic and deliberately small: big enough to catch regressions and report honest per-field accuracy, small enough that the rarest categories carry real uncertainty. It grows as live misses get labelled and folded in.
The paper trail
The complete methodology document and the latest eval report, exactly as they ship in the repository alongside the classifier they describe.
Read the full methodology
How Earshot classifies mentions, and how we measure that it works
Every mention Earshot collects passes through an AI classifier before it reaches your dashboard. This page explains how that classifier is built, why we trust its output enough to alert you on it, and exactly how we measure its accuracy. It answers the question a careful buyer should ask: is this one generic call to a language model, or an engineered, measured system? It is the second, and here is what that means in practice.
Throughout, "the model" means Claude, the language model family from Anthropic. Earshot uses two of them: a small fast model for filtering and a large model for the judgment calls.
What the classifier decides
For every mention, the classifier returns one structured record with these fields:
- relevant: is this genuinely about your brand? A coffee chain named Acme should not see cartoon anvils, dictionary uses of the word, or a plumbing company that happens to share the name. Mentions of a competitor count only when they also say something about your brand, such as a switching story.
- sentiment: the writer's stance toward your brand: positive, negative, or neutral. Stance, not topic. A calm, factual complaint is negative. Sarcasm and backhanded praise are negative. A warning that protects your customers from a scam is neutral toward you.
- topic: a short phrase naming what the mention is about, such as "refund delay" or "store service", so mentions cluster into themes you can act on.
- severity: a 0 to 3 scale. 0 is noise, 1 is a mild grumble, 2 is a serious complaint or service failure, and 3 is reserved for crisis: safety or legal risk, regulator involvement, a data breach, a scam wave, or a clearly viral negative.
- language: the dominant language of the text. This matters because real mentions are not always written in one language at a time.
- needsAttention: should a human look at this? True for genuine complaints, safety and legal risks, scams and impersonation, churn threats, and crisis stories. False for praise, questions, and mild grumbles.
- confidence: the model's own calibrated certainty in the whole record. Low confidence is not hidden; it triggers a re-check.
The output is constrained to a fixed schema. The model cannot return a malformed record, an invented field, or a severity of 7. Anything that fails validation is rejected and retried later rather than guessed at.
How classification works: two stages plus a verify pass
Stage 1: the relevance gate. A fast, inexpensive model (claude-haiku-4-5) reads each mention and answers one question only: is this genuinely about the monitored brand? Homonyms, namesakes, and lookalike businesses are filtered out here. This is also what keeps costs sane: irrelevant mentions never reach the expensive model.
Stage 2: full classification. Mentions that pass the gate go to a more capable model (claude-opus-4-8) that returns the full record above. Its instructions carry three things beyond the field definitions:
- Your project's context. Your brand terms, your competitor terms, and your industry are part of the prompt, so relevance and attribution are judged against your actual setup, not a generic idea of a brand.
- Language and slang lexicons. A general English lexicon teaches internet slang and sarcasm cues, the patterns that flip a positive-looking sentence into a complaint. Projects monitoring South African audiences also get a South African pack covering code-switching between English, Afrikaans, isiZulu, isiXhosa, Sesotho, Sepedi, Setswana, and tsotsitaal, the urban township slang. A mention that reads as calm English can carry a serious complaint in the embedded vernacular, and the classifier is taught to read it.
- Worked hard cases. The prompt includes a curated set of examples chosen because naive classifiers get them wrong: sarcastic praise, code-switched complaints, homonyms, competitor switching stories, scam warnings, and mixed-sentiment mentions. The model learns the decision boundary from boundary cases.
Stage 3: the verify pass. Any mention flagged as severity 2 or higher, marked as needing attention, or classified with low confidence is not trusted on a single pass. It is re-examined individually by the large model, which confirms or corrects every field. When the two passes disagree, the careful second look wins. This catches the expensive mistakes, a missed crisis or a false alarm, while leaving the bulk of easy, confident mentions to a single efficient pass.
How accuracy is measured
Engineering without measurement is just assertion. Earshot ships its evaluation harness in the same repository as the classifier, and the numbers on this page's companion report come from running it.
The gold set. Sixty-three mentions, each labelled by hand against the full schema. It is a synthetic-but-realistic benchmark: written to mirror real public mentions, grounded in real failure modes, and de-identified by design, so no real person's post is quoted. It deliberately over-samples the hard cases: sarcasm, mixed sentiment, competitor confusion, irrelevant homonyms, scams, crisis mentions, and a substantial South African code-switched slice, because those are the mentions where a classifier earns or loses its keep.
What the harness reports. It runs the real production pipeline, both stages and the verify pass, over the gold set and compares every field to the human label:
- accuracy per field, separately, so a weak field cannot hide behind a strong one: relevance, sentiment, severity (exact and within one band), language (including on the code-switched slice), and needs-attention;
- sentiment macro-F1, a score that weights each sentiment class equally so the rare classes count as much as the common ones;
- a confusion matrix showing exactly which sentiment gets mistaken for which;
- every single miss, printed with the text, the expected record, and what the classifier said instead.
That last item matters most. The harness does not just print a score; it shows its homework. Every mistake is inspectable, and every inspected mistake is a candidate for a new worked example in the prompt, which is how the classifier improves.
Where the numbers live. We do not print accuracy numbers on this page on purpose. They change whenever the model, the prompt, or the gold set changes, and a number frozen into marketing copy goes stale silently. The harness writes its results, dated and versioned with the model names, to a companion report, and that report is the only source of published accuracy figures. If you see an Earshot accuracy number anywhere, it came from a harness run.
How your corrections make it better, without training a model
The classifier is good out of the box, but no generic system knows your brand's edge cases the way you do: your slang, your regulars' sarcasm, the lookalike business that shares your name. So Earshot lets anyone on your team correct a mention that was read wrong, straight from the mentions list: fix whether it was really about you, fix the sentiment, fix the severity. Three honest things happen when you do.
- Your correction becomes your private labelled set. It is your data, scoped to your workspace, and no other customer ever sees it. Over time it becomes a benchmark of how your brand's mentions should read, written by the people who know.
- The classifier reads your recent corrections as worked examples. When Earshot next classifies your mentions, your own corrected hard cases are added to the prompt alongside the general ones, so the model sees exactly how your team wants these calls made. This is the same mechanism as the curated worked examples above, pointed at your brand.
- We can measure agreement on your brand specifically. Corrections let Earshot compare, on your real mentions, how often the AI's first read matches what your team decided.
To be exact about what this is and is not: we do not fine-tune or retrain an AI model on your data. The model's weights never change; Claude is not fine-tunable through its API, and we do not pretend otherwise. Your corrections improve results by better prompting, plain-text examples the model reads, not by frozen training. That keeps the whole thing transparent and reversible, and it means the asset you are building is your labelled data, which stays yours.
Honest limitations
- The model is not fine-tuned. The domain knowledge lives in the prompt, in plain text anyone can read and audit, rather than frozen into model weights. We accept a somewhat lower ceiling in exchange for transparency and fast iteration.
- The gold set is sixty-three items. Large enough to catch real regressions and report honest per-field accuracy, small enough that the rarest categories carry wide uncertainty. It is a starting benchmark that grows from real classified mentions, not a finished one.
- The benchmark is synthetic. It mirrors real mention patterns but cannot capture every quirk of live data. As real mentions flow through, misses and edge cases are labelled and folded into the gold set.
- Some judgments are genuinely close. Two careful people can disagree on whether a complaint is a 1 or a 2, or whether a mixed mention is negative or neutral. The harness measures against one consistent labelling, and the misses list is where those close calls get debated and the rules tightened.
- Topics are free text. Topic phrasing is guided, not locked to a fixed list, so the harness reports topics for inspection rather than scoring them as right or wrong.
What this buys you
When Earshot alerts you at 2am about a severity 3 mention, that alert has passed a relevance gate, a full classification against your brand's own context, and an independent second look. When our methodology page cites an accuracy figure, you can trace it to a dated harness run over a labelled benchmark, with every miss on the record. That is the standard we think brand monitoring should be held to, and it is the one we hold ourselves to.
Latest eval report
Earshot classifier eval
Generated by npm run eval (lib/ai/eval/run.ts). Do not edit by hand; re-run the harness to refresh. These are the only accuracy numbers that may be published anywhere.
- Date: 2026-07-08
- Gold set: n=63 (63 scored, 0 dropped; 59 expected-relevant; code-switched slice n=17)
- Models: relevance gate
claude-haiku-4-5, classification + verifyclaude-opus-4-8
Per-field accuracy
| Field | Accuracy |
|---|---|
| relevant | 98.4% |
| sentiment | 94.9% |
| severity (exact) | 96.6% |
| severity (within 1) | 100.0% |
| language (prefix match) | 98.3% |
| language, code-switched slice | 100.0% |
| needsAttention | 96.6% |
Sentiment macro-F1: 0.927
Sentiment confusion matrix
Rows are the gold label, columns are the prediction.
| gold / pred | positive | neutral | negative |
|---|---|---|---|
| positive | 16 | 1 | 0 |
| neutral | 0 | 8 | 1 |
| negative | 0 | 1 | 32 |
Misses
6 of 63 scored rows had at least one field off. Run npm run eval for the full inspectable list with texts.
See it judge a real brand, live
The public demo monitors a brand you know, on genuinely pulled public data, refreshed daily.