Accuracy
Watch our AI read a review
We gave it 63 posts a person had already sorted by hand. Here is every answer it gave.
- 62 of 63
- Is it about the business?
- 56 of 59
- Positive, neutral or negative?
- 57 of 59
- Does someone need to act?
Last tested 8 July 2026
Post 1 of 63
Test post, Review site
Ordered an espresso machine from Acme Coffee three weeks ago. Still no delivery and still no refund. Every time I call they say the request is under review. I paid R4,000 for silence.
Every answer matched the person's
How the AI read it
- About the business?Yes
- ToneNegative
- How serious2, serious
- Main languageEnglish
- Does someone need to act?Yes
- What it is aboutRefund delay
How the test works
Nobody types these numbers in. They come straight from the test run.
Step 1
A person sorts the posts
Each test post is labelled by hand: about you or not, positive or negative, how serious, which language.
Step 2
The AI sorts them blind
The same AI that reads your reviews sorts every post without seeing the answers.
Step 3
We publish every miss
Each mistake is shown in full next to the answer the person gave.
The posts were written to mirror real reviews and mentions, including hard cases: sarcasm, lookalike business names and posts that switch language mid-sentence.
For technical readers
How a post is read
A fast model filters, a stronger one classifies, a third read checks.
Reading a mention correctly is harder than collecting it. So a cheap model filters the noise, a stronger model does the judgment, and a third read double-checks the mentions that matter most.
Intake
Every mention
News, reviews, social, forums
The raw stream we collect for your brand, before anything has been read or judged.
Stage 1
Relevance gate
claude-haiku-4-5
A fast, low-cost first pass that decides one thing only: is this genuinely about you, or a namesake?
Stage 2
Classification
claude-opus-4-8
A stronger model returns the full record: stance, topic, urgency, language, and whether a human should see it.
Stage 3
Verify pass
claude-opus-4-8
Anything urgent or uncertain is read a second time. When the two reads disagree, the careful second look wins.
Measured
Published metric
Versioned test set
The same pipeline runs over a hand-labelled test set, scored field by field, and the score is published here.
Cost. Most of what gets collected is noise or namesakes. The cheap model filters it first, so the expensive model only reads mentions really about you.
Speed. The gate clears the easy calls in bulk, so the slower, careful model is saved for the mentions that need judgment.
Accuracy where it counts. A missed crisis or a false alarm is the costly mistake. The verify pass spends extra effort on exactly those.
How it reads local language
A word list and worked examples for posts that mix languages.
A sentence that reads like calm English can carry a serious complaint in the vernacular underneath. So the stronger model is briefed with your brand context, a lexicon and worked examples.
A lexicon that reads code-switching
Code-switching is mixing languages inside one sentence, the way South African mentions often read. Earshot ships two cue packs in plain text: 48 English slang and sarcasm cues, and 41 South African cues spanning Afrikaans, isiZulu, isiXhosa, Sesotho, Sepedi, Setswana and tsotsitaal, the urban township slang.
Test set mention g004
“Bafo, Acme Coffee is giving me the runaround nge-order yami. Three weeks now, no delivery, the call centre just says 'we are escalating'. Ngeke, this is too much for a coffee machine.”
Cues the pack catches
ngeke never, no way (isiZulu; refusal or outrage)
giving me the runaround being deflected repeatedly (South African English complaint idiom)
Read as
- Sentiment
- Negative
- Severity
- 2
- Needs attention
- Yes
- Language
- English
Output that cannot be malformed
Every call returns one fixed record: relevant, sentiment, topic, severity, language, needs-attention and a confidence. A schema and a validator reject anything malformed, so a severity of 7 is retried, never shown to you.
Worked hard cases in the prompt
Twelve examples that naive classifiers get wrong ride in every call: sarcastic praise, code-switched complaints, namesakes, competitor switching, scam warnings and mixed sentiment.
How the labels were made
One labeller, a fixed rule set, and no test post in the prompt.
One labeller wrote the answer key, working to a fixed rule set so the calls are consistent. A mention is relevant when it is genuinely about the brand, not a namesake or a competitor. Sentiment is the writer’s stance toward the brand, so sarcasm counts as negative and a scam warning that protects customers counts as neutral. Severity runs 0 to 3: 0 is noise, 1 a mild grumble, 2 a real complaint, 3 a crisis such as a safety, legal, regulator or scam-wave case. Language is the primary language of the mention.
A single labeller is the honest limit of the current set. A second rater and a measured agreement score are planned, not yet done, so we do not quote one.
No answers leak into the prompt
The prompt carries a dozen worked examples. None of them is a test post, and a test fails the build if one ever turns up, so the accuracy comes from judgment rather than memory.
Built from the mentions AI gets wrong
63 real-style mentions, written to mirror how public posts behave and weighted towards the hard ones: sarcasm that reads like praise, complaints that switch between English, Afrikaans, isiZulu and tsotsitaal mid-sentence, lookalike businesses that share your name, and scam warnings that protect your customers.
Easy mentions prove nothing, so the set over-samples the boundary cases.
The full scorecard
Every measure with its likely range, and every post in one table.
The whole scorecard from the test run on 8 July 2026: all 63 scored posts, the label a person gave each one, and the answer the AI returned. Every count on this page comes from this list.
About the business or not
62 of 63
Likely 92% to 100%
Positive, neutral or negative
56 of 59
Likely 86% to 98%
Seriousness, exact score
57 of 59
Likely 88% to 99%
Seriousness, within one step
59 of 59
Likely 94% to 100%
Main language
58 of 59
Likely 91% to 100%
Main language, mixed posts
17 of 17
Likely 82% to 100%
Needs someone to act
57 of 59
Likely 88% to 99%
Every answer right
57 of 63
Likely 81% to 96%
“Likely” is the 95% confidence interval: the range the true rate probably sits in, given the number of posts tested.
Judging the alert as an alert
For alerts, two things matter: catching every post that needs action, and not crying wolf. Recall measures the first and precision the second. Both are counted from the scorecard below.
Recall
22 of 22
posts that needed action were flagged. That is 100%, likely 85% to 100%.
Precision
22 of 24
flagged posts really needed action. The other 2 were routine, flagged to be safe. That is 92%, likely 74% to 98%.
The 6 misses, in full
None of the misses failed to flag a post that needed action. 2 flagged a routine post as needing action, the safer kind (g014, g027). The rest did not change whether a post would be flagged.
“Quick question, how long before Acme Beans rewards points expire? Trying to understand the programme before I sign up.”
- About the business: the person said yes, the AI said no
- Language: the person said en, the AI said und
“Nobody explained that Acme Coffee's 'free month' auto-renews at full price. The signup screen rushed me through. I understand it now but it was not upfront, and that leaves a bad taste.”
- Seriousness: the person said 1, the AI said 2
- Needs action: the person said no, the AI said yes
“I understand prices go up with inflation, but a clear email from Acme Coffee explaining the increase would help. I only noticed when the debit was higher. Communication could be better.”
- Sentiment: the person said neutral, the AI said negative
“I hit gold tier on Acme Beans rewards last month and was told I qualify for the birthday voucher, but nothing arrived. Emailed twice, no reply. It was a big reason I stayed with Acme Coffee.”
- Seriousness: the person said 1, the AI said 2
- Needs action: the person said no, the AI said yes
“URGENT: due to the Acme Coffee system upgrade you must reverify your account here to avoid losing your rewards points. Click the link and confirm your ID and card details now. Limited time only.”
- Sentiment: the person said negative, the AI said neutral
“Acme Coffee has donated equipment to twelve township coffee startups as part of its incubator programme, the company announced at a ceremony in Khayelitsha. The programme also offers mentorship from senior roasters.”
- Sentiment: the person said positive, the AI said neutral
Open the full scorecard, all 63 posts
A crossed-out label is the person’s answer, and the red one beside it is what the AI said instead. Everything else matched.
| Post | About the business | Sentiment | Seriousness | Language | Needs action |
|---|---|---|---|---|---|
g001 reviews Ordered an espresso machine from Acme Coffee three weeks ago. Still no delivery and still no refund. Every time I call they say the request is under review. I paid R4,000 for silence. | yes | negative | 2 | en | yes |
g002 reviews, mixed language Sengikhathele. Bengithenga ku-Acme Coffee iminyaka eyisi-9 kodwa manje bayala ukungibuyisela imali, bathi isikhokhiwe. Asikholwa lokhu, sinabo ubufakazi bokukhokha. Shame on you AcmeCoffee. | yes | negative | 2 | zu | yes |
g003 news Complaint escalated to the Consumer Goods and Services Ombud. Complainant alleges Acme Coffee continued charging a 12-month subscription after cancellation and refuses to reverse the debits. The company has been asked to respond within the prescribed period. | yes | negative | 3 | en | yes |
g004 facebook, mixed language Bafo, Acme Coffee is giving me the runaround nge-order yami. Three weeks now, no delivery, the call centre just says 'we are escalating'. Ngeke, this is too much for a coffee machine. | yes | negative | 2 | en | yes |
g005 reviews, mixed language Ndidiniwe yi-Acme Coffee. Umntu wabo wathi i-subscription ayinayo i-cancellation fee, kodwa ngoku bafuna i-R600 ukuyirhoxisa. Sasingazi ukuba kukho le fee, akukho apho yayibhalwe khona. | yes | negative | 2 | xh | yes |
g006 google reviews, mixed language Acme Coffee lebenkeleng la Maseru le re emisitse hora e le nngwe bakeng sa dicappuccino tse pedi feela. Re ne re hloka thuso ka pele empa ho ne ho sa tsamaye letho ho fihlela re belaela ka lentswe le phahameng. | yes | negative | 1 | st | no |
g007 x Reporting @acmecoffee to my bank. Subscription debit went off twice this month and support has ignored three emails. Cancelling today and never coming back. | yes | negative | 2 | en | yes |
g008 x Order 'in preparation' for 45 minutes while my whole office waited. App says delivered, nothing arrived, no driver, no call. How is this acceptable, Acme Coffee? | yes | negative | 2 | en | yes |
g009 facebook, mixed language Acme Coffee ba nkentse hampe, the bill was R200 more than the menu price. Ke kopa thuso, who do I escalate to? They keep saying prices changed mara we have the old receipt right here. | yes | negative | 2 | st | yes |
g010 reviews Acme Coffee catering failed us completely. Booked and paid for my sister's baby shower, the order arrived two hours late and stone cold. The catering service is the whole reason we chose them. | yes | negative | 2 | en | yes |
g011 reviews Ek het Acme Coffee se boonmasjien teruggestuur omdat dit stukkend aangekom het, en hulle weier om my geld terug te betaal. Agt weke se gestryery al. Ons gaan dit nie net los nie. | yes | negative | 2 | af | yes |
g012 google reviews Store confirmed my refund was approved but two weeks later there is still no money in my account. The phone goes unanswered and the manager is 'in a meeting' every time. Exhausting. | yes | negative | 2 | en | yes |
g013 facebook Quick question, how long before Acme Beans rewards points expire? Trying to understand the programme before I sign up. | person said yes AI said no | neutral | 0 | person said en AI said und | no |
g014 reviews Nobody explained that Acme Coffee's 'free month' auto-renews at full price. The signup screen rushed me through. I understand it now but it was not upfront, and that leaves a bad taste. | yes | negative | person said 1 AI said 2 | en | person said no AI said yes |
g015 x, mixed language Le subscription ya Acme Coffee e bitsa bokae ka kgwedi? Ke batla ho tseba pele ke saena le bona. Anyone who knows please ngiazise. | yes | neutral | 0 | st | no |
g016 google reviews, mixed language The Acme Coffee Soweto staff were so patient when I came in to swap a bag of beans I bought by mistake. Sorted in ten minutes with a smile. Ngiyabonga, great service. | yes | positive | 0 | en | no |
g017 google reviews Queued at the Sandton Acme Coffee for 25 minutes this morning with only two baristas working. The coffee was fine but please add more staff at peak times. | yes | negative | 1 | en | no |
g018 google reviews Uitstekende diens by die Acme Coffee in Bloemfontein. Die barista het my bestelling perfek gemaak en die nuwe bone geduldig verduidelik. Baie dankie, ons kom beslis terug. | yes | positive | 0 | af | no |
g019 reviews Barista at the Pretoria Acme Coffee was rude and dismissive when I asked about decaf options. Made me feel like a nuisance for asking. Do better with how you treat walk-ins. | yes | negative | 1 | en | no |
g020 facebook, mixed language Acme Coffee increased the subscription price again this year and I am on a pension. Every year it goes up but the beans are the same. Eish, these increases are getting hard to afford. | yes | negative | 1 | en | no |
g021 x My Acme Coffee-intekening het met meer as 10 persent gestyg hierdie jaar. Niemand het my vooraf laat weet nie. Hoekom so 'n groot sprong sonder kennisgewing? | yes | negative | 1 | af | no |
g022 reviews I understand prices go up with inflation, but a clear email from Acme Coffee explaining the increase would help. I only noticed when the debit was higher. Communication could be better. | yes | person said neutral AI said negative | 1 | en | no |
g023 facebook Acme Coffee took the subscription debit twice off my account this month, two days apart. Double debit on a tight month. Please refund the extra amount immediately. | yes | negative | 2 | en | yes |
g024 reviews, mixed language Acme Coffee e ntse e nkgula chelete le ha ke khansetse subscription likhoeli tse tharo tse fetileng. Ke ntse ke lefa for coffee I no longer get. Please stop the deduction and pay me back. | yes | negative | 2 | st | yes |
g025 x How do I set up an Acme Coffee office subscription for a team of 12? Do I go through the app or is there a business team? Just need the process, thanks. | yes | neutral | 0 | en | no |
g026 facebook, mixed language Got my Acme Beans rewards voucher this month and I am so happy. After a year of buying it feels good to get something back for loyalty. Ngiyabonga Acme Coffee. | yes | positive | 0 | en | no |
g027 reviews I hit gold tier on Acme Beans rewards last month and was told I qualify for the birthday voucher, but nothing arrived. Emailed twice, no reply. It was a big reason I stayed with Acme Coffee. | yes | negative | person said 1 AI said 2 | en | person said no AI said yes |
g028 x, mixed language Le Acme Beans rewards ya bona e sebetsa jwang? Motswala wa ka o re o fumana i-free coffee after 10 orders. Is this true kapa ke leshano feela? | yes | neutral | 0 | st | no |
g029 google reviews, mixed language I-Acme Coffee indithumelele umatshini we-espresso ukusuka eRhawutini ukuya eLusikisiki ngaphandle kwengxaki, kwaye umnxeba wokuseta wawuluncedo kakhulu. Enkosi kakhulu, yiyo le nto sibathembayo. | yes | positive | 0 | xh | no |
g030 reviews The replacement-part voucher from Acme Coffee took forever to be issued and the approved courier list is very limited. We wanted a weekend delivery slot but were told it is not covered. More flexibility please. | yes | negative | 1 | en | no |
g031 reviews When I phoned Acme Coffee in a panic because my machine died the day before I was hosting a family lunch, the consultant stayed on the line, talked me through a fix, and arranged a loaner. Small thing but it mattered. | yes | positive | 0 | en | no |
g032 facebook, mixed language Phoned Acme Coffee support about my missing order and the agent was cold and just kept reading a script. Bekungasekho lutho lobubele. Sometimes you need a human, not a robot. Please train your line better. | yes | negative | 1 | en | no |
g033 news Acme Coffee confirms it is investigating a payment system outage that disrupted its app this week. The company says it took systems offline as a precaution while services are restored. No comment yet on whether any card data was affected. | yes | neutral | 2 | en | yes |
g034 x Been trying to order on the Acme Coffee app for three days and it just spins. The store says systems are down. I need to change my delivery address and I cannot do anything. Sort it out please. | yes | negative | 2 | en | yes |
g035 facebook URGENT: due to the Acme Coffee system upgrade you must reverify your account here to avoid losing your rewards points. Click the link and confirm your ID and card details now. Limited time only. | yes | person said negative AI said neutral | 3 | en | yes |
g036 x, mixed language Heads up to all Acme Coffee customers: after the outage there are fake SMSes going around asking for card details. Acme will never ask for your PIN. Ningakhohliswa, do not click those links. | yes | neutral | 2 | en | yes |
g037 instagram Loved seeing the Acme Coffee youth barista academy graduation this year. Fifty young people trained and placed in jobs across three provinces. This is what giving back looks like. | yes | positive | 0 | en | no |
g038 news Acme Coffee has donated equipment to twelve township coffee startups as part of its incubator programme, the company announced at a ceremony in Khayelitsha. The programme also offers mentorship from senior roasters. | yes | person said positive AI said neutral | 0 | en | no |
g039 x Applied for the Acme Coffee incubator but the portal kept timing out and I never got a confirmation my entry was received. Frustrating, the deadline is close. Did anyone else struggle? | yes | negative | 1 | en | no |
g040 facebook, mixed language Ndiyayithanda i-Acme Coffee incubator. Abantu abatsha bakuthi bafumana amathuba okuqala amashishini ekofu kwiilali zethu. Halala, oku kukuphuhlisa uluntu ngenene. | yes | positive | 0 | xh | no |
g041 google reviews The Acme Beans rewards have actually saved me money on my morning coffee and even on groceries at partner stores. Did not expect that from a coffee chain. Pleasantly surprised. | yes | positive | 0 | en | no |
g042 facebook, mixed language Sebenzisani i-Acme Beans rewards bantu, ama-voucher ayasebenza ngempela. Ngithenge ikhofi ngama-points inyanga yonke. Worth it for something we already buy every day. Highly recommend. | yes | positive | 0 | zu | no |
g043 reviews My family has bought from Acme Coffee for three generations, back when it was one roastery in Durban. When my machine died they replaced it under warranty with zero fuss. Loyal for life. | yes | positive | 0 | en | no |
g044 google reviews, mixed language Lilemo tse 20 le Acme Coffee mme ha ho mohla ke bileng le bothata. Ke ba tshepa ka pelo yohle. Ha order ya ka e ile ya lahleha hang, ba e lokisitse ka lona letsatsi leo. Nke ke ka ya kae kapa kae. | yes | positive | 0 | st | no |
g045 x Honestly Acme Coffee has been solid for my family for years. People only post when they are angry but I want to say thank you, the subscription has never missed a delivery. Credit where it is due. | yes | positive | 0 | en | no |
g046 x Wow Acme Coffee, only three weeks to refund a cancelled order, truly record breaking speed. Absolutely worth the four phone calls and two 'escalations'. 10/10 would wait again. | yes | negative | 2 | en | yes |
g047 x The new Acme Coffee app update is 'amazing'. Love logging in three separate times to order one flat white. Great job guys, really nailed it. | yes | negative | 1 | en | no |
g048 reviews Acme Coffee's beans are genuinely the best I have had, but the checkout crashes every single time I try to pay with Apple Pay. Two updates now and still broken. Fix it and you are perfect. | yes | negative | 1 | en | no |
g049 x Switched from Acme Coffee to Bravo Beans last month. Half the price, twice the quality, and their app actually works. Should have done it years ago. | yes | negative | 2 | en | yes |
g050 x Bravo Beans just opened a second location in Austin and the queue was out the door all morning. Best cold brew in town, no contest. | no | not scored | not scored | not scored | not scored |
g051 reddit Rewatching old Looney Tunes and the Acme anvil catalogue still makes me laugh every time. Wile E. Coyote never learns, and I hope he never does. | no | not scored | not scored | not scored | not scored |
g052 news Reaching the acme of his career, the striker signed a five-year deal last night and will captain the side from August. Fans queued overnight for shirts. | no | not scored | not scored | not scored | not scored |
g053 google reviews Acme Plumbing quoted me $900 to replace a single valve and then charged a callout fee on top. Absolute rip-off, avoid these guys. | no | not scored | not scored | not scored | not scored |
g054 news The specialty coffee market remains crowded, with chains like Acme Coffee and Java Junction both expanding subscription offerings, according to an industry review released on Tuesday. | yes | neutral | 0 | en | no |
g055 facebook My daughter's hot chocolate from Acme Coffee had a shard of glass in it. The manager shrugged and offered a free muffin. Doctor visit done, reporting to trading standards, and I will post this everywhere until someone senior responds. | yes | negative | 3 | en | yes |
g056 x This Acme Coffee thread is wild, 4k retweets in an hour. They charged a food bank triple the quoted price for an event and refused to refund the difference. Read the whole thing before you order again. | yes | negative | 3 | en | yes |
g057 google reviews Popped into the Acme Coffee on Shoreditch High Street and the barista remembered my order from last week. Lovely little chain, deserves the hype. | yes | positive | 0 | en | no |
g058 instagram Queue was long at Acme Coffee this morning, not gonna lie, but the flat white was worth every minute. The new Rwandan roast is superb, easily their best this year. | yes | positive | 0 | en | no |
g059 x Ngl I came here to complain about my late Acme Coffee order but support refunded me before I even finished typing the DM. Okay, that is how you do customer service. | yes | positive | 0 | en | no |
g060 reddit Does the Acme Coffee subscription ship to Ireland? Their site only lists UK postcodes but I have seen Irish unboxing videos. Anyone here actually receiving it in Dublin? | yes | neutral | 0 | en | no |
g061 app store Rated 1/5 stars. The app charged me twice for the monthly subscription and three weeks later I still have no refund. Every ticket gets closed with a canned reply. Do not link a card to this. | yes | negative | 2 | en | yes |
g062 app store Rated 5/5 stars. Ordering my Acme Coffee is one tap, rewards apply automatically, and the pickup timing is always spot on. Easily the smoothest coffee app I have used. Keep it up. | yes | positive | 0 | en | no |
g063 google play Rated 3/5 stars. The Acme Coffee beans are genuinely excellent and delivery is quick, but the app logs me out every couple of days and I have to re-enter my card each time. Fix the login and this is five stars. | yes | negative | 1 | en | no |
Seriousness is scored on the strict 0 to 3 scale in the scorecard; the “within one step” figure counts a score one step off as close. Language is scored by main language, so a post the AI judged not to be about the business shows its language unset. Both are the test harness’s real rules, shown as they run.
The full documents
The methodology document and the latest test report.
The methodology document
How Earshot classifies mentions, and how we measure that it works
Every mention Earshot collects passes through an AI classifier before it reaches your dashboard. This page explains how that classifier is built, why we trust its output enough to alert you on it, and exactly how we measure its accuracy. It answers the question a careful buyer should ask: is this one generic call to a language model, or an engineered, measured system? It is the second, and here is what that means in practice.
Throughout, "the model" means Claude, the language model family from Anthropic. Earshot uses two of them: a small fast model for filtering and a large model for the judgment calls.
What the classifier decides
For every mention, the classifier returns one structured record with these fields:
- relevant: is this genuinely about your brand? A coffee chain named Acme should not see cartoon anvils, dictionary uses of the word, or a plumbing company that happens to share the name. Mentions of a competitor count only when they also say something about your brand, such as a switching story.
- sentiment: the writer's stance toward your brand: positive, negative, or neutral. Stance, not topic. A calm, factual complaint is negative. Sarcasm and backhanded praise are negative. A warning that protects your customers from a scam is neutral toward you.
- topic: a short phrase naming what the mention is about, such as "refund delay" or "store service", so mentions cluster into themes you can act on.
- severity: a 0 to 3 scale. 0 is noise, 1 is a mild grumble, 2 is a serious complaint or service failure, and 3 is reserved for crisis: safety or legal risk, regulator involvement, a data breach, a scam wave, or a clearly viral negative.
- language: the dominant language of the text. This matters because real mentions are not always written in one language at a time.
- needsAttention: should a human look at this? True for genuine complaints, safety and legal risks, scams and impersonation, churn threats, and crisis stories. False for praise, questions, and mild grumbles.
- confidence: the model's own calibrated certainty in the whole record. Low confidence is not hidden; it triggers a re-check.
The output is constrained to a fixed schema. The model cannot return a malformed record, an invented field, or a severity of 7. Anything that fails validation is rejected and retried later rather than guessed at.
How classification works: two stages plus a verify pass
Stage 1: the relevance gate. A fast, inexpensive model (claude-haiku-4-5) reads each mention and answers one question only: is this genuinely about the monitored brand? Homonyms, namesakes, and lookalike businesses are filtered out here. This is also what keeps costs sane: irrelevant mentions never reach the expensive model.
Stage 2: full classification. Mentions that pass the gate go to a more capable model (claude-opus-4-8) that returns the full record above. Its instructions carry three things beyond the field definitions:
- Your project's context. Your brand terms, your competitor terms, and your industry are part of the prompt, so relevance and attribution are judged against your actual setup, not a generic idea of a brand.
- Language and slang lexicons. A general English lexicon teaches internet slang and sarcasm cues, the patterns that flip a positive-looking sentence into a complaint. Projects monitoring South African audiences also get a South African pack covering code-switching between English, Afrikaans, isiZulu, isiXhosa, Sesotho, Sepedi, Setswana, and tsotsitaal, the urban township slang. A mention that reads as calm English can carry a serious complaint in the embedded vernacular, and the classifier is taught to read it.
- Worked hard cases. The prompt includes a curated set of examples chosen because naive classifiers get them wrong: sarcastic praise, code-switched complaints, homonyms, competitor switching stories, scam warnings, and mixed-sentiment mentions. The model learns the decision boundary from boundary cases.
Stage 3: the verify pass. Any mention flagged as severity 2 or higher, marked as needing attention, or classified with low confidence is not trusted on a single pass. It is re-examined individually by the large model, which confirms or corrects every field. When the two passes disagree, the careful second look wins. This catches the expensive mistakes, a missed crisis or a false alarm, while leaving the bulk of easy, confident mentions to a single efficient pass.
How accuracy is measured
Engineering without measurement is just assertion. Earshot ships its evaluation harness in the same repository as the classifier, and the numbers on this page's companion report come from running it.
The gold set. Sixty-three mentions, each labelled by hand against the full schema. It is a synthetic-but-realistic benchmark: written to mirror real public mentions, grounded in real failure modes, and de-identified by design, so no real person's post is quoted. It deliberately over-samples the hard cases: sarcasm, mixed sentiment, competitor confusion, irrelevant homonyms, scams, crisis mentions, and a substantial South African code-switched slice, because those are the mentions where a classifier earns or loses its keep.
What the harness reports. It runs the real production pipeline, both stages and the verify pass, over the gold set and compares every field to the human label:
- accuracy per field, separately, so a weak field cannot hide behind a strong one: relevance, sentiment, severity (exact and within one band), language (including on the code-switched slice), and needs-attention;
- sentiment macro-F1, a score that weights each sentiment class equally so the rare classes count as much as the common ones;
- a confusion matrix showing exactly which sentiment gets mistaken for which;
- every single miss, printed with the text, the expected record, and what the classifier said instead.
That last item matters most. The harness does not just print a score; it shows its homework. Every mistake is inspectable, and every inspected mistake is a candidate for a new worked example in the prompt, which is how the classifier improves.
Where the numbers live. We do not print accuracy numbers on this page on purpose. They change whenever the model, the prompt, or the gold set changes, and a number frozen into marketing copy goes stale silently. The harness writes its results, dated and versioned with the model names, to a companion report, and that report is the only source of published accuracy figures. If you see an Earshot accuracy number anywhere, it came from a harness run.
How your corrections make it better, without training a model
The classifier is good out of the box, but no generic system knows your brand's edge cases the way you do: your slang, your regulars' sarcasm, the lookalike business that shares your name. So Earshot lets anyone on your team correct a mention that was read wrong, straight from the mentions list: fix whether it was really about you, fix the sentiment, fix the severity. Three honest things happen when you do.
- Your correction becomes your private labelled set. It is your data, scoped to your workspace, and no other customer ever sees it. Over time it becomes a benchmark of how your brand's mentions should read, written by the people who know.
- The classifier reads your recent corrections as worked examples. When Earshot next classifies your mentions, your own corrected hard cases are added to the prompt alongside the general ones, so the model sees exactly how your team wants these calls made. This is the same mechanism as the curated worked examples above, pointed at your brand.
- We can measure agreement on your brand specifically. Corrections let Earshot compare, on your real mentions, how often the AI's first read matches what your team decided.
To be exact about what this is and is not: we do not fine-tune or retrain an AI model on your data. The model's weights never change; Claude is not fine-tunable through its API, and we do not pretend otherwise. Your corrections improve results by better prompting, plain-text examples the model reads, not by frozen training. That keeps the whole thing transparent and reversible, and it means the asset you are building is your labelled data, which stays yours.
Honest limitations
- The model is not fine-tuned. The domain knowledge lives in the prompt, in plain text anyone can read and audit, rather than frozen into model weights. We accept a somewhat lower ceiling in exchange for transparency and fast iteration.
- The gold set is sixty-three items. Large enough to catch real regressions and report honest per-field accuracy, small enough that the rarest categories carry wide uncertainty. It is a starting benchmark that grows from real classified mentions, not a finished one.
- The benchmark is synthetic. It mirrors real mention patterns but cannot capture every quirk of live data. As real mentions flow through, misses and edge cases are labelled and folded into the gold set.
- Some judgments are genuinely close. Two careful people can disagree on whether a complaint is a 1 or a 2, or whether a mixed mention is negative or neutral. The harness measures against one consistent labelling, and the misses list is where those close calls get debated and the rules tightened.
- Topics are free text. Topic phrasing is guided, not locked to a fixed list, so the harness reports topics for inspection rather than scoring them as right or wrong.
What this buys you
When Earshot alerts you at 2am about a severity 3 mention, that alert has passed a relevance gate, a full classification against your brand's own context, and an independent second look. When our methodology page cites an accuracy figure, you can trace it to a dated harness run over a labelled benchmark, with every miss on the record. That is the standard we think brand monitoring should be held to, and it is the one we hold ourselves to.
The latest test report
Earshot classifier eval
Generated by npm run eval (lib/ai/eval/run.ts). Do not edit by hand; re-run the harness to refresh. These are the only accuracy numbers that may be published anywhere.
- Date: 2026-07-08
- Gold set: n=63 (63 scored, 0 dropped; 59 expected-relevant; code-switched slice n=17)
- Models: relevance gate
claude-haiku-4-5, classification + verifyclaude-opus-4-8
Per-field accuracy
| Field | Accuracy |
|---|---|
| relevant | 98.4% |
| sentiment | 94.9% |
| severity (exact) | 96.6% |
| severity (within 1) | 100.0% |
| language (prefix match) | 98.3% |
| language, code-switched slice | 100.0% |
| needsAttention | 96.6% |
Sentiment macro-F1: 0.927
Sentiment confusion matrix
Rows are the gold label, columns are the prediction.
| gold / pred | positive | neutral | negative |
|---|---|---|---|
| positive | 16 | 1 | 0 |
| neutral | 0 | 8 | 1 |
| negative | 0 | 1 | 32 |
Misses
6 of 63 scored rows had at least one field off. Run npm run eval for the full inspectable list with texts.
See it read your own reviews
No card, no account. Or talk to us