Disclaimer: This post was generated by Claude Code (Anthropic's AI coding agent) with support, interaction, and direction by Sheun Aluko, MD.

From 70% to 95% Accuracy: Embedding Similarity Beats Zero-Shot NLI for Client-Side Classification

Previously

In our first post, we explored using zero-shot NLI classification to predict conversational acknowledgement categories from user text — entirely client-side, in the browser. The goal: play the right acknowledgement audio ("Sure.", "Hmm.", "On it.") before the LLM even starts responding.

After four experiments across two NLI models, three quantization levels, and two rounds of label engineering, the best we achieved was:

  NLI Zero-Shot (Best Config — Run 4)
  ══════════════════════════════════════
  Model:     Xenova/nli-deberta-v3-xsmall (q8, ~87MB)
  Accuracy:  70% (14/20)
  Latency:   861ms avg
  Failure:   greeting as catch-all for ambiguous inputs

The bottleneck was architectural: NLI runs 8 forward passes (one per candidate label). No amount of label tuning could fix an 850ms latency floor or push past a 70% accuracy ceiling.

The Pivot: Embedding Similarity

The alternative approach is conceptually simple:

Pre-compute embeddings for example sentences in each category
At runtime: embed the user's text (single forward pass), compute cosine similarity against all stored embeddings
Pick the category whose exemplar has the highest similarity

"send an email to my boss"
     ↓ embed (one forward pass, ~20ms)
  cosine similarity vs 281 stored embeddings (<1ms):
    "send an email to my manager"         → 0.94 (affirmative_action)
    "draft a response to that message"    → 0.82 (affirmative_action)
    "hey whats up"                        → 0.18 (greeting)
  Winner: affirmative_action

One forward pass instead of eight. Cosine math instead of entailment inference. And crucially: accuracy scales with exemplar count, not label engineering skill.

Architecture

apps/smartchats/src/classifier/
├── ack_cat.ts                # 8 categories with exemplar sentences
├── embedding_classifier.ts   # Pipeline, cosine similarity, telemetry
├── text_classifier.ts        # NLI classifier (kept for comparison)
├── benchmark.ts              # 20-case standardized test suite
└── index.ts                  # Default → embedding, NLI as named export

The embedding classifier uses Transformers.js with all-MiniLM-L6-v2 (~23MB, q8) — a sentence embedding model purpose-built for semantic similarity. The pipeline:

// Init (once, on load)
const embedder = await pipeline('feature-extraction', 'Xenova/all-MiniLM-L6-v2', { dtype: 'q8' });

// Pre-compute all exemplar embeddings
for (const category of ACK_CATEGORIES) {
  for (const sentence of category.exemplars) {
    storedEmbeddings.set(category.id, await embed(sentence));  // 384-dim Float32Array
  }
}

// Runtime (per classification)
const inputEmbedding = await embed(userText);           // ~20ms
const scores = cosineSimilarity(inputEmbedding, all);   // <1ms
return categoryWithHighestMax(scores);

Run 5: Baseline Embedding (5 Exemplars per Category)

First test — 5 exemplar sentences per category (40 total), directly comparable to the NLI benchmark:

  Embedding Similarity — Run 5 (5 exemplars/category)
  ══════════════════════════════════════════════════════
  Model:       Xenova/all-MiniLM-L6-v2 (q8, ~23MB)
  Exemplars:   40 (5 per category)
  Accuracy:    70% (14/20)
  Latency:     19ms avg | 14ms min | 24ms max
  Load:        1,352ms (cached)
  Embed init:  664ms (all 40 exemplars)

Same 70% accuracy as the best NLI run — but at 19ms instead of 861ms. A 45x latency improvement.

Different Failure Modes

The misses were interesting. NLI defaulted to greeting on ambiguous inputs. Embeddings defaulted to buying_time:

  NLI Misses (Run 4):                          Embedding Misses (Run 5):
  ───────────────────                           ─────────────────────────
  "i got a job promotion"    → greeting         "i failed my exam"           → enthusiasm
  "send an email to boss"    → greeting         "computer keeps crashing"    → buying_time
  "show me my calendar"      → greeting         "we just launched product"   → buying_time
  "meaning of life"          → soft_transition  "grammatical gender"         → buying_time
  "quantum computing"        → soft_transition  "quantum computing"          → buying_time
  "computer keeps crashing"  → quick_confirm    "so what else is new"        → buying_time

Every architecture has a catch-all category — the one that absorbs ambiguous inputs. For NLI, it was greeting (the broadest hypothesis). For embeddings with sparse exemplars, it was buying_time (the broadest semantic bucket given only 5 lookup/search examples).

The key insight: embedding misses had low similarity scores (0.17–0.33), meaning the model knew it wasn't confident. With NLI, scores were also low on misses (0.26–0.35), but there was nothing we could do about it — the architecture was the limit.

With embeddings, the fix is obvious: add more exemplars.

Run 6: Expanded Exemplars (35 per Category)

We expanded from 5 to 35 exemplars per category (281 total), specifically targeting the failure patterns:

thinking: Added "how does X work" and "explain the concept of" patterns that were bleeding into buying_time
empathy: Added tech frustration ("my computer keeps crashing and i lost all my work"), failure, burnout, health concerns
enthusiasm: Added achievement + product success patterns ("we launched and its doing amazing")
soft_transition: Added casual topic-changers ("so anyway", "enough of that", "different topic")
buying_time: Strengthened with explicit lookup/search/check phrasing to sharpen boundaries

  Embedding Similarity — Run 6 (35 exemplars/category)
  ══════════════════════════════════════════════════════
  Model:       Xenova/all-MiniLM-L6-v2 (q8, ~23MB)
  Exemplars:   281 (35 per category)
  Accuracy:    95% (19/20)
  Latency:     21ms avg | 15ms min | 29ms max
  Load:        1,810ms (cached)
  Embed init:  4,578ms (all 281 exemplars)

95% accuracy — up from 70% — with only 2ms additional runtime latency.

Per-Category Results

Category	Run 4 (NLI)	Run 5 (5 ex.)	Run 6 (35 ex.)
greeting	100%	100%	100%
quick_confirm	100%	100%	100%
affirmative_action	67%	100%	100%
buying_time	50%	100%	100%
empathy	67%	33%	100%
thinking	33%	33%	100%
enthusiasm	50%	50%	100%
soft_transition	100%	50%	50%

Every category hit 100% except soft_transition. The single remaining miss:

"so what else is new" → greeting (expected soft_transition, score 0.777)

This is a genuine semantic ambiguity — "so what else is new" functions as both a greeting and a topic change depending on context. At 0.777 similarity to greeting exemplars, the model is making a reasonable call.

The Scoreboard

  Accuracy Progression
  ═══════════════════════════════════════════════════════════════

  Run 1-3  NLI (various)    ░░░░░░░░░░░░                     ~33-40% (manual)
  Run 4    NLI (tuned)      ██████████████████████████████░    70%
  Run 5    Embed (5/cat)    ██████████████████████████████░    70%
  Run 6    Embed (35/cat)   █████████████████████████████████████████░  95%
                            ├────────┼────────┼────────┼────────┼────────┤
                            0%      25%      50%      75%     100%


  Inference Latency
  ═══════════════════════════════════════════════════════════════

  Run 1    NLI DeBERTa      █████████████████████████████████████████████████  ~1,150ms
  Run 2    NLI MobileBERT   ███████████████████████████████░                    ~650ms
  Run 4    NLI DeBERTa      █████████████████████████████████████████░          ~861ms
  Run 5    Embed (5/cat)    █░                                                   19ms
  Run 6    Embed (35/cat)   █░                                                   21ms
                            ├────────┼────────┼────────┼────────┤
                            0      250ms    500ms    750ms   1,000ms

Why Exemplar Count Is (Almost) Free

The runtime cost breakdown tells the story:

Phase	Run 5 (40 exemplars)	Run 6 (281 exemplars)	Change
Embedding forward pass	19ms	21ms	+2ms
Cosine similarity	<1ms	<1ms	~0ms
Total runtime	19ms	21ms	+2ms
Init: exemplar embedding	664ms	4,578ms	+3,914ms

The embedding forward pass (the neural network inference) dominates runtime — and it runs exactly once regardless of how many exemplars exist. Cosine similarity over 281 384-dimensional vectors is pure arithmetic: multiply, sum, divide. At ~1,100 floating-point operations per comparison and 281 comparisons, it's roughly 300K FLOPs — trivial for any modern CPU.

The cost goes into initialization: embedding 281 sentences takes ~4.6 seconds vs ~0.7 seconds for 40. But this happens once at page load (or lazily on first classification), and the exemplar embeddings are just Float32Arrays that could be pre-computed and shipped as a static asset.

The Tuning Loop: Exemplars vs. Labels

With NLI zero-shot, the tuning lever was label engineering — rewriting hypothesis sentences to be more distinctive:

Before: "The user is making a simple request or statement that needs a brief confirmation"
After:  "The user is agreeing, saying yes, or confirming something"

This required understanding how NLI entailment works, testing against failure modes, and carefully choosing language that would separate categories. It was slow, unintuitive work.

With embedding similarity, the tuning lever is exemplar curation — adding more example sentences that cover the semantic space of each category:

empathy (5 exemplars):   "im feeling really down", "my dog passed away", ...
                         → misses: "my computer keeps crashing" (0.29)

empathy (35 exemplars):  + "my computer keeps crashing and i lost all my work",
                         + "i failed my exam again", + "im exhausted and burned out",
                         + "nothing i do seems to work", ...
                         → all 3 test cases correct

The feedback loop is faster: run the benchmark, look at what missed, add exemplars that cover the missed semantic region. No need to reason about entailment or understand model internals.

Tradeoffs

Embedding similarity isn't strictly superior to NLI — it trades some properties for others:

Property	NLI Zero-Shot	Embedding Similarity
Accuracy	70% (ceiling)	95% (with exemplar tuning)
Latency	~850ms (floor)	~21ms
Model size	~87MB (DeBERTa)	~23MB (MiniLM)
New category cost	Write 1 label	Write 35 exemplars
Tuning difficulty	Hard (label engineering)	Easy (add examples)
Zero-shot capable	Yes (any label)	No (needs exemplars)
Scales with data	No (1 label per category)	Yes (more exemplars = better)

The biggest tradeoff: NLI can classify against any label with zero training data. Embedding similarity requires upfront exemplar work. But for a fixed set of 8 categories that don't change often, 35 sentences per category is a tiny investment for a 25-point accuracy gain.

Implementation Details

Cosine Similarity

The core math is straightforward — no libraries needed:

function cosineSimilarity(a: Float32Array, b: Float32Array): number {
  let dot = 0, normA = 0, normB = 0;
  for (let i = 0; i < a.length; i++) {
    dot += a[i] * b[i];
    normA += a[i] * a[i];
    normB += b[i] * b[i];
  }
  return dot / (Math.sqrt(normA) * Math.sqrt(normB));
}

For each classification, we compute similarity against all 281 stored embeddings, then aggregate per category (min, max, avg, median). The winning category is the one with the highest max — a single exemplar that closely matches is enough to claim the category.

Scoring Strategy: Max vs. Avg

We track all four aggregations per category but use max for the winner. Why not average?

Consider "i failed my exam" against the empathy category with 35 exemplars. Most empathy exemplars are about sadness, loss, or frustration — not specifically about academic failure. The average similarity might be moderate. But one exemplar — "i failed my exam again" — will be very high. Max captures that single strong match.

Average would penalize categories with broad semantic coverage (empathy covers sadness, frustration, health, loss, work, relationships) in favor of narrow categories (greeting is always some variation of "hi").

Telemetry

Every classification emits an embedding_classifier_result event with the full score breakdown:

{
  "input_text": "how does quantum computing work",
  "winning_category": "thinking",
  "top_max_score": 0.892,
  "embed_ms": 20,
  "similarity_ms": 0,
  "latency_ms": 21,
  "scores": [
    { "categoryId": "thinking", "max": 0.892, "avg": 0.614, "median": 0.601, "min": 0.389 },
    { "categoryId": "buying_time", "max": 0.512, "avg": 0.301, ... },
    ...
  ]
}

The benchmark also emits its full report as a single embedding_classifier_benchmark event, making it trivially retrievable from session exports without post-processing.

Raw Data

All session telemetry is preserved in .session_data/classifier_experiment/ with a manifest linking each run:

Run	Approach	Model	Exemplars	Accuracy	Latency
1	NLI	DeBERTa xsmall	—	~manual	~1,150ms
2	NLI	MobileBERT	—	~33%	~650ms
3	NLI	MobileBERT q4f16	—	~40%	~1,300ms
4	NLI	DeBERTa xsmall (tuned)	—	70%	861ms
5	Embedding	all-MiniLM-L6-v2	40 (5/cat)	70%	19ms
6	Embedding	all-MiniLM-L6-v2	281 (35/cat)	95%	21ms

Takeaways

Architecture > tuning. Switching from NLI (8 forward passes) to embedding similarity (1 forward pass) eliminated the latency floor entirely. 850ms → 21ms isn't an optimization — it's a different algorithm.
Exemplar count is the new hyperparameter. With embeddings, accuracy scales with data, not engineering cleverness. 5 exemplars gave 70% accuracy. 35 exemplars gave 95%. The marginal cost of each exemplar is near-zero at runtime.
Different architectures fail differently. NLI defaulted to greeting. Embeddings defaulted to buying_time. Understanding how a classifier fails is as important as knowing its accuracy number.
Cosine math is free. Computing similarity over 281 384-dim vectors takes <1ms. The entire runtime cost is the single embedding forward pass. You can scale exemplars to thousands before cosine becomes a measurable cost.
Pre-computation amortizes the real cost. Embedding 281 exemplars takes ~4.6 seconds, but it's a one-time init cost. These embeddings could be pre-computed at build time and shipped as a static JSON file, reducing init to a fetch + parse.
21ms beats the LLM every time. The whole point was to classify user intent before the LLM starts responding. At 21ms, we have over 100ms of headroom before even the fastest LLM stream begins — plenty of time to select and start playing the right acknowledgement audio.

All measurements collected using InsightsClient telemetry during live browser sessions. Session data exported via bin/save_session and analyzed programmatically. Raw session files and the experiment manifest are preserved in .session_data/classifier_experiment/.