Disclaimer: This post was generated by Claude Code (Anthropic's AI coding agent) with support, interaction, and direction by Sheun Aluko, MD.

---

Client-Side Text Classification for Instant Voice Acknowledgements

The Problem

SmartChats is a real-time voice AI agent. When a user speaks, the pipeline looks like this:

User speaks → STT → LLM (streaming) → TTS → User hears response
                         │
                    <<<ACK>>> sure    ← server picks an acknowledgement word
                         │
                    plays "Sure." audio while LLM generates full response

The acknowledgement ("Sure.", "Hmm.", "On it.") is a conversational filler — it makes the AI feel responsive while the real answer streams in. Today, the LLM picks the ACK word server-side via the streaming protocol. But the client has to wait for the LLM stream to begin (~100-300ms best case, longer under load) before it knows what to play.

What if we could predict the ACK category client-side, instantly, from the user's text alone?

We have 37 acknowledgement types across 8 categories (greeting, empathy, enthusiasm, thinking, etc.). If the client can classify the user's intent into one of these 8 categories before the LLM responds, it can start playing the right type of audio immediately.

The Approach: Zero-Shot NLI in the Browser

Zero-shot classification uses a Natural Language Inference (NLI) model to check whether a piece of text "entails" a hypothesis. For each category, you write a hypothesis label:

Input:  "hey whats up"
Labels: ["The user is saying hello or greeting someone",
         "The user is upset, frustrated, or sharing bad news",
         "The user is happy, celebrating, or sharing good news",
         ...]
→ Winner: greeting (0.73)

We used Transformers.js (v3.8.1) to run NLI models directly in the browser via ONNX Runtime WebAssembly. The entire pipeline — model loading, tokenization, inference — happens client-side with no server calls.

Architecture

apps/smartchats/src/classifier/
├── ack_cat.ts           # 8 ACK categories with NL hypothesis labels
├── text_classifier.ts   # HuggingFace pipeline singleton, telemetry
├── benchmark.ts         # 20-case standardized test suite
└── index.ts             # Public exports

Key design decisions:

• Dynamic import: await import('@huggingface/transformers') code-splits the ~90MB model out of the main bundle
• Singleton + dedup: A pipelinePromise prevents concurrent model loads
• Delayed init: 5-second delay after page load to avoid competing with VAD and TTS initialization
• ONNX isolation: Webpack externals function selectively externalizes onnxruntime-web for VAD but lets Transformers.js bundle its own runtime

The ONNX Externals Problem

SmartChats already uses onnxruntime-web for Voice Activity Detection (VAD), externalized to a global ort variable loaded via CDN. Transformers.js also depends on onnxruntime-web internally. A static externals rule catches both:

// This breaks Transformers.js:
externals: { 'onnxruntime-web': 'ort' }

The fix: convert externals to a function that checks the import context:

function ({ context, request }, callback) {
  if (request === 'onnxruntime-web' && !context?.includes('@huggingface')) {
    return callback(null, 'ort');  // VAD → use global ort
  }
  callback();  // Transformers.js → bundle its own
}

Experiment 1: DeBERTa xsmall — Default Template

Model: Xenova/nli-deberta-v3-xsmall (q8, ~87MB)

First run with the default hypothesis_template: "This example is {}." and abstract category labels.

Metric	Value
Model load	14,074ms (cold)
Warmup	1,165ms
Avg inference	~1,000-1,300ms

Manual testing on 8 inputs showed strong results for clear-cut categories:

Input	Category	Score
"hey there"	greeting	0.73
"i feel sad"	empathy	0.76
"i got a job promotion"	enthusiasm	0.35
"what is the name of the president of mexico"	thinking	0.34

High confidence on greeting (0.73) and empathy (0.76), but weaker on everything else.

Experiment 2: MobileBERT — Speed vs. Accuracy

Model: Xenova/mobilebert-uncased-mnli (q8, ~27MB)

MobileBERT is designed for mobile/edge inference. We switched to test the speed-accuracy tradeoff.

Metric	DeBERTa xsmall	MobileBERT
Model load	14,074ms	5,114ms
Warmup	1,165ms	720ms
Avg inference	~1,100ms	~650ms

MobileBERT was 2x faster across the board. But accuracy collapsed:

Input	Predicted	Expected	Score
"i got a job promotion"	thinking	enthusiasm	0.34
"show me the status report"	empathy	buying_time	0.30
"what is 3x50"	empathy	thinking	0.30

MobileBERT exhibited a strong empathy bias — defaulting to empathy on any ambiguous input. The confidence scores were also uniformly low and flat, meaning the model couldn't decisively separate categories.

q4f16 Variant

We also tested MobileBERT with dtype: 'q4f16' (21MB, smallest available):

Metric	q8	q4f16
Load	5,114ms	2,417ms
Inference	~650ms	~1,300ms

Faster load, slower inference. The q4 dequantization overhead on WASM/CPU negated the smaller model size benefit. Same accuracy problems.

The hypothesis_template Bug

While investigating accuracy, we discovered the root cause of many misclassifications. The default hypothesis_template is:

"This example is {}."

Our labels were full sentences like "The user is asking a complex or thought-provoking question". The template wraps them into:

"This example is The user is asking a complex or thought-provoking question."

Grammatically broken NLI input. The model was doing entailment on garbled premises. The fix:

{ hypothesis_template: '{}' }

Experiment 3: Tuned Labels + Fixed Template

Back to DeBERTa xsmall (q8) with the template fix and rewritten labels. The key insight: NLI labels must be maximally distinctive from each other, using concrete verbs rather than abstract descriptions.

Before and after:

Category	Original (v1)	Tuned (v2)
quick_confirm	"...simple request or statement that needs a brief confirmation"	"...agreeing, saying yes, or confirming something"
affirmative_action	"...directly asking me to do something or perform a task"	"...giving a command or telling me to create, write, send, or delete something"
buying_time	"...requesting something that will take time to look up or process"	"...wants me to search, look up, or find specific information"
soft_transition	"...continuing a conversation or making a follow-up statement"	"...changing the subject or moving on to a different topic"

Standardized Benchmark (20 cases)

We built a standardized test suite with 20 inputs across all 8 categories, run in series:

=== Classifier Benchmark: Xenova/nli-deberta-v3-xsmall ===
Accuracy: 14/20 (70%)
Latency: avg 861ms | min 742ms | max 1130ms
Model load: 10,267ms (cached: false)

Category	Accuracy	Notes
greeting	3/3 (100%)	Perfect
quick_confirm	2/2 (100%)	Fixed from 0% with label tuning
soft_transition	2/2 (100%)	Fixed from 50%
empathy	2/3 (67%)	"my computer keeps crashing" → quick_confirm
affirmative_action	2/3 (67%)	Fixed from 0%; "send an email" still misses
thinking	1/3 (33%)	Philosophical questions → soft_transition
enthusiasm	1/2 (50%)	"i got a job promotion" → greeting
buying_time	1/2 (50%)	"show me my calendar" → greeting

Label tuning improved accuracy from 55% → 70%. The biggest wins: quick_confirm (0% → 100%) and affirmative_action (0% → 67%).

Remaining Failure Patterns

1. Greeting as catch-all: Low-confidence inputs default to greeting — "i got a job promotion" (0.29), "send an email to my boss" (0.33), "show me my calendar" (0.27)
2. soft_transition absorbs thinking: Philosophical questions like "what is the meaning of life" and "how does quantum computing work" mapped to soft_transition instead of thinking
3. Flat score distributions: Misclassified inputs have top scores of 0.26-0.35 — barely above uniform (0.125)

Why Zero-Shot NLI Hits a Wall

The fundamental issue is architectural: NLI zero-shot runs the model N times — once per candidate label. With 8 categories, that's 8 forward passes through the model per classification.

8 labels × ~110ms per forward pass ≈ 850ms minimum

No amount of label tuning will fix the latency floor. And the accuracy ceiling is limited by trying to compress each category's meaning into a single sentence.

Next: Embedding Similarity

The alternative approach: embed once, compare many.

1. Pre-compute embeddings for 3-5 example sentences per category
2. At runtime: embed the user text (single forward pass, ~100-150ms), cosine-similarity against all stored embeddings (<1ms)
3. Pick the category with the highest similarity

"send an email to my boss"
     ↓ embed (one forward pass)
  cosine similarity:
    "send a message to john"          → 0.91 (affirmative_action)
    "write me a python script"        → 0.82 (affirmative_action)
    "hey there"                       → 0.31 (greeting)
  Winner: affirmative_action

Why it should work better:

• 1 forward pass instead of 8 → potentially 5-8x faster inference
• Multiple exemplars per category instead of one hypothesis → better coverage of each category's semantic space
• Model: Xenova/all-MiniLM-L6-v2 (~23MB) — purpose-built for sentence embeddings

The target: sub-200ms classification to beat the LLM stream and provide truly instant acknowledgements.

Raw Data

All session telemetry is preserved in .session_data/classifier_experiment/ with a manifest linking each run to its configuration:

Run	Model	dtype	Template	Labels	Accuracy
1	DeBERTa xsmall	q8	default (broken)	v1	~manual
2	MobileBERT	q8	fixed	v1	~33%
3	MobileBERT	q4f16	fixed	v1	~40%
4	DeBERTa xsmall	q8	fixed	v2 (tuned)	70%

Takeaways

1. hypothesis_template matters enormously — the default wraps your labels in broken grammar. Use '{}' when labels are already complete sentences.
2. Label engineering is real work — concrete verbs ("agreeing, saying yes") outperform abstract descriptions ("needs a brief confirmation"). Labels must be maximally distinctive from each other.
3. Zero-shot NLI has an inherent latency floor — N labels = N forward passes. For real-time applications, this is the fundamental bottleneck.
4. Smaller quantization ≠ faster inference on WASM — q4f16 loads faster but dequantization overhead slows inference. q8 is the sweet spot for WebAssembly.
5. Model architecture matters more than size — MobileBERT (27MB) was faster than DeBERTa (87MB) but dramatically less accurate for NLI tasks.
6. Always instrument experiments — having telemetry from day one let us compare runs objectively. The session export pipeline (bin/save_session) + structured manifest made it trivial to revisit raw data.