Jev for Coyu Query Understanding — Technical Evaluation

Executive summary

Jev-only strict accuracy85.71%120/140 complete outputs correct

Hybrid strict accuracy100.00%140/140 on challenge suite

Jev-only latency442 msmedian · p99 602 ms

Hybrid model calls21.43%30/140 requests invoked Jev

Measured result: deterministic span precedence plus selective Jev invocation improved strict accuracy by 14.29 percentage points, reduced model calls by 78.57%, reduced mean model-path latency from 445.9 ms to 95.1 ms, and reduced mean Jev cost from $0.00007681 to $0.00000787 per challenge request.

Interpretation: 100% is a regression-suite result, not a production-accuracy claim. The suite is deliberately small and some parser rules were developed from earlier errors. The larger 110-query held-out silver test scored 96.36% strict / 98.47% traffic-weighted before those fixes.

Experimental setup

Model: typesafe/jev-1.13-20260917 through OpenRouter Decisions API
Challenge suite: 28 observed Coyu-style queries, seven per difficulty band, repeated five times in randomized order
Model-heavy request: Four independent typed choices per request: canonical brand, color, product category, shopping relevance
Candidate retrieval: Same frozen fuzzy retriever; at most six brand candidates plus none
Hybrid request: URL cleanup → longest exact brand → consume span → exact facets/modifiers → Jev only for unresolved brand-like text
Strict correctness: All four output fields must match: one incorrect field makes the entire query incorrect
Latency: Client wall-clock HTTP time. Percentiles use nearest-rank over 140 requests; no fastest-run selection
Cost: OpenRouter-reported request cost, including every repeated call

Query classification derived from the search history

Why the first schema was insufficient: brand, color, category, and relevance are useful fields, but they conflate what the shopper mentioned with how search should behave. The replacement is multi-label facet extraction plus one exclusive operational route. Facet shares overlap; route shares sum to 100%.

Facet traffic coverage

Product type

30.56%

Brand

15.34%

Color

12.38%

Silhouette/detail

9.15%

Material

5.41%

Occasion/context

4.72%

Pattern/print

2.03%

Style/aesthetic

1.53%

Audience/size/fit

1.10%

Price/promotion

0.16%

Lexical census across 130,393 rows / 358,030 searches. Shares overlap because one query can contain several facets.

What the extra facets recover

Query	Old interpretation risk	v2 grounded meaning
`denim`	Category = jeans	material=denim; attribute discovery
`beach`	Brand = Beach Bum / noise	occasion/context=beach
`polka dot`	Brand = Pinq Polka	pattern=polka_dot
`farshi salwar`	Unknown/category only	silhouette=farshi; product=salwar
`party dress`	Category only	occasion=party; product=dress
`under 5k`	Unknown text	price.max=5000 INR
`neet`	Unknown text	autocomplete prefix candidates
`femella`	Potential noise	unresolved; preserve lexical text

Exclusive operational route — share of actual traffic

Route	Traffic	Examples	Search action
product_search	26.89%	`gown`, `white dress`	canonical category + co-occurring filters
unresolved_lexical_search	17.97%	`femella`, `pearl`	preserve text; lexical/embedding retrieval; annotate
repair_or_autocomplete	16.11%	`neet`, `inchin`	prefix completion or bounded correction
attribute_discovery	14.50%	`denim`, `farshi`	cross-category facet retrieval
brand_navigation	13.47%	`ganga`, `sage by mala`	brand page / high-confidence brand filter
occasion_solution	4.67%	`party`, `beach`	occasion collection + semantic/merchandising boost
noise_out_of_domain	4.01%	`s`, phone leakage	suppress request; fix instrumentation
brand_filtered_shop	1.17%	`our love dresses`	brand lock + residual filters
audience_or_fit_constrained	1.04%	`plus size`, `maternity`	retain explicit audience/availability constraints
deal_or_newness_seeking	0.16%	`sale`, `under 5k`	numeric price/promotion/newness controls

1 · GROUNDED MENTIONS

Brand, product type, color, material, pattern, silhouette, length, sleeve, neckline, occasion, context, aesthetic, audience, size, fit, and price—each with a raw span and canonical ID.

2 · SPAN ACCOUNTING

Every character is consumed, preserved, or deliberately dropped. Unknown meaningful text is never silently discarded. One span cannot become incompatible entities.

3 · EXECUTABLE SEARCH PLAN

Separate must-filters, boosts, lexical text, semantic text, retrieval channels, zero-result relaxation, fallback, and machine-readable explanation codes.

Jev's role in v2: resolve bounded ambiguity only—fuzzy brand candidate choice, unresolved-span role, or conflicting taxonomy nodes. Exact facets, price operators, span ownership, query route, and relaxation policy remain deterministic. This prevents Jev from turning beach into a brand or denim into jeans.

Accuracy by query difficulty

Jev-only HybridEach band: 7 unique queries × 5 runs

Easysingle exact entity

100%Jev

100%Hybrid

dressgownpinkblackgangaashicosaree

Mediummultiple facets or brand + category

100%Jev

100%Hybrid

white dressblack dressparty dressour love dressesganga fashion suitsetgreen kurta setfloral dress

Hardmisspellings and fuzzy aliases

85.71%Jev

100%Hybrid

gamgainchiniching indianitika choprakarren millewhote stuffsage by malla

Extremely toughoverlap, common words, tracking noise

57.14%Jev

100%Hybrid

white stuffmini dresspolka dotbeachdenimdot and dashISOPURE&destinationCoords=…

Stability: Jev returned the same field tuple in all five runs for every query. Errors were systematic interpretation errors, not sampling randomness. Repeating the same prompt does not repair them.

Response-time distribution

Jev-only: 140 network requests

442median ms

496p90 ms

548p95 ms

602p99 ms

796max ms

request count

350–400

400–450

450–500

500–600

600–800

800–1000

>1000

92.1% of requests completed between 350 and 500 ms. Two requests define most of the p99 tail; observed maximum was 796 ms.

Percentile profile

Jev-only Hybrid including bypassed queries

The hybrid curve is bimodal: 110/140 requests bypassed Jev in under 1 ms of measured decision time; invoked calls remained in the 350–532 ms range.

Latency by difficulty

Band	Jev p50	Jev p90	Jev p99	Hybrid p50	Hybrid p90	Hybrid p99	Hybrid invocation
Easy	445 ms	497 ms	601 ms	0.002 ms	0.033 ms	0.037 ms	0%
Medium	441 ms	496 ms	602 ms	0.002 ms	0.038 ms	0.042 ms	0%
Hard	429 ms	494 ms	796 ms	426 ms	511 ms	532 ms	85.7%
Extremely tough	444 ms	496 ms	583 ms	0.003 ms	0.034 ms	0.039 ms	0%

Sub-millisecond hybrid figures are parser decision time only; they exclude application/network/search-engine overhead. They show avoided model latency, not literal end-user page latency.

Where direct Jev failed

iching india

Jev: brand=inching_india
relevance=noise

Hybrid: fuzzy brand accepted
relevance derived=shopping

beach

Jev: brand=beach_bum
relevance=noise

Hybrid: modifier=beach
brand=none · shopping

denim

Jev: category=jeans
material lost

Hybrid: modifier=denim
category=none · shopping

polka dot

Jev: brand=Pinq Polka
pattern lost

Hybrid: pattern=polka dot
brand=none · shopping

Root cause: Jev is strong at choosing among candidates, but it cannot know Coyu's precedence policy by itself. Common shopping words that resemble catalogue brands or categories create stable, confident semantic mistakes. Span consumption removes those misleading candidates before inference.

Recommended Jev pipeline

01 CLEANURL-decode, strip coordinates and tracking noise.

02 BRAND LOCKLongest exact canonical name or curated alias.

03 CONSUMERemove the accepted brand span before facets.

04 FACETSExtract colors, categories, materials, patterns, styles.

05 RETRIEVETop-six canonical brands from unresolved text.

06 JEVResolve only ambiguous candidate choice.

07 DERIVEBuild query type and search filters in code.

input: "white stuff dress"
lock.brand: "White Stuff"
residual: "dress"
category: "dress"
color: null
jev_called: false

input: "gamga"
lock.brand: null
candidates: ["Ganga Fashions", …]
jev.choice: "Ganga Fashions"
query_type: "brand_only"
jev_called: true

Measured pipeline improvement

Metric	Direct Jev	Hybrid + Jev	Change
Challenge strict accuracy	85.71%	100.00%	+14.29 pp
Extremely-tough accuracy	57.14%	100.00%	+42.86 pp
Model invocation	100%	21.43%	−78.57%
Mean model-path latency	445.9 ms	95.1 ms	−78.68%
p50 model-path latency	442.4 ms	0.003 ms	bimodal / mostly bypassed
p99 model-path latency	602.0 ms	517.1 ms	−14.10%
Mean Jev cost/query	$0.00007681	$0.00000787	−89.75%
Projected cost / 1M challenge-mix queries	$76.81	$7.87	−$68.94

Production estimate from the broader held-out fixture: model invocation was 7.27%, implying approximately $2.68 / million queries at the observed mix. The challenge suite intentionally contains more misspellings, so its 21.43% invocation rate and $7.87/million estimate are conservative.

Limitations and next measurements

The 28-query challenge suite measures repeatability and known edge classes; it is not an independent production holdout.
Difficulty bands were assigned operationally, not learned from user behavior. “Extremely tough” means high collision risk between brands, facets, and noisy text.
The reported hybrid sub-millisecond path measures parser decision time, not database retrieval, application serialization, or frontend render time.
p99 over 140 direct Jev requests is useful but still sensitive to a few observations. Production monitoring should retain at least 10,000 calls by region and provider route.
Jev confidence should not be interpreted as calibrated probability until evaluated on a fresh, double-annotated 800–2,000-query sample.
Next launch gate: candidate recall@6, strict accuracy, selective accuracy at an abstention threshold, p50/p90/p95/p99 under expected concurrency, and cost per million actual searches.