Can a System One model replace the LLM call in your agent loop?

A measured head-to-head on Indian-language inbound messages: TypeSafe Jev against a cheap chat model, 24 labelled examples, Gujarati and Hinglish included.

Labels are mine, the sample is small, and where I think my own labelling is arguable I say so. Reproduce it: dataset.py and harness.py.

Check a case

Case

Access code, optional - with it the buttons make live calls, without it they show the recorded run:

The three decisions

Intent (choice: pick one)

95.8% Jev

cheap LLM 91.7% 23/24 vs 22/24

Needs a human fast (noul: yes/no)

66.7% Jev

cheap LLM 62.5% 16/24 vs 15/24

Urgency (score: low to high)

37.5% Jev

cheap LLM 70.8% 9/24 vs 17/24

Latency, mean

929 ms Jev p95 1164

cheap LLM 754 ms (p95 928)

Cost per 1,000 decisions

$0.023

Jev, output billed free (537 input tokens each)

Unsure enough to route

2 of 24

baseline 0 of 24 under 80% confidence

Did it admit uncertainty?

A confidence threshold is only useful if some answers are honestly unsure. Histogram of stated confidence, all 24 decisions.

cheap LLM (median 95, min 85)

0-60%0

60-70%0

70-80%0

80-90%1

90-95%7

95-100%16

Jev (median 99, min 58)

0-60%1

60-70%0

70-80%1

80-90%1

90-95%1

95-100%20

The cheap LLM never went below 85% (nothing to route on). Jev produced a real tail - 2 of 24 under 80% - and both of those calls were still right.

Does it hold up in Gujarati and Hinglish?

Language	n	Intent (choice) baseline / Jev	Needs a human (noul) baseline / Jev	Urgency (score) baseline / Jev
gujarati	11	10/11 / 10/11	8/11 / 7/11	9/11 / 2/11
hinglish	5	4/5 / 5/5	1/5 / 3/5	2/5 / 4/5
english	8	8/8 / 8/8	6/8 / 6/8	6/8 / 3/8

Every example, both answers, my label

Message	My label	Cheap LLM	Jev	Jev intent probabilities
નમસ્તે, મારે મારી દુકાન માટે GST રજિસ્ટ્રેશન કરાવવું છે. શું તમે આ કામ કરો છો? ચાર્જ કેટલો લાગશે? gujarati	new enquiry fast: True urgency: high	new enquiry fast: True · medium conf 95	new enquiry fast: False · low conf 95	new enquiry0.96 existing custo0.00 billing or pay0.04 spam or promot0.00
મેં ગઈકાલે ઓર્ડર કરેલો સામાન હજુ સુધી નથી આવ્યો. ટ્રેકિંગ નંબર પણ કામ નથી કરતો. gujarati	existing customer support fast: True urgency: high	existing customer support fast: True · high conf 95	existing customer support fast: False · medium conf 100	new enquiry0.00 existing custo1.00 billing or pay0.00 spam or promot0.00
તમારો લેટેસ્ટ પ્રાઇસ લિસ્ટ મોકલો જરા gujarati	new enquiry fast: False urgency: medium	new enquiry fast: True · medium conf 95	new enquiry fast: False · medium conf 90	new enquiry0.93 existing custo0.07 billing or pay0.00 spam or promot0.00
Hi, I was charged twice for the same invoice number 8841. Please check and refund. english	billing or payment fast: True urgency: high	billing or payment fast: True · high conf 95	billing or payment fast: False · medium conf 99	new enquiry0.00 existing custo0.01 billing or pay0.99 spam or promot0.00
Congratulations! You have won a free iPhone 16. Click this link to claim your prize now. english	spam or promotion fast: False urgency: low	spam or promotion fast: False · low conf 98	spam or promotion fast: False · medium conf 100	new enquiry0.00 existing custo0.00 billing or pay0.00 spam or promot1.00
bhai mera order 3 din se pending hai, kya problem hai? mera kaam ruk gaya hai hinglish	existing customer support fast: True urgency: high	existing customer support fast: True · high conf 95	existing customer support fast: True · high conf 100	new enquiry0.00 existing custo1.00 billing or pay0.00 spam or promot0.00
અમારી કંપનીને તમારી સર્વિસ જોઈએ છે. અમે 15 લોકોની ટીમ માટે મંથલી પ્લાન લેવા માંગીએ છીએ. કોઈ સેલ્સ પર્સન સાથે વાત કરાવો. gujarati	new enquiry fast: True urgency: high	new enquiry fast: True · high conf 95	new enquiry fast: True · medium conf 100	new enquiry1.00 existing custo0.00 billing or pay0.00 spam or promot0.00
Please update my delivery address for the order placed yesterday. english	existing customer support fast: False urgency: medium	existing customer support fast: True · high conf 95	existing customer support fast: False · medium conf 100	new enquiry0.00 existing custo1.00 billing or pay0.00 spam or promot0.00
ફ્રી લોટરી જીતો, આજે જ રજિસ્ટર કરો, 50000 રૂપિયા જીતવાની તક gujarati	spam or promotion fast: False urgency: low	spam or promotion fast: False · low conf 98	spam or promotion fast: False · medium conf 100	new enquiry0.00 existing custo0.00 billing or pay0.00 spam or promot1.00
payment ho gaya hai mera, UTR number chahiye ya nahi bill ke liye? hinglish	billing or payment fast: False urgency: medium	billing or payment fast: True · high conf 92	billing or payment fast: False · medium conf 98	new enquiry0.00 existing custo0.01 billing or pay0.99 spam or promot0.00
તમારું પેકેજ સારું લાગે છે પણ બજેટ ઓછું છે. કોઈ ડિસ્કાઉન્ટ મળી શકે? gujarati	new enquiry fast: False urgency: medium	new enquiry fast: True · medium conf 92	new enquiry fast: False · low conf 97	new enquiry0.98 existing custo0.01 billing or pay0.01 spam or promot0.00
The app is not opening since this morning, I have an important meeting in 2 hours and need the report. english	existing customer support fast: True urgency: high	existing customer support fast: True · high conf 95	existing customer support fast: True · high conf 100	new enquiry0.00 existing custo1.00 billing or pay0.00 spam or promot0.00
બિલમાં મને 18% ને બદલે 28% GST લાગ્યો છે, આ ખોટું છે. gujarati	billing or payment fast: True urgency: high	billing or payment fast: True · high conf 95	billing or payment fast: False · medium conf 99	new enquiry0.00 existing custo0.01 billing or pay0.99 spam or promot0.00
kya aap mumbai me delivery karte ho? minimum order kitna hai? hinglish	new enquiry fast: False urgency: medium	new enquiry fast: True · medium conf 95	new enquiry fast: False · low conf 100	new enquiry1.00 existing custo0.00 billing or pay0.00 spam or promot0.00
We are a distributor of similar products and would like to discuss a partnership. Who is the right person to speak with? english	new enquiry fast: True urgency: high	new enquiry fast: False · medium conf 90	new enquiry fast: False · low conf 98	new enquiry0.98 existing custo0.00 billing or pay0.00 spam or promot0.02
સર, મારી ફ્રેન્ચાઇઝીની પેમેન્ટ રોકાઈ ગઈ છે, ત્રીજા વખત પૂછું છું. જવાબ નથી મળતો. gujarati	billing or payment fast: True urgency: high	billing or payment fast: True · high conf 95	billing or payment fast: True · high conf 97	new enquiry0.00 existing custo0.02 billing or pay0.98 spam or promot0.00
Increase your sales by 500% with our guaranteed marketing service, limited offer this week only!!! english	spam or promotion fast: False urgency: low	spam or promotion fast: False · low conf 98	spam or promotion fast: False · medium conf 100	new enquiry0.00 existing custo0.00 billing or pay0.00 spam or promot1.00
કાલે સાંજે બંધ રહેશો? અમારે ઓફિસમાંથી બે વ્યક્તિ મોકલવી છે. gujarati	existing customer support fast: False urgency: low	new enquiry fast: True · high conf 90	new enquiry fast: False · medium conf 81	new enquiry0.86 existing custo0.14 billing or pay0.00 spam or promot0.00
bhai invoice me company ka naam galat likha hai, dobara bhejo hinglish	billing or payment fast: False urgency: medium	billing or payment fast: True · high conf 95	billing or payment fast: True · medium conf 98	new enquiry0.00 existing custo0.02 billing or pay0.98 spam or promot0.00
Is there a student discount available? I want to buy the annual plan. english	new enquiry fast: False urgency: medium	new enquiry fast: False · medium conf 90	new enquiry fast: False · low conf 100	new enquiry1.00 existing custo0.00 billing or pay0.00 spam or promot0.00
મારો ડેટા ડિલીટ કરવો છે તમારી સિસ્ટમમાંથી, આ કેવી રીતે થાય? gujarati	existing customer support fast: True urgency: high	existing customer support fast: True · high conf 90	existing customer support fast: False · medium conf 99	new enquiry0.00 existing custo1.00 billing or pay0.00 spam or promot0.00
Thanks, the issue is resolved now. Appreciate the quick help! english	existing customer support fast: False urgency: low	existing customer support fast: False · low conf 95	existing customer support fast: False · low conf 100	new enquiry0.00 existing custo1.00 billing or pay0.00 spam or promot0.00
અમારે તમારી સાથે લાંબા ગાળાનો કોન્ટ્રાક્ટ કરવો છે, વોલ્યુમ મોટું રહેશે. કોઈ સિનિયર સાથે વાત કરાવો. gujarati	new enquiry fast: True urgency: high	new enquiry fast: True · high conf 92	new enquiry fast: True · low conf 73	new enquiry0.79 existing custo0.03 billing or pay0.00 spam or promot0.18
aapka competitor hume 20% sasta de raha hai, aap match kar sakte ho? hinglish	new enquiry fast: False urgency: medium	billing or payment fast: True · high conf 85	new enquiry fast: True · medium conf 58	new enquiry0.68 existing custo0.29 billing or pay0.01 spam or promot0.02

Try it live

Both providers are asked the same three questions. Available: cheap LLM.

Jev is at $0.042 per million input tokens with output billed free, which is where the cost figure comes from. Latency here is against a cheap flash model; against a large reasoning model the gap is much bigger.