Use Jev to Route Cheap Models Without Guessing

A task that reads “bump the parser pin and fix the one failing test” does not need your frontier model. A task that reads “tidy up the billing tables” might need it plus a human. Both are one line long, and a prompt rubric that says “use the small model for small tasks” routes them the same way, because a rubric has no way to say it isn’t sure.

Jev model routing replaces the rubric with a typed decision: one Choice over tiers you wrote down, probabilities you can log, and a confidence number you can put a floor under. At TypeSafe’s list price the classifier call costs $0.042 per million input tokens, with output free, so the pick itself is close to free. The wrong pick is not.

By the end you have a route table in version control: written criteria, an unclear exit, a floor below which work stays on frontier, a never-route list in code, three fallback drills, and one number, cost per merged PR, that says whether any of it saved money. This is the pick made before the task starts; escalating after a failed gate already has a home.

Sep 17: LangChain merges a Jev model router with no floor

On Sep 17, 2026, LangChain merged PR #40543, “feat(typesafe): experimental ModelRouterMiddleware”, into its langchain-typesafe partner package, which is still an alpha on PyPI (0.0.1a1 that day, 0.0.1a3 on Sep 20). The router’s source states the mechanism in one sentence: “The middleware classifies the latest human message once before an agent run, stores the complete ChoiceAnswer in agent state, and uses its selected label for every model call in the run.” Each option is a ModelChoice(model, criteria), and the question is a single Jev Choice.

Screenshot: GitHub, “feat(typesafe): experimental ModelRouterMiddleware” (Sep 17, 2026), captured Sep 21, 2026.

What the middleware leaves out matters more than what it does. It has no confidence floor, no unclear option and no fallback route. And when the classifier fails, it stops: “Classifier failures propagate and terminate the run rather than silently selecting a different model.” A crash beats a silent swap, but it means the routing fail mode you inherit is “the run dies” unless you write a different one.

Two issues, both open as of Sep 21, sharpen the edges. Issue #40700 reports that when ModelFallbackMiddleware wraps the router, the router puts the originally routed model back on the fallback’s retry, so the configured fallback never runs; reversing the order works. Issue #40651 reports that a run whose state holds no human message, such as one resumed from stored state, dies with a bare StopIteration or an opaque RuntimeError.

The yardstick came earlier: on Sep 4, 2026, GitHub’s Project HydraFusion arrived as a research preview in Copilot CLI behind /experimental, an orchestrator that picks Single, Critique or Cascade per task. Its Cascade is the pattern this piece feeds: “An efficient model drafts a solution and a quality gate decides whether to accept it or escalate to a stronger model.” GitHub bounds its own claim: “For this preview, first-turn, single-prompt coding tasks are the best place to start.” HydraFusion contains no Jev, and its numbers, charted in step 6, are the ones you’ll hear when you propose a router.

Why a tier pick costs more when the model acts

In a chat, a wrong tier costs one weak answer and a retry. In an agent run the label is reused for every model call, so one pick sets the model for dozens of tool calls, a branch, a diff and a reviewer’s afternoon. A cheap model that half-finishes a refactor produces the most expensive artifact in software: a plausible PR that is wrong in one place.

A chatbot router needs to be right on average. A fleet router needs to know when it isn’t sure, send those tasks to the model you trust, and prove with a ledger that the savings survive retries and escalations. A tier pick is a decision seat, not a reply seat; the ten-second rule for Jev explains why a routing question qualifies and a code review doesn’t.

Step 1: Write the route table as code, with criteria a stranger could apply

Put the route table in version control beside the rest of your fleet policy, not in a system prompt. It names the tiers, the model behind each, the literal criteria Jev reads, the floor, the default route and the rules that run before Jev is ever called.

# route-table.yaml (illustrative shape; the lane launcher enforces it, no prompt reads it)
version: 2026-09-21.1
classifier:
  model: jev-1.13.0            # the pinned versioned ID, never jev-latest
  question_id: model_route
  state_fields: [task_title, task_body, touched_paths, labels]
  deadline_ms: 1500            # derive yours from measured latency on your route
tiers:
  efficient:
    model: <your-efficient-model>
    criteria: >-
      The task names the files or symbols to change, the change is mechanical
      (rename, version bump, lint fix, docstring, one failing test with a
      stated cause) and the task states the expected result.
  frontier:
    model: <your-frontier-model>
    criteria: >-
      The task needs design choices, changes behaviour across modules, adds a
      feature, or leaves the expected result to judgement.
  unclear:
    action: ask_for_clarification
    criteria: >-
      The task does not say what should change, or a careful reader could
      place it in either tier above.
floor: 0.70                    # below this confidence the task stays on frontier
default_route: frontier        # classifier error, 429, timeout, empty state
never_route_down:              # evaluated in code before any classifier call
  labels: [migration, deploy, infra, security, data-deletion]
  paths: ["migrations/**", "infra/**", ".github/workflows/**"]
  max_expected_sittings: 1

Write criteria a stranger could apply in ten seconds, because the classifier reads them literally. TypeSafe’s jaggedness page for jev-1.13 puts it plainly: “jev-1.13 answers the question you wrote, not the one you meant.” The same page says the model reads dates as text and should not do arithmetic, so diff size, file count and due dates are features you compute in code and put in the never-route rules. They never go in criteria.

Name tiers by what they do, not by vendor, so a model swap is a one-line diff; provider-forced routing swaps have their own continuity drill.

Step 2: Give the classifier an unclear exit

A Choice answer is always one of the options you supplied, per TypeSafe’s primitives page. Offer only efficient and frontier and every ambiguous task gets forced into one of them, often with a probability that looks decisive enough to trust. The unclear option gives ambiguity somewhere to go.

What unclear does depends on the lane:

• Interactive lane: return one clarifying question to the person who filed the task. Jev returns typed answers, not text, so the question comes from a template or the frontier model.
• Unattended lane: nobody is there to answer, so unclear routes to frontier and tags the task for the weekly review.
• Both: log it. A rising unclear share is the first sign your intake is producing tasks nobody could route, including a person.

Step 3: Set a confidence floor and keep low-confidence work on frontier

The floor is where “not sure” becomes a route. Every Choice answer carries a confidence computed from its probability distribution. TypeSafe’s confidence page opens with “A confidence threshold is not one number,” and its example code sends anything under 0.5 to a human. For a tier pick, the frontier model plays the human’s part: below the floor, the task goes where it would have gone without a router.

On a three-option Choice, the demo on that page puts confidence at roughly (3 × top probability − 1) / 2, so a 0.70 floor asks for about 0.80 probability on the winning tier. I’d start there and move it only on outcomes, as step 7 describes.

Treat the floor as the safe default, not a fix. Low confidence on a routing question usually means one of two things, and lowering the number solves neither:

• Thin state. LangChain’s router reads only the latest human message, so a task that arrives as “same as before, but for invoices” gives Jev nothing to judge. Fix intake: put the ticket body, touched paths and labels in state.
• Overlapping criteria. If efficient and frontier both plausibly describe a “small refactor”, probability splits and confidence sinks across the board. Rewrite the criteria until a stranger could separate them.

# route.py (illustrative); ask_choice() wraps whichever Jev door you use
def route(task, table):
    if hits_never_route(task, table.never_route_down):        # code, not Jev
        return record(task, "frontier", reason="never_route")
    state = {f: task.get(f) for f in table.classifier.state_fields}
    if not state.get("task_body"):                             # the #40651 case
        return record(task, "frontier", reason="no_state")
    try:
        ans = ask_choice(state, table.route_question(),
                         model=table.classifier.model,
                         deadline_ms=table.classifier.deadline_ms)
    except ClassifierError as err:                             # 4xx, 5xx, 429, timeout
        return record(task, table.default_route, reason=f"error:{err.kind}")
    if ans.choice == "unclear":
        return record(task, "clarify", answer=ans, reason="unclear")
    if ans.confidence < table.floor:
        return record(task, "frontier", answer=ans, reason="below_floor")
    return record(task, ans.choice, answer=ans, reason="routed")

Every exit writes a record, including the ones that never called Jev. The error branch is the fail mode LangChain leaves to you; the fail-open or fail-closed decision for Jev gates sets the timeout and retry budget, and for routing the answer is one fixed, logged default route.

Step 4: Hard-code the never-route list before Jev sees the task

Destructive and long-horizon tasks never route down, and that rule lives in code that runs before the classifier. A classifier’s pick is a probability. A migration on the cheap tier is a certainty you find out about in production.

Four kinds of task go on the list:

1. Anything that deletes or rewrites data: migrations, backfills, retention jobs.
2. Anything that deploys, changes infrastructure, touches CI workflows or handles secrets.
3. Anything longer than one sitting: multi-PR refactors, overnight lanes, goal-mode runs.
4. Anything that touches paths you’ve marked protected.

Long-horizon work stays up for a second reason: HydraFusion’s published results cover first-turn, single-prompt tasks only, and a cheap model’s small mistakes compound with every turn it takes.

Routing is a cost control, not a permission control. A routed-down model inherits the lane’s tools, so the deny rules and sandbox that fence a frontier run must fence this one too. The route table decides which model does the work; the lane’s permissions still decide what the work can touch.

Step 5: Test the escalation path in the exact middleware order you ship

Routing down only pays if the way back up works: a post-draft gate (tests, a type check, a reviewer model) catches what the efficient tier got wrong and escalates. The cascade’s per-task arithmetic is already worked through in the DeepSeek agent-economics piece; the job here is proving the escalation fires.

Screenshot: GitHub, “langchain-typesafe: ModelRouterMiddleware overrides ModelFallbackMiddleware fallback models” (Sep 21, 2026), captured Sep 21, 2026.

Issue #40700 is the reason to test end to end instead of trusting the diagram. A fallback that wraps the router re-enters it and gets overwritten, and nothing errors. Run three drills in CI before the router takes real traffic, whatever framework you use:

1. Tier outage. Point the efficient tier at a model ID that doesn’t exist on a sandbox lane. Assert the frontier model appears in the trace and in the ledger, with the failed attempt counted.
2. Gate failure. Feed a task whose test the efficient model can’t pass. Assert the escalation attempt is recorded with its own cost and that the final PR came from the frontier tier.
3. No human message. Start a run from stored state with no human turn. Assert it routes to frontier with reason: no_state instead of dying.

Every exit that isn’t a confident efficient pick lands on frontier, including a failed gate. Drill each arrow.

Failover when your default model itself goes down is a separate discipline. Keep that drill apart from these three, so a green failover test can’t hide a dead escalation path.

Step 6: Measure cost per merged PR on a fixed task set

Pick a fixed task set before you tune anything: 100 to 200 real, closed tasks from your own backlog, with the merged diff as the reference. Run it through two arms, all-frontier and routed, and count every dollar either arm spent getting to a mergeable PR: classifier calls, every attempt, every retry, every escalation. Divide by merged PRs. Abandoned and reverted work counts in spend and not in merges, which is the point.

Illustrative, 200 tasks	All-frontier arm	Routed arm, floor 0.70
Classifier calls	none	170 calls × ~3K tokens × $0.042/M ≈ $0.02
Efficient-tier attempts, incl. retries	none	145 × $0.50 = $72.50
Frontier attempts, incl. escalations	200 × $3.00 = $600	84 × $3.00 = $252
Total spend	$600	≈ $324.52
Merged PRs	170	160
Cost per merged PR	$3.53	$2.03

Illustrative assumptions: 30 never-route tasks skip the classifier; of the 170 classified, 40 fall below the floor or come back unclear and stay up, and 130 route down, of which 15 need a retry and 14 escalate. Only the Jev price is real. The classifier line is a rounding error; retries, escalations and merges move the result. If the routed arm isn’t clearly cheaper per merged PR on your own set, the router is overhead with a dashboard.

GitHub’s best-tuned HydraFusion system against Opus 5, first-turn tasks, estimated costs. A yardstick for your ledger, not a forecast for a pre-task pick.

Read HydraFusion as a yardstick. Against Opus 5, the best-tuned configuration cut estimated cost 67% on TerminalBench 2.1 with a 4.9-point quality gain and 36% on DeepSWE with a 1.5-point loss, every model at medium reasoning. That is a whole system: three patterns per task, and in Cascade a gate after the draft. A tier pick made before the task, with no gate behind it, is a smaller machine, so expect a smaller number and let your ledger say how much smaller.

Log one decision record per routed task: task ID, route-table version, the pinned classifier model, choice, probabilities, confidence, floor, reason, the model actually served, attempts, retries, escalations, spend and outcome. The record’s full shape, and why it must carry a pinned version, is in the Jev decision-log piece.

Step 7: Tune the floor on outcomes, then review routed-down failures weekly

Confidence tells you how sure Jev was, not whether the routed task went well. Tune the floor on outcomes instead:

1. Bucket routed-down tasks by confidence band: 0.70 to 0.80, 0.80 to 0.90, above 0.90.
2. For each band, compute the efficient tier’s merge rate, the escalation rate and cost per merged PR.
3. Raise the floor when the lowest band’s cost per merged PR reaches the frontier arm’s.
4. Lower it one step only after replaying the band just under the floor through the efficient tier on the fixed task set. You never see live outcomes for tasks you didn’t route down, so the replay is the only evidence.
5. Ship every floor change as a route-table version bump with the band table attached. Never edit it in place.

Then hold a weekly review of routed-down failures. Pull every routed-down task that failed its gate, escalated, needed a retry or was reverted after merge, and put each in exactly one bucket:

Bucket	What happened	What changes
Criteria	The task matched the criteria as written, and the criteria were wrong	Rewrite the criteria; bump the version
State	Jev had too little to read	Fix intake so state carries the ticket body and paths
Floor	Confidence sat in the lowest band and the band loses money	Raise the floor
Never-route miss	The task was destructive or long-horizon and the pre-filter missed it	Add the label or path; treat it as an incident
Model	Genuine efficient-tier work, and the model failed anyway	Note it; if the rate climbs, swap the tier’s model

Keep the review to thirty minutes with one owner. The output is a diff to the route table or nothing.

Where Jev model routing breaks, and the signal for each

The dead escalation path. Fallback wraps router, or the gate’s escalate branch was never wired. Signal: zero escalations in a week that had efficient-tier gate failures. Fix: the step 5 drills, run in CI on every router change.

The run that dies at the router. LangChain terminates the run on any classifier error, and TypeSafe’s models page lists rate limits of 250,000 tokens per second and 1,200 requests per minute that it says can change without notice. Signal: runs that end before their first model call with a classifier error. Fix: catch the error, take default_route, and log the reason.

The alias that moved. TypeSafe’s models page is explicit: “An alias moves when a new release ships, so the answers behind it can change without a change on your side.” A floor tuned on jev-1.13.0 means nothing on a later version. Signal: the confidence distribution shifts with no route-table change. Fix: pin the versioned ID and replay the fixed task set on every bump.

Scope that changes mid-run. The label is chosen once and reused, so a later message that adds “and deploy it” rides on a cheap-tier pick. Signal: runs whose later human messages hit never-route terms. Fix: re-check the never-route rules in code on each new human message; on a hit, stop the run and relaunch the work on frontier as a new task rather than switching models mid-session, which is the cache cost Daniel San’s mod avoids by picking the main model only at session start.

The floor that drifts toward zero. Budget pressure lowers the floor without outcome data. Signal: a floor change with no band table in the commit. Fix: a review rule that rejects floor changes without one.

Savings on paper. Cost per task falls while cost per merged PR rises, because cheap attempts multiply. Signal: the two lines diverge on the weekly report. Fix: report cost per merged PR and nothing else to whoever approves the router.

The route table belongs to the fleet, not the prompt

A rubric in a system prompt is one model’s advice about another. A route table is policy: versioned, reviewed, enforced by the code that launches the lane, and logged per decision, so “which model shipped this PR” has an answer. That puts routing in the same layer as spend caps, kill switches and lane permissions, the layer AgentOps reserves for every control that has to survive a model’s opinion.

If you’d rather buy than build, the questions stay the same: can you see which model ran, pin it, export the record and compute cost per merged PR on your own tasks? Hosted routers you can’t see inside turns those questions into acceptance criteria for Fugu Max, Copilot Auto and Cursor Auto.

FAQ

Does LangChain’s ModelRouterMiddleware have a confidence threshold?

No. As of the 0.0.1a3 alpha, it classifies the latest human message once per agent run with one Jev Choice, keeps the full answer, including probabilities and confidence, in agent state, and uses the chosen label for every model call. Adding a floor means wrapping it or writing your own routing step.

Sources

• LangChain PR #40543: experimental ModelRouterMiddleware — merged Sep 17, 2026
• LangChain model_router.py source — once-per-run classification
• LangChain issue #40700 — router overrides fallback models
• LangChain issue #40651 — no human message ends the run
• PyPI: langchain-typesafe — alpha, 0.0.1a3 on Sep 20, 2026
• GitHub blog: Project HydraFusion — Sep 4, 2026; offline results vs Opus 5
• TypeSafe docs: Confidence — confidence on Choice answers
• TypeSafe docs: Models — pricing, aliases, rate limits
• TypeSafe docs: Primitives — Choice answers and options
• TypeSafe docs: jev-1.13 jaggedness — literal reading, dates, arithmetic