Log in
AI agents spend a ridiculous amount of compute generating text nobody actually needs. The decisions an agent makes along the way don’t require a written answer and yet, agents still send them to generative models, wait for an answer while burning through tokens and then parse that output back. The overhead is already drawing scrutiny — OpenAI’s own researchers recently disclosed spending $7,000 a day running agent workloads.
Kev, a new family of open decision models built on Qwen 3.5, takes a different approach and skips the generation entirely.
Developer Jared Palmer released a new generation of Kev on Sunday, with 0.8 billion, 4 billion, and 9 billion parameter models built on Qwen 3.5. Kev is prefill-only, processing the state, questions, and candidates in a single forward pass before reading the decisions from a pointer head without an autoregressive decoding loop.
Kev, a new family of open decision models built on Qwen 3.5, takes a different approach and skips the generation entirely.
Decisions without generated text
Kev supports three decision types: Noul for yes/no, Choice for selecting among candidates, and Score for ordered levels, mirroring TypeSafe’s System One API. Developers provide the state and questions, and the pointer head returns probabilities across the available candidates.
For a tool-routing decision, the output could look like this:
search: 0.82
database: 0.13
calculator: 0.05
Kev can still choose the wrong tool, but because it scores only the candidates it’s given, it can’t introduce an option that isn’t on the list.
Routing, safety checks, escalation, and ranking can then move to the decision layer, leaving larger reasoning models to handle the open-ended work.
Kev can still choose the wrong tool, but because it scores only the candidates it’s given, it can’t introduce an option that isn’t on the list.
Batching choices, one pass
Multiple decisions can also be made against the same state in a single forward pass, with a block-causal attention mask isolating the questions while the pointer head scores each set of candidates independently.
Palmer’s documentation shows the 4B model processing three questions in 277 milliseconds in bf16 on an M5, although without a controlled comparison against Qwen generating equivalent answers on the same hardware, the result doesn’t establish how much faster the approach is in practice.
The ability to evaluate several decisions against the same context could become more useful as agent loops grow more complex, but skipping generation doesn’t make the resulting decisions inherently better.
Calibration limits and tradeoffs
The largest model, Kev-9B, reached 83.7% accuracy on the project’s locked out-of-domain test, according to Palmer’s model card. That’s a developer-reported benchmark, and Palmer documents some limitations alongside it.
The probabilities Kev returns don’t always reflect how confident developers should be in the result. Palmer found that temperature calibration can drift on unseen source distributions, a problem for agents that use probability thresholds to decide whether to execute an action or escalate it, since even a high-probability choice can still be wrong.
Fine-tuning also changes some of the capabilities inherited from the underlying model. Palmer’s evaluations show declines on general-knowledge and arithmetic tests, particularly among the smaller models. That’s consistent with Kev’s more specialized role alongside a general-purpose model, although its performance in dynamic agent environments will also depend on how well it handles tools, choices, and labels it never encountered during training — and debugging agent failures often points to infrastructure rather than the model itself.
The approach predates Kev. TypeSafe introduced Jev earlier this month as part of its System One platform, using the same Noul, Choice and Score primitives, and Kev implements its /v1/systemone request and response format so applications built against the API can point to a local Kev server instead.
Open weights, open training
The biggest difference is that Palmer released Kev under Apache 2.0 with the model weights, training code, and evaluation tooling, giving developers the option to run and train it on their own infrastructure. Jev’s weights and training data aren’t public, however, which makes direct performance comparisons difficult because differences between the models can’t be isolated to architecture, size, or training.
For applications that make only a handful of bounded decisions, constrained decoding on a model that’s already running may be simpler than adding another model to the stack. Agent loops can make those decisions constantly, however, moving through routing, ranking, safety checks, tool selection, and escalation before generating much user-facing text. It’s a pattern showing up across model architectures — stripping out unnecessary computation when the task doesn’t require it.
When those steps only require a choice or probability, Kev can handle the decision directly while leaving open-ended reasoning and final responses to the larger generative model.
When those steps only require a choice or probability, Kev can handle the decision directly while leaving open-ended reasoning and final responses to the larger generative model.
The post Your AI agent is burning tokens on choices that don’t need words appeared first on The New Stack.
