Broken tool calls on vLLM, the generic hf parser and the community's #1 pain
Your agent calls a function, the model responds with the tool-call JSON, and your server parses it wrong. The agent retries, the user waits, and the support thread fills with logs. That scenario was the most discussed inference topic in October. Ubicloud's post on fixing tool-call accuracy in vLLM topped the month's discussions. - Source Fixing Tool-Call Accuracy in vLLM
The good news is that vLLM v0.32 shipped the structural change this problem needed, and this guide covers why tool calls break, what changed, and how to debug the ones that still do.
Why tool calls break
The model doesn't emit clean tool-call JSON. It emits text in a specific conversation format, and that format varies per model family. Llama uses one dialect, Qwen another, Mistral its own. The tool-call parser's job is translating that text into the OpenAI schema your client expects.
vLLM's historical problem was the parser matrix. One parser per model family, hand-selected via --tool-call-parser, and a default that was wrong for your model unless you read the docs. The known outcome, tool calls that work in development with one model and break in production with another, or that break after updating the model to a version that changed its template.
The root cause doesn't live in vLLM but in the checkpoint. Most models already declare their response format in the tokenizer config or a response_template. Asking the operator to know that format and pick the right parser duplicated information and guaranteed misalignment. - Source PR #58604
What changed in v0.32
The generic hf parser reads the format straight from the checkpoint. The same treatment for tool calls and reasoning, with --tool-call-parser hf and --reasoning-parser hf.
vllm serve $MODEL \
--tool-call-parser hf \
--reasoning-parser hf
The per-model guessing game ends there. The parser trusts what the model's trainer declared, and new models work without waiting for the specific parser release. For models whose formats the template doesn't cover, dedicated parsers remain available, now as exception rather than rule.
The same version shipped /v1/systemone, a structured-decisions endpoint over a state, and v0.33 added /v1/decisions on by default for supported models, extending the idea to native classification and scoring. If your agent picked between options with a hand-rolled prompt and text parsing, there's an endpoint for that now. - Source v0.32.0 release, v0.33.0 release
The debugging that's still necessary
The generic parser solves selection, not quality. These are the four failure points I keep seeing, in frequency order.
The checkpoint's template is wrong or ambiguous. Happens with community fine-tunes. Check the tokenizer's actual response_template, and if it's wrong, fix it in your model repo instead of patching with a different parser.
The model hallucinates the format under pressure. With long outputs or forced structure, some models mix tool-call format with plain text. vLLM's official structured outputs guide covers forcing the schema at the sampler with guided decoding, which removes the entire class of format errors at some throughput cost. - Source vLLM structured outputs
Your client speaks a different dialect. The parser produces the OpenAI format. If your agent speaks Anthropic's tool-use dialect, the intermediate translation is your bug, not the engine's. Providers solve it with native endpoints, and vLLM exposes the Messages API alongside the OpenAI-compatible one.
The model version changed its template. The version pinning I apply to vLLM applies to the model too. A checkpoint update can change the response_template and your configured parser is now stale. The canary smoke test from my migration guide exists to catch exactly this. - Source vLLM v0.33.0
How to measure your tool-call accuracy
What isn't measured doesn't get fixed. The minimal harness I run per model and per vLLM version.
# tool-call-smoke.py — 20 cases, 3 tool families
import json, requests
CASES = [
("find flights from CDMX to LIM on December 12", {"tool": "search_flights", "args": {"from": "CDMX", "to": "LIM", "date": "2026-12-12"}}),
("cancel order 8842", {"tool": "cancel_order", "args": {"order_id": "8842"}}),
# ... your own set with your agent's real tools
]
def check(resp_json, expected):
calls = resp_json["choices"][0]["message"].get("tool_calls") or []
if not calls:
return False
c = calls[0]["function"]
return c["name"] == expected["tool"] and json.loads(c["arguments"]) == expected["args"]
passed = 0
for prompt, expected in CASES:
r = requests.post(f"{BASE}/v1/chat/completions", json={
"model": MODEL,
"messages": [{"role": "user", "content": prompt}],
"tools": TOOLS,
}).json()
passed += check(r, expected)
print(f"{passed}/{len(CASES)} tool calls correct")
Twenty cases cover 80% of the regressions I've seen. Run the harness on every version change of vLLM, model and parser, and keep the results next to the version pin. When the number drops, the model's template diff tells you why.
Final tips
- Switch to the
hfparser as soon as you're on v0.32+. It removes per-model configuration and aligns the engine with what the checkpoint declares. - Keep your checkpoints' templates in your repo. A
response_templatediff explains most mysterious tool-call regressions. - Guided decoding for critical schemas. If a malformed tool call triggers a payment or a deletion, forcing the schema is worth the throughput.
- Run the 20-case harness on every canary. The three places tool calling breaks are versions, templates and clients. All three show up in a 30-request canary.
- Report the missing parsers. The vLLM team prioritizes by usage signals, and the right parser is sometimes just missing for your model family. - Source vLLM GitHub
Broken tool calls stopped being a model mystery and became a verifiable pipeline problem. That's real progress.
How many of your agent bugs ended up being a mis-chosen parser? Tell me in the comments.