Akash Kokare/Notes

Research note · 30 September 2026 · EvalForge

An agent that does nothing passed 40% of my benchmark

A benchmark means something only if an agent with no ability scores zero. Mine didn’t.

My benchmark has 20 customer-support cases for tool-using agents, graded on what the agent did, not just what it said. Others have found do-nothing agents passing 38% of τ-bench airline tasks (Zhu et al., NeurIPS 2025), so I ran the same check on mine.

Method

Six agents with no ability, scored by the real evaluators, with no model called. They do nothing, refuse, ask a generic question, or do one lookup and then say “Done.”, falsely claim success, or give a canned cautious reply.

Finding

8 of 20 cases passed. All eight were cases where the right move is to change nothing, such as refunding an order that doesn’t exist. Doing nothing satisfies every state check, and nothing read the reply. So “order #9999 has been refunded” passed, as long as no refund was made.

Fix

A deterministic reply check: each case lists what the reply must mention and what it must not claim. My first version accepted generic phrases, and the canned-reply agent passed four cases with one sentence. The rule that worked: a required mention must be a case-specific fact, such as an order id, that only an agent that looked could know.

v1v2
Cases a do-nothing agent passes8/200/20
GPT-4o17/2017/20
GPT-4o-mini19/2019/20

No real model’s verdict changed. Every failure the new check catches was a false claim.

Does it pass every time?

pass^k is the chance that all k tries of a case succeed (Yao et al., τ-bench). I estimate it per case without bias, C(c,k)/C(n,k), and bootstrap over cases rather than runs. gpt-4o-mini, 8 tries per case, cost $0.134: pass^8 = 95%, the same as pass^1. No case was flaky, so a harder probe is needed before consistency shows up.

Limits

← Akash Kokare Counterexample? Email me