Skip to content
Aerobic Model
Deep dives Article #05

How I test that the model picks the right tool: 96 scenarios, 96 first calls right

A small eval for an MCP server: one prompt, one expected first call, the real server with a fake data layer, and what its misses taught me about tool descriptions.

9 min read Any sport
  • Code & method included
  • AI-assisted, author-checked
Key numbers #05
96/96first tool calls right

A drawer organizer with about twenty small tool tiles; a glowing line runs from a speech bubble to one wrench tile lifted out of its slot, next to a clipboard with ticked check boxes.

One question, one tool, and a checklist that says whether it was the right one. (AI-generated illustration)

In the previous article of this series I wrote about the tool list of Garmin MCP, my unofficial, open-source MCP server for Garmin Connect: 21 tools built around questions, about 17,100 tokens of context, and a test that fails the build when the list grows. But a small tool list can still be a confusing one, and the only way I know to find out is to ask the model.

So the repository has an eval. It is a file of 96 user messages, each with the tool call I expect the model to make first. In the last full run, on 18 November 2026 with Sonnet, the first call was right in 96 of 96. This article is about how that eval works, why it checks so little on purpose, the misses that changed my tool descriptions, and what the number does not tell you.

The descriptions are the interface

When a user writes "how did I sleep this week?", the model sees the message, the server instructions and the tool list: names, descriptions, input schemas. Nothing else tells it that query_health is the tool for sleep hours across a week and get_sleep is the one for the stages of one night. If two descriptions both say "sleep", the model guesses.

Unit tests check that get_sleep returns the right night. They say nothing about whether the model will ever call it. The eval tests the part of the server written in English prose.

What a scenario says

Each line of docs/evals/tool-selection.jsonl is one JSON object: a prompt, the expected first call and, usually, the calls that would be a mistake.

Five scenario lines from the eval file, wrapped for reading and colour-coded: one tool with arguments; a list of four acceptable tools with arguments per tool; null for a delete request; null or get_activity with update_activity forbidden; query_health with args_exclude granularity hour.

The five shapes a scenario can take. All five are real lines of the file.

The format grew one field at a time, whenever a scenario could not say what I meant:

  • expect.tool is a tool name. It can be a list when several first calls are right: for "Zrób mi trening 6x400 m" (build me 6x400 m) the model may build the workout straight away or first read thresholds, training status or the calendar.
  • null means no tool fits. With delete tools switched off, "Usuń aktywność 123" (delete activity 123) should get an answer, not a call. null can also sit in a list: [null, "get_activity"] means "answer directly, or read first".
  • args_include is a subset of the call's input. Extra keys are fine, arrays match element by element in any order, and a number matches a numeric string, since the server accepts ids both ways. With a list of tools it can be keyed by tool name.
  • args_exclude fails a call that contains it. A question about stress "day by day this week" must not go to query_health with granularity: "hour", which is the right tool with the wrong answer.
  • forbid lists tools that fail the scenario. A tool cannot be both expected and forbidden; the parser rejects that line.

Before anything runs, the runner checks every tool name in the file against the server's real tool list, so a renamed tool breaks the eval loudly.

Why only the first call

The eval looks at one thing: the first tool_use block of the first assistant message. Everything after it is ignored, and claude runs with --max-turns 1.

That sounds like too little, but the first call is where the descriptions do their work. After it, the next call depends on the data that came back, and in this setup there is none, as you will see in a moment. It also keeps a scenario cheap and the verdict simple: right tool, right key arguments, nothing forbidden. In my small runs a scenario took a few seconds.

The real server, a fake Garmin

The eval does not mock the MCP layer. It starts the real server over stdio: the same tools, descriptions, input schemas, server instructions and resources a user gets. Only the data layer is replaced, by a GarminService that answers every method with an empty envelope:

const empty = async () => ({ data: {}, meta: { source: 'cache' } });
const service = new Proxy(
  { withAutoSync: (read) => read() },
  { get: (target, key) => (key in target ? target[key] : key === 'then' ? undefined : empty) },
);

Nothing reaches Garmin, and nothing touches my local cache. The destructive tools are off, as they are by default, so the scenarios see the same 21 tools a new user sees.

The model side is a signed-in Claude Code, called as claude -p once per scenario, with flags that strip away everything that is not the server:

  • --tools "": no built-in tools, so the model cannot reach for a file read or a web search instead;
  • --setting-sources "": no user or project settings files;
  • --mcp-config with only the fake server, plus --strict-mcp-config, so my other MCP servers are not in the list;
  • --output-format stream-json, so the runner can read the tool call as data;
  • and a fresh temporary directory as the working directory, so the repository's CLAUDE.md and project memory never reach the prompt.

That last point matters. Run the eval from inside the repository and the model may pick the right tool because it read my notes, not my descriptions.

Diagram: tool-selection.jsonl feeds eval-tools.mjs, which starts claude -p with its flags in a fresh temporary directory; claude talks over stdio to eval-fake-server.mjs, the real server with a fake GarminService that never touches Garmin Connect or the local cache; stream-json events go to firstToolCall, then judge, then a table and pass rate with an exit code.

One scenario, end to end. The runner repeats this for every line of the file.

Polish and English

Of the 96 scenarios, 50 are in Polish and 46 in English. I ask my training questions in Polish, while the tool descriptions are in English only. The eval checks that a question like "O której się budziłem dziś w nocy?" (when did I wake up last night?) still finds get_sleep, not the health table.

Language has not been the cause of the misses I have seen. Ambiguity has, although one of them below happened in Polish while its English twin passed.

What the misses taught me

The earliest run in the history, on 12 October, was 31 of 31. Since then new features have brought new scenarios, and I run the whole file after changes to the descriptions and at the end of each larger batch of work.

Bar chart of every full run: 31/31 on Oct 12, then 41/42 twice, 42/42, 48/48 on Oct 30, 51/51, 58/58 and 61/61 on Nov 17, 90/94 after 33 new scenarios, 94/94 and 96/96 on Nov 18. Misses are shown in red.

Every full run in the repository's history. Three of them had misses, and each one changed something.

A scenario that was wrong. After I merged the workout preview into create_workout with dry_run: true, the run came back 41 of 42. For "Zrób mi trening 6x400 m" the model first called get_thresholds, to know my threshold pace before building intervals. It did so every time, also before my change. A coach would ask the same question, so I widened the scenario to accept it.

A description that was wrong. The next run was again 41 of 42, with a new miss: "Ile procent czasu biegania i roweru spędziłem w tym miesiącu w Z1, poniżej LT1?" (what share of my running and cycling this month was in Z1, below LT1?). The model called get_period_summary instead of get_load, in the full run and in three reruns of that scenario alone. The month summary advertised an "HR zone distribution", and that was close enough.

Diff of two tool descriptions: get_period_summary replaces "HR zone distribution" with "time in the Garmin HR zones 1-5", pointing the 3-zone split against LT1/LTHR to get_load; get_load adds "against LT1/LTHR" and "answers time below LT1, % in Z1, polarisation". Before: get_period_summary in 4 of 4 tries. After: get_load in 3 of 3 reruns, full run 42/42.

The fix was in the descriptions, not in the scenario. It cost 175 bytes of context.

The fix follows a pattern I try to keep in every description: say exactly what your numbers are, and name the neighbouring tool for the neighbouring question.

Trimming with a safety net. In November I shortened the descriptions, taking out text repeated between a tool and its parameters. Removing words that route the model is how you break tool selection without noticing, so with the trim came seven scenarios aimed at the removed phrases. One of them: "On activity 21001 set my perceived effort to 7 and how I felt to strong." The mapping of "strong" to feel: 75 moved from the tool description into the parameter description, and the scenario's args_include checks it still arrives. The run after the trim was 58 of 58.

When the scenarios are new, suspect the scenarios. A larger batch of features added 33 scenarios, and the next full run was 90 of 94. All 61 older scenarios passed; all four misses were new. In two, about fixing the exercises of a strength workout while overwrites are switched off, I expected no call, and the model first read the sets with get_activity. Reading is careful, not wrong, and it never tried the write, so those scenarios became [null, "get_activity"] with update_activity forbidden. In one, comparing the best 20-minute power of two rides went to get_activity with the power curve instead of compare_activities; both give the right answer, so both are accepted. In the last, a Polish prompt with "na sobotnim trailu 21001" (on Saturday's trail 21001) did not read 21001 as an id, while the English version did. I rewrote the prompt so the id is unambiguous. The descriptions did not change by a single byte, and the next run was 94 of 94.

That is a judgment call, so I write it down in the commit message: what the model called, and why the expectation moved instead of the description.

Where the scenarios point

The file is not balanced and does not need to be. Every one of the 21 tools is an accepted first call in at least one scenario, and the busy ones get more: get_activity in 14, query_health in 12, update_activity in 11.

Heatmap of scenarios per tool: columns for Polish and English scenarios that accept the tool as first call, their sum, and how many scenarios forbid it. get_activity is accepted in 14, query_health in 12; garmin_sync is forbidden in 53 scenarios and get_activities in 21.

Which first call each scenario accepts, and which tools would fail it.

The right column is more useful to me. 84 scenarios have a forbid list, and garmin_sync is in 53 of them. The read tools download missing data on their own, and the server instructions say there is no need to sync first. A model that calls garmin_sync before answering "how did I sleep?" makes the user wait for nothing, and the eval catches that. get_activities is forbidden in 21: questions about planned races and workouts, where past activities are the wrong place to look, and questions that already name an activity id.

Running it

pnpm nx build garmin-mcp-server
node scripts/eval-tools.mjs [scenarios.jsonl]

The runner takes any scenario file, which is how I check a few new lines without a full run. EVAL_MODEL picks the model (Sonnet by default) and EVAL_MIN_PASS the threshold, 0.9 by default: exit code 0 at or above it, 1 below, 2 when claude is not installed.

Terminal: node scripts/eval-tools.mjs on a file of three scenarios prints "3 scenarios, claude -p --model sonnet, 21 tools", a table where get_load, get_sleep and "no tool" are all ok, and "pass rate 3/3 = 100.0% (minimum 90.0%)".

Three scenarios from the file, run while writing this article: the zone question from above, a night named by its morning date, and a delete request with delete tools off.

The eval is not part of CI. It needs a signed-in Claude Code, not an API key, and every scenario counts against my plan's limits. CI runs offline tests of the harness instead: the scenario parser, the matcher and the stream-json parser against two fixture files. A broken harness that passes everything would be worse than no eval.

What 96 of 96 does not mean

I like the number, and I do not trust it far.

  • One model. Every run is Sonnet through Claude Code. Other clients send the tool list differently, and other models read descriptions differently. I have not measured them.
  • One run per scenario. Model output varies. When a scenario fails, I rerun it alone, usually three times, before deciding anything, but a full run is one sample per line. The default threshold of 0.9 leaves room for that.
  • One call. The eval does not see the second call, the arguments the model fills in later, or whether it gives up after an empty answer.
  • Not the answer. It says nothing about whether the final reply is correct or useful. With empty data there is no reply worth judging.
  • My own scenarios. I wrote all 96, knowing the tools. Real users phrase things in ways I do not, and a scenario file only covers the questions someone thought of.

What it does do is turn "I think the descriptions are clear" into a file that fails when they are not.

If you are building an MCP server

  • Write prompts with the tool you expect first. Start with the questions where two tools are close.
  • Check only the first call, against the real server with a fake data layer.
  • Run the client with nothing else: no built-in tools, no settings, no other servers, an empty working directory.
  • Use forbid for the cheap wrong turns, such as syncing first or listing when the user gave an id.
  • Before trimming a description, add scenarios for the words you are about to remove.
  • When a scenario fails, rerun it alone, then decide whether the description or the scenario is wrong, and write down which.

The scenarios are in docs/evals/tool-selection.jsonl, the runner in scripts/eval-tools.mjs and the fake server in scripts/eval-fake-server.mjs at https://github.com/endurancetools/garmin-mcp. The documentation is at https://endurancetools.github.io/garmin-mcp/. I would like to hear how others test tool selection, and whether anyone has found a good way to test the answer, not just the call.

Unofficial; not affiliated with or endorsed by Garmin.

Some illustrations in this article were generated with ChatGPT.

marcin_running mark
Written and signed by

Marcin @marcin_running

Runner, triathlete and ultrarunner. I build open-source tools that let AI read training data.

Follow @marcin_running on Instagram