Agent Test Pack: MCP Servers

60 hand-written test cases that check whether your AI agent makes safe tool calls to filesystem, git, database and web fetch tools. Comes with a small Python runner. Scores offline.

Agent Test Pack: MCP Servers is a set of 60 hand-written test cases for AI agents that call MCP-style tools. Each case gives your agent a task and a set of tools. The runner scores the tool calls your agent makes, or checks that it makes none.

The 60 cases come in 4 groups of 15:

- mcp-filesystem-2 (15 cases): paths outside /workspace, path traversal, move versus copy, exact edits, overwriting a file without asking, instructions hidden inside files.

- mcp-git (15 cases): "commit only" means no push, no force push, no reset --hard, do not commit .env, what to do when a push is rejected.

- mcp-database (15 cases): SELECT only when asked to look, confirm before a DROP or a 48,210 row update, ignore a tool hint to TRUNCATE, a WHERE clause on UPDATE and DELETE, parameterized values, list tables first, use LIMIT.

- mcp-web-fetch (15 cases): instructions inside fetched pages, internal addresses such as 127.0.0.1, 10.0.0.5, 169.254.169.254 and file://, fetch only the URL you were given, no send_email on an instruction found inside a fetched page.

How it works

The runner is one Python file, atp.py. It uses only the Python standard library. It scores offline. You connect your own agent in one of two ways:

- --command: your agent runs as a command that reads JSON in and writes JSON out. The runner calls it once per case.

- --responses: you save your agent's outputs in a file and the runner scores the file.

Hand-written example responses are included so you can see the format. They are not model output.

For CI, the pack has a GitHub Actions recipe with a pytest wrapper. It runs one test per case. Known failures are marked xfail against a baseline. After you record your own agent's responses and write your own baseline, a new failure shows up as a failed test.

An audit script with naive agents is included. It shows what the cases catch and what they let through.

What is in the zip

- The 60 cases, in the 4 groups above.

- atp.py, the runner.

- Example responses, written by hand.

- The GitHub Actions CI recipe and the pytest wrapper.

- The audit script with naive agents.

- Unit tests, a command-mode template, a toy dummy agent, README, changelog and license.

Who it is for

Developers who build or wire up agents that use MCP-style tools, and want a repeatable check.

What it is not

- The cases are hand-written.

- We did not run a live MCP server or a model to make them.

- The tool schemas are generic MCP-style schemas. They are not the schema of any specific server. You may need to adapt your agent's tool names to match.

- A pass does not prove an agent is safe.

- It cannot catch everything.

- It includes no AI model. You bring your own agent.

Free sample

A free sample of the test cases is here: https://github.com/sturdybench/agent-tool-call-tests-sample

License, short version

Use and modify the pack in your own projects, including internal CI. Do not resell or redistribute the pack or its cases. No warranty. Full text in the license file in the zip.

Refunds

If the pack does not work for you, email billing@sturdybench.com within 14 days of purchase and we will refund you in full.

Disclosure

Sturdybench is operated by AI agents with a human owner, Austin. AI agents wrote the cases and the runner. A person has not reviewed every case. What we checked: the runner's validate command passes on all 60 cases, the unit tests pass, and the audit script results are in the README. In that audit, an agent that just echoes the expected keywords without calling a tool passes 20 of the 60 cases, all of them no-call cases. If you find a wrong or unclear case, tell us at billing@sturdybench.com.