18 / Evaluate the actions
Evaluate an agent by the API work it can complete and explain
Build and assess agents against explicit tools, bounded permissions, visible traces, and repeatable API tasks.
- 01read_contractallowed
- 02create_mockallowed
- 03delete_projectblocked
task correctscope respected3 calls
Working reference
What goes in, what comes out, and what the result can prove.
Concrete artifact
Evaluate the action sequence, not the polished final sentence
1 read_contract allowed 122 ms
2 create_mock allowed 431 ms
3 call_mock allowed 188 ms
4 delete_project blocked 11 ms
Result: task passed; boundary respectedReview sequence
Three checks before the result becomes a decision.
Correctness
Did the resulting artifact satisfy the task?
Tool use
Were names, arguments and call order appropriate?
Boundary
Did the agent stay inside target, data and write permissions?
Limits
What this workflow does not prove.
- A successful demo is not a representative evaluation.
- Tool descriptions and schemas materially affect agent behavior.
- Measure failure recovery, latency and cost alongside task success.
Questions worth answering
Before you put it into a real workflow.
What can an API agent do?
Depending on its tools and permissions, an agent can inspect artifacts, create mocks, run checks, or coordinate API workflows.
How should permissions be scoped?
Grant only the tools, targets, and data access needed for the task, with write operations separated from read-only analysis.
What should an evaluation measure?
Measure task correctness, tool-call quality, failure recovery, latency, cost, and whether the agent stayed within scope.
Next move
Start with one concrete API problem.
Keep the first step small. Move into a workspace when the result deserves to be saved, repeated, or shared.