coder_eval
Sandboxed, reproducible evaluation framework for coding agents and their skills.
Open source
coder_eval is an open-source UiPath framework for authoring coding-agent tasks in YAML, running Claude Code, Codex, or Google Antigravity with selectable sandbox drivers, and scoring their files, commands, tool use, cost, and token data.
Official resources
Selection advice
Choose coder_eval when the evaluation target is a coding agent or reusable agent skill and the score must reflect real files, commands, and tool behavior. Choose a general LLM evaluator when text-output metrics are the primary requirement.
Quick comparison
coder_eval targets complete coding-agent runs and skill behavior. DeepEval is organized around Python test cases and metrics, while Promptfoo covers config-driven prompt evaluation and red teaming.
| coder_eval | DeepEval | Promptfoo | |
|---|---|---|---|
| Best for | Reproducible coding-agent and skill evaluation | Python LLM and agent test suites | Prompt, model, and red-team regression tests |
| Evaluation unit | Full agent run with files, commands, and tool calls | Test cases with metric assertions | Configured prompts, providers, and assertions |
| Workflow | YAML tasks, sandbox drivers, weighted criteria, and CI gates | pytest-style Python tests and built-in metrics | YAML or JavaScript config with a local CLI |
| Tradeoff | Requires Python 3.13+ and your own agent credentials | Python-centric rather than coding-agent-sandbox specific | Broader eval scope with less focus on coding-agent sandboxes |
Best for
- coding-agent benchmarks
- agent skill evaluation
- A/B testing agent configurations
- CI quality gates
Not ideal for
- teams seeking a managed hosted evaluation service
- runtimes that cannot use Python 3.13 or newer
- untrusted tasks run with the temp-directory driver
Core concepts
declarative YAML taskssandbox driversweighted success criteriaagent pluginsA/B experiments
Minimal implementation shape
Define the prompt, agent, sandbox, and weighted success criteria in a YAML task; validate it with `coder-eval plan`, run it with `coder-eval run`, then inspect the generated report.