Promptfoo
Local-first LLM evaluation and red teaming CLI for agent safety testing.
Open source
Best for teams that want to run evals locally, in CI, or before deploying agents—covering prompt quality, safety red teaming, and regression testing.
Selection advice
Choose Promptfoo when you want to catch prompt and model regressions before deployment. It's the testing framework for LLM prompts that CI/CD pipelines need.
Quick comparison
Promptfoo is the default pick for YAML-driven prompt evals and red teaming in CI. DeepEval and Braintrust are common alternatives when pytest-style agent tests or a hosted eval platform is the priority.
| Promptfoo | DeepEval | Braintrust | |
|---|---|---|---|
| Best for | Prompt evals, red teaming, and model comparison in CI | Python pytest-style LLM and agent test cases | Hosted eval workflows with dataset versioning |
| Workflow fit | YAML configs and CLI in local or CI pipelines | pytest assertions with built-in quality metrics | Cloud UI for experiments and production monitoring |
| Red teaming | First-class red team guides and attack suites | Safety metrics; less attack-library focus | Eval platform features; red team varies by setup |
| Tradeoff | CLI-first; less hosted platform out of the box | Python-only ergonomics for many teams | Strong platform; more vendor surface than a CLI |
Best for
- local evals
- CI/CD testing
- red teaming
- prompt comparison
Not ideal for
- teams that need a hosted evaluation platform only
- projects where production monitoring is the main need
Core concepts
evalsred teamingpromptsassertionsCI integration
Minimal implementation shape
Define eval assertions in YAML, run promptfoo eval in CI, automatically compare outputs across models, and block deployment on regressions.