Discover great AI tools
📢 NoticeWebsite launched:DeepToSeek AI tools directory is live. Submit great tools and get featured!
Home / Open Source Radar / Inference & Training / promptfoo

Test your prompts, agents, and RAGs. Red teaming/pentesting/vulnerability scanning for AI. Compare performance of GPT, Claude, Gemini, DeepSeek, and more. Simple declarative configs with command line and CI/CD integration. Used by OpenAI and Anthropic.

Stars27.4k
Forks2.8k
LanguageTypeScript
LicenseMIT
Last push2026-11-18

About this project

promptfoo is an open-source inference and training project on GitHub: Test your prompts, agents, and RAGs. Red teaming/pentesting/vulnerability scanning for AI. Compare performance of GPT, Claude, Gemini, DeepSeek, and more. Simple declarative configs with command line and CI/CD integration. Used by OpenAI and Anthropic. It is infrastructure rather than a finished product: the layer that serves, quantises, trains or orchestrates models.

On GitHub the repository is tagged ci, ci-cd, cicd, evaluation, evaluation-framework, llm, llm-eval, llm-evaluation, llm-evaluation-framework and llmops, which is a fair summary of what it covers. That generally means an API surface, batching and memory management, and often an OpenAI-compatible endpoint so existing clients keep working unchanged.

It is written mainly in TypeScript and released under MIT, a permissive license — commercial use is fine as long as copyright notices are kept. It is actively maintained: the most recent commit landed on November 18, 2026.

To try it, clone the repository and follow its README; that file is the canonical install guide. There is also a project page at https://promptfoo.dev for documentation or a hosted demo. Because it runs on your own infrastructure, budget time for environment setup: dependencies, model weights and configuration usually take longer than the first successful run.

Key facts as recorded on GitHub: 27,403 stars; 2,790 forks; primary language TypeScript; license MIT; last commit on November 18, 2026; topics ci, ci-cd, cicd, evaluation, evaluation-framework, llm.