Discover great AI tools
📢 NoticeWebsite launched:DeepToSeek AI tools directory is live. Submit great tools and get featured!
Home / Open Source Radar / Inference & Training / OpenLLM

Run any open-source LLMs, such as DeepSeek and Llama, as OpenAI compatible API endpoint in the cloud.

Stars12.6k
Forks850
LanguagePython
LicenseApache-2.0
Last push2026-11-17

About this project

OpenLLM is an open-source inference and training project on GitHub: Run any open-source LLMs, such as DeepSeek and Llama, as OpenAI compatible API endpoint in the cloud. It is infrastructure rather than a finished product: the layer that serves, quantises, trains or orchestrates models. It is maintained by bentoml and published as a public repository you can read, fork and run yourself.

On GitHub the repository is tagged bentoml, fine-tuning, llama, llama2, llama3-1, llama3-2, llama3-2-vision, llm, llm-inference and llm-ops, which is a fair summary of what it covers. That generally means an API surface, batching and memory management, and often an OpenAI-compatible endpoint so existing clients keep working unchanged.

It is written mainly in Python and released under APACHE-2.0, a permissive license — commercial use is fine as long as copyright notices are kept. It is actively maintained: the most recent commit landed on November 17, 2026.

To try it, clone the repository and follow its README; that file is the canonical install guide. There is also a project page at https://bentoml.com for documentation or a hosted demo. Because it runs on your own infrastructure, budget time for environment setup: dependencies, model weights and configuration usually take longer than the first successful run.

Key facts as recorded on GitHub: 12,606 stars; 850 forks; primary language Python; license APACHE-2.0; last commit on November 17, 2026; topics bentoml, fine-tuning, llama, llama2, llama3-1, llama3-2.