Orca
Microsoft Research's explanation-tuned small models, now a dated research release
Overview
Orca is a Microsoft Research line of small language models trained on the step-by-step explanations of larger models. Only Orca 2, released in November 2023 in 7B and 13B sizes on a Llama 2 base, has downloadable weights, under a non-commercial research licence; the later Orca-Math and AgentInstruct work released training data rather than models.
Pricing
Pricing shown for reference only. These figures reflect RECATOOLS research as of 14 Oct 2026 and may be out of date or incomplete. This is not financial or purchasing advice — always confirm the current price on the provider’s official website before making any decision.
Use cases
What you can produce with Orca
- Orca 2 7B and 13B checkpoints for research on reasoning distillation
- A reference implementation of explanation tuning to compare new methods against
- The 200,000-problem Orca-Math training set, under the MIT licence
- A one-million-pair AgentInstruct instruction dataset, under CDLA-Permissive-2.0
- Local experiments through Ollama's 7B and 13B builds with a 4K context
- A paper trail (Orca, Orca 2, Orca-Math, AgentInstruct) with reported results to reproduce
Orca matters for the technique it popularised: training small models on a teacher's reasoning explanations, an approach now common in synthetic-data pipelines. The Orca 2 checkpoints remain a clean, documented reference for that research, and the Orca-Math and AgentInstruct datasets are usable under permissive licences.
As a model to deploy it is a non-starter. The licence forbids commercial use, the Llama 2 base is two generations old, and the weights have not changed since November 2023. Teams that need a small open model should look at Microsoft's MIT-licensed Phi models instead.
What we found
Orca is a line of research models from Microsoft Research, and this page describes it as it stands in October 2026: a finished set of papers and two downloadable checkpoints, not a maintained product. The idea behind it, which Microsoft calls explanation tuning, is to train a small model on a large model's step-by-step explanations rather than only its final answers. The first Orca paper, in June 2023, described a 13-billion-parameter model trained on GPT-4 explanation traces that reached parity with ChatGPT on the Big-Bench Hard benchmark. Its weights were never published.
What can be downloaded is Orca 2, released in November 2023 in 7B and 13B sizes as fine-tunes of Meta's Llama 2. Orca 2 taught the small model to choose a reasoning strategy for each task, and the paper reports results similar to or better than models five to ten times larger on zero-shot reasoning tests. Both checkpoints sit on Hugging Face, ungated and unchanged since November 2023, under the Microsoft Research License, which allows only non-commercial, non-revenue research use. Llama 2's own licence applies on top. Microsoft's project page adds that the models should not be used in downstream applications.
Two later papers released data but not models. Orca-Math, in February 2024, fine-tuned Mistral-7B to 86.81% on the GSM8K maths benchmark, and Microsoft published its 200,000-problem training set under the MIT licence. AgentInstruct, in July 2024, trained a model called Orca-3 and reported gains of 40% on AGIEval and 54% on GSM8K over Mistral-7B-Instruct; a one-million-pair slice of its data is public under a permissive CDLA licence.
No hosted provider serves Orca 2 today, so running it means downloading the weights, for example through Ollama's 4K-context builds. For anything beyond research, Microsoft's small open models now ship as the MIT-licensed Phi family, and Orca is best read as the research record it is.
Compiled by RECATOOLS from public sources: vendor documentation and pricing, code repositories and published reporting.
Notable facts
- Orca was the first model to systematically train on GPT-4's reasoning traces rather than just final answers — a methodology that became widely adopted across the research community.
- Orca-Math 7B outperformed GPT-4 on the elementary school math benchmark MATH — a remarkable result for a model 100x smaller.
- The Orca research paper was cited over 500 times within 6 months of publication, making it one of the most impactful small LLM papers of 2023.
Frequently asked questions
About this listing
This entry was compiled from publicly available data including Orca's official website, press releases, documentation, and reputable third-party publications. RECATOOLS is not affiliated with Orca unless explicitly stated.
Third-party AI tools update their pricing, features, availability, and policies frequently. Information here may be outdated by the time you read this — we make reasonable efforts to keep listings current, but cannot guarantee absolute accuracy.
For the latest details, please refer to Orca directly →
Spotted something out of date? Suggest an update →
Related reading
Health & Wellness
Fine-Particle Air Pollution Linked to Higher Hospital Admission Risk for Up to 21 Days, St...
Cybersecurity
Microsoft Patched a Record 1,411 Flaws in October and Suggested a New Way to Sort Them
Health & Wellness
A Guide for New Mothers Missed Its Main Goal but Nearly Doubled Primary Care Visits
Alternatives to Orca
More in LLMs & Chat
