Training data
that has to earn
its place.
OpenAutodata runs Meta's Agentic Self-Instruct loop on open-weight models. A question ships only when a 3B model fails it and a strong model solves it. Everything else gets rewritten or thrown out.
- Best Open-Source AI Project
- Snowflake track
- MIT licensed
- Weak/strong gap, single-shot vs agentic, in Meta's paper
- 1.9 → 34pt
Weak/strong gap, single-shot vs agentic, in Meta's paper
- Every one open-weight, from five labs
- 5models
Every one open-weight, from five labs
- Graded per question: 3 weak, 3 strong, rubric by rubric
- 6answers
Graded per question: 3 weak, 3 strong, rubric by rubric
- Human annotation required to ship a dataset
- 0labels
Human annotation required to ship a dataset
Most synthetic data is never checked.
Big labs pay human annotators to write and review fine-tuning pairs. Everyone else prompts an LLM and ships whatever comes back. Teams report row counts. Nobody reports a pass rate.
- 01 Too easy
The model already knows the answer. It trains on nothing.
“What was Q1 revenue?” Pure lookup.
- 02 Unanswerable
The source text cannot support the answer, so the model learns to guess.
Ambiguous base for a 97/3 split.
- 03 Wrong key
The reference answer itself is off, and fine-tuning bakes the error in.
Gross price read as net sales.
Five checks. All must pass.
Each solver answers three times and the judge scores every answer on its own. The weak model has to struggle without zeroing out, the strong model has to land without acing it, and the gap has to be at least 20 points. These are the exact thresholds from Meta's paper. Drag the runs and watch the gate decide.
node_evaluate
- passWeak avg ≤ 6548.3
- passWeak max ≤ 7555
- passNo zero scoresclean
- passStrong avg in [60, 95)85.0
- passGap ≥ 2036.7
Verdict
Accepted
Ships to the dataset.
From a 10-Q to a fine-tune file.
A LangGraph state machine with one conditional edge that matters: revise or ship.
- 01
Ingest
SQL pulls the Results of Operations section of real 10-Q filings from Snowflake Marketplace, cleans them and splits them into chunks.
SELECT filing text → data/chunks.json - 02
Challenge
GLM-5.3 reads one chunk and writes four things: a context with only the facts needed, a multi-step question, a reference answer, and a 10 to 15 criterion rubric with weights from 1 to 7.
chunk → { context, question, reference_answer, rubric } - 03
Quality check
Before any solving, Nemotron 3 Super checks that the context does not leak the answer, the question tests reasoning rather than recall, and every calculation criterion states its expected value.
package → PASS | FAIL - 04
Solve
Llama 3.2 3B answers three times from the context alone, never the filing or the reference answer. DeepSeek V4.1 Flash gets the identical prompt three times, every round, so each round has a gap.
context + question → 3 weak, then 3 strong answers - 05
Judge
Nemotron 3 Super grades each answer on its own: met or not met, criterion by criterion. It never sees the reference answer. Code turns the verdicts into a weighted 0 to 100 score.
each answer → met / not met per criterion → score - 06
Gate
node_evaluate runs the paper's checks. A failure is labelled too easy, too hard, failed on strong or failed quality check, and the Challenger writes an entirely new question with the judge's notes. Three rounds, then the chunk is discarded with its reason logged.
scores → ACCEPTED | REVISE | REJECTED - 07
Persist
Every round lands in OPENAUTODATA_TRAJECTORIES. Accepted pairs land in OPENAUTODATA_ACCEPTED_SET, ready to export for fine-tuning.
→ Snowflake + data/accepted.json
Five open-weight models. Six roles.
Different labs, different sizes, on purpose. The weak solver is a small language model, 3B parameters, small enough to struggle. Every other role runs a large model. The benchmark judge comes from a separate family so the pipeline never grades its own homework.
- LLM · large
Reads the filing chunk and writes a context, a question, a reference answer and a 10 to 15 criterion rubric. Rewrites from scratch when a round fails.
License: GLM-5.3 License - LLM · 120B, 12B active
Before anything is solved, checks that the context does not leak the answer, the question tests reasoning not recall, and the rubric is checkable.
License: NVIDIA Nemotron Open Model License - SLM · 3B
Answers 3 times from the context alone. Should miss. If it aces the question, the question is too easy.
License: Llama 3.2 Community License - LLM · large
Same prompt, 3 times. Only runs if the weak solver struggled. Should land the answer without acing it.
License: MIT - LLM · 120B, 12B active
Grades each answer on its own, criterion by criterion, met or not met. Never sees the reference answer. Code turns verdicts into a 0 to 100 score.
License: NVIDIA Nemotron Open Model License - LLM · 2.4T, 95B active
Blind A/B judge for the final benchmark. Never touches the loop, so the pipeline never grades itself.
License: Qwen3.8-Max License
Runs on
- LangGraph
- OpenRouter
- Snowflake
- Hugging Face
- Meta FAIR
Measured, not claimed.
Meta's number sets the bar. Our blind benchmark, judged by a model that never touches the loop, shows whether we clear it.
Reference · Meta FAIR, AutoData
Weak/strong gap, in points
single-shot generation
the loop OpenAutodata implements
Ours · blind A/B, Qwen3.8 judge
Same SEC chunks. Single-shot vs OpenAutodata.
difficulty, 1-10, independent judge
+3.5 harder
The judge picked OpenAutodata as harder in 2 of 2 pairs and preferred it overall in 2 of 2, but rated it more valid in 0 of 2.
First run · only 2 pairs judged · a larger run is next

Run log
data/trajectories.json · polling every 5s- Rounds logged
- 0
- Accepted pairs
- 0 of 0 chunks
- Rounds per accepted pair
- n/a
- Mean accepted gap
- n/a
Your documents. Your key. Your dataset.
Clone it, add an OpenRouter key, point it at any CSV of text chunks. SEC filings are the demo, not the limit.
$ git clone https://github.com/ritvikreddygangula/Open-AutoData && cd Open-AutoData$ python -m venv .venv && source .venv/bin/activate$ pip install -r requirements.txt$ cp .env.example .env # set OPEN_ROUTER=sk-or-...$ python -m src.chunks my_filings.csv # any CSV with CHUNK_ID, CHUNK_TEXT$ python -m src.agent_graph # full loop, lands with feat/agent-graph
Standing on Meta FAIR's Agentic Self-Instruct.
AutoData treats data creation as an agent loop: a challenger proposes, a weak and a strong solver attempt, a judge scores against a rubric, and the loop revises until the example splits the two solvers. Meta measured a 34 point weak/strong gap with this method versus 1.9 for single-shot CoT Self-Instruct. We rebuilt the loop as an open-source LangGraph harness, swapped in a different open-weight model set, and pointed it at SEC filings in Snowflake.
Kulikov, Whitehouse, Wu, Nie, Saha, Helenowski, Yuan, Golovneva, Lanchantin, Bachrach, Foerster, Li, Fang, Sukhbaatar, Weston