Skip to content
OpenAutodataGitHub
Built on Meta FAIR's AutoData research

Training data
that has to earn
its place.

OpenAutodata runs Meta's Agentic Self-Instruct loop on open-weight models. A question ships only when a 3B model fails it and a strong model solves it. Everything else gets rewritten or thrown out.

  • Best Open-Source AI Project
  • Snowflake track
  • MIT licensed
Live replaywaiting for trajectories
Weak/strong gap, single-shot vs agentic, in Meta's paper
1.9 → 34pt

Weak/strong gap, single-shot vs agentic, in Meta's paper

Every one open-weight, from five labs
5models

Every one open-weight, from five labs

Graded per question: 3 weak, 3 strong, rubric by rubric
6answers

Graded per question: 3 weak, 3 strong, rubric by rubric

Human annotation required to ship a dataset
0labels

Human annotation required to ship a dataset

01The problem

Most synthetic data is never checked.

Big labs pay human annotators to write and review fine-tuning pairs. Everyone else prompts an LLM and ships whatever comes back. Teams report row counts. Nobody reports a pass rate.

  1. 01 Too easy

    The model already knows the answer. It trains on nothing.

    “What was Q1 revenue?” Pure lookup.

  2. 02 Unanswerable

    The source text cannot support the answer, so the model learns to guess.

    Ambiguous base for a 97/3 split.

  3. 03 Wrong key

    The reference answer itself is off, and fine-tuning bakes the error in.

    Gross price read as net sales.

02The math gate

Five checks. All must pass.

Each solver answers three times and the judge scores every answer on its own. The weak model has to struggle without zeroing out, the strong model has to land without acing it, and the gap has to be at least 20 points. These are the exact thresholds from Meta's paper. Drag the runs and watch the gate decide.

Weak solverLlama 3.2 3B
Strong solverDeepSeek V4.1 Flash
Average positions0 to 100
60
65
95
W
S

node_evaluate

  • passWeak avg ≤ 6548.3
  • passWeak max ≤ 7555
  • passNo zero scoresclean
  • passStrong avg in [60, 95)85.0
  • passGap ≥ 2036.7

Verdict

Accepted

Ships to the dataset.

03The pipeline

From a 10-Q to a fine-tune file.

A LangGraph state machine with one conditional edge that matters: revise or ship.

  1. 01

    Ingest

    SQL pulls the Results of Operations section of real 10-Q filings from Snowflake Marketplace, cleans them and splits them into chunks.

    SELECT filing text → data/chunks.json
  2. 02

    Challenge

    GLM-5.3 reads one chunk and writes four things: a context with only the facts needed, a multi-step question, a reference answer, and a 10 to 15 criterion rubric with weights from 1 to 7.

    chunk → { context, question, reference_answer, rubric }
  3. 03

    Quality check

    Before any solving, Nemotron 3 Super checks that the context does not leak the answer, the question tests reasoning rather than recall, and every calculation criterion states its expected value.

    package → PASS | FAIL
  4. 04

    Solve

    Llama 3.2 3B answers three times from the context alone, never the filing or the reference answer. DeepSeek V4.1 Flash gets the identical prompt three times, every round, so each round has a gap.

    context + question → 3 weak, then 3 strong answers
  5. 05

    Judge

    Nemotron 3 Super grades each answer on its own: met or not met, criterion by criterion. It never sees the reference answer. Code turns the verdicts into a weighted 0 to 100 score.

    each answer → met / not met per criterion → score
  6. 06

    Gate

    node_evaluate runs the paper's checks. A failure is labelled too easy, too hard, failed on strong or failed quality check, and the Challenger writes an entirely new question with the judge's notes. Three rounds, then the chunk is discarded with its reason logged.

    scores → ACCEPTED | REVISE | REJECTED
  7. 07

    Persist

    Every round lands in OPENAUTODATA_TRAJECTORIES. Accepted pairs land in OPENAUTODATA_ACCEPTED_SET, ready to export for fine-tuning.

    → Snowflake + data/accepted.json
04The agents

Five open-weight models. Six roles.

Different labs, different sizes, on purpose. The weak solver is a small language model, 3B parameters, small enough to struggle. Every other role runs a large model. The benchmark judge comes from a separate family so the pipeline never grades its own homework.

Runs on

  • LangGraph
  • OpenRouter
  • Snowflake
  • Hugging Face
  • Meta FAIR
05The results

Measured, not claimed.

Meta's number sets the bar. Our blind benchmark, judged by a model that never touches the loop, shows whether we clear it.

Reference · Meta FAIR, AutoData

Weak/strong gap, in points

CoT Self-Instruct1.9

single-shot generation

Agentic Self-Instruct31.4

the loop OpenAutodata implements

Ours · blind A/B, Qwen3.8 judge

Same SEC chunks. Single-shot vs OpenAutodata.

Single-shot baseline2.5

difficulty, 1-10, independent judge

OpenAutodata6

+3.5 harder

The judge picked OpenAutodata as harder in 2 of 2 pairs and preferred it overall in 2 of 2, but rated it more valid in 0 of 2.

First run · only 2 pairs judged · a larger run is next

Benchmark: difficulty 2.5 for single-shot baseline vs 6.0 for OpenAutodata; blind judge preferred OpenAutodata in 100% of pairs, harder in 100%, more valid in 0%; 2 pairs judged.
Blind A/B on 2 SEC chunks. Baseline items have no rubric, so they are compared by the judge only.

Run log

data/trajectories.json · polling every 5s
Rounds logged
0
Accepted pairs
0 of 0 chunks
Rounds per accepted pair
n/a
Mean accepted gap
n/a
    06Use it

    Your documents. Your key. Your dataset.

    Clone it, add an OpenRouter key, point it at any CSV of text chunks. SEC filings are the demo, not the limit.

    $ git clone https://github.com/ritvikreddygangula/Open-AutoData && cd Open-AutoData
    $ python -m venv .venv && source .venv/bin/activate
    $ pip install -r requirements.txt
    $ cp .env.example .env # set OPEN_ROUTER=sk-or-...
    $ python -m src.chunks my_filings.csv # any CSV with CHUNK_ID, CHUNK_TEXT
    $ python -m src.agent_graph # full loop, lands with feat/agent-graph
    07 The research

    Standing on Meta FAIR's Agentic Self-Instruct.

    AutoData treats data creation as an agent loop: a challenger proposes, a weak and a strong solver attempt, a judge scores against a rubric, and the loop revises until the example splits the two solvers. Meta measured a 34 point weak/strong gap with this method versus 1.9 for single-shot CoT Self-Instruct. We rebuilt the loop as an open-source LangGraph harness, swapped in a different open-weight model set, and pointed it at SEC filings in Snowflake.

    Kulikov, Whitehouse, Wu, Nie, Saha, Helenowski, Yuan, Golovneva, Lanchantin, Bachrach, Foerster, Li, Fang, Sukhbaatar, Weston