Every number, with the run behind it.
We do not ask you to trust a skill. Each one was measured on held-out cases it never saw, by a machine, and the results are here. Every number here has a run behind it.
Office skills
Sixteen held-out specifications per row, best of two samples. Cheap model: DeepSeek-chat. Strong model: claude-sonnet-5. Prices at the time of the run.
| skill, context | cheap model alone | cheap model + skill | strong model alone | cents per success, skill vs strong | skill |
|---|---|---|---|---|---|
| docx, portrait then landscape section with header and table | 25% | 100% | 87.5% | 0.15 vs 4.83 (32x) | docx_landscape_rungs |
| pptx, combo chart on a secondary axis | 12.5% | 100% | 100% | 0.32 vs 4.66 (15x) | pptx_combo_chart_rung |
| pdf, fillable AcroForm | 43.8% | 100% | 100% | 0.15 vs 2.92 (19x) | pdf_fillable_form_rungs |
| xlsx, validated sheet: frozen header, dropdown, named range, conditional format (checker forged, not hand-written) | 56.2% | 87.5% | 100% | 0.18 vs 3.58 (20x) | xlsx_validated_sheet_rungs |
| pdf, real form, the real CRA TD1 tax form, filled from human labels | 18.8% | 93.8% | 100% | 0.34 vs 27.17 (80x) | real_form_fill_td1_rungs |
How the shelf grows
13 contexts were described in plain words. No code was written for any of them: a strong model wrote the reference, the exact checker, deliberate corruptions that must fail, and a specification generator, and a mutation gate decided whether to trust the result. 12 checkers passed, for $1.35 in total. A skill is then built and listed only where it measurably beats the model working on its own, so the shelf stays worth reading.
| context | cheap model alone | cheap model + skill | interval on the lift | checker |
|---|---|---|---|---|
| calendar, RFC 5545 with a weekly recurrence and a display alarm | 38% | 88% | +0.250..+0.750 | $0.13 |
| email, MIME with a non-ASCII subject and an attachment | 31% | 75% | +0.188..+0.688 | $0.17 |
Small and local models
With a skill wrapped as a single call, the model only fills in a JSON argument schema. Qwen3 runs in bf16 with thinking off on one consumer GPU.
| skill | executor | held-out set | cents per item |
|---|---|---|---|
| pdf fillable form | DeepSeek-chat (API) | 16/16 | 0.059 |
| pptx combo chart | DeepSeek-chat (API) | 16/16 | 0.065 |
| pdf fillable form | Qwen3-1.7B, local, no API | 15/16 | 0 |
| pptx combo chart | Qwen3-1.7B, local, no API | 16/16 | 0 |
| pdf fillable form | Qwen3-0.6B, local, no API | 16/16 | 0 |
| pptx combo chart | Qwen3-0.6B, local, no API | 16/16 | 0 |
| calendar with recurrence | Qwen3-0.6B, local, no API | 8/8 | 0 |
A public benchmark: 134 unseen games
A public benchmark of household tasks, run on its own unseen evaluation split. Our library was built from 24 training games and never touched the unseen split.
| executor | success | model turns per game | cents per game |
|---|---|---|---|
| DeepSeek cold (ReAct over admissible commands, 50 steps) | 70/134 | 31.4 | 0.389 |
| DeepSeek + rungs v4 | 134/134 | 1.00 | 0.033 |
| Local Qwen3-1.7B + rungs v4, no API | 133/134 | 1.07 | 0 |
| Local Qwen3-1.7B + rungs v4 + parser rung | 134/134 | 1.00 | 0 |
| Parser rung alone, no model | 134/134 | 0 | 0 |
| The best published baseline on this split (GPT-4o with a skill library) | 99.3% | 3.2 | n/a |
Autopilot
60 mixed office tasks with no human in the loop: 58 correct for 8.3 cents in total, 0.15 cents per task from first to last, because every skill the workload needed had already been forged once.