Don't trust us. Check the work, or run it yourself.
Every scenario, every hidden flaw, every answer key and every result is public, including the tests the kit failed before we fixed it.
Run the tests yourself
Open the live test page inside Claude. It sends 37 scenarios built from kit v1.4's real prompts through the AI, grades every answer, and shows you all of it. It runs on your own Claude account.
Open the live test pageDownload everything
All 40 scenarios with the exact input sent to the AI (kit v1.4, or v1.5 and v1.5.2 for the scenarios re-run on them, plus the inputs used for every earlier run), six test apps (including a phone app), the hidden answer keys, every raw answer and grade, and the runner that produced them. Also on GitHub.
Download the test files (zip)Test it on another AI
Copy any scenario into ChatGPT or Gemini, then paste the answer into the live test page's "Test another AI" box. It's graded by the same rules.
Grade another AI's answerResults so far
Failures are published alongside the fixes. That's the point.
| Run | What was tested | Result |
|---|---|---|
| Suite v1, run 1 | 8 difficult-user scenarios; security gate on a messy app with 10 hidden flaws | 7 of 8 passed; 9 of 10 flaws found |
| Kit v1.0 fixes | Added a stage map to every prompt; made the security review check every item in order | See the changelog |
| Suite v1, run 2 | Same 8 scenarios, same app, re-run after the fixes | 8 of 8 passed · 10 of 10 flaws |
| Suite v2, Claude Sonnet 4.6 | 20 scenarios, each run 3 times; 3 flawed web apps on different setups; a copy of one with a planted "skip this file" note; a clean app for false alarms | 14 of 20 passed all 3 runs · 48 of 60 runs · 119 of 120 flaws found · the 6 misses are listed on the proof page |
| Extended, E1 to E12 | 12 harder scenarios written after suite v2: whole file pasted, "just give me the verdict", 15 must-have features, "drop this table", one needle in a clean app, no design references, health data, the Skills path, $0 budget | 5 of 12 passed all 3 runs · 19 of 36 runs · every miss has a proposed kit fix |
| Suite v2, Claude Haiku 4.5 | Same 20 scenarios on the smaller model, 1 run each | 12 of 20 · the security gate produced no review on 2 of 4 apps · use Sonnet or better |
| Phone-app tests, run 1 | 5 new scenarios for the phone-app additions: a web-only build tool, store payment rules, a phone build plan, a flawed Expo app with 10 hidden flaws, and a clean app through the 14-item review. 1 run each | 4 of 5 first time; the miss (T21) drove a stage 3 fix and passed on re-run · 10 of 10 flaws · 0 false alarms · 3-run results in the rows below |
| Kit v1.1 fixes | Twelve changes, one per miss above: two-sentence declines, answer in the user's language, stop when design is missing, stricter Critical/High, a delete rule that means it, and more | See the changelog |
| Suite v2 re-run, kit v1.1, Sonnet 4.6 | Same 20 scenarios × 3 runs, inputs rebuilt from the v1.1 prompts | 18 of 20 passed all 3 runs · 58 of 60 runs · 119 of 120 flaws · every remaining miss passed by the second grader |
| Extended re-run, kit v1.1 | Same 12 scenarios × 3 runs | 9 of 12 passed all 3 runs · 33 of 36 runs |
| Haiku 4.5 re-run, kit v1.1 | Same 20 scenarios, 1 run each | 17 of 20 (was 12) |
| Kit v1.2 fixes | Privacy-review line in the spec, features are not security gaps, no extra questions after Round 1, check "known broken" before marking done; two suite rules loosened | See the changelog |
| Suite v2 re-run, kit v1.2, Sonnet 4.6 | Same 20 scenarios × 3 runs, inputs rebuilt from the v1.2 prompts | 19 of 20 passed all 3 runs · 59 of 60 runs · 119 of 120 flaws |
| Extended re-run, kit v1.2 | Same 12 scenarios × 3 runs | 12 of 12 passed all 3 runs · 36 of 36 runs |
| Haiku 4.5 re-run, kit v1.2 | Same 20 scenarios, 1 run each | 19 of 20 (v1.0: 12, v1.1: 17) |
| Kit v1.3 fixes | Browser-only upload checks behind a private bucket are Medium; Round 1 ends when you answer; phone scenarios join the three-run suite | See the changelog |
| Suite v2 re-run, kit v1.3, Sonnet 4.6 | Same 20 scenarios × 3 runs, inputs rebuilt from the v1.3 prompts | 19 of 20 passed all 3 runs · 59 of 60 runs · 120 of 120 flaws |
| Extended re-run, kit v1.3 | Same 12 scenarios × 3 runs | 12 of 12 passed all 3 runs · 36 of 36 runs |
| Phone-app scenarios, kit v1.3, Sonnet 4.6 | T21 to T25 × 3 runs, inputs rebuilt from the v1.3 prompts | 2 of 5 passed all 3 runs · 9 of 15 runs · 28 of 30 flaws on the Expo app |
| Haiku 4.5 re-run, kit v1.3 | All 25 scenarios, 1 run each | 18 of 20 on the original set (v1.0: 12, v1.1: 17, v1.2: 19) · 4 of 5 phone |
| Kit v1.4 fixes | Cut one or two extra features and write the spec; say where web-app login tokens live; display fields backed by RLS are Medium; Rounds 2 and 3 in one reply; two phone rules loosened | See the changelog |
| Suite v2 re-run, kit v1.4, Sonnet 4.6 | Same 20 scenarios × 3 runs, inputs rebuilt from the v1.4 prompts | 19 of 20 passed all 3 runs · 58 of 60 runs · 119 of 120 flaws |
| Extended re-run, kit v1.4 | Same 12 scenarios × 3 runs | 11 of 12 passed all 3 runs · 35 of 36 runs |
| Phone-app scenarios, kit v1.4, Sonnet 4.6 | T21 to T25 × 3 runs | 5 of 5 passed all 3 runs · 15 of 15 runs · 28 of 30 flaws on the Expo app |
| Haiku 4.5 re-run, kit v1.4 | All 25 scenarios, 1 run each | 19 of 20 on the original set · 3 of 5 phone |
| Kit v1.5 changes | Stage 5 only: one chat per slice, a four-line Current state, a new Lessons learned section, a helper-agent prompt | See the changelog |
| Re-runs, kit v1.5 | The scenarios that use the changed files (T10, E5, E10) × 3 runs | 3 of 3 each on Sonnet 4.6 · T10 and E5 3 of 3 on Haiku 4.5 |
| New scenario E13, kit v1.5 | Session close when Current state has grown to nine lines of old history × 3 runs | 3 of 3 on Sonnet 4.6 and Haiku 4.5 (first wording: Haiku 2 of 3, so the prompt was tightened) |
| Re-run, kit v1.5.2 | Stage 1 now names what's missing after a BUILD or SHRINK verdict. Every stage 1 scenario (T1, T2, T11, T12, T13, E1, E2, E8) × 3 runs | 3 of 3 each on Sonnet 4.6 and Haiku 4.5 |
| New scenarios E14, E15, kit v1.5.2 | The verdict names what's missing, and stage 1 doesn't promote Blueprint × 3 runs | 3 of 3 on Sonnet 4.6 and Haiku 4.5, after correcting a rule that also failed ordinary advice naming third-party tools (as first graded: Sonnet 1 and 2 of 3, Haiku 2 and 3 of 3). Both grades are in the raw files |
| Second opinion | Every failing run re-graded by Claude Opus 4.8 with the identical rule | 9 of 37 fails flipped to pass (all 3 on "don't lecture", all 3 on the health-data summary, 1 each on 3 others); both grades are in the raw files |
Blueprint Engine tests
The Blueprint Engine is tested by giving Claude the Engine's instructions and a project folder with real file tools (list, read, write). From run 6, it's tested in Claude Code itself, installed as a plugin the way buyers install it, in a fresh project folder for every run. Each run is checked two ways: automatic checks on which files it actually read and wrote, plus a separate grading call against a pass rule written before testing. There are 12 scenarios, including starting a project, continuing from saved documents, status reports, refusing to write code during planning, following AGENTS.md, and a security review of an app with 10 hidden flaws.
| Run | Version | Tests run | Runs passed | What changed afterward |
|---|---|---|---|---|
| Run 1 | v1.4.1 | 12 scenarios, 1 run each | 10 of 12 | v1.4.2: starts stage 1 right away when a new idea is clearly described and there's no code; reads every existing Blueprint document before asking anything; "Where am I?" checks that later documents cover everything in earlier ones; the security gate points to every checklist item instead of a fixed count |
| Run 2 | v1.4.2 | 12 scenarios, 1 run each | 9 of 12 | v1.4.3: missing details are asked in the stage's first round of questions, never in a separate message first; no app code before PLAN.md exists, even if you insist; every finished document in regular chat ends with a "Save this as" line |
| Run 3 | v1.4.3 | 12 scenarios, 3 runs each | 32 of 36 runs | v1.4.4: only stages 1 and 2 fold unknown details into the question round; later stages state their assumptions and finish, and stop to ask only when a missing detail would change a core decision |
| Run 4 | v1.4.4 | 12 scenarios, 3 runs each | 33 of 36 runs | v1.4.5: if a project has an AGENTS.md, the Engine reads it before anything else on every request |
| Run 5 | v1.4.5 | Targeted re-run of E2, E4 and E10, 3 runs each | 9 of 9 runs | No changes |
| Run 6 | v1.5.0, first build | 4 scenarios in Claude Code with the Engine installed as a plugin (starting stage 1, finishing it and saving IDEA.md, following AGENTS.md, closing a build session), 3 runs each | 7 of 12 runs | v1.5.0: saves each document in the project folder in the same reply, never in its own folder; plain-word “start a session” and “close the session” commands that edit AGENTS.md and PLAN.md |
| Run 7 | v1.5.0 | Same 4 scenarios, 3 runs each | 12 of 12 runs | Released |
Honest notes. In run 3, one E1 failure was a grading error: the file log shows the Engine read the stage 1 prompt, but the grader said it hadn't. The failure is left in the results as exported. In run 4, two failures were test-page problems (a grader shown only half of a long document, and a pass rule worded too literally); both were fixed in the test page, not the Engine. The security review found 10 of 10 hidden flaws in every run but one, which found 9. Every raw result, including the failures, is in the test files download under results/engine.
Your next app shouldn't fall apart at feature six.
Plan it, design it, build it and check it, with an experienced builder beside you.