What does the score mean?
An AI judge assessed how well each plan used relevant constraints and rationale, stayed faithful to sources, and prepared for implementation.
This measures planning quality. Delivery speed, working-code correctness and customer outcomes were not measured.
How we ran the benchmark +
Five history-dependent engineering tasks were selected from real work before answer generation. Three controls are reported separately and excluded from the headline. Both conditions used fresh isolated Codex sessions, GPT-6.1 Sol at low reasoning, identical task inputs and one answer per condition. The with-Shifu condition also received the current integration skill and read-only production context retrieval. Project-specific expected constraints and source references were held out from the answering models, except in the full-context control. Explicit records from earlier benchmark activities were filtered from retrieved context; the filter is heuristic. Jev 1.13.0 scored each answer on five anchored dimensions, normalized to 0–10. Each answer score is the equal-weight mean of those dimensions; the headline gives each of the five primary tasks equal weight. Condition labels were hidden from the judge. Both answers were judged using the same task-specific reference packet, retrieved evidence and integration instructions. Those materials verify assertions; expected requirements and current user instructions remain authoritative.
This internal pilot covers one work history and five selected primary tasks, with one answer per condition. It does not establish statistical significance or represent all engineers, repositories or tasks.
Scores estimate the quality of pre-coding engineering plans. Generated implementations and tests were not executed; productivity, production safety and customer outcomes were not measured.
The with-Shifu condition combines integration instructions and retrieval, so the result does not isolate memory alone. Production retrieval is bounded and the live corpus was not frozen.
Reference documents can contain local, uncommitted contracts or proposals. Their presence in production memory and their deployed behavior were not independently established. A retrieval gap may reflect ingestion or selection.
The full-context control repeats one primary scenario and is not an independent task domain. Setup gains may reflect instructions supplied by the integration skill.
Scenario descriptions are anonymized paraphrases, not verbatim benchmark prompts. Private prompts, generated answers, source records, account identifiers and implementation details are not published. Scores are unchanged.
Evaluated 24 Oct 2026. Model: GPT-6.1 Sol, low reasoning. Judge: Jev 1.13.0.