03 / The Disrupt challenge
Don’t take our word for it.
Compare the answers.
At the stand, watch two Codex instances in VS Code tackle the same question about the same source code. The difference: one can use Mechantic.
Same questionSame source codeSame coding assistant
With Mechantic
Codex.
With a head start.
Source code access, plus Mechantic’s codebase understanding and source evidence.
Without Mechantic
Codex.
Working from the code.
Source code access, using its usual code-reading and investigation tools.
Independent comparison
A third Codex compares both answers.
The judge runs in VS Code with access to the source. Open a completed question to see both answers, why the judge preferred one, and the supporting findings and evidence.
What we compare
Speed is only one part of the result.
- Time to answer
- Elapsed time from question dispatch to the final answer. Includes the assisted answer’s full Mechantic consultation; excludes judging.
- Incorrect claims
- Selected consequential claims the judge finds contradicted by source code. Missing evidence alone does not make a claim incorrect.
- Unsupported claims
- Selected consequential code claims with missing or invalid supporting source references. An unsupported claim may still be true.
- Less helpful
- One point to the answer the judge finds materially less helpful for this question. Ties and unresolved comparisons receive no point.
Lower is better on all four measures. Incorrect and unsupported claims are selected findings, not exhaustive defect counts.
Simple questions may tie. That’s fine. The hard questions are the test.
A comparison is evidence about that question and that run—not a promise that every answer will be faster or correct.