Use for bounded acceleration, not universal delivery productivity.
AI value · Completion time
Verified
71.17 minutes treatment; 55.8% faster, P=.0017
Conditioned on completion; narrow task; author affiliations.
Before
Reads the HTTP-server specification and writes JavaScript without Copilot. → Runs tests, debugs, and submits the implementation.
After
Offers inline code completions while the developer implements the same server task. → Integrates suggestions, runs tests, debugs, and submits.
Human boundary
Developer owns code; tests check requirements.
Why it matters
Inline generation can compress a bounded task but not prove lifecycle productivity.
How the work changed
Before
How the work ran before the change.
Step 1 of 2
Randomized control developer
Reads the HTTP-server specification and writes JavaScript without Copilot.
ControlSame instructions and JavaScript familiarity requirements as treatment.
Step 2 of 2
Developer
Runs tests, debugs, and submits the implementation.
ControlCorrectness/completeness test suite; no production deployment.
What changed
Inline generation can compress a bounded task but not prove lifecycle productivity.
Decision rightHuman moves from creator to judge
After
How the same work runs now.
Step 1 of 2
GitHub Copilot
Offers inline code completions while the developer implements the same server task.
ControlSuggestion only; developer may accept, edit, or reject.
Step 2 of 2
Developer
Integrates suggestions, runs tests, debugs, and submits.
ControlDeveloper owns code; automated tests score correctness and completeness.
Process model built from the published workflow evidence for GitHub Copilot research experiment. Every step, actor, and control appears in full below.Every step, actor, and control
Exception path
Incorrect suggestions are rejected or corrected.
Work removed
Some boilerplate typing and lookup
Decision authority
Developer owns code; tests check requirements.
Before
01
Randomized control developer
Reads the HTTP-server specification and writes JavaScript without Copilot.
Control: Same instructions and JavaScript familiarity requirements as treatment.
02
Developer
Runs tests, debugs, and submits the implementation.
Control: Correctness/completeness test suite; no production deployment.
After
01
GitHub Copilot
Offers inline code completions while the developer implements the same server task.
Control: Suggestion only; developer may accept, edit, or reject.
02
Developer
Integrates suggestions, runs tests, debugs, and submits.
Control: Developer owns code; automated tests score correctness and completeness.
Work that left the path
Some boilerplate typing and lookup
Human role before
Control-group developers implemented the HTTP-server specification, ran automated tests, debugged failures, and submitted the code to the study evaluator.
Human role after
Developers accept, edit or reject, debug and submit.
AI roleSuggests code inline during JavaScript implementation.
