That access to a general model improves all apparently similar knowledge tasks.
Collections: Negative results
Executive brief
The operating control is task classification, not generic prompt training; users need a way to recognize when the model is outside its frontier.
AI value · Task completion speed and correctness by task type
VerifiedInside the frontier: 25.1% faster, 12.2% more tasks, over 40% higher quality; outside the frontier: 19 percentage points less likely to be correct
Experimental tasks approximated consulting work but were not client delivery; the negative out-of-frontier result is as important as the positive averages.
Before
Analyze source materials, generate ideas, and draft recommendations unaided by GPT-4. → Check coherence and submit the work.
After
Delegate or interleave ideation, analysis, and drafting with the model. → Verify, revise, and submit model-assisted work.
Human boundary
The model proposes content and analysis; the consultant owns verification and the submitted answer.
Why it matters
That access to a general model improves all apparently similar knowledge tasks.
Before
How the work ran before the change.
Consultant
Analyze source materials, generate ideas, and draft recommendations unaided by GPT-4.
Consultant
Check coherence and submit the work.
What changed
That access to a general model improves all apparently similar knowledge tasks.
Decision rightHuman moves from creator to judge
After
How the same work runs now.
Consultant with GPT-4
Delegate or interleave ideation, analysis, and drafting with the model.
ControlUser prompting and task choice
Consultant
Verify, revise, and submit model-assisted work.
ControlHuman final decision
Exception path
For tasks outside the model's capability frontier, consultants must solve independently or validate against source evidence rather than accept the model's confident answer.
Decision authority
The model proposes content and analysis; the consultant owns verification and the submitted answer.
| # | Actor | Action | Control |
|---|---|---|---|
| 01 | Consultant | Analyze source materials, generate ideas, and draft recommendations unaided by GPT-4. | Professional judgment |
| 02 | Consultant | Check coherence and submit the work. | Human accountability |
| # | Actor | Action | Control |
|---|---|---|---|
| 01 | Consultant with GPT-4 | Delegate or interleave ideation, analysis, and drafting with the model. | User prompting and task choice |
| 02 | Consultant | Verify, revise, and submit model-assisted work. | Human final decision |
Work that left the path
Human role before
Consultants performed every analytical and drafting step directly.
Human role after
Consultants decide when to delegate, when to integrate, and when to distrust the model.
AI role
General-purpose GPT-4 assistance across realistic consulting tasks.
Randomized no-AI control consultantsInside the frontier: 25.1% faster, 12.2% more tasks, over 40% higher quality; outside the frontier: 19 percentage points less likely to be correct
2023 preregistered experiment · 758 BCG consultants, about 7% of individual-contributor consultants
Experimental tasks approximated consulting work but were not client delivery; the negative out-of-frontier result is as important as the positive averages.
Anti-pattern
Applying the positive average to all knowledge work or assuming prompt training eliminates frontier risk.
Questions
Portability conditions
Reputation risk
low