SWE-bench Pro public set passes 90% by 2027

grid-and-power-planner · openai/gpt-5.6-sol

I start from the supplied threshold-event reference class of 0.654. The September 2026 score of 80.3%, up from roughly 23% in one year, supports an upward adjustment: only 9.7 points remain, and accepted scaffolds plus listed self-reported entries widen the paths to qualification. However, the final ten points generally face diminishing returns, and there is no newer indicator confirming continued progress. I estimate a 0.76 conditional chance of crossing by year-end 2027 and about 0.94 that the required leaderboard or same-metric successor remains resolvable; compounded, that is approximately 0.71. The forecaster’s 0.80 relies too heavily on straight-line extrapolation.

semiconductor-supply-chain-analyst · openai/gpt-5.6-sol

My reference class is public coding-agent benchmarks that have already exceeded 80%: many gain the next 10 points within 12–24 months as models, inference budgets, and accepted scaffolds improve, though progress slows near saturation. The required gain is 9.7 points over roughly 16 months from the dated baseline, with several plausible model and agent iterations. The public set and permissive accepted-scaffold rule increase crossing odds. I remain below 0.80 because no current indicator confirms continued gains, halving the remaining error rate is difficult, and leaderboard retirement, methodology changes, or delayed listing can cause a literal miss.

enterprise-cio · openai/gpt-5.6-sol

I use the reference class of frontier coding benchmarks already above 0.80 and given roughly 16 months to cross 0.90. Public tasks, accepted scaffolds, and competitive benchmark optimization create a high crossing hazard, but the final ten points usually face diminishing returns and greater reliability demands. I estimate about 0.77 probability that a qualifying score is achieved, multiplied by roughly 0.94 for the named leaderboard remaining available and listing it under the required metric. With no fresh indicator reading, I discount the forecaster’s 0.80 and place modest weight on the 0.6538 structural prior.

credit-analyst · openai/gpt-5.6-sol

Using the reference class of frontier software-engineering benchmarks during the first two years after adoption, I estimate roughly 0.80 probability that accepted models or scaffolds gain the remaining 9.7 points within 15 months. The public problem set, permissive scaffolding, and “any entry” criterion help. However, progress should slow near saturation, and 90% requires solving a difficult residual rather than extending the earlier linear trend. I further discount for the conjunctive requirement that Scale retain and load the named public leaderboard, preserve the headline metric, and list the qualifying result by the cutoff. With both indicators unavailable, there is no fresh evidence supporting the forecaster’s higher estimate.

geopolitics-policy-analyst · openai/gpt-5.6-sol

I use the reference class of prominent public coding benchmarks after frontier-lab adoption: rapid early gains followed by diminishing returns, often reaching saturation within roughly 1–2 years. Moving from the supplied 80.3% baseline to 90.0% in 16 months is plausible because public problems permit targeted scaffold and inference-time optimization, and only one accepted entry is required. However, the final 9.7 points are materially harder than earlier gains; no current indicator confirms continued progress. Literal-resolution risks include methodology changes, delayed leaderboard publication, or retirement without a qualifying successor. I estimate roughly 0.76 for a qualifying result and 0.95 for compliant publication/leaderboard continuity, yielding about 0.72.

superforecaster-statistician · openai/gpt-5.6-sol

I use the engine’s threshold-question reference-class base rate of 0.654. The supplied 80.3% baseline leaves 9.7 points in about 16 months; the prior year’s rapid rise and acceptance of any listed scaffold justify an upward adjustment. However, linear extrapolation from 23% to 80.3% is unreliable near saturation: reaching 90% requires roughly halving the remaining error rate, and benchmark gains commonly slow after easier failures are removed. Literal resolution also requires Scale to retain or replace the page and list a qualifying, properly dated entry. With no fresh indicators, 0.80 appears overconfident.