brianletort.ai
← Library

Google

Verified evidenceContinuous decisioningQueue elimination

Postsubmit Speculative Cycles with Transition Prediction

Novel breakages need not wait for a comprehensive post-submit test cycle.

Software engineering · Global (Google Test Automation Platform / monorepo CI)

Collections: Queue eliminated

An editorial scene for Google contrasts submits a change after limited presubmit testing. with submits a change after limited presubmit testing. in the postsubmit speculative cycles with transition prediction workflow.

Executive brief

The operating-model shift, in one view.

When the expensive complete pass cannot get faster, the transformation is to give a cheaper, more frequent pass the right to choose what to run. Google moved postsubmit ordering from 'all affected tests when capacity allows' to a ranked speculative slice, and median novel-breakage detection fell from 107 minutes to 37.

AI value · Median (p50) time to detect novel breaking targets

Verified

37 minutes with Speculative Cycles (approximately 65% / 70-minute reduction)

Comparison is to the prior comprehensive scheduler on empirically observed production timings at a 25% target budget, not a randomized controlled experiment.

Before

Submits a change after limited presubmit testing. → Waits for capacity, then runs all tests affected since the last definitive repository status. → Triage the first detected failure, identify the culprit change, and roll back when evidence thresholds are met.

After

Submits a change after limited presubmit testing. → Every ~20 minutes, after static affected-target filtering, scores targets for pass-to-fail transition risk and ranks. → Schedules the top-risk targets within a 25% test budget to find novel breakages sooner. → Consume earlier failure signals for investigation and…

Human boundary

Transition Prediction decides speculative scheduling order within a human-set budget. Humans retain rollback policy, feature/training configuration, and the continued existence of comprehensive cycles. The model does not skip required comprehensive coverage by itself.

Why it matters

Novel breakages need not wait for a comprehensive post-submit test cycle.

How the work changed

Before

How the work ran before the change.

  1. Step 1 of 3

    Developer

    Submits a change after limited presubmit testing.

    ControlPresubmit checks and code review; comprehensive testing is deferred

  2. Step 2 of 3

    TAP Postsubmit comprehensive cycle

    Waits for capacity, then runs all tests affected since the last definitive repository status.

    ControlStatic build-graph dependence; no ML ranking of which tests to run first

  3. Step 3 of 3

    Culprit finding / autorollback / developers

    Triage the first detected failure, identify the culprit change, and roll back when evidence thresholds are met.

    ControlAutorollback currently requires at least 10 distinct failing targets

What changed

Novel breakages need not wait for a comprehensive post-submit test cycle.

Decision rightSelection moves from a fixed rule to the model

After

How the same work runs now.

  1. Step 1 of 4

    Developer

    Submits a change after limited presubmit testing.

    ControlSame presubmit and review gates

  2. Step 2 of 4

    Transition Prediction (TRANSPRED)

    Every ~20 minutes, after static affected-target filtering, scores targets for pass-to-fail transition risk and ranks.

    ControlShallow gradient-boosted trees on coarse metadata; scores are ranks, not calibrated probabilities

  3. Step 3 of 4

    Speculative Cycle scheduler

    Schedules the top-risk targets within a 25% test budget to find novel breakages sooner.

    ControlFixed top-k cost cap; comprehensive cycles still run

  4. Step 4 of 4

    Culprit finding / autorollback / developers

    Consume earlier failure signals for investigation and rollback.

    ControlSame autorollback evidence threshold (AtLeast 10 targets)

Process model built from the published workflow evidence for Google. Every step, actor, and control appears in full below.
Every step, actor, and control

Exception path

Missed breakages wait for the next comprehensive cycle. Flaky or low-evidence culprits are filtered by culprit verification. Late speculative detections that do not beat comprehensive timing still fall back to the existing gardening workflow.

Work removed

  • Waiting solely on the next capacity-gated comprehensive cycle to surface a novel breakage
  • Treating all affected tests as equal priority in the first postsubmit slice

Decision authority

Transition Prediction decides speculative scheduling order within a human-set budget. Humans retain rollback policy, feature/training configuration, and the continued existence of comprehensive cycles. The model does not skip required comprehensive coverage by itself.

Before

  1. 01

    Developer

    Submits a change after limited presubmit testing.

    Control: Presubmit checks and code review; comprehensive testing is deferred

  2. 02

    TAP Postsubmit comprehensive cycle

    Waits for capacity, then runs all tests affected since the last definitive repository status.

    Control: Static build-graph dependence; no ML ranking of which tests to run first

  3. 03

    Culprit finding / autorollback / developers

    Triage the first detected failure, identify the culprit change, and roll back when evidence thresholds are met.

    Control: Autorollback currently requires at least 10 distinct failing targets

After

  1. 01

    Developer

    Submits a change after limited presubmit testing.

    Control: Same presubmit and review gates

  2. 02

    Transition Prediction (TRANSPRED)

    Every ~20 minutes, after static affected-target filtering, scores targets for pass-to-fail transition risk and ranks them.

    Control: Shallow gradient-boosted trees on coarse metadata; scores are ranks, not calibrated probabilities

  3. 03

    Speculative Cycle scheduler

    Schedules the top-risk targets within a 25% test budget to find novel breakages sooner.

    Control: Fixed top-k cost cap; comprehensive cycles still run

  4. 04

    Culprit finding / autorollback / developers

    Consume earlier failure signals for investigation and rollback.

    Control: Same autorollback evidence threshold (AtLeast 10 targets)

Work that left the path

  • Waiting solely on the next capacity-gated comprehensive cycle to surface a novel breakage
  • Treating all affected tests as equal priority in the first postsubmit slice

Human role before

TAP Postsubmit used capacity-gated comprehensive batching of all affected tests. Developers did not set per-change test execution order; they waited on hour-scale postsubmit signal.

Human role after

Humans set scheduler policy (top-k budget, rollback thresholds, model refresh). The ML scheduler owns which postsubmit tests run in the fast speculative slice.

AI rolePredict which targets are newly likely to fail and feed a ranked top-k speculative cycle so novel breakages are found before the next comprehensive cycle.

Outcomes

Median (p50) time to detect novel breaking targets

Verified

107 minutes with comprehensive cycles only→37 minutes with Speculative Cycles (approximately 65% / 70-minute reduction)

Three months of production TAP Postsubmit data contemporaneous with the 2025 study · 120 billion test×cycle pairs, 7.7 million breaking targets, ~20,000 unique breakages

Comparison is to the prior comprehensive scheduler on empirically observed production timings at a 25% target budget, not a randomized controlled experiment. The paper says Google 'is implementing' Speculative Cycles; the evaluated production-data window does not state the fleet fraction live during every day of the three months. The Google Research abstract reports an approximately 70-minute median reduction; the ICST paper states 65% from 107 to 37 minutes, which is the same difference.

What leaders can reuse

Anti-pattern

Claiming that developers used to pick test order by hand, or reporting a 70-minute win without stating the 107-to-37-minute comprehensive-cycle baseline and the 25% budget.

Questions

  1. 01What is the first pass that is allowed to be incomplete in order to be faster?
  2. 02If the model is wrong, does a slower complete cycle still run, or did we just create silent coverage gaps?
  3. 03Is the outcome median time-to-detect on production traffic, or a replay against a synthetic scheduler?

Portability conditions

  • A high-accuracy history of which commits actually broke which targets, not just which tests failed
  • Enough volume that skipping 75% of a cycle is still cheaper than buying more machines
  • A remaining comprehensive cycle so missed breakages still surface
  • Human-set rollback evidence thresholds the speculative slice can actually meet

Reputation risk

low

Evidence and authority

What the public record supports.

Current · updated

1 peer reviewed, 1 primary; publication outcomes are verified.

Bundle 1.0.0 · reviewed 2026-08-23 · stable ID dd6d683bf4ef1237

Related transformations

More in Software engineering →

Google DeepMind and EMBL-EBI · Continuous decisioning

AlphaFold Protein Structure Database precomputed structure access

Researchers can use precomputed structure predictions, then decide what experimental validation is required.

Google Research Flood Hub · Continuous decisioning

AI forecasts for ungauged basins

A global model can transfer hydrology into ungauged basins.

Mayo Clinic primary-care practices · Continuous decisioning

EAGLE AI-ECG clinical decision support

A routine ECG can trigger evaluation for asymptomatic low ejection fraction before specialist suspicion.

Google and Google DeepMind · Continuous decisioning

ML-recommended hourly wind-energy delivery commitments

That variable wind generation cannot be committed to the grid on a useful day-ahead schedule.

Sources

Read the evidence, freshness, caveat, and version policy.