Meta (Facebook)

Verified evidenceDecision rights movedQueue eliminationContinuous decisioning

Predictive Test Selection for change-based regression testing

That every code change must run every test that transitively depends on modified files in order to catch regressions before trunk. Meta's predictive model estimates each test's likelihood of catching a regression for a specific change, so a much smaller learned subset can be run without loss of regression signal.

Software engineering · Global engineering organisation; workflow applies to Facebook's mobile codebase and continuous-integration system

Collections: Queue eliminated · Embodied work

An exhaustive field of regression tests contrasts with a model-selected subset monitored by a software engineer.

Executive brief

The operating-model shift, in one view.

Meta did not add another layer of dashboards on top of its regression tests; it moved the allocation decision from a build-graph rule to a learned model, subject to two published correctness thresholds. The instructive part is the discipline of the guardrails: over 99.9% faulty-change catch and over 95% individual outcome accuracy are published production floors, not aspirations. That is what makes 'run less to catch more' defensible to an engineering audience — a floor a skeptical reader can test against.

AI value · Pre-submit test load (share of transitively-impacted tests run per change)

Verified

Runs fewer than one-third of dependency-selected tests

The one-third figure is a comparison against the build-dependency selection strategy — not a comparison against literally all tests in the repository. Meta does not publish per-change absolute test counts or infrastructure-cost figures.

Before

Submits a code change for pre-submit testing before acceptance into the monolithic repository trunk. → Analyses build dependencies and marks every test that transitively depends on any modified source file as impacted. → Runs the full set of transitively-impacted tests — as much as roughly one-quarter of all available…

After

Submits a code change to the monolithic mobile-code repository for pre-submit testing. → Scores each affected test's regression-catching probability and selects the highest-value subset. → Runs only the model-selected subset — approximately one-third of the transitively-impacted tests — for the change. → Reviews any…

Human boundary

The model decides which tests execute for a pre-submit run within its selection budget. Meta engineering sets and audits the production correctness thresholds (more than 95% individual-outcome accuracy; catch at least one failing test on more than 99.9% of problematic changes). Test authorship, failure interpretation,…

Why it matters

Every impacted test does not need to run for every code change.

How the work changed

Before

How the work ran before the change.

Step 1 of 4
Facebook engineer
Submits a code change for pre-submit testing before acceptance into the monolithic repository trunk.
Step 2 of 4
Regression test-selection tool based on build metadata
Analyses build dependencies and marks every test that transitively depends on any modified source file as impacted.
ControlTest-selection rule based on transitive build dependency.
Step 3 of 4
Testing infrastructure
Runs the full set of transitively-impacted tests — as much as roughly one-quarter of all available mobile-code tests.
ControlAll impacted tests must pass before merge.
Step 4 of 4
Facebook engineer
Reviews any failing test, repairs the change or the test, and resubmits until the impacted set is green.

What changed

Every impacted test does not need to run for every code change.

Decision rightThe model selects tests; engineers own thresholds and failures

After

How the same work runs now.

Step 1 of 4
Facebook engineer
Submits a code change to the monolithic mobile-code repository for pre-submit testing.
Step 2 of 4
Predictive test-selection model
Scores each affected test's regression-catching probability and selects the highest-value subset.
ControlProduction correctness thresholds: the deployed model must predict more than 95% of individual test outcomes correctly and catch at least one failing test for more than 99.9% of problematic changes; the model is retrained regularly from recent code changes; retries mitigate flaky failures during training.
Step 3 of 4
Testing infrastructure
Runs only the model-selected subset — approximately one-third of the transitively-impacted tests — for the change.
ControlSelected tests must pass before merge; unselected tests are relied on for later, longer-cycle validation, not for pre-submit.
Step 4 of 4
Facebook engineer
Reviews any failing test in the selected subset, repairs the change or the test, and resubmits.

Process model built from the published workflow evidence for Meta (Facebook). Every step, actor, and control appears in full below.

Every step, actor, and control

Exception path

If the model would fall below its production correctness thresholds on validation code changes, it is retrained on recent history before promotion; flaky failures are handled by aggressive retries during training-data collection so that the model does not learn a flakiness signal. Any failing test in the selected subset routes to the change author for repair; unselected tests are still available for later, longer-cycle validation outside pre-submit.

Work removed

Executing the roughly two-thirds of transitively-impacted tests that the learned model estimates are extremely unlikely to catch a regression for a given change
Engineer waiting time on those unnecessary pre-submit test runs
Manual tuning of build-dependency-based selection rules as the codebase evolves

Decision authority

Before

01
Facebook engineer
Submits a code change for pre-submit testing before acceptance into the monolithic repository trunk.
Control: Trunk-based development model; no regression may enter the trunk.
02
Regression test-selection tool based on build metadata
Analyses build dependencies and marks every test that transitively depends on any modified source file as impacted.
Control: Test-selection rule based on transitive build dependency.
03
Testing infrastructure
Runs the full set of transitively-impacted tests — as much as roughly one-quarter of all available mobile-code tests per change on average — including many tests unlikely to reveal a regression.
Control: All impacted tests must pass before merge.
04
Facebook engineer
Reviews any failing test, repairs the change or the test, and resubmits until the impacted set is green.
Control: Human review of every failure.

After

01
Facebook engineer
Submits a code change to the monolithic mobile-code repository for pre-submit testing.
Control: Same trunk-based development model.
02
Predictive test-selection model
Scores each affected test's regression-catching probability and selects the highest-value subset.
Control: Production correctness thresholds: the deployed model must predict more than 95% of individual test outcomes correctly and catch at least one failing test for more than 99.9% of problematic changes; the model is retrained regularly from recent code changes; retries mitigate flaky failures during training.
03
Testing infrastructure
Runs only the model-selected subset — approximately one-third of the transitively-impacted tests — for the change.
Control: Selected tests must pass before merge; unselected tests are relied on for later, longer-cycle validation, not for pre-submit.
04
Facebook engineer
Reviews any failing test in the selected subset, repairs the change or the test, and resubmits.
Control: Human review of every failure; test authorship and repair remain the engineer's job.

Work that left the path

Executing the roughly two-thirds of transitively-impacted tests that the learned model estimates are extremely unlikely to catch a regression for a given change
Engineer waiting time on those unnecessary pre-submit test runs
Manual tuning of build-dependency-based selection rules as the codebase evolves
Learning a flakiness signal from noisy test outcomes because retries during training suppress it

Human role before

Engineers wrote and maintained tests, and the continuous-integration system exhaustively ran every transitively-impacted test on every change; engineers spent time waiting on and triaging failures in tests that had no realistic chance of catching a regression for the change in front of them.

Human role after

Engineers still author and maintain tests and are still the sole reviewers of failures. What has moved is the allocation of finite testing capacity: engineers no longer implicitly allocate exhaustive regression capacity per change through the build graph; the learned model selects the pre-submit subset, subject to Meta-set production thresholds that engineers monitor rather than execute.

AI roleAutonomous per-change test selector. A gradient-boosted decision-tree model trained on a large history of code changes and test outcomes ranks each transitively-impacted test by its probability of catching a regression for the proposed change, and the top subset is run. The model retrains as the codebase evolves and explicitly accounts for test flakiness during training.

Outcomes

Pre-submit test load (share of transitively-impacted tests run per change)

Verified

Build-dependency selection would exercise all tests that transitively depend on modified files — on average as much as approximately one-quarter of all available mobile-code tests per change; treated as 100% of the transitively-impacted set for pre-submit runs.→The deployed system runs 'just a third of all tests that transitively depend on modified code' per change (Engineering at Meta); the ICSE 2019 industry paper reports 'fewer than a third of the tests that would be selected on the basis of build dependencies'.

Deployed at Facebook for more than a year at 2018-11-21; also stated as 'operational at Facebook for several months' in the ICSE 2019 industry-paper text. · Facebook's monolithic mobile codebase and continuous-integration system — exact monthly change or test counts are not disclosed in either source.

Faulty-change catch rate (fraction of regressions detected pre-trunk)

Verified

The prior build-dependency strategy is treated as the reliability floor; catch rate is not separately quantified in Meta's public disclosure.→'Catches more than 99.9 percent of all regressions before they are visible to other engineers in the trunk code' (blog); a required production floor of 'catch at least one failing test for more than 99.9 percent of problematic changes' (blog) and 'over 99.9% of faulty changes are still reported back to developers' (ICSE paper).

Deployed at Facebook for more than a year at 2018-11-21. · All pre-submit changes to Facebook's monolithic mobile codebase during the deployed period; the absolute change and regression counts are not disclosed.

The 99.9% is a production correctness threshold enforced by Meta on the deployed model, not a third-party audit. The catch-rate metric is Meta-defined ('regressions visible in trunk') and Meta-measured.

Change-based testing infrastructure cost efficiency

Verified

Under the prior transitive-dependency selection strategy, treated as the baseline cost of change-based testing (index = 1x).→'Doubled the efficiency of our testing infrastructure' (blog); the ICSE 2019 paper phrases the same result as reducing 'the total infrastructure cost of testing code changes by a factor of two'.

Reported for the more-than-a-year Facebook production deployment as of 2018-11-21. · Facebook's change-based testing infrastructure for the mobile codebase; absolute compute or dollar figures are not disclosed.

'Efficiency' is defined by Meta as change-based testing infrastructure cost; the blog phrasing is operator-narrated, but the ICSE 2019 paper restates it as 'reduces the total infrastructure cost of testing code changes by a factor of two' in the industry-refereed abstract, so the two-times figure is corroborated by a peer/industry-reviewed source. No independent audit.

What leaders can reuse

Anti-pattern

Treating 'we ship less-often to be safe' as the only lever. The pre-Meta baseline in the blog is exactly that: run every transitively-impacted test on every change, because a build-graph rule feels safe. Meta's move was to accept that most of those tests never fire and to publish thresholds that make it possible to run a much smaller set without giving up regression signal. Skipping that discipline — running fewer tests without stating the catch-rate and per-test-accuracy floor — is where organisations get in trouble.

Questions

01What is the current implicit rule that decides which regression tests we run per change, and how much of that work is actually load-bearing?
02What catch-rate and per-signal accuracy would we be willing to publish as a floor for any model that replaces that rule?
03How would we detect quality drift as our codebase evolves — who is on the hook when the floor is missed?
04Do we treat flaky tests as noise or as a first-class training signal, and how would we know if we were confusing the two?
05If we cut change-based testing cost in half, where does the freed capacity get reinvested — more changes per day, deeper long-cycle testing, or savings?

Portability conditions

A large, well-instrumented history of code changes with test outcomes attached at change granularity
Trunk-based development, or an equivalent model where pre-submit is the primary regression gate
A retraining and monitoring pipeline that can hold the model to its published correctness thresholds as the codebase evolves
Handling of flaky tests as a first-class training signal, not as noise to be ignored
Sufficient repository scale that the difference between running all impacted tests and one-third of them is economically meaningful

Reputation risk

low: this is a Meta-owned engineering system with no direct customer-safety impact; the load-bearing claims (one-third of tests, over 99.9% catch, doubled efficiency) are stated verbatim in both a first-party engineering blog and an ICSE 2019 industry-refereed paper, and the workflow implications are conservative (engineers keep test authorship, failure interpretation, and change acceptance).

Evidence and authority

What the public record supports.

Current · updated 2026-08-22

1 peer reviewed, 1 primary; publication outcomes are verified.

Bundle 1.0.0 · reviewed 2026-08-23 · stable ID 953dda76e1d11526

Sources

Predictive test selection to ensure reliable code changesEngineering at Meta (Facebook) · 2018-11-21 · primary
Predictive Test Selection (ICSE 2019 SEIP industry paper)Meta Research / IEEE-ACM International Conference on Software Engineering, Software Engineering in Practice track · 2019-05 · peer reviewed

Read the evidence, freshness, caveat, and version policy.

Predictive Test Selection for change-based regression testing

The operating-model shift, in one view.

How the work changed

Outcomes

Pre-submit test load (share of transitively-impacted tests run per change)

Faulty-change catch rate (fraction of regressions detected pre-trunk)

Change-based testing infrastructure cost efficiency

What leaders can reuse

What the public record supports.

Related transformations

First-line customer-service errand resolution

Postsubmit Speculative Cycles with Transition Prediction

LLM-assisted 32-bit to 64-bit identifier migration

AI pair-programmed HTTP server

Sources