---
title: 'Meta (Facebook): Predictive Test Selection for change-based regression testing'
slug: meta-predictive-test-selection
stable_id: 953dda76e1d11526
company: Meta (Facebook)
function_code: software_engineering
pattern_codes:
  - decision_right_transfer
  - queue_elimination
  - continuous_decisioning
evidence_strength: verified
publication_tier: showcase
freshness: current
reviewed_at: '2026-08-23'
updated_at: '2026-08-22'
source_quality_summary: 1 peer reviewed, 1 primary; publication outcomes are verified.
caveat_summary: >-
  The one-third figure is a comparison against the build-dependency selection strategy —
  not a comparison against literally all tests in the repository. Meta does not publish
  per-change absolute test counts or infrastructure-cost figures. The 99.9% is a
  production correctness threshold enforced by Meta on the deployed model, not a
  third-party audit. The catch-rate metric is Meta-defined ('regressions visible in
  trunk') and Meta-measured. 'Efficiency' is defined by Meta as change-based testing
  infrastructure cost; the blog phrasing is operator-narrated, but the ICSE 2019 paper
  restates it as 'reduces the total infrastructure cost of testing code changes by a
  factor of two' in the industry-refereed abstract, so the two-times figure is
  corroborated by a peer/industry-reviewed source. No indepen
collections:
  - queue-eliminated
  - embodied-work
bundle_version: 1.0.0
bundle_fingerprint: sha256:c23c6cc2b88153d008ea8fda928f632ce0011fc2d4c5036672a16e5d895bab93
canonical_url: https://brianletort.ai/transformations/meta-predictive-test-selection
---

# Meta (Facebook): Predictive Test Selection for change-based regression testing

That every code change must run every test that transitively depends on modified files in order to catch regressions before trunk. Meta's predictive model estimates each test's likelihood of catching a regression for a specific change, so a much smaller learned subset can be run without loss of regression signal.

Function: Software engineering. Patterns: Decision-right transfer; Queue elimination; Continuous decisioning. Evidence: verified.

Freshness: current. Reviewed: 2026-08-23. Updated: 2026-08-22.


Source quality: 1 peer reviewed, 1 primary; publication outcomes are verified.

## Before

1. **Facebook engineer** — Submits a code change for pre-submit testing before acceptance into the monolithic repository trunk. (control: Trunk-based development model; no regression may enter the trunk.)
2. **Regression test-selection tool based on build metadata** — Analyses build dependencies and marks every test that transitively depends on any modified source file as impacted. (control: Test-selection rule based on transitive build dependency.)
3. **Testing infrastructure** — Runs the full set of transitively-impacted tests — as much as roughly one-quarter of all available mobile-code tests per change on average — including many tests unlikely to reveal a regression. (control: All impacted tests must pass before merge.)
4. **Facebook engineer** — Reviews any failing test, repairs the change or the test, and resubmits until the impacted set is green. (control: Human review of every failure.)

## After

1. **Facebook engineer** — Submits a code change to the monolithic mobile-code repository for pre-submit testing. (control: Same trunk-based development model.)
2. **Predictive test-selection model** — Scores each affected test's regression-catching probability and selects the highest-value subset. (control: Production correctness thresholds: the deployed model must predict more than 95% of individual test outcomes correctly and catch at least one failing test for more than 99.9% of problematic changes; the model is retrained regularly from recent code changes; retries mitigate flaky failures during training.)
3. **Testing infrastructure** — Runs only the model-selected subset — approximately one-third of the transitively-impacted tests — for the change. (control: Selected tests must pass before merge; unselected tests are relied on for later, longer-cycle validation, not for pre-submit.)
4. **Facebook engineer** — Reviews any failing test in the selected subset, repairs the change or the test, and resubmits. (control: Human review of every failure; test authorship and repair remain the engineer's job.)

## Decision rights

The model decides which tests execute for a pre-submit run within its selection budget. Meta engineering sets and audits the production correctness thresholds (more than 95% individual-outcome accuracy; catch at least one failing test on more than 99.9% of problematic changes). Test authorship, failure interpretation, and change acceptance remain with the engineer; the model does not repair tests or accept changes.

## Exception path

If the model would fall below its production correctness thresholds on validation code changes, it is retrained on recent history before promotion; flaky failures are handled by aggressive retries during training-data collection so that the model does not learn a flakiness signal. Any failing test in the selected subset routes to the change author for repair; unselected tests are still available for later, longer-cycle validation outside pre-submit.

## Outcomes

- **Pre-submit test load (share of transitively-impacted tests run per change)** (verified): Build-dependency selection would exercise all tests that transitively depend on modified files — on average as much as approximately one-quarter of all available mobile-code tests per change; treated as 100% of the transitively-impacted set for pre-submit runs. → The deployed system runs 'just a third of all tests that transitively depend on modified code' per change (Engineering at Meta); the ICSE 2019 industry paper reports 'fewer than a third of the tests that would be selected on the basis of build dependencies'.. The one-third figure is a comparison against the build-dependency selection strategy — not a comparison against literally all tests in the repository. Meta does not publish per-change absolute test counts or infrastructure-cost figures.
- **Faulty-change catch rate (fraction of regressions detected pre-trunk)** (verified): The prior build-dependency strategy is treated as the reliability floor; catch rate is not separately quantified in Meta's public disclosure. → 'Catches more than 99.9 percent of all regressions before they are visible to other engineers in the trunk code' (blog); a required production floor of 'catch at least one failing test for more than 99.9 percent of problematic changes' (blog) and 'over 99.9% of faulty changes are still reported back to developers' (ICSE paper).. The 99.9% is a production correctness threshold enforced by Meta on the deployed model, not a third-party audit. The catch-rate metric is Meta-defined ('regressions visible in trunk') and Meta-measured.
- **Change-based testing infrastructure cost efficiency** (verified): Under the prior transitive-dependency selection strategy, treated as the baseline cost of change-based testing (index = 1x). → 'Doubled the efficiency of our testing infrastructure' (blog); the ICSE 2019 paper phrases the same result as reducing 'the total infrastructure cost of testing code changes by a factor of two'.. 'Efficiency' is defined by Meta as change-based testing infrastructure cost; the blog phrasing is operator-narrated, but the ICSE 2019 paper restates it as 'reduces the total infrastructure cost of testing code changes by a factor of two' in the industry-refereed abstract, so the two-times figure is corroborated by a peer/industry-reviewed source. No independent audit.

## Executive lesson

Meta did not add another layer of dashboards on top of its regression tests; it moved the allocation decision from a build-graph rule to a learned model, subject to two published correctness thresholds. The instructive part is the discipline of the guardrails: over 99.9% faulty-change catch and over 95% individual outcome accuracy are published production floors, not aspirations. That is what makes 'run less to catch more' defensible to an engineering audience — a floor a skeptical reader can test against.

## Anti-pattern

Treating 'we ship less-often to be safe' as the only lever. The pre-Meta baseline in the blog is exactly that: run every transitively-impacted test on every change, because a build-graph rule feels safe. Meta's move was to accept that most of those tests never fire and to publish thresholds that make it possible to run a much smaller set without giving up regression signal. Skipping that discipline — running fewer tests without stating the catch-rate and per-test-accuracy floor — is where organisations get in trouble.

## Questions for leaders

- What is the current implicit rule that decides which regression tests we run per change, and how much of that work is actually load-bearing?
- What catch-rate and per-signal accuracy would we be willing to publish as a floor for any model that replaces that rule?
- How would we detect quality drift as our codebase evolves — who is on the hook when the floor is missed?
- Do we treat flaky tests as noise or as a first-class training signal, and how would we know if we were confusing the two?
- If we cut change-based testing cost in half, where does the freed capacity get reinvested — more changes per day, deeper long-cycle testing, or savings?

## Sources

- [Predictive test selection to ensure reliable code changes](https://engineering.fb.com/2018/11/21/developer-tools/predictive-test-selection/) — Engineering at Meta (Facebook)
- [Predictive Test Selection (ICSE 2019 SEIP industry paper)](https://research.facebook.com/publications/predictive-test-selection/) — Meta Research / IEEE-ACM International Conference on Software Engineering, Software Engineering in Practice track