{"stable_id":"953dda76e1d11526","slug":"meta-predictive-test-selection","company":"Meta (Facebook)","workflow_name":"Predictive Test Selection for change-based regression testing","function_code":"software_engineering","pattern_codes":["decision_right_transfer","queue_elimination","continuous_decisioning"],"changed_assumption":"That every code change must run every test that transitively depends on modified files in order to catch regressions before trunk. Meta's predictive model estimates each test's likelihood of catching a regression for a specific change, so a much smaller learned subset can be run without loss of regression signal.","evidence_strength":"verified","publication_tier":"showcase","freshness":"current","reviewed_at":"2026-08-23","updated_at":"2026-08-22","source_quality_summary":"1 peer reviewed, 1 primary; publication outcomes are verified.","caveat_summary":"The one-third figure is a comparison against the build-dependency selection strategy — not a comparison against literally all tests in the repository. Meta does not publish per-change absolute test counts or infrastructure-cost figures. The 99.9% is a production correctness threshold enforced by Meta on the deployed model, not a third-party audit. The catch-rate metric is Meta-defined ('regressions visible in trunk') and Meta-measured. 'Efficiency' is defined by Meta as change-based testing infrastructure cost; the blog phrasing is operator-narrated, but the ICSE 2019 paper restates it as 'reduces the total infrastructure cost of testing code changes by a factor of two' in the industry-refereed abstract, so the two-times figure is corroborated by a peer/industry-reviewed source. No indepen","freshness_caveat":null,"workflow_summary":{"before":[{"actor":"Facebook engineer","action":"Submits a code change for pre-submit testing before acceptance into the monolithic repository trunk.","actor_type":"human"},{"actor":"Regression test-selection tool based on build metadata","action":"Analyses build dependencies and marks every test that transitively depends on any modified source file as impacted.","actor_type":"control"},{"actor":"Testing infrastructure","action":"Runs the full set of transitively-impacted tests — as much as roughly one-quarter of all available mobile-code tests.","actor_type":"control"},{"actor":"Facebook engineer","action":"Reviews any failing test, repairs the change or the test, and resubmits until the impacted set is green.","actor_type":"human"}],"hinge":"Every impacted test does not need to run for every code change.","after":[{"actor":"Facebook engineer","action":"Submits a code change to the monolithic mobile-code repository for pre-submit testing.","actor_type":"human"},{"actor":"Predictive test-selection model","action":"Scores each affected test's regression-catching probability and selects the highest-value subset.","actor_type":"ai"},{"actor":"Testing infrastructure","action":"Runs only the model-selected subset — approximately one-third of the transitively-impacted tests — for the change.","actor_type":"system"},{"actor":"Facebook engineer","action":"Reviews any failing test in the selected subset, repairs the change or the test, and resubmits.","actor_type":"human"}],"decision_mode":"moved","decision_marker":"The model selects tests; engineers own thresholds and failures"},"before":[{"order":1,"actor":"Facebook engineer","action":"Submits a code change for pre-submit testing before acceptance into the monolithic repository trunk.","handoff_to":"Continuous-integration system","control":"Trunk-based development model; no regression may enter the trunk."},{"order":2,"actor":"Regression test-selection tool based on build metadata","action":"Analyses build dependencies and marks every test that transitively depends on any modified source file as impacted.","handoff_to":"Testing infrastructure","control":"Test-selection rule based on transitive build dependency."},{"order":3,"actor":"Testing infrastructure","action":"Runs the full set of transitively-impacted tests — as much as roughly one-quarter of all available mobile-code tests per change on average — including many tests unlikely to reveal a regression.","handoff_to":"Engineer","control":"All impacted tests must pass before merge."},{"order":4,"actor":"Facebook engineer","action":"Reviews any failing test, repairs the change or the test, and resubmits until the impacted set is green.","handoff_to":"Trunk","control":"Human review of every failure."}],"after":[{"order":1,"actor":"Facebook engineer","action":"Submits a code change to the monolithic mobile-code repository for pre-submit testing.","handoff_to":"Predictive test-selection system","control":"Same trunk-based development model."},{"order":2,"actor":"Predictive test-selection model","action":"Scores each affected test's regression-catching probability and selects the highest-value subset.","handoff_to":"Testing infrastructure","control":"Production correctness thresholds: the deployed model must predict more than 95% of individual test outcomes correctly and catch at least one failing test for more than 99.9% of problematic changes; the model is retrained regularly from recent code changes; retries mitigate flaky failures during training."},{"order":3,"actor":"Testing infrastructure","action":"Runs only the model-selected subset — approximately one-third of the transitively-impacted tests — for the change.","handoff_to":"Engineer","control":"Selected tests must pass before merge; unselected tests are relied on for later, longer-cycle validation, not for pre-submit."},{"order":4,"actor":"Facebook engineer","action":"Reviews any failing test in the selected subset, repairs the change or the test, and resubmits.","handoff_to":"Trunk","control":"Human review of every failure; test authorship and repair remain the engineer's job."}],"decision_rights":"The model decides which tests execute for a pre-submit run within its selection budget. Meta engineering sets and audits the production correctness thresholds (more than 95% individual-outcome accuracy; catch at least one failing test on more than 99.9% of problematic changes). Test authorship, failure interpretation, and change acceptance remain with the engineer; the model does not repair tests or accept changes.","exception_path":"If the model would fall below its production correctness thresholds on validation code changes, it is retrained on recent history before promotion; flaky failures are handled by aggressive retries during training-data collection so that the model does not learn a flakiness signal. Any failing test in the selected subset routes to the change author for repair; unselected tests are still available for later, longer-cycle validation outside pre-submit.","removed_work":["Executing the roughly two-thirds of transitively-impacted tests that the learned model estimates are extremely unlikely to catch a regression for a given change","Engineer waiting time on those unnecessary pre-submit test runs","Manual tuning of build-dependency-based selection rules as the codebase evolves","Learning a flakiness signal from noisy test outcomes because retries during training suppress it"],"outcomes":[{"metric":"Pre-submit test load (share of transitively-impacted tests run per change)","baseline":"Build-dependency selection would exercise all tests that transitively depend on modified files — on average as much as approximately one-quarter of all available mobile-code tests per change; treated as 100% of the transitively-impacted set for pre-submit runs.","result":"The deployed system runs 'just a third of all tests that transitively depend on modified code' per change (Engineering at Meta); the ICSE 2019 industry paper reports 'fewer than a third of the tests that would be selected on the basis of build dependencies'.","period":"Deployed at Facebook for more than a year at 2018-11-21; also stated as 'operational at Facebook for several months' in the ICSE 2019 industry-paper text.","scale":"Facebook's monolithic mobile codebase and continuous-integration system — exact monthly change or test counts are not disclosed in either source.","attribution_caveat":"The one-third figure is a comparison against the build-dependency selection strategy — not a comparison against literally all tests in the repository. Meta does not publish per-change absolute test counts or infrastructure-cost figures.","evidence_label":"verified"},{"metric":"Faulty-change catch rate (fraction of regressions detected pre-trunk)","baseline":"The prior build-dependency strategy is treated as the reliability floor; catch rate is not separately quantified in Meta's public disclosure.","result":"'Catches more than 99.9 percent of all regressions before they are visible to other engineers in the trunk code' (blog); a required production floor of 'catch at least one failing test for more than 99.9 percent of problematic changes' (blog) and 'over 99.9% of faulty changes are still reported back to developers' (ICSE paper).","period":"Deployed at Facebook for more than a year at 2018-11-21.","scale":"All pre-submit changes to Facebook's monolithic mobile codebase during the deployed period; the absolute change and regression counts are not disclosed.","attribution_caveat":"The 99.9% is a production correctness threshold enforced by Meta on the deployed model, not a third-party audit. The catch-rate metric is Meta-defined ('regressions visible in trunk') and Meta-measured.","evidence_label":"verified"},{"metric":"Change-based testing infrastructure cost efficiency","baseline":"Under the prior transitive-dependency selection strategy, treated as the baseline cost of change-based testing (index = 1x).","result":"'Doubled the efficiency of our testing infrastructure' (blog); the ICSE 2019 paper phrases the same result as reducing 'the total infrastructure cost of testing code changes by a factor of two'.","period":"Reported for the more-than-a-year Facebook production deployment as of 2018-11-21.","scale":"Facebook's change-based testing infrastructure for the mobile codebase; absolute compute or dollar figures are not disclosed.","attribution_caveat":"'Efficiency' is defined by Meta as change-based testing infrastructure cost; the blog phrasing is operator-narrated, but the ICSE 2019 paper restates it as 'reduces the total infrastructure cost of testing code changes by a factor of two' in the industry-refereed abstract, so the two-times figure is corroborated by a peer/industry-reviewed source. No independent audit.","evidence_label":"verified"}],"executive_lesson":"Meta did not add another layer of dashboards on top of its regression tests; it moved the allocation decision from a build-graph rule to a learned model, subject to two published correctness thresholds. The instructive part is the discipline of the guardrails: over 99.9% faulty-change catch and over 95% individual outcome accuracy are published production floors, not aspirations. That is what makes 'run less to catch more' defensible to an engineering audience — a floor a skeptical reader can test against.","anti_pattern":"Treating 'we ship less-often to be safe' as the only lever. The pre-Meta baseline in the blog is exactly that: run every transitively-impacted test on every change, because a build-graph rule feels safe. Meta's move was to accept that most of those tests never fire and to publish thresholds that make it possible to run a much smaller set without giving up regression signal. Skipping that discipline — running fewer tests without stating the catch-rate and per-test-accuracy floor — is where organisations get in trouble.","questions_for_leaders":["What is the current implicit rule that decides which regression tests we run per change, and how much of that work is actually load-bearing?","What catch-rate and per-signal accuracy would we be willing to publish as a floor for any model that replaces that rule?","How would we detect quality drift as our codebase evolves — who is on the hook when the floor is missed?","Do we treat flaky tests as noise or as a first-class training signal, and how would we know if we were confusing the two?","If we cut change-based testing cost in half, where does the freed capacity get reinvested — more changes per day, deeper long-cycle testing, or savings?"],"collections":["queue-eliminated","embodied-work"],"bundle_version":"1.0.0","bundle_fingerprint":"sha256:c23c6cc2b88153d008ea8fda928f632ce0011fc2d4c5036672a16e5d895bab93","canonical_url":"https://brianletort.ai/transformations/meta-predictive-test-selection"}