Page MenuHomePhabricator

[GrowthBook Integration] Preserve TK processing and cached GrowthBook experiments when GrowthBook API is unavailable
Closed, ResolvedPublic2 Estimated Story Points

Description

Background

Because we set a default for TK-registered experiments via config in T428017: [GrowthBook Integration] Replace run_poller with Test Kitchen and GrowthBook source config variables while we iterate on GrowthBook experiments validation/adaptation, we need a way to retain TK experiments when the GB API is unavailable.

The GrowthBook poller currently aborts the entire poll cycle if fetchExperiments() errors. This means a temporary GrowthBook outage (like during deployments) blocks TK-registered experiments, freezes experiments_cache, and registers growthbook_polls_total{outcome="failure"} which is indistinguishable from a real pipeline fail.

Instead we should implement the following:

  • TK experiments should keep flowing on every poll regardless of GrowthBook reachability (for now until we later disable TK experiments)
  • Existing GB-sourced rows in experiments_cache should survive a GB outage instead of going stale and (if anything updated) being mis-evicted.
  • A distinct metric outcome so logging can tell the difference between "GB unreachable, TK fine" from "the whole pipeline is broken."

This mirrors the existing saved-groups graceful-degradation pattern i.e. savedGroupsAvailable >> per-experiment preservedExperiments one level up at the poller.

Description

Create a flag gbAvailable that runs through poll() > process() > store() such that:

  • poll() catches fetchExperiments errors, logs them, and passes an empty experiments array and gbAvailable=false to processConfig - the cycle should be tagged as growthbook_polls_total{outcome="partial"} instead of failure and growthbook_last_successful_poll_timestamp_seconds only proceeds on full success
  • process() skips validateAllExperiments + adaptAllExperiments when gbAvailable is false and passes preserveAllGb: true to store()
  • store() adds .whereNot('source', 'growthbook') to the delete query when preserveAllGb is set which preserves existing GB experiments while TK experiments are still inserted/updated/deleted

Acceptance Criteria

  • GB fetch failure means: outcome="partial", TK still processed, existing GB rows survive
  • Full success: outcome="success", timestamp advances
  • TK load or store failure: outcome="failure" (same as before)
  • Existing saved-groups preservation still works and runs alongside new preserveAllGb
  • Add/update relevant tests

Details

Related Changes in Gerrit:
Related Changes in GitLab:
TitleReferenceAuthorSource BranchDest Branch
Keep processing TK experiments when GrowthBook is unavailablerepos/data-engineering/test-kitchen!341cjmingT428222/preserve-tk-experiments-when-gb-n/amain
Customize query in GitLab

Event Timeline

Change #1298823 had a related patch set uploaded (by Santiago Faci; author: Santiago Faci):

[operations/deployment-charts@master] Test Kitchen UI: Deploy v1.4.1 release to staging

https://gerrit.wikimedia.org/r/1298823

Change #1298823 merged by jenkins-bot:

[operations/deployment-charts@master] Test Kitchen UI: Deploy v1.4.1 release to staging

https://gerrit.wikimedia.org/r/1298823

Change #1299575 had a related patch set uploaded (by Santiago Faci; author: Santiago Faci):

[operations/deployment-charts@master] Test Kitchen UI: Deploy v1.4.1 release to production

https://gerrit.wikimedia.org/r/1299575

Change #1299575 merged by jenkins-bot:

[operations/deployment-charts@master] Test Kitchen UI: Deploy v1.4.1 release to production

https://gerrit.wikimedia.org/r/1299575