With the introduction of a new testwiki-targeted httpbb test suite in sibling task T428972, we'll now have active probes to evaluate health of the Pretrain environment during deployments. The other half of our automated supervision plan is to introduce logstash checks for the canaries and production deployment stages.
Specifically, we will need to add:
- Two new logstash checks with error thresholds appropriate for testwiki during the canaries and production stages, respectively. These values will need to be inferred from historical data for errors with wiki: testwiki in some way (i.e., presumably requiring some modifications to scap analyze-logstash in order to support filtering on the wiki field).
- Note: Unlike scope: train, where canaries capacity (and thus share of traffic) is ~ 3% of total production-stage capacity, expect that scope: pretrain will instead be up to 25% in the near term.
- A new logstash check targeting scope: train during the scope: pretrain production stage, using the same production-stage threshold as used elsewhere (150 errors). This will function as our backstop "system-wide health" signal.
See also TODOs in https://gerrit.wikimedia.org/r/1327168.
Open question: As a separate matter that we may want to keep in mind (but does not block this work), we may want to assess whether a canary_wait_time of 20s is appropriate / viable for Pretrain given the modest amount of traffic testwiki receives. This is primarily relevant to unattended deployments (i.e., timers) which are a bit less latency sensitive than human-attended ones (e.g., backports).