Background

One of Growth-Team's maintenance jobs is defined as follows:

# Track task pool size
profile::mediawiki::periodic_job { 'growthexperiments-listTaskCounts':
    command                 => '/usr/local/bin/foreachwikiindblist /srv/mediawiki/dblists/growthexperiments.dblist extensions/GrowthExperiments/maintenance/listTaskCounts.php --topictype ores --statsd --output none',
    interval                => '*-*-* *:11:00',
    cron_schedule           => '11 * * * *',
    kubernetes              => true,
    team                    => $team_name,
    script_label            => 'listTaskCounts.php',
    description             => 'Track ores task pool size',
    concurrency_policy      => 'Forbid',
    startingdeadlineseconds => 1800, # 30 minutes deadline
    helmfile_defaults_dir   => $helmfile_defaults_dir,
}

This job is powering Grafana charts related to the number of tasks we have. As of now, this includes:

• Total Link Recommendation pool size per wiki on the Special:Homepage and Suggested Edits dashboard
• Revise Tone tasks pool per wiki on the Revise Tone Structured Task dashboard
• Task type selector on the Growth team product KPIs dashboard

None of those dashboards make use of the per-topic data (and neither can they; we do not push this to Grafana at all). I did not find instances of any other dashboard.

As of writing, Grafana only receives per-task type counts. This means Grafana can render how many AddLink or Revise Tone tasks a wiki has, but it has no idea how they are distributed across topics.

Problems

There are actually multiple problems with this setup:

• listTaskCounts.php computes per-topic assignment only to discard it: As of now, listTaskCounts.php actually does compute how many tasks are in every task type and topic. However, this information is thrown away (since it runs with --output none), and it is not reported back to Grafana. This essentially means we are running thousands of search queries every hour, the results of which we do not need. If we implement the suggestions listed, we could do the same work with 2% of the queries we use now.
• listTaskCounts.php is the only code that consumes raw ORES topics without grouping: listTaskCounts.php can compute per-topic counts in two ways: using Growth topics and using raw ORES topics. Growth topics is what is visible to the user in Special:Homepage (and in Special:NewcomerTasksInfo). Raw ORES topics are topics defined by ORES; Growth topics are groupings of ORES topics (meaning one Growth topic matches one or more ORES topics). The per-raw-ORES-topic data was once useful when Add Link was setup to have at least 500 suggestions per raw ORES topic (it would give us a way to monitor it). We even had a metric for it in Grafana; this was removed in T345210. As part of T392944, we fully removed the per-topic population of Add Link task pool, so this capability will not be needed in the future too. This is (a) unnecessary code we can remove (b) also much slower, given there is much more raw ORES topics than Growth topics.

Acceptance Criteria

• Do not pass --topictype=ores to listTaskCounts.php at all: the per-topic numbers are ignored anyway, so we can just as well use the cheaper of the two; this should be a quick Puppet change, but it still means we compute numbers we do not use
• Do not compute per-topics values when they are not needed: We can introduce something like --topictype=none to not compute per-topic counts at all. Alternatively, we can declare that --statsd is not using those values, and when --statsd --output=none is used, we just don't compute them silently.

Implementing both changes should save about 98% of search queries the script makes.