Change #1135753 had a related patch set uploaded (by Clément Goubert; author: Clément Goubert):

[operations/puppet@production] alertmanager: Add team/project receivers for Phab

https://gerrit.wikimedia.org/r/1135753

Change #1135754 had a related patch set uploaded (by Clément Goubert; author: Clément Goubert):

[operations/puppet@production] alertmanager: Add routing for task alerts

https://gerrit.wikimedia.org/r/1135754

Change #1135753 merged by Clément Goubert:

[operations/puppet@production] alertmanager: Add team/project receivers for Phab

https://gerrit.wikimedia.org/r/1135753

Change #1135754 merged by Clément Goubert:

[operations/puppet@production] alertmanager: Add routing for task alerts

https://gerrit.wikimedia.org/r/1135754

These jobs are running with the "--dfn-only" flag set so we should at the very least rename these jobs or update the description to outline that they are only deleting links from nonexistent articles rather than doing more general refreshLinks.php actions

Hi, I am pretty sure nobody knows about those jobs anymore.

The cronjobs were added to Puppet by @Dzahn back in 2012 with https://gerrit.wikimedia.org/r/c/operations/puppet/+/5104 . I can almost guarantee this are cronjobs that were untracked flat files on zwinger/whatever the deployment/maint host was at the time. Daniel was most probably Puppetizing the crontab so that we can track the config.

The --dfn-only change was introduced by @tstarling in 2005 (twenty years ago) by https://static-codereview.wikimedia.org/MediaWiki/9473.html , and that refers to REL1_4. The change there was http://mediawiki.org/wiki/Special:Code/MediaWiki/8663 with the comment Assorted improvements.

There were less than a dozen of people having cluster access at that time. It was definitely not me, I had no root. It might have been Jeluf, Brooke or Tim :) I would expect Tim have set the cronjob roughly at the same time he wrote the code.

What I suspect is that maintenance/refreshLinks.php --dfn-only is to remove obsolete links when an article is deleted and old entries are leaking cause at the time we had no database transactions (iirc) and even so, we could have a transaction to delete the article and another one to delete the links. While the first can succeed (and delete the article), the second could fail (and leave the link behind).

I suspect those jobs are there to garbage collects links. But that is really pure guessing :]

In T388530#10780909, @hashar wrote:

Hi, I am pretty sure nobody knows about those jobs anymore.

The cronjobs were added to Puppet by @Dzahn back in 2012 with https://gerrit.wikimedia.org/r/c/operations/puppet/+/5104 . I can almost guarantee this are cronjobs that were untracked flat files on zwinger/whatever the deployment/maint host was at the time. Daniel was most probably Puppetizing the crontab so that we can track the config.

There is a reference to ticket RT-2355 in the gerrit change from 2012. So that was in our old ticket system.

Using the "reference" field and searching for "rt2355" in advanced Phabricator search leads to T80599 which is the imported bug under a new number.

This then links to ticket Bugzilla 16112, haha. And that is https://static-bugzilla.wikimedia.org/show_bug.cgi?id=16112

The comments there start at 2008-10-25 and should tell more of the story.

See https://static-bugzilla.wikimedia.org/show_bug.cgi?id=16112#c3 and around it.

Note: Any BZ tasks you can add +2000 to get the phab task T18112: Run "refreshLinks.php --dfn-only" on all wikis periodically

Yep, unfortunately you can't for RT and you had to get to RT first to then get to Bugzilla from there.

Change #1143121 had a related patch set uploaded (by Scott French; author: Scott French):

[operations/puppet@production] P:mw::maintenance::refreshlinks: rename and prepare for mw-cron

https://gerrit.wikimedia.org/r/1143121

Change #1143122 had a related patch set uploaded (by Scott French; author: Scott French):

[operations/puppet@production] P:mw::maintenance::refreshlinks: migrate s8 to mw-cron

https://gerrit.wikimedia.org/r/1143122

I'll be driving the migration of these jobs.

From a quick read through [0] for the --dfn-only case, it looks like what this script does is fairly straightforward, if rather slow. Indeed, it looks like most shards (sections) take at least an hour.

Given that these run only monthly, and all occurrences for this month will have already passed by the time the necessary changes are ready to merge, and there any concerns about picking one shard and triggering a manual re-run to confirm things work as expected? (or at least appear to)

FWIW, the queries issued by this script do at least target the vslow group.

[0] https://gerrit.wikimedia.org/r/plugins/gitiles/mediawiki/core/+/refs/heads/master/maintenance/refreshLinks.php

Change #1143121 merged by Scott French:

[operations/puppet@production] P:mw::maintenance::refreshlinks: rename and prepare for mw-cron

https://gerrit.wikimedia.org/r/1143121

In T388530#10802405, @Scott_French wrote:

[…]
FWIW, the queries issued by this script do at least target the vslow group.

Indeed, although separate from where refreshLinks jobs read from, they are can be write-heavy. These jobs essentially re-index the link tables of an entire wiki database (pagelinks, templatelinks, categorylinks, etc.) — Those writes are the main reason we'd want to keep frequency down.

Although given that this is the --dfn-only variant, a re-run will most likely not discover any discrepancies and thus perform few if any writes.

If we want to be extra cautious (or sparing with wasting resources) I'd avoid re-runs for s1 (enwiki), s3 (default), s4 (commons), and s8 (wikidata) as those are particularly large.

I think re-runs for s6 or s7 should be fine to do. Mapping of DB sections can be found at https://noc.wikimedia.org/db.php.

In T388530#10732991, @hnowlan wrote:

These jobs are running with the "--dfn-only" flag set so we should at the very least rename these jobs or update the description to outline that they are only deleting links from nonexistent articles rather than doing more general refreshLinks.php actions

+1. Especially since T157670: Periodically run refreshLinks.php on production sites exists, which would actually "run refreshLinks.php" on all wikis, whereas this cron does not.

Change #1143122 merged by Scott French:

[operations/puppet@production] P:mw::maintenance::refreshlinks: migrate s6 to mw-cron

https://gerrit.wikimedia.org/r/1143122

After the s6 shard was migrated to k8s, I stated a manual pilot run of the job using this procedure. Thanks to @Krinkle for the discussion on safety concerns around this.

The s6 pilot completed without issue, producing logs broadly consistent with those from the last systemd timer trigger on May 6th and with a similar runtime (30m vs. 25m).

Given that, I would propose that we migrate the remaining shards.

Change #1144637 had a related patch set uploaded (by Scott French; author: Scott French):

[operations/puppet@production] P:mw::maintenance::refreshlinks: migrate remaining shards to mw-cron

https://gerrit.wikimedia.org/r/1144637

Change #1144637 merged by Scott French:

[operations/puppet@production] P:mw::maintenance::refreshlinks: migrate remaining shards to mw-cron

https://gerrit.wikimedia.org/r/1144637