We should probably fix this and determine the counts prior to enabling Add Link again, so that we can be sure the task would work. This would also (re)add those numbers to the Grafana board.

This is a good task to fix this properly, but see T372333#10085138 🙂

Oh, very well then :). Thanks for the info! We should do this regardless, so the numbers are in Grafana et al, but not necessarily before Add Link gets re-enabled at de.

Change #1063841 had a related patch set uploaded (by Michael Große; author: Michael Große):

[mediawiki/extensions/GrowthExperiments@master] Run fixLinkRecommendationData even when disabled in CC

https://gerrit.wikimedia.org/r/1063841

Change #1063841 merged by jenkins-bot:

[mediawiki/extensions/GrowthExperiments@master] Run fixLinkRecommendationData even when disabled in CC

https://gerrit.wikimedia.org/r/1063841

Change #1079925 had a related patch set uploaded (by Michael Große; author: Michael Große):

[mediawiki/extensions/GrowthExperiments@wmf/1.43.0-wmf.26] Run fixLinkRecommendationData even when disabled in CC

https://gerrit.wikimedia.org/r/1079925

Change #1079925 merged by jenkins-bot:

[mediawiki/extensions/GrowthExperiments@wmf/1.43.0-wmf.26] Run fixLinkRecommendationData even when disabled in CC

https://gerrit.wikimedia.org/r/1079925

Mentioned in SAL (#wikimedia-operations) [2024-10-14T14:02:42Z] <lucaswerkmeister-wmde@deploy2002> Started scap sync-world: Backport for [[gerrit:1079923|refactor(tests): don't use per-method coverage annotation]], [[gerrit:1079894|refactor(HomepageHooks): extract method for simpler modifyability]], [[gerrit:1079915|Clear LinkRecommendation suggestions on page save (T364341 T372337)]], [[gerrit:1079925|Run fixLinkRecommendationData even when disabled in CC (T373176)]]

Mentioned in SAL (#wikimedia-operations) [2024-10-14T14:04:49Z] <lucaswerkmeister-wmde@deploy2002> migr, lucaswerkmeister-wmde: Backport for [[gerrit:1079923|refactor(tests): don't use per-method coverage annotation]], [[gerrit:1079894|refactor(HomepageHooks): extract method for simpler modifyability]], [[gerrit:1079915|Clear LinkRecommendation suggestions on page save (T364341 T372337)]], [[gerrit:1079925|Run fixLinkRecommendationData even when disabled in CC (T373176)]] synced to

Mentioned in SAL (#wikimedia-operations) [2024-10-14T14:09:31Z] <lucaswerkmeister-wmde@deploy2002> Finished scap sync-world: Backport for [[gerrit:1079923|refactor(tests): don't use per-method coverage annotation]], [[gerrit:1079894|refactor(HomepageHooks): extract method for simpler modifyability]], [[gerrit:1079915|Clear LinkRecommendation suggestions on page save (T364341 T372337)]], [[gerrit:1079925|Run fixLinkRecommendationData even when disabled in CC (T373176)]] (duration: 0

QA Note: I'm not sure how to QA this task, but one way to see its effects is to look at the panels for the dangling records and now also see dots for the wikis that have this task disabled in CommunityConfiguration.

• Dangling db records: https://grafana.wikimedia.org/d/vGq7hbnMz/special-homepage-and-suggested-edits?from=now-7d&orgId=1&to=now&viewPanel=37
• Dangling Search Index records: https://grafana.wikimedia.org/d/vGq7hbnMz/special-homepage-and-suggested-edits?from=now-7d&orgId=1&to=now&viewPanel=38

(Note that I have adjusted the Grafana panel to also show us always the dots for a data-point, not only the connecting lines. This highlights where we are missing data. It is still unclear why that happens.)

(Note 2: there might be more dots than usual for eswiki and frwiki, that comes from the work for T372337)

In T373176#10230691, @Michael wrote:

QA Note: I'm not sure how to QA this task, but one way to see its effects is to look at the panels for the dangling records and now also see dots for the wikis that have this task disabled in CommunityConfiguration.

• Dangling db records: https://grafana.wikimedia.org/d/vGq7hbnMz/special-homepage-and-suggested-edits?from=now-7d&orgId=1&to=now&viewPanel=37
• Dangling Search Index records: https://grafana.wikimedia.org/d/vGq7hbnMz/special-homepage-and-suggested-edits?from=now-7d&orgId=1&to=now&viewPanel=38

(Note that I have adjusted the Grafana panel to also show us always the dots for a data-point, not only the connecting lines. This highlights where we are missing data. It is still unclear why that happens.)

(Note 2: there might be more dots than usual for eswiki and frwiki, that comes from the work for T372337)

Thanks, @Michael! I looked at the grafana dashboards and the numbers for dangling db records/index is still high.

Dangling DB record:enwiki (195K), arwiki(49.8K), and dewiki (60.4K)
Dangling search index records: eswiki(54K) and frwiki (49K) .
I checked for link-recommendation on eswiki wmf.26 and frwiki wmf.26 - the link-recommendation suggestions are not available on Special:Homepage.

Questions:

• the number of dangling records on grafana boards should decrease as a result of the script run? Or it's out of the scope of this task?
• should we wait with this task to check after T372337: High number of dangling search index results at fr.wikipedia or it.wikipedia will be done?

In T373176#10236296, @Etonkovidova wrote:

[...]
Dangling DB record:enwiki (195K), arwiki(49.8K), and dewiki (60.4K)
Dangling search index records: eswiki(54K) and frwiki (49K) .

For eswiki, when looking at the last few days, you should see things (slowly) improving. Especially when compared to frwiki for which we did not yet change the behavior and only increased the frequency with which we record the number of dangling records.

[...]
Questions:

• the number of dangling records on grafana boards should decrease as a result of the script run? Or it's out of the scope of this task?

Not via what this task is about in particular. We also use this script in dry-run mode to report the numbers of dangling records (without doing anything about them). And that is what can be verified by looking at enwiki or ruwiki or arwiki, all of which have the structured add-a-link task currently disabled.

• should we wait with this task to check after T372337: High number of dangling search index results at fr.wikipedia or it.wikipedia will be done?

No, I don't think so. This task here is just about the script working on wikis where the task is available but disabled. That it does. Fixing the dangling records is a different problem that we can address in T372337.