Revalidate Add Link suggestions on enwiki
Closed, ResolvedPublic
Actions

Assigned To

Authored By

	Urbanecm_WMF
	Aug 7 2026, 4:17 PM

Project Tags

Referenced Files

	F98078895: image.png
	Aug 11 2026, 4:20 PM

Subscribers

Description

According to conversation on WMF Slack (internal WMF link), the enwiki community recently changed the minimum required score. However, the task pool was not refreshed:

mysql:research@dbstore1009.eqiad.wmnet [enwiki]> select sum(jt.score >= 0.7), count(*) from growthexperiments_link_recommendations r, JSON_TABLE(r.gelr_data, '$.links[*]' COLUMNS (score DOUBLE PATH '$.score')) AS jt;
+----------------------+----------+
| sum(jt.score >= 0.7) | count(*) |
+----------------------+----------+
|              2719146 |  4336334 |
+----------------------+----------+
1 row in set (15.150 sec)

mysql:research@dbstore1009.eqiad.wmnet [enwiki]> select 2719146 / 4336334 * 100;
+-------------------------+
| 2719146 / 4336334 * 100 |
+-------------------------+
|                 62.7061 |
+-------------------------+
1 row in set (0.000 sec)

mysql:research@dbstore1009.eqiad.wmnet [enwiki]>

and it only has ~63% of suggestions that actually meet the requirements. Let's revalidate all of them via revalidateLinkRecommendations.php.

Related Objects
Search...

		Status	Subtype	Assigned	Task
		Resolved		Urbanecm_WMF	T434319 Revalidate Add Link suggestions on enwiki
		Resolved		Urbanecm_WMF	T437158 Revalidate Add Link suggestions on enwiki

Event Timeline

Urbanecm_WMF created this task.Aug 7 2026, 4:17 PM

Restricted Application added a subscriber: Aklapper. · View Herald TranscriptAug 7 2026, 4:17 PM

Urbanecm_WMF triaged this task as High priority.Aug 7 2026, 4:17 PM

While at it, let's also make sure it uses the up to date model:

urbanecm@wmf3345 ~ % curl -s 'https://api.wikimedia.org/service/linkrecommendation/v1/linkrecommendations/wikipedia/en/Prague' | jq .meta.dataset_checksums.model
# the most up to date model
"c25a9fb36577a272dd7ca101655d6169bf8a86dbfb1edce7c9b6ee445dec8e71"
urbanecm@wmf3345 ~ % 

# the actual distribution
mysql:research@dbstore1009.eqiad.wmnet [enwiki]> select JSON_VALUE(gelr_data, '$.meta.dataset_checksums.model') AS model, count(*) from growthexperiments_link_recommendations where gelr_data is not null group by JSON_VALUE(gelr_data, '$.meta.dataset_checksums.model');
+------------------------------------------------------------------+----------+
| model                                                            | count(*) |
+------------------------------------------------------------------+----------+
| 60ec6ebaaf35995b2131c895e6d19ec29c7f239a99d2c5a319703af3c46b9c77 |      174 |
| af64a2773f50c6e2a075968e1cf3abb588baa31f029a7b973beee387727483f5 |   294386 |
| c25a9fb36577a272dd7ca101655d6169bf8a86dbfb1edce7c9b6ee445dec8e71 |   846720 |
+------------------------------------------------------------------+----------+
3 rows in set (8.838 sec)

mysql:research@dbstore1009.eqiad.wmnet [enwiki]>

Mentioned in SAL (#wikimedia-operations) [2026-08-07T16:27:41Z] <urbanecm@deploy1003> mwscript-k8s job started: GrowthExperiments:revalidateLinkRecommendations.php --wiki=enwiki --verbose --scoreLessThan=0.7 --exceptDatasetChecksums=T434319-enwiki-models.txt # T434319

In T434319#12195979, @Stashbot wrote:

Mentioned in SAL (#wikimedia-operations) [2026-08-07T16:27:41Z] <urbanecm@deploy1003> mwscript-k8s job started: GrowthExperiments:revalidateLinkRecommendations.php --wiki=enwiki --verbose --scoreLessThan=0.7 --exceptDatasetChecksums=T434319-enwiki-models.txt # T434319

Content of the file is:

P95954 (An Untitled Masterwork)

1	c25a9fb36577a272dd7ca101655d6169bf8a86dbfb1edce7c9b6ee445dec8e71

Randomly checked on an updated page:

mysql:research@dbstore1009.eqiad.wmnet [enwiki]> select gelr_page, gelr_revision, jt.score, JSON_VALUE(gelr_data, '$.meta.dataset_checksums.model') from growthexperiments_link_recommendations r, JSON_TABLE(r.gelr_data, '$.links[*]' COLUMNS (score DOUBLE PATH '$.score')) AS jt where gelr_page=656;
+-----------+---------------+--------------------+------------------------------------------------------------------+
| gelr_page | gelr_revision | score              | JSON_VALUE(gelr_data, '$.meta.dataset_checksums.model')          |
+-----------+---------------+--------------------+------------------------------------------------------------------+
|       656 |    1355113911 | 0.7443436980247498 | c25a9fb36577a272dd7ca101655d6169bf8a86dbfb1edce7c9b6ee445dec8e71 |
|       656 |    1355113911 | 0.8823658227920532 | c25a9fb36577a272dd7ca101655d6169bf8a86dbfb1edce7c9b6ee445dec8e71 |
|       656 |    1355113911 |  0.753072202205658 | c25a9fb36577a272dd7ca101655d6169bf8a86dbfb1edce7c9b6ee445dec8e71 |
|       656 |    1355113911 | 0.7678349018096924 | c25a9fb36577a272dd7ca101655d6169bf8a86dbfb1edce7c9b6ee445dec8e71 |
|       656 |    1355113911 |  0.714238703250885 | c25a9fb36577a272dd7ca101655d6169bf8a86dbfb1edce7c9b6ee445dec8e71 |
|       656 |    1355113911 | 0.8766732811927795 | c25a9fb36577a272dd7ca101655d6169bf8a86dbfb1edce7c9b6ee445dec8e71 |
|       656 |    1355113911 | 0.7996238470077515 | c25a9fb36577a272dd7ca101655d6169bf8a86dbfb1edce7c9b6ee445dec8e71 |
|       656 |    1355113911 | 0.9001020193099976 | c25a9fb36577a272dd7ca101655d6169bf8a86dbfb1edce7c9b6ee445dec8e71 |
|       656 |    1355113911 | 0.7177906036376953 | c25a9fb36577a272dd7ca101655d6169bf8a86dbfb1edce7c9b6ee445dec8e71 |
|       656 |    1355113911 | 0.7429404258728027 | c25a9fb36577a272dd7ca101655d6169bf8a86dbfb1edce7c9b6ee445dec8e71 |
+-----------+---------------+--------------------+------------------------------------------------------------------+
10 rows in set (0.002 sec)

mysql:research@dbstore1009.eqiad.wmnet [enwiki]>

Seems correct.

For anyone interested in the logs:

🔁 To resume streaming logs, run:
K8S_CLUSTER=eqiad KUBECONFIG=/etc/kubernetes/mw-script-eqiad.config kubectl logs -f job/mw-script.eqiad.ae74h9jw mediawiki-ae74h9jw-app

Urbanecm_WMF moved this task from Incoming to Doing on the Growth-Team (FY2026-27 Q1 Sprint 3) board.Aug 7 2026, 4:33 PM

Hi @Sdkb and @asilvering, I just wanted to make sure to keep you both in the loop:

Certain Community Configuration changes to Suggested Edits, such as increasing the minimumLinkScore, will ensure that newly created suggestions meet the higher threshold. They don’t, however, automatically remove suggestions that were created under the previous threshold.

Since enwiki had more than 1,000,000 Add a Link suggestions, it makes sense to also revalidate the existing pool and remove suggestions that no longer meet the updated confidence score. That work is already underway (this task), and hopefully by next week we’ll have a smaller, higher-confidence pool of suggestions.

Longer term, I’d also love to see us make progress on T317290: Provide a mechanism to regenerate link recommendation task pool after configuration changes, which would give us a more durable way to address this. But, as always, we have more good ideas than time to tackle them all.

Let me know if you have any questions.

Some errors during the revalidation:

Muslim conquests of Afghanistan is outdated, regenerating... There was a problem during the HTTP request: 504 Gateway Timeout
Black is outdated, regenerating... There was a problem during the HTTP request: 504 Gateway Timeout
Cinema of Italy is outdated, regenerating... There was a problem during the HTTP request: 504 Gateway Timeout

Checking what do we do on errors:

mysql:research@dbstore1008.eqiad.wmnet [enwiki]> select page_id, page_namespace, page_title from page where page_namespace=0 and page_title='Muslim_conquests_of_Afghanistan';
+---------+----------------+---------------------------------+
| page_id | page_namespace | page_title                      |
+---------+----------------+---------------------------------+
|    3068 |              0 | Muslim_conquests_of_Afghanistan |
+---------+----------------+---------------------------------+
1 row in set (0.001 sec)

mysql:research@dbstore1008.eqiad.wmnet [enwiki]> 

mysql:research@dbstore1009.eqiad.wmnet [enwiki]> select * from growthexperiments_link_recommendations where gelr_page=3068;
Empty set (0.001 sec)

mysql:research@dbstore1009.eqiad.wmnet [enwiki]>

Apparently, suggestions were dropped (we delete all invalid recommendations, and only insert back whenever a good recommendation is found). This is sensible enough.

As a small visual status update: we can see in our Specia:Homepage performance tracking data how the number of available tasks for add link is dropping: https://grafana.wikimedia.org/d/vGq7hbnMz/special3a-homepage-and-suggested-edits?orgId=1&from=now-30d&to=now&timezone=utc&var-platform=$__all&var-UserImpactHandlerPingLimiter=$__all&var-impactrendermode=$__all

The number of dangling records are not unexpected for such an operation. If they are not transient, we can clean them up later with a maintenance script.

DMburugu edited projects, added: Growth-Team (FY2026-27 Q1 Sprint 4); removed: Growth-Team (FY2026-27 Q1 Sprint 3).Aug 11 2026, 4:42 PM

DMburugu moved this task from Incoming to Doing on the Growth-Team (FY2026-27 Q1 Sprint 4) board.

This can be QA'd in production since the revalidation appears to be moving in the right direction

The script is not finished yet though.

An update on the numbers:

mysql:research@dbstore1009.eqiad.wmnet [enwiki]> select sum(jt.score >= 0.7), count(*) from growthexperiments_link_recommendations r, JSON_TABLE(r.gelr_data, '$.links[*]' COLUMNS (score DOUBLE PATH '$.score')) AS jt;
+----------------------+----------+
| sum(jt.score >= 0.7) | count(*) |
+----------------------+----------+
|              2488452 |  3025702 |
+----------------------+----------+
1 row in set (10.832 sec)

mysql:research@dbstore1009.eqiad.wmnet [enwiki]> select 2488452 / 3025702 * 100;
+-------------------------+
| 2488452 / 3025702 * 100 |
+-------------------------+
|                 82.2438 |
+-------------------------+
1 row in set (0.000 sec)

mysql:research@dbstore1009.eqiad.wmnet [enwiki]> select JSON_VALUE(gelr_data, '$.meta.dataset_checksums.model') AS model, count(*) from growthexperiments_link_recommendations where gelr_data is not null group by JSON_VALUE(gelr_data, '$.meta.dataset_checksums.model');

+------------------------------------------------------------------+----------+
| model                                                            | count(*) |
+------------------------------------------------------------------+----------+
| 60ec6ebaaf35995b2131c895e6d19ec29c7f239a99d2c5a319703af3c46b9c77 |      174 |
| af64a2773f50c6e2a075968e1cf3abb588baa31f029a7b973beee387727483f5 |   107384 |
| c25a9fb36577a272dd7ca101655d6169bf8a86dbfb1edce7c9b6ee445dec8e71 |   742695 |
+------------------------------------------------------------------+----------+
3 rows in set (6.714 sec)

mysql:research@dbstore1009.eqiad.wmnet [enwiki]>

About 82% of links now have a confidence score of 0.7 and above, and about 87% of pages have recommendations generated using the most up to date model.

It has failed:

[urbanecm@deploy1003 ~]$ kubectl get jobs -l username=urbanecm -L script
NAME                       STATUS   COMPLETIONS   DURATION   AGE   SCRIPT
mw-script.eqiad.ae74h9jw   Failed   0/1           12d        12d   revalidateLinkRecommendations.php
[urbanecm@deploy1003 ~]$

Logs are timeouting:

[urbanecm@deploy1003 ~]$ kubectl logs job/mw-script.eqiad.ae74h9jw
error: timed out waiting for the condition
[urbanecm@deploy1003 ~]$

so I don't really know what it failed on, but I can rerun numbers...

mysql:research@dbstore1009.eqiad.wmnet [enwiki]> select sum(jt.score >= 0.7), count(*) from growthexperiments_link_recommendations r, JSON_TABLE(r.gelr_data, '$.links[*]' COLUMNS (score DOUBLE PATH '$.score')) AS jt;
+----------------------+----------+
| sum(jt.score >= 0.7) | count(*) |
+----------------------+----------+
|              2381754 |  2570294 |
+----------------------+----------+
1 row in set (9.413 sec)

mysql:research@dbstore1009.eqiad.wmnet [enwiki]> select 2381754 / 2570294 * 100;
+-------------------------+
| 2381754 / 2570294 * 100 |
+-------------------------+
|                 92.6647 |
+-------------------------+
1 row in set (0.001 sec)

mysql:research@dbstore1009.eqiad.wmnet [enwiki]> select JSON_VALUE(gelr_data, '$.meta.dataset_checksums.model') AS model, count(*) from growthexperiments_link_recommendations where gelr_data is not null group by JSON_VALUE(gelr_data, '$.meta.dataset_checksums.model');
+------------------------------------------------------------------+----------+
| model                                                            | count(*) |
+------------------------------------------------------------------+----------+
| 60ec6ebaaf35995b2131c895e6d19ec29c7f239a99d2c5a319703af3c46b9c77 |      174 |
| af64a2773f50c6e2a075968e1cf3abb588baa31f029a7b973beee387727483f5 |    49155 |
| c25a9fb36577a272dd7ca101655d6169bf8a86dbfb1edce7c9b6ee445dec8e71 |   689566 |
+------------------------------------------------------------------+----------+
3 rows in set (6.282 sec)

mysql:research@dbstore1009.eqiad.wmnet [enwiki]>

We are now at 92%. The script skips pages that don't need an update, so I'll rerun it...

Mentioned in SAL (#wikimedia-operations) [2026-08-19T17:37:45Z] <urbanecm@deploy1003> mwscript-k8s job started: GrowthExperiments:revalidateLinkRecommendations.php --wiki=enwiki --verbose --scoreLessThan=0.7 --exceptDatasetChecksums=T434319-enwiki-models.txt # T434319

[urbanecm@deploy1003 ~]$ mwscript-k8s --file=T434319-enwiki-models.txt --sal -f --comment=T434319 -- GrowthExperiments:revalidateLinkRecommendations.php --wiki=enwiki --verbose --scoreLessThan=0.7 --exceptDatasetChecksums=T434319-enwiki-models.txt                      
⏳ Starting GrowthExperiments:revalidateLinkRecommendations.php on Kubernetes as job mw-script.eqiad.77zn2nnw ...                                                                                                                                                            
⏳ Waiting for the container to start...                                                                                                                                                                                                                                     
🚀 Job is running.                                                                                                                                                                                                                                                           
📜 Streaming logs:                                                                                                                                                                                                                                                           
Revalidating link recommendations:                                                                                                                                                                                                                                           
  fetching task batch starting with page 0
[...]
^C🔁 To resume streaming logs, run:
K8S_CLUSTER=eqiad KUBECONFIG=/etc/kubernetes/mw-script-eqiad.config kubectl logs -f job/mw-script.eqiad.77zn2nnw mediawiki-77zn2nnw-app
ℹ️ To terminate your job and delete it, run:
K8S_CLUSTER=eqiad KUBECONFIG=/etc/kubernetes/mw-script-deploy-eqiad.config kubectl delete job mw-script.eqiad.77zn2nnw

It's now running again. Waiting until it gets to pages it doesn't decide to skip.

See T435384: Getting logs for a long-lived job timeouts for the timeout.

RLazarus mentioned this in T435384: Getting logs for a long-lived job timeouts.Aug 21 2026, 12:18 AM

It's now running again. Waiting until it gets to pages it doesn't decide to skip.

It got to that stage pretty quickly, and is evaluating pages again. We're at 99%!

mysql:research@dbstore1009.eqiad.wmnet [enwiki]> select sum(jt.score >= 0.7), count(*) from growthexperiments_link_recommendations r, JSON_TABLE(r.gelr_data, '$.links[*]' COLUMNS (score DOUBLE PATH '$.score')) AS jt;
+----------------------+----------+
| sum(jt.score >= 0.7) | count(*) |
+----------------------+----------+
|              2381754 |  2570294 |
+----------------------+----------+
1 row in set (9.413 sec)

mysql:research@dbstore1009.eqiad.wmnet [enwiki]> select 2347377 / 2365057 * 100;
+-------------------------+
| 2347377 / 2365057 * 100 |
+-------------------------+
|                 99.2524 |
+-------------------------+
1 row in set (0.001 sec)

mysql:research@dbstore1009.eqiad.wmnet [enwiki]> select JSON_VALUE(gelr_data, '$.meta.dataset_checksums.model') AS model, count(*) from growthexperiments_link_recommendations where gelr_data is not null group by JSON_VALUE(gelr_data, '$.meta.dataset_checksums.model');
+------------------------------------------------------------------+----------+
| model                                                            | count(*) |
+------------------------------------------------------------------+----------+
| 60ec6ebaaf35995b2131c895e6d19ec29c7f239a99d2c5a319703af3c46b9c77 |      174 |
| af64a2773f50c6e2a075968e1cf3abb588baa31f029a7b973beee387727483f5 |     4988 |
| c25a9fb36577a272dd7ca101655d6169bf8a86dbfb1edce7c9b6ee445dec8e71 |   676715 |
+------------------------------------------------------------------+----------+
3 rows in set (5.747 sec)

mysql:research@dbstore1009.eqiad.wmnet [enwiki]>

And we're done!

  Vijay Chintakayala is outdated, regenerating... All of the links in the recommendation have been pruned
  Natica fasciata is outdated, regenerating... All of the links in the recommendation have been pruned
  Chervoniy is outdated, regenerating... number of good links too small (1)
Done; replaced 57726, discarded 63105, null recommendations deleted 0
[urbanecm@deploy1003 ~]$

(ignore the total numbers, as the script was terminated in the middle, cf. T435384)

Checking the numbers:

mysql:research@dbstore1009.eqiad.wmnet [enwiki]> select sum(jt.score >= 0.7), count(*) from growthexperiments_link_recommendations r, JSON_TABLE(r.gelr_data, '$.links[*]' COLUMNS (score DOUBLE PATH '$.score')) AS jt;
+----------------------+----------+
| sum(jt.score >= 0.7) | count(*) |
+----------------------+----------+
|              2312047 |  2312794 |
+----------------------+----------+
1 row in set (8.513 sec)

mysql:research@dbstore1009.eqiad.wmnet [enwiki]> select 2312047 / 2312794 * 100;
+-------------------------+
| 2312047 / 2312794 * 100 |
+-------------------------+
|                 99.9677 |
+-------------------------+
1 row in set (0.001 sec)

mysql:research@dbstore1009.eqiad.wmnet [enwiki]> select JSON_VALUE(gelr_data, '$.meta.dataset_checksums.model') AS model, count(*) from growthexperiments_link_recommendations where gelr_data is not null group by JSON_VALUE(gelr_data, '$.meta.dataset_checksums.model');
+------------------------------------------------------------------+----------+
| model                                                            | count(*) |
+------------------------------------------------------------------+----------+
| 60ec6ebaaf35995b2131c895e6d19ec29c7f239a99d2c5a319703af3c46b9c77 |      174 |
| af64a2773f50c6e2a075968e1cf3abb588baa31f029a7b973beee387727483f5 |      254 |
| c25a9fb36577a272dd7ca101655d6169bf8a86dbfb1edce7c9b6ee445dec8e71 |   667792 |
+------------------------------------------------------------------+----------+
3 rows in set (5.735 sec)

mysql:research@dbstore1009.eqiad.wmnet [enwiki]>

Something's wrong... We should be at 100% and only having one model. Hypothesis on the cause:

Dangling records: For some reason, the script never sees pages where the DB record is dangling, so it fails to renew it.
[for the link score] Recommendations with a high-enough link score are available: Perhaps for all the pages with a <0.7 recommendation, there is also a >=0.7 recommendation, which might be enough for the script (the SQL query is looking at individual recommended links, not pages)? This would not explain the model though, because that is at the page-level...
Old revision data: Potentially, the script might be only revalidating data for the newest revision. We should be dropping them, but it is possible we failed, and those records should just be dropped?
An error in the SQL: I don't work with JSON management functions too often, and there might very well be an issue in the query, not in the revalidation.

Example page for the link score:

mysql:research@dbstore1009.eqiad.wmnet [enwiki]> select gelr_page from growthexperiments_link_recommendations r, JSON_TABLE(r.gelr_data, '$.links[*]' COLUMNS (score DOUBLE PATH '$.score')) AS jt WHERE jt.score < 0.7 limit 1;
+-----------+
| gelr_page |
+-----------+
|  21061576 |
+-----------+
1 row in set (0.001 sec)

mysql:research@dbstore1009.eqiad.wmnet [enwiki]> select jt.score, gelr_page, gelr_revision from growthexperiments_link_recommendations r, JSON_TABLE(r.gelr_data, '$.links[*]' COLUMNS (score DOUBLE PATH '$.score')) AS jt where gelr_page=21061576;
+--------------------+-----------+---------------+
| score              | gelr_page | gelr_revision |
+--------------------+-----------+---------------+
| 0.6227237582206726 |  21061576 |     625881408 |
| 0.8780578374862671 |  21061576 |     625881408 |
+--------------------+-----------+---------------+
2 rows in set (0.001 sec)

mysql:research@dbstore1009.eqiad.wmnet [enwiki]>

Checking the revision on-wiki gives:

The revision #625881408 belongs to a deleted page; details can be found in the deletion log.

Indeed (querying a public replica, as archive has sensitive data):

MariaDB [enwiki_p]> select * from revision where rev_id=625881408;
Empty set (0.001 sec)

MariaDB [enwiki_p]> select * from archive where ar_rev_id=625881408\G
*************************** 1. row ***************************
        ar_id: 123620296
 ar_namespace: 0
     ar_title: Cable_Car_(cocktail)
      ar_text: NULL
ar_comment_id: NULL
     ar_actor: 23738107
 ar_timestamp: 20140917000458
ar_minor_edit: 0
     ar_flags: NULL
    ar_rev_id: 625881408
   ar_deleted: 0
       ar_len: 1022
   ar_page_id: 21061576
 ar_parent_id: 532073799
1 row in set (0.003 sec)

MariaDB [enwiki_p]>

Checking more pages:

mysql:research@dbstore1009.eqiad.wmnet [enwiki]> select distinct gelr_page from growthexperiments_link_recommendations r, JSON_TABLE(r.gelr_data, '$.links[*]' COLUMNS (score DOUBLE PATH '$.score')) AS jt WHERE jt.score < 0.7 limit 3;
+-----------+
| gelr_page |
+-----------+
|  21061576 |
|   6815834 |
|   5495323 |
+-----------+
3 rows in set (0.025 sec)

mysql:research@dbstore1009.eqiad.wmnet [enwiki]> select jt.score, gelr_page, gelr_revision from growthexperiments_link_recommendations r, JSON_TABLE(r.gelr_data, '$.links[*]' COLUMNS (score DOUBLE PATH '$.score')) AS jt where gelr_page in (6815834, 5495323);
+--------------------+-----------+---------------+
| score              | gelr_page | gelr_revision |
+--------------------+-----------+---------------+
| 0.7074320912361145 |   5495323 |     907960353 |
|  0.650580883026123 |   5495323 |     907960353 |
| 0.6510672569274902 |   5495323 |     907960353 |
| 0.8591067790985107 |   5495323 |     907960353 |
|  0.636380672454834 |   6815834 |     895686470 |
| 0.9273339509963989 |   6815834 |     895686470 |
| 0.6354208588600159 |   6815834 |     895686470 |
| 0.6355598568916321 |   6815834 |     895686470 |
+--------------------+-----------+---------------+
8 rows in set (0.002 sec)

mysql:research@dbstore1009.eqiad.wmnet [enwiki]> select distinct gelr_revision from growthexperiments_link_recommendations r, JSON_TABLE(r.gelr_data, '$.links[*]' COLUMNS (score DOUBLE PATH '$.score')) AS jt where gelr_page in (6815834,
 5495323);
+---------------+
| gelr_revision |
+---------------+
|     907960353 |
|     895686470 |
+---------------+
2 rows in set (0.001 sec)

mysql:research@dbstore1009.eqiad.wmnet [enwiki]>

Both revisions:

give the same "belongs to a deleted page".

Let's check whether the outdated models are caused by the same thing:

mysql:research@dbstore1009.eqiad.wmnet [enwiki]> select gelr_page, gelr_revision from growthexperiments_link_recommendations where gelr_data is not null and JSON_VALUE(gelr_data, '$.meta.dataset_checksums.model') != 'c25a9fb36577a272dd7ca101655d6169bf8a86dbfb1edce7c9b6ee445dec8e71' limit 10;
+-----------+---------------+
| gelr_page | gelr_revision |
+-----------+---------------+
|  21061576 |     625881408 |
|   6815834 |     895686470 |
|   5495323 |     907960353 |
|  11380692 |     910491221 |
|  38833860 |     919613528 |
|   7151832 |     950965947 |
|  49382329 |     951354027 |
|  53834871 |     995200068 |
|   9037620 |    1015154453 |
|  64156472 |    1039624842 |
+-----------+---------------+
10 rows in set (0.053 sec)

mysql:research@dbstore1009.eqiad.wmnet [enwiki]>

Spot checked a few revisions, all are belonging to a deleted page.

\GrowthExperiments\NewcomerTasks\MediaWikiEventIngress\PageLatestRevisionChangedIngress is supposed to clean revision records for edited (and ideally, also deleted) pages, but that doesn't seem to happen.

Urbanecm_WMF mentioned this in T435729: GrowthExperiments never deletes rows with gelr_data IS NULL.Aug 23 2026, 4:48 PM

Filled followups:

Mentioned in SAL (#wikimedia-operations) [2026-08-23T17:01:06Z] <urbanecm@deploy1003> mwscript-k8s job started: GrowthExperiments:fixLinkRecommendationData.php --wiki=enwiki --db-table --search-index --force --verbose # T434319

Mentioned in SAL (#wikimedia-operations) [2026-08-23T17:26:55Z] <urbanecm@deploy1003> mwscript-k8s job started: GrowthExperiments:fixLinkRecommendationData.php --wiki=enwiki --db-table --force --verbose --verbose # T434319

Let's call it done.

KStoller-WMF created subtask T437158: Revalidate Add Link suggestions on enwiki.Sun, Sep 6, 2:31 AM

KStoller-WMF mentioned this in T437158: Revalidate Add Link suggestions on enwiki.

KStoller-WMF closed subtask T437158: Revalidate Add Link suggestions on enwiki as Resolved.Tue, Sep 22, 10:29 PM

Revalidate Add Link suggestions on enwikiClosed, ResolvedPublicActions

Description

Related ObjectsSearch...

Event Timeline

Revalidate Add Link suggestions on enwiki
Closed, ResolvedPublic
Actions

Related Objects
Search...