Mentioned in SAL (#wikimedia-operations) [2024-08-12T19:13:20Z] <urbanecm> [urbanecm@mwmaint1002 ~]$ mwscript extensions/GrowthExperiments/maintenance/fixLinkRecommendationData.php --wiki=dewiki --search-index --verbose # T372333, logs available as P67277

Mentioned in SAL (#wikimedia-operations) [2024-08-12T19:15:19Z] <urbanecm> [urbanecm@mwmaint1002 ~]$ mwscript extensions/GrowthExperiments/maintenance/fixLinkRecommendationData.php --wiki=dewiki --force --db-table --verbose # T372333, script started

Mentioned in SAL (#wikimedia-operations) [2024-08-12T19:21:48Z] <urbanecm> [urbanecm@mwmaint1002 ~]$ mwscript extensions/GrowthExperiments/maintenance/fixLinkRecommendationData.php --wiki=dewiki --force --db-table --verbose # T372333, script finished, logs are (gzipped) at F57269843

In T372333#10059303, @Stashbot wrote:

Mentioned in SAL (#wikimedia-operations) [2024-08-12T19:13:20Z] <urbanecm> [urbanecm@mwmaint1002 ~]$ mwscript extensions/GrowthExperiments/maintenance/fixLinkRecommendationData.php --wiki=dewiki --search-index --verbose # T372333, logs available as P67277

[urbanecm@mwmaint1002 ~]$ grep 'Fixing' fixLinkRecommendationData-dewiki-aug-2024.log | wc -l
3289
[urbanecm@mwmaint1002 ~]$

Indeed, virtually no of the suggestions had an entry in the DB table (and the same way around, of course). No idea how or why this happened, but something is clearly very wrong.

Because of the very high number of dangling DB records, it wouldn't be possible for the refreshing script to generate new recommendations. For that reason, I dropped all dangling rows (see SAL entries above). This is the new DB size:

wikiadmin2023@10.64.48.161(dewiki)> select count(*) from growthexperiments_link_recommendations;
+----------+
| count(*) |
+----------+
|        7 |
+----------+
1 row in set (0.000 sec)

wikiadmin2023@10.64.48.161(dewiki)>

Clearly, something is very wrong – only seven recommendations survived?

Based on the information above, it is pretty clear to me what happened (and my actions above should've already fixed that). Within the next hours, fresh recommendations should be generated to the pool, making Add Link working again.

As a side-note, I noticed during the investigation that there is an abnormally high number of dangling records at frwiki and itwiki (AFAICS, Add Link works there though). I logged that as T372337.

After some time of the refresh script running, there are recommendations in the DB again:

wikiadmin2023@10.64.0.214(dewiki)> select count(*) from growthexperiments_link_recommendations;
+----------+
| count(*) |
+----------+
|      843 |
+----------+
1 row in set (0.001 sec)

wikiadmin2023@10.64.0.214(dewiki)>

Still nothing in the Homepage or search. Hopefully that would clear by tomorrow.

In T372333#10059660, @Urbanecm_WMF wrote:

Still nothing in the Homepage or search. Hopefully that would clear by tomorrow.

There's something possibly wrong on Search's side. We discussed a bit via IRC and I logged T372362: link-recommendations suggestions are not getting processed by Search infrastructure per Search's request.

With T372362 having made progress, I've created {DNM}: Run fixLinkRecommendationData even when disabled in CC to get a bit of insight into the tasks that we have available in a way that accounts for both Cirrus Search index as well as the database. I've chatted with @Tgr, the original principal author, about it and got a LGTM.
Always as a dry-run, since this we're only interested in the output, not actually doing actual changes. I will run it against testwiki first and if that give the expected output, then I will run it against dewiki and add the output here.

Mentioned in SAL (#wikimedia-operations) [2024-08-21T15:57:13Z] <MichaelG_WMF> T372333, with I431d2aba14db9ab8931e21260cb2005c7276e2b8 checked out, running mwscript /home/migr/GrowthExperiments/maintenance/fixLinkRecommendationData.php --dry-run --wiki=testwiki --search-index --db-table

Yesterday, running the modified maint script against testwiki did reveal a bug, because this output is obviously wrong:

migr@mwmaint1002:~/GrowthExperiments$ mwscript /home/migr/GrowthExperiments/maintenance/fixLinkRecommendationData.php --wiki=testwiki --dry-run --search-index --db-table
Total number of OK search index entries: -4532
Total number of dangling search-index entries: 159
Total number of OK db-table entries: 987
Total number of dangling db-table entries: 5

Let's try again with a new patchset today.

Mentioned in SAL (#wikimedia-operations) [2024-08-22T14:19:20Z] <MichaelG_WMF> T372333, with I431d2aba14db9ab8931e21260cb2005c7276e2b8 checked out, running mwscript /home/migr/GrowthExperiments/maintenance/fixLinkRecommendationData.php --wiki=dewiki --dry-run --search-index --db-table

Now running the modified fixLinkRecommendationData.php script against testwiki and dewiki delivered some interesting results:

migr@mwmaint1002:~/GrowthExperiments$ mwscript /home/migr/GrowthExperiments/maintenance/fixLinkRecommendationData.php --wiki=testwiki --dry-run --search-index --db-table
Total number of OK search index entries: 2738
Total number of dangling search-index entries: 160
Total number of OK db-table entries: 1019
Total number of dangling db-table entries: 5
migr@mwmaint1002:~/GrowthExperiments$ mwscript /home/migr/GrowthExperiments/maintenance/fixLinkRecommendationData.php --wiki=dewiki --dry-run --search-index --db-table
Total number of OK search index entries: 46309
Total number of dangling search-index entries: 855
Total number of OK db-table entries: 15059
Total number of dangling db-table entries: 61273
migr@mwmaint1002:~/GrowthExperiments$

I assume that there are more OK search index entries than OK db-table entries comes from double counting. The script goes through every topic individually and some pages are in multiple topics and so are counted multiple times. I assume the high number of dangling db-table records in dewiki comes from when we tried to add the tags to cirrus search but failed because of T372362?

Either way, I think the line Total number of OK db-table entries: 15059 means that we should now have many tasks available if link-recommendations are enabled again now on dewiki 🤞

In T372333#10085138, @Michael wrote:

Now running the modified fixLinkRecommendationData.php script against testwiki and dewiki delivered some interesting results:

migr@mwmaint1002:~/GrowthExperiments$ mwscript /home/migr/GrowthExperiments/maintenance/fixLinkRecommendationData.php --wiki=testwiki --dry-run --search-index --db-table
Total number of OK search index entries: 2738
Total number of dangling search-index entries: 160
Total number of OK db-table entries: 1019
Total number of dangling db-table entries: 5
migr@mwmaint1002:~/GrowthExperiments$ mwscript /home/migr/GrowthExperiments/maintenance/fixLinkRecommendationData.php --wiki=dewiki --dry-run --search-index --db-table
Total number of OK search index entries: 46309
Total number of dangling search-index entries: 855
Total number of OK db-table entries: 15059
Total number of dangling db-table entries: 61273
migr@mwmaint1002:~/GrowthExperiments$

Thanks for the data!

I assume that there are more OK search index entries than OK db-table entries comes from double counting. The script goes through every topic individually and some pages are in multiple topics and so are counted multiple times. I assume the high number of dangling db-table records in dewiki comes from when we tried to add the tags to cirrus search but failed because of T372362?

Both seem plausible to me.

In retrospective, running the fixing script so that it dropped both DB records and search index records wasn't a good idea (if we haven't run it, the number presumably wouldn't be that high). TBH, I was convinced the link refreshing logic just didn't work properly with the task disabled, and only when even refreshed recommendations didn't appear as well I started to suspect something else.

Either way, I think the line Total number of OK db-table entries: 15059 means that we should now have many tasks available if link-recommendations are enabled again now on dewiki 🤞

Yep. The other number to pay closer attention to is "dangling search-index entries". If we have "too many" of those tasks, then newcomers would be likely presented invalid tasks. We only consult the DB table when a newcomer actually clicks the task, and if we find the entry dangling, the newcomer receives an error message and is sent back to the Homepage. 855 out of ~16k seems low enough in that regard, so re-enabling seems like a good idea at this point.

Since the numbers are promising, I followed up with @Johannnes89, letting them know the problem is now resolved. @Johannnes89 re-enabled Add Link at de.wikipedia, and the feature now appears to work just fine. With that, moving to QA, as the scope of this task is now resolved.

Checked dewiki wmf.21 - the add link tasks are present. However, the availability of tasks is limited to first 20 tasks or so and sometime even to two. The issue is filed as filed as T373234: [wmf.21] Special:Homepage - SE counter for articles with structured Add link tasks displays incorrect numbers
{F57465823}