Page MenuHomePhabricator

Create database tables for permalinks in production wikis, and enable the feature
Open, Needs TriagePublic

Related Objects

StatusSubtypeAssignedTask
OpenNone
OpenNone
OpenNone
OpenNone
OpenNone
OpenNone
OpenNone
OpenNone
Openmatmarex
Resolvedmatmarex
Resolvedmatmarex
ResolvedPRODUCTION ERRORmatmarex
ResolvedPRODUCTION ERRORmatmarex
Resolvedmatmarex
ResolvedPRODUCTION ERRORmatmarex
ResolvedLadsgroup
Resolvedmatmarex
Resolvedmatmarex
Resolvedmatmarex
Resolvedmatmarex
Resolvedmatmarex
ResolvedPRODUCTION ERRORmatmarex
Openmatmarex
Resolvedmatmarex
Invalidmatmarex
ResolvedPRODUCTION ERRORmatmarex
Resolvedmatmarex
Resolvedmatmarex
Resolvedmatmarex
Resolvedmatmarex
Resolvedmatmarex
ResolvedSecuritymatmarex

Event Timeline

There are a very large number of changes, so older changes are hidden. Show Older Changes

Mentioned in SAL (#wikimedia-operations) [2022-09-07T20:21:20Z] <samtar@deploy1002> Finished scap: Backport for [[gerrit:830687|Enable wgDiscussionToolsEnablePermalinksBackend on all wikis (T315353)]] (duration: 06m 57s)

Does the deletion work correctly? I'm seeing this:

wikiadmin@10.64.16.208(testwiki)> select * from discussiontools_item_revisions  order by itr_id desc limit 50;
+--------+---------------+-----------------+--------------+---------------+---------------------+-----------+------------------+
| itr_id | itr_itemid_id | itr_revision_id | itr_items_id | itr_parent_id | itr_transcludedfrom | itr_level | itr_headinglevel |
+--------+---------------+-----------------+--------------+---------------+---------------------+-----------+------------------+
|  46963 |         25262 |          542560 |        18019 |         46962 |                NULL |         4 |             NULL |
|  46962 |         25261 |          542560 |        18019 |         46961 |                NULL |         3 |             NULL |
|  46961 |         25260 |          542560 |        18019 |         46960 |                NULL |         2 |             NULL |
|  46960 |         25259 |          542560 |        18018 |         46959 |                NULL |         1 |             NULL |
|  46959 |         25258 |          542560 |        18017 |          NULL |                NULL |         0 |                2 |
|  46954 |         25260 |          542558 |        18019 |         46953 |                NULL |         2 |             NULL |
|  46951 |         25259 |          542557 |        18018 |         46950 |                NULL |         1 |             NULL |
|  46950 |         25258 |          542557 |        18017 |          NULL |                NULL |         0 |                2 |

https://test.wikipedia.org/w/index.php?title=Talk:Rivers_in_mexico/en&action=history

I'm slightly worried we actually might run out of auto increment value if it inserts and deletes this much (and generally it puts pressure on replication with this number of writes). Let me think a bit.

Having DT persistence for page makes sense and is good and easy but I'm not sure I understand problem space of persistence on revision.

I think at least it shouldn't insert + delete. Go with update as much as possible (I'm seeing it for page but not for revision)

@Ladsgroup Can you explain what about these numbers is concerning? Note that this is after running the maintenance script on testwiki (T315510#8200829), and that the largest numbers in itr_revision_id are just keys to the revision table. What I'm seeing here is 46963 discussion items encountered after processing 7761 discussion pages, which seems entirely reasonable.

We should try using 'update' if you think that would help.

@Ladsgroup Can you explain what about these numbers is concerning? Note that this is after running the maintenance script on testwiki (T315510#8200829), and that the largest numbers in itr_revision_id are just keys to the revision table. What I'm seeing here is 46963 discussion items encountered after processing 7761 discussion pages, which seems entirely reasonable.

But the maint script doesn't run for revisions, I assume it'll be much higher number if it were to run on revisions. In other words, It looks like after edits, it's growing really fast.

For example, in this case, in a small discussion page with three comments, three edits led to a jump of 13 in itr_id, in a large discussion page, each edit would a cause an even larger jump. On top of that, deletion and insertion like this has larger write overhead in replication.

We should try using 'update' if you think that would help.

It will help here. Also, I suggest splitting the transaction into smaller pieces.

Change 854097 had a related patch set uploaded (by Bartosz Dziewoński; author: Bartosz Dziewoński):

[operations/mediawiki-config@master] Enable wgDiscussionToolsEnablePermalinksBackend on group0 wikis

https://gerrit.wikimedia.org/r/854097

Change 854097 merged by jenkins-bot:

[operations/mediawiki-config@master] Enable wgDiscussionToolsEnablePermalinksBackend on group0 wikis

https://gerrit.wikimedia.org/r/854097

Mentioned in SAL (#wikimedia-operations) [2022-11-07T21:17:34Z] <urbanecm@deploy1002> Started scap: Backport for [[gerrit:854097|Enable wgDiscussionToolsEnablePermalinksBackend on group0 wikis (T315353)]]

Mentioned in SAL (#wikimedia-operations) [2022-11-07T21:17:54Z] <urbanecm@deploy1002> urbanecm and matmarex: Backport for [[gerrit:854097|Enable wgDiscussionToolsEnablePermalinksBackend on group0 wikis (T315353)]] synced to the testservers: mwdebug1001.eqiad.wmnet, mwdebug1002.eqiad.wmnet, mwdebug2002.codfw.wmnet, mwdebug2001.codfw.wmnet

Mentioned in SAL (#wikimedia-operations) [2022-11-07T21:23:22Z] <urbanecm@deploy1002> Finished scap: Backport for [[gerrit:854097|Enable wgDiscussionToolsEnablePermalinksBackend on group0 wikis (T315353)]] (duration: 05m 47s)

Change 854606 had a related patch set uploaded (by Bartosz Dziewoński; author: Bartosz Dziewoński):

[operations/mediawiki-config@master] Enable wgDiscussionToolsEnablePermalinksBackend on group1 wikis

https://gerrit.wikimedia.org/r/854606

Change 854606 merged by jenkins-bot:

[operations/mediawiki-config@master] Enable wgDiscussionToolsEnablePermalinksBackend on group1 wikis

https://gerrit.wikimedia.org/r/854606

Mentioned in SAL (#wikimedia-operations) [2022-11-08T21:35:15Z] <urbanecm@deploy1002> Started scap: Backport for [[gerrit:854606|Enable wgDiscussionToolsEnablePermalinksBackend on group1 wikis (T315353)]]

Mentioned in SAL (#wikimedia-operations) [2022-11-08T21:35:34Z] <urbanecm@deploy1002> urbanecm and matmarex: Backport for [[gerrit:854606|Enable wgDiscussionToolsEnablePermalinksBackend on group1 wikis (T315353)]] synced to the testservers: mwdebug1001.eqiad.wmnet, mwdebug1002.eqiad.wmnet, mwdebug2002.codfw.wmnet, mwdebug2001.codfw.wmnet

Mentioned in SAL (#wikimedia-operations) [2022-11-08T21:41:53Z] <urbanecm@deploy1002> Finished scap: Backport for [[gerrit:854606|Enable wgDiscussionToolsEnablePermalinksBackend on group1 wikis (T315353)]] (duration: 06m 36s)

Change 856566 had a related patch set uploaded (by Bartosz Dziewoński; author: Bartosz Dziewoński):

[operations/mediawiki-config@master] Enable wgDiscussionToolsEnablePermalinksBackend on group1 wikis (#2)

https://gerrit.wikimedia.org/r/856566

Change 856566 merged by jenkins-bot:

[operations/mediawiki-config@master] Enable wgDiscussionToolsEnablePermalinksBackend on group1 wikis (#2)

https://gerrit.wikimedia.org/r/856566

Mentioned in SAL (#wikimedia-operations) [2022-11-14T14:23:05Z] <taavi@deploy1002> Started scap: Backport for [[gerrit:856566|Enable wgDiscussionToolsEnablePermalinksBackend on group1 wikis (#2) (T315353)]]

Mentioned in SAL (#wikimedia-operations) [2022-11-14T14:23:26Z] <taavi@deploy1002> taavi and matmarex: Backport for [[gerrit:856566|Enable wgDiscussionToolsEnablePermalinksBackend on group1 wikis (#2) (T315353)]] synced to the testservers: mwdebug1002.eqiad.wmnet, mwdebug2002.codfw.wmnet, mwdebug1001.eqiad.wmnet, mwdebug2001.codfw.wmnet

Mentioned in SAL (#wikimedia-operations) [2022-11-14T14:30:11Z] <taavi@deploy1002> Finished scap: Backport for [[gerrit:856566|Enable wgDiscussionToolsEnablePermalinksBackend on group1 wikis (#2) (T315353)]] (duration: 07m 05s)

I think I might have found the underlying problem here.
As I suspected, I think the DT updates (and probably any other secondary data update) are running twice in the same context.

For example, these two look similar if not almost identical:

But look at the start of stack trace. One is explicit call to $page->doSecondaryDataUpdates( $options ); in the job and one is deferred update being ran implicitly after the job is done which JobExecuter runs regardless of whether secondary data updates are done or not (see JobExecutor.php(105)). The only other possible explanation is that we have another job that completely unrelatedly queues secondary data updates in its context.

That was a red herring but looking at the data, refreshlinks jobs are mostly useless and won't make any change to the output (assuming dt storage is filled) but ThreadItemStore::insertThreadItems is being called regardless, maybe we can do a short-circuit when revision id already exists in dt.

Change 901697 had a related patch set uploaded (by Bartosz Dziewoński; author: Bartosz Dziewoński):

[operations/mediawiki-config@master] Enable wgDiscussionToolsEnablePermalinksBackend on group2 wikis

https://gerrit.wikimedia.org/r/901697

Change 901697 merged by jenkins-bot:

[operations/mediawiki-config@master] Enable wgDiscussionToolsEnablePermalinksBackend on group2 wikis

https://gerrit.wikimedia.org/r/901697

Mentioned in SAL (#wikimedia-operations) [2023-03-21T20:02:55Z] <taavi@deploy2002> Started scap: Backport for [[gerrit:900331|Enable DiscussionTools_visualenhancements_newsectionlink_enable on labs for testing]], [[gerrit:901697|Enable wgDiscussionToolsEnablePermalinksBackend on group2 wikis (T315353)]]

Mentioned in SAL (#wikimedia-operations) [2023-03-21T20:04:27Z] <taavi@deploy2002> esanders and taavi and matmarex: Backport for [[gerrit:900331|Enable DiscussionTools_visualenhancements_newsectionlink_enable on labs for testing]], [[gerrit:901697|Enable wgDiscussionToolsEnablePermalinksBackend on group2 wikis (T315353)]] synced to the testservers: mwdebug1002.eqiad.wmnet, mwdebug2001.codfw.wmnet, mwdebug1001.eqiad.wmnet, mwdebug2002.codfw.wmnet

Mentioned in SAL (#wikimedia-operations) [2023-03-21T20:20:35Z] <taavi@deploy2002> Finished scap: Backport for [[gerrit:900331|Enable DiscussionTools_visualenhancements_newsectionlink_enable on labs for testing]], [[gerrit:901697|Enable wgDiscussionToolsEnablePermalinksBackend on group2 wikis (T315353)]] (duration: 17m 40s)

matmarex added a project: Skipped QA.

This is done now: all new comments being posted are being indexed in the permalink database. Old comments will be handled in T315510 by the maintenance script runs.

The new plan to re-enable the feature without repeating this incident is: run the maintenance script first, then enable the config option, then run the maintenance script again on just the changes that will have occurred between these two steps and thus wouldn't have been processed.

ppelberg added a subscriber: ppelberg.

This ticket being resolved depends on permalinks being enabled and permalinks being enabled depends on T315510 being resolved.

Change 946998 had a related patch set uploaded (by Bartosz Dziewoński; author: Bartosz Dziewoński):

[operations/mediawiki-config@master] Enable wgDiscussionToolsEnablePermalinksBackend on s2/s3/s5/s6 group2

https://gerrit.wikimedia.org/r/946998

Change 946998 merged by jenkins-bot:

[operations/mediawiki-config@master] Enable wgDiscussionToolsEnablePermalinksBackend on s2/s3/s5/s6 group2

https://gerrit.wikimedia.org/r/946998

Mentioned in SAL (#wikimedia-operations) [2023-08-08T20:13:26Z] <urbanecm@deploy1002> Started scap: Backport for [[gerrit:946998|Enable wgDiscussionToolsEnablePermalinksBackend on s2/s3/s5/s6 group2 (T315353)]]

Mentioned in SAL (#wikimedia-operations) [2023-08-08T20:14:57Z] <urbanecm@deploy1002> urbanecm and matmarex: Backport for [[gerrit:946998|Enable wgDiscussionToolsEnablePermalinksBackend on s2/s3/s5/s6 group2 (T315353)]] synced to the testservers mwdebug2002.codfw.wmnet, mwdebug1001.eqiad.wmnet, mwdebug1002.eqiad.wmnet, mwdebug2001.codfw.wmnet, and mw-debug kubernetes deployment (accessible via k8s-experimental XWD option)

Mentioned in SAL (#wikimedia-operations) [2023-08-08T20:24:22Z] <urbanecm@deploy1002> Finished scap: Backport for [[gerrit:946998|Enable wgDiscussionToolsEnablePermalinksBackend on s2/s3/s5/s6 group2 (T315353)]] (duration: 10m 55s)

Mentioned in SAL (#wikimedia-operations) [2023-08-08T20:29:13Z] <urbanecm> mwmaint1002: foreachwikiindblist 'group2 & s2' extensions/DiscussionTools/maintenance/persistRevisionThreadItems.php --current --all --touched-after=20230615000000 (T315353)

Mentioned in SAL (#wikimedia-operations) [2023-08-08T20:30:26Z] <urbanecm> mwmaint1002: foreachwikiindblist 'group2 & s3' extensions/DiscussionTools/maintenance/persistRevisionThreadItems.php --current --all --touched-after=20230615000000 (T315353)

Mentioned in SAL (#wikimedia-operations) [2023-08-08T20:30:37Z] <urbanecm> mwmaint1002: foreachwikiindblist 'group2 & s5' extensions/DiscussionTools/maintenance/persistRevisionThreadItems.php --current --all --touched-after=20230615000000 (T315353)

Change 965562 had a related patch set uploaded (by Bartosz Dziewoński; author: Bartosz Dziewoński):

[operations/mediawiki-config@master] Enable wgDiscussionToolsEnablePermalinksBackend on s7 group2

https://gerrit.wikimedia.org/r/965562

Change 965562 merged by jenkins-bot:

[operations/mediawiki-config@master] Enable wgDiscussionToolsEnablePermalinksBackend on s7 group2

https://gerrit.wikimedia.org/r/965562

Mentioned in SAL (#wikimedia-operations) [2023-10-12T20:58:22Z] <thcipriani@deploy2002> Started scap: Backport for [[gerrit:965562|Enable wgDiscussionToolsEnablePermalinksBackend on s7 group2 (T315353)]]

Mentioned in SAL (#wikimedia-operations) [2023-10-12T20:59:36Z] <thcipriani@deploy2002> thcipriani and matmarex: Backport for [[gerrit:965562|Enable wgDiscussionToolsEnablePermalinksBackend on s7 group2 (T315353)]] synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug)

Mentioned in SAL (#wikimedia-operations) [2023-10-12T21:06:17Z] <thcipriani@deploy2002> Finished scap: Backport for [[gerrit:965562|Enable wgDiscussionToolsEnablePermalinksBackend on s7 group2 (T315353)]] (duration: 07m 55s)