Page MenuHomePhabricator

Opening Special:EditWatchlist with a large watchlist hits server timeout (Create watchlist pager)
Closed, ResolvedPublicPRODUCTION ERROR

Description

From T220245#5105063 (April 2019) where @kostajh wrote:

Plan as discussed with @Catrope and @SBisson:

  1. Implement a pager that respects the groupings displayed on EditWatchlist. Max number of items rendered per page would be 500.
  2. Add a namespace filter at the top of the page which defaults to "(all)"
Original 2012 report by @Nemo_bis

A user on translatewiki.net: "And I cannot now edit my watchlist due to its huge volume. The server does not allow me to do that (HTTP error 500)". From the log:
I don't know if he tried Special:EditWatchlist or Special:EditWatchlist/raw, he also mentioned having 15.000 items in watchlist but that may be an hyperbole (works for me with only 3000 watchlisted pages).

Examples of (possibly?) caused errors in the logs:

-rakkaus:#mediawiki-i18n- [20-Aug-2012 14:52:34] PHP Fatal error: Allowed memory size of 204472320 bytes exhausted (tried to allocate 32 bytes) in /www/w/includes/db/Database.php on line 1746

-rakkaus:#mediawiki-i18n- [20-Aug-2012 14:53:06] PHP Fatal error: Allowed memory size of 204472320 bytes exhausted (tried to allocate 32 bytes) in /www/w/includes/db/Database.php on line 1733

-rakkaus:#mediawiki-i18n- [20-Aug-2012 14:56:14] PHP Fatal error: Allowed memory size of 204472320 bytes exhausted (tried to allocate 83 bytes) in /www/w/includes/db/DatabaseMysql.php on line 210


See Also:

SeverityUsers with large watchlists (anything that requires more than 60 seconds for Special:EditWatchlist to load, anecdotally seems to be 10k items and above) experience request timeout errors, and are unable to use Special:EditWatchlist
Frequency648 instances in last 90 days
Level of effort to fixMedium/High . There's a patch that needs to be rebased, and substantially reworked for the namespace/title ordering proposed by Platform Engineering
Consequences if we don't fixEditors with large watchlists are frustrated that key functionality (Special:EditWatchlist) is not usable.

Related Objects

Event Timeline

There are a very large number of changes, so older changes are hidden. Show Older Changes

As this does not seem to be solved in the near future it might be useful to add a link to Special:EditWatchlist/raw to the GUI of the watchlist page.

For a warning message to be timely and not-too-intrusive it would need to be at Special:Preferences#mw-prefsection-watchlist "Watched pages". The six checkboxes there provide for automatic additions to the watchlist. But a real solution would be much better, such as the possibility of downloading a complete watchlist in editable form and allowing uploading of a properly formatted list, possibly only to an empty watchlist.

Change 505784 abandoned by Kosta Harlan:

[mediawiki/core@master] (wip) Introduce alphabetic pager for Special:EditWatchlist

Reason:

https://gerrit.wikimedia.org/r/505784

Restricted Application changed the subtype of this task from "Task" to "Production Error". · View Herald TranscriptMar 25 2025, 10:54 AM

Change #1169702 had a related patch set uploaded (by Cparle; author: Cparle):

[mediawiki/core@master] [WIP] Paging on Special:EditWatchlist

https://gerrit.wikimedia.org/r/1169702

Hello @Cparle,
For Tech News, what wording would you suggest as the content, and When should it be included?

Communicating detail is important. Note the Commit message is rather lacking in detail too https://gerrit.wikimedia.org/r/c/mediawiki/core/+/1169702/58//COMMIT_MSG

Hello @Cparle,
For Tech News, what wording would you suggest as the content,

Something along the lines [[Special:EditWatchlist]] is now paginated, showing only a limited amount of watched pages at once. This should solve page load issues for people who watch thousands of pages. (Reading it back, I fear there are a few too technical terms, so feel free to rephrase it.)

and When should it be included?

Not yet: it’s not even merged yet; and as the patch currently stands, it’ll also need a config change (or a follow-up patch) to be available without tweaking the URL.

Some suggested followups now that I've approved the main patch:

  • clearUserWatchedItemsNow() calls getWatchlistFull() just for the message that says how many items were cleared. The query is redundant with the one in WatchedItemStore::clearUserWatchedItems().
  • WatchedItemStore::clearUserWatchedItems() has the potential to cause replication lag and needs to be batched.
  • Submitting the form should take you back to the paged list so that the user can review the next page. Going to a success page without even a link to the correct offset makes this special page pretty much unusable for cleaning out a large watchlist.

Context: https://gerrit.wikimedia.org/r/c/mediawiki/core/+/1169702 (Paging on Special:EditWatchlist) is the "main patch" referred to above. It was +2'd, but is currently failing CI so did not finish getting merged.

Change #1169702 merged by jenkins-bot:

[mediawiki/core@master] Paging on Special:EditWatchlist

https://gerrit.wikimedia.org/r/1169702

Change #1196703 had a related patch set uploaded (by Cparle; author: Cparle):

[operations/mediawiki-config@master] Enable Special:Watchlist pagination on beta

https://gerrit.wikimedia.org/r/1196703

Change #1196703 merged by jenkins-bot:

[operations/mediawiki-config@master] Enable Special:EditWatchlist pagination on beta

https://gerrit.wikimedia.org/r/1196703

Mentioned in SAL (#wikimedia-operations) [2025-10-20T13:46:46Z] <lucaswerkmeister-wmde@deploy2002> Started scap sync-world: Backport for [[gerrit:1196492|Enable REST Sandbox on all wikis (T389409)]], [[gerrit:1192913|Growth: remove no longer in use GENewcomerTasksStarterDifficultyEnabled (T396382)]], [[gerrit:1196857|MetricsPlatform: Initialize $wgMetricsPlatformExperimentStreamNames (T406332)]], [[gerrit:1196703|Enable Special:EditWatchlist pagination on beta (T41510)]]

Mentioned in SAL (#wikimedia-operations) [2025-10-20T14:02:31Z] <lucaswerkmeister-wmde@deploy2002> Finished scap sync-world: Backport for [[gerrit:1196492|Enable REST Sandbox on all wikis (T389409)]], [[gerrit:1192913|Growth: remove no longer in use GENewcomerTasksStarterDifficultyEnabled (T396382)]], [[gerrit:1196857|MetricsPlatform: Initialize $wgMetricsPlatformExperimentStreamNames (T406332)]], [[gerrit:1196703|Enable Special:EditWatchlist pagination on beta (T41510)]] (duration

Testing this out on Beta! Some thoughts:

  • If I filter for a namespace for which I'm watching zero pages, I see the message "You have no items on your watchlist." That's confusing, because I do have pages on my watchlist, just not in this namespace. Should we perhaps change this to something like "There are no pages on your watchlist in this namespace" or similar?
  • If I change the pagination limit to a number greater than the current number of pages on my watchlist, the pagination links disappear, so I can't easily change it back to a lower number (without modifying the URL). I found that unexpected, but I don't have strong opinions about it.
  • After removing titles from my watchlist, I still see the Namespace filter above the confirmation message. I don't think it should be visible on this screen.

Screenshot 2025-10-21 at 13.52.14.png (718×1 px, 103 KB)

I tried it on my watchlist in home wiki with paginate=1 and it adds the namespace filter but it doesn't paginate at all. I have 11K pages watched and it tries to load all of them. Maybe I'm missing something obvious?

@Ladsgroup the train hasn't rolled yet I don't think

Testing this out on Beta! Some thoughts:

  • If I filter for a namespace for which I'm watching zero pages, I see the message "You have no items on your watchlist." That's confusing, because I do have pages on my watchlist, just not in this namespace. Should we perhaps change this to something like "There are no pages on your watchlist in this namespace" or similar?

Thanks. I raised T407965.

  • If I change the pagination limit to a number greater than the current number of pages on my watchlist, the pagination links disappear, so I can't easily change it back to a lower number (without modifying the URL). I found that unexpected, but I don't have strong opinions about it.

I think this is how pagination works in other parts of MediaWiki. Of course, that does not necessarily mean it is desirable.

  • After removing titles from my watchlist, I still see the Namespace filter above the confirmation message. I don't think it should be visible on this screen.

Raised T407964.

I tested this on beta, focusing mainly around the risk of the incorrect watched pages being shown. I wrote a script to check that the results from Special:EditWatchlist matched the results from action=query&list=watchlistraw for various different namespace, limit, offset and direction parameters, and that, from there, paginating through the next and prev buttons returned all the users watched pages (in the chosen namespace, as appropriate). I repeated this for users with watched pages of various expiry lengths and those with expired watched pages.

I checked that the various links on the page go to the right place, that the checkboxes work as they did before and that I can remove watched pages (and raised T407853).

I checked that Special:EditWatchlist/raw and Special:EditWatchlist/clear continue to work, although I don't think they have been touched in this work.

I didn't do much testing on skins other than Vector 2022, except for briefly with Minerva and MobileFrontend (which has its own implementation that does not appear to be effected by this work).

I didn't do any cross-browser or cross-device testing or testing without JavaScript.

Some things to note:

  • When you change the namespace filter, any other parameters (limit, offset, direction) are cleared. This may be desirable. For example, if the offset parameter is for a namespace different from the one you are trying to filter to you may not get any results, which would confuse a user and it would not be obvious how to rectify it.
  • We no longer order results in expiry order (shortest expiry first). This includes when the pagination feature is turned off ($wgEditWatchlistPaginate = false). This makes sense as I guess it would be hard if not impossible to do this with pagination.

There is also T407958, but I guess it is not related to this work.

Not resolving this task just yet as there might still be some discussion going on.

There's a comment (at https://meta.wikimedia.org/wiki/Tech#Editing_Watchlists_on_different_Wikimedia_projects) that I believe is related to this task. The editor there (Sabelöga) notes that: On some wikis [I see Group0 & Group1 wikis] in the Special:EditWatchlist page there is no longer a TableOfContents shown for the different sections [regression?], and Temporarily-watchlisted pages are no longer grouped at the top of each section [which Dom mentioned in the last bulletpoint above]. I will reply there, pointing them here.

There's a comment (at https://meta.wikimedia.org/wiki/Tech#Editing_Watchlists_on_different_Wikimedia_projects) that I believe is related to this task. The editor there (Sabelöga) notes that: On some wikis [I see Group0 & Group1 wikis] in the Special:EditWatchlist page there is no longer a TableOfContents shown for the different sections [regression?], and Temporarily-watchlisted pages are no longer grouped at the top of each section [which Dom mentioned in the last bulletpoint above]. I will reply there, pointing them here.

Changing of the order is intentional to drastically improve performance of loading the page.

For Tech News, if I understand correctly this is almost ready to go but still waiting on a configuration patch (therefore I will wait till next week to include this).

I'm not seeing Special:EditWatchlist pagination on enwiki unless I add ?paginate=1, but even then, the namespace filter doesn't work. And the namespace TOC is gone - I assume this was replaced by the filter, but now I have nothing for looking at specific namespaces except scrolling down the page.

Setting a filter navigates to e.g. https://en.wikipedia.org/wiki/Special:EditWatchlist?namespace=2. If I then add paginate=1 again, the entries are filtered (e.g. https://en.wikipedia.org/wiki/Special:EditWatchlist?namespace=2&paginate=1). But clearly that's not how it's supposed to work.

Change #1200105 had a related patch set uploaded (by Cparle; author: Cparle):

[operations/mediawiki-config@master] Enable pagination on Special:Watchlist everywhere

https://gerrit.wikimedia.org/r/1200105

Deployment of the full feature is scheduled for Monday Nov 3 at 2pm UTC https://wikitech.wikimedia.org/wiki/Deployments#Monday,_November_3

Also note that I created this wish https://meta.wikimedia.org/wiki/Community_Wishlist/W454 for re-implementing sort-by-expiry, just so we can get some idea of how important it is to people.

As it says in the wish we can't re-enable sort-by-expiry as it was because the queries were just too slow, but we might be able to figure out some alternative way of doing it instead

If it's important to you (or to anyone you know), please support the wish!

@Cparle I want to include this in the Tech News going out on Monday Nov 3. How should I word this for an entry?

Here's a draft

We've had a long-standing problem where users with large watchlists can't edit their watchlist because the Special:EditWatchlist page times out. To address this we've introduced paging on Special:EditWatchlist and once it's deployed everyone should be able to edit their watchlist without difficulty

Note that as part of the database adjustments that we needed to make this work, we've changed the sorting on the page so that it no longer sorts by expiry. This was necessary because sorting by expiry is >100x slower than sorting by title. We've had some feedback that this negatively affects some user workflows, so we're created a community wish to re-implement sort-by-expiry to try and understand user demand for it. As it says in the wish - we can't just re-enable sort-by-expiry because the queries were way too slow, but we might be able to figure out some alternative way of doing it instead. If this is important to you please support the wish!

@Cparle I will rewrite this into one paragraph, thanks for the prompt response!

Would also suggest a different word than "paging", since non-techies probably won't know what that is.

I believe the more familiar term is 'pagination' but I guess non-techies could have confusion about that as well. Might be best to describe it.

Thanks for this significant improvement! I suspect most people will be familiar with spreading things over multiple pages, unless they've young enough to have only encountered endless scrolling?

Is this wording better?

Users with large watchlists have long faced timeouts when editing Special:EditWatchlist. The page now loads entries in smaller sections instead of all at once due to a paging update, allowing everyone to edit their watchlists smoothly. As part of the database update, sorting by expiry has been removed because it was over 100× slower than sorting by title. A community wish has been created to explore alternative ways to restore sort-by-expiry. If this feature is important to you, please support the wish!

How about this, but with the page size specified...

Users with large watchlists have long faced timeouts when editing Special:EditWatchlist. Following this update, the watchlist is now displayed in pages of 999 entries, instead of all at once. This should allow everyone to edit their watchlists. ...

Like more-or-less everything on the wikis the page size defaults to 50, but you can change it (standard values are 20, 50, 100, 500 iirc)

Change #1200105 merged by jenkins-bot:

[operations/mediawiki-config@master] Enable pagination on Special:EditWatchlist everywhere

https://gerrit.wikimedia.org/r/1200105

Mentioned in SAL (#wikimedia-operations) [2025-11-03T14:22:18Z] <lucaswerkmeister-wmde@deploy2002> Started scap sync-world: Backport for [[gerrit:1200105|Enable pagination on Special:EditWatchlist everywhere (T41510)]]

Mentioned in SAL (#wikimedia-operations) [2025-11-03T14:26:16Z] <lucaswerkmeister-wmde@deploy2002> cparle, lucaswerkmeister-wmde: Backport for [[gerrit:1200105|Enable pagination on Special:EditWatchlist everywhere (T41510)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.

Mentioned in SAL (#wikimedia-operations) [2025-11-03T14:34:26Z] <lucaswerkmeister-wmde@deploy2002> Finished scap sync-world: Backport for [[gerrit:1200105|Enable pagination on Special:EditWatchlist everywhere (T41510)]] (duration: 12m 08s)

Cparle claimed this task.