Page MenuHomePhabricator

After opening a diff, entry on Special:Watchlist sometimes stays unread (bold)
Open, HighPublicBUG REPORT

Description

on march 15th (around 18:00 (CET) ) a brief bug was noticed concerning special:watchlist on fr:wikipedia.

Steps to Reproduce:
open a diff of your watchlist, reload the watchlist.

Actual Results:
the article title stay in bold.

Expected Results:
the article title should not be bold anymore.

the bug was "resolved" 25-30 minutes after I first spotted it.

Today (march 17th) around 13:30 (CET) I noticed the bug was strinking again. This time, it lasted around 20 mn.

Hewiki comment:
We already saw it every couple of hours. The API query for unread edits returns all rhe bold ones all the waiting time.

Event Timeline

There are a very large number of changes, so older changes are hidden. Show Older Changes

This has become considerably worse as of this week.

Yes, significant degradation in the past few days is known (and a sub-set of the generic problem here); see T240518: Some jobs are not being processed / are processed slowly, where hopefully my colleagues should find and get a fix soon.

Chaddy added a subscriber: Chaddy.Dec 16 2019, 2:35 PM

The problems occurs almost every time I reload my watchlist. Often entrys are bold I already have seen, even edits I did myself. Also sometimes entrys are not bold although I never have seen them. Furthermore in article histories sometimes diffs are marked as unseen although I already have seen them. Something is really broken here. And it is not a new problem, it occurs since several months! But a short time ago it has become worse. Please fix it asap!

greg removed Catrope as the assignee of this task.Dec 16 2019, 7:22 PM
greg added a subscriber: greg.

Remove Roan from assignee; he worked on this previously, but then this task was reopened and there's no update from him since.

YBG added a subscriber: YBG.Dec 16 2019, 11:25 PM
MBH added a subscriber: MBH.Dec 17 2019, 2:46 AM

While the actions taken to address T240518: Some jobs are not being processed / are processed slowly remedied the very noticeable spike, this is still an ongoing issue. This is a Regression against the previously working as intended behavior.

Comment. Many pages I routinely watch remain on my watchlist, but are disappearing from visibility. Template:Did You Know has been on my watchlist for a very long time. When the DYKUpdate bot loaded the new batch on 22 December, the template totally disappeared from visibility on my watchlist, even though the page itself said I had it watched. When an editor made an edit, the DYK template magically became visible on my watches pages again. As of the 23 December DYKUpdate happened, the template once again disappeared from my watched list visibility. I got it to reappear by doing a test edit and then reverting myself. The same phenonemon happened to me with watching Village pump (technical). No way of knowing how many are not displaying (but should be) if I'm not expecting a specific one to change.

Stryn added a comment.EditedDec 23 2019, 6:46 AM

Sounds like you have some filters enabled on your watchlist to not show edits made by bots.

Someone mentioned to me that it might have to do with the bot edits suppression on my wachlist. I've not unchecked that option, so perhaps my issue has been solved.

Has this still been a problem recently? Asking as T240518 got fixed.

MBH added a comment.Feb 9 2020, 5:03 AM

Looks like it was fixed many weeks ago.

JJMC89 added a comment.Feb 9 2020, 7:39 AM

Has this still been a problem recently? Asking as T240518 got fixed.

Yes. I experienced it within the last week.

While the actions taken to address T240518: Some jobs are not being processed / are processed slowly remedied the very noticeable spike, this is still an ongoing issue. This is a Regression against the previously working as intended behavior.

Me either, three times last week.

No problems since weeks ago

For weeks I had no problems. But today again the error occured. So unfortunately it is NOT fixed! Maybe a new update destroyed the older fix?

No, I'm seeing it constantly.

JM added a subscriber: JM.Feb 14 2020, 11:31 AM

Yes, it still happens from time to time, although much less frequently. Sometimes even my own edits are marked as unread.

WDoranWMF added subscribers: Pchelolo, WDoranWMF.

Performance-Team Is this an issue you can input on? Is there a way CPT can help out? Going to untag us for now but @Pchelolo will take a quick look to review.

I've checked the JobQueue performance for clearUserWatchlist job and the delay is less then one second. I could find any evidence the problem is in the job queue itself, so I'm not sure what CPT can help with here.

I've checked the JobQueue performance for clearUserWatchlist job and the delay is less then one second. I could find any evidence the problem is in the job queue itself, so I'm not sure what CPT can help with here.

You can try it now, on cswiki I have bold almost all edits from today, even though all diff links are colored as visited by my browser, see screenshot:

Dvorapa added a comment.EditedMar 2 2020, 2:10 PM

After 5 minutes from the original screenshot and 15 minutes from me visiting that particular page, still one left:

The last link took 2 hours to be marked as read. It seems to me like the clear user watchlist job leaves some revisions out

Yes, it's become more prominent again in the last day or two on en.wikipedia.org.

Maile66 removed a subscriber: Maile66.Mar 3 2020, 3:04 AM

Happening a lot in the last few days, including right now, on en.wikipedia.org

Happening a lot in the last few days, including right now, on en.wikipedia.org

Same for me, not just enwiki

Izno added a comment.Apr 12 2020, 2:20 AM

I've been having general connection issues here and there today where Wikipedia doesn't load (to various failings; some of it was "hit go in browser and got nothing" and some of it was "got HTML but no CSS/JS") and also experienced this issue at the same time. Maybe related?

It happened to me as well in the last few days, and I noticed that while it is happening I can not upload files on Commons (I am using the upload wizard, and the upload gets queued forever and then says something like "server did not respond on time".

@Izno: General connection issues are a different topic than the watchlist staying bold, hence it is off-topic here. See T250025 - thanks.
@Ymblanter: "Uploading files to Commons is a different topic than the watchlist staying bold, hence it is off-topic here. Please file separate issues - thanks.

My only point was that the uploading problems occur consistently at the same time as the watchlist problems. In the situation when the issues are happening over a year and nobody understands why it might be helpful. If it is not, fine with me.

Happening a lot in the last few days, including right now, on en.wikipedia.org

It is also still happening very often in German Wikipedia. I am disappointed that the developers still did not find a solution for this quite old problem.

Krinkle added a comment.EditedApr 14 2020, 2:42 PM

This problem is mainly due to JobQueue stalling in production which happens from time to time, but usually resolves itself within a few minutes.

However, in practice this should not be an issue given we only use the JobQueue for long-term storing of the seen markers. There is a 1 hour limited storage of upto 300 reviewed diffs in MW's "MainStash", which should have solved thid problem.

But, I too see this probelm still from time to time and it is consistently "forgetting" items within a minute. Not after an hour, not after 300 items.

This suggests to me what regardless of whether the JobQueue is working, there is likely a bug in the WatchedItemStore's logic for the stashed "1 hour / upto 300 item" value.

kostajh added subscribers: Krinkle, kostajh.

Here's a way to reproduce this locally with MediaWiki-Docker, you can put this in LocalSettings.php (h/t to @Krinkle for the tips)

$wgJobRunRate = 0;
$wgMainCacheType = CACHE_ACCEL;
$wgMainStash = 'db-replicated';

Then, create TestOne and TestTwo articles with an anonymous user, and as another user add them to your watchlist. Now edit the articles with the anonymous user and reload the watchlist. You'll see both appear bold (unseen), but then if you click to open them you'll start seeing the silliness noted above where sometimes the revisions are bold, sometimes they're not, etc. As far as I can tell, running php maintenance/runJobs.php always brings the revisions back to the expected seen/unseen state.

In debugging this, I ended up in WatchedItemStore.php, in the resetNotificationTimestamp() method, where eventually you get to this bit of code

// Mark the item as read immediately in lightweight 
$this->stash->merge(
	$this->getPageSeenTimestampsKey( $user ),
		function ( $cache, $key, $current ) use ( $title, $seenTime ) {
				$value = $current ?: new MapCacheLRU( 300 );

Some odd things about this: the documentation for merge() says the arguments to the method are BagOStuff, cache key, current value, TTL., we don't know what types current value could be, but OK, let's look at at the specific implementation in MediumSpecificBagOStuff. There we end up in mergeViaCas() where current value (that gets passed as an argument to merge() is defined as

$currentValue = $this->resolveSegments(
	$key,
	$this->doGet( $key, $flags, $token )
);

And resolveSegments() says it returns * @return string|null|bool The combined string, false if missing, null on error.

But, back to the code above:

function ( $cache, $key, $current ) use ( $title, $seenTime ) {
	$value = $current ?: new MapCacheLRU( 300 );

When the unseen/seen bug occurs, $current is not string|false|null, but is instead an instance of MapCacheLRU, and the catch is that its maxCacheKeys value is set to null, instead of 300 (as we set it when $current is null). So, if the MapCacheLRU is holding the seen data of a recent item, it is promptly expunged when we do $value->set( $subKey, $seenTime ); later in that method, because MapCacheLRU's set() implementation will empty the cache if the new size is greater than $this->maxCacheKeys, which is null in this case.

This doesn't totally explain the all the problems noted here but seems to be part of the puzzle anyway.

$current is the value stored in the cache key, which can be anything. In this case, it looks like a MapCacheLRU object is stored in the cache key, which presumably gets serialized when stored in the cache and unserialized when retrieved from the cache. MapCacheLRU::serialize() does not return, and MapCacheLRU::unserialize() does not restore, the maxKeys property, so it would seem that that's (part of) the problem.

Change 589183 had a related patch set uploaded (by Catrope; owner: Catrope):
[mediawiki/core@master] MapCacheLRU: Serialize maxCacheKeys property

https://gerrit.wikimedia.org/r/589183

Change 589183 merged by jenkins-bot:
[mediawiki/core@master] MapCacheLRU: Serialize maxCacheKeys property

https://gerrit.wikimedia.org/r/589183

Change 589262 had a related patch set uploaded (by Kosta Harlan; owner: Kosta Harlan):
[mediawiki/core@master] Allow updating notification timestamp where timestamp is not null

https://gerrit.wikimedia.org/r/589262

Change 589358 had a related patch set uploaded (by Krinkle; owner: Krinkle):
[mediawiki/core@master] objectcache: Add regression test for MapCacheLRU serialization

https://gerrit.wikimedia.org/r/589358

Change 589358 merged by jenkins-bot:
[mediawiki/core@master] objectcache: Add regression test for MapCacheLRU serialization

https://gerrit.wikimedia.org/r/589358

A quick update for those watching this task; https://gerrit.wikimedia.org/r/589183 was merged and will ride the next train. Next week there are no deployments so it should be in group 2 wikis on Thursday April 30. At that point I would expect that the bug in this task description will no longer be present.

There is a remaining issue, that might be deserving of its own task:

  • you are watching an article
  • someone makes three edits to that article with revisions 2, 3 and 4
  • revisions 2, 3 and 4 show up on your watchlist because you have the "Expand watchlist to show all changes, not just the most recent" preference enabled
  • revisions 2, 3 and 4 are in bold / unseen, as expected
  • you click on the diff for revision 2
  • you reload the watchlist, before the updateWatchlistNotification job has executed
  • expected behavior: when reloading watchlist, revision 2 is marked as "seen" and revisions 3 and 4 remain "unseen"
  • actual behavior: revisions 2, 3 and 4 are marked as "seen"
    • addendum to actual behavior: when the updateWatchlistNotification job (which was created when you clicked on the diff for revision 2) runs, the watchlist is now in the correct, expected state where revision 2 is marked as "seen" and revisions 3 and 4 are marked as "unseen"

Change 589262 abandoned by Kosta Harlan:
Allow updating notification timestamp where timestamp is not null

https://gerrit.wikimedia.org/r/589262

Izno added a comment.Apr 17 2020, 3:41 PM

There is a remaining issue, that might be deserving of its own task: [snip]

Yes please, file one! This one has also been driving me nuts and I've been just on the wrong side of the lazy mark to file it myself. (It also occurs a lot due to interaction with RevisionTimeline.)

There is a remaining issue, that might be deserving of its own task: [snip]

Yes please, file one! This one has also been driving me nuts and I've been just on the wrong side of the lazy mark to file it myself. (It also occurs a lot due to interaction with RevisionTimeline.)

Filed as T250770: Watchlist: `updateWatchlistNotification` job incorrectly marks revisions as "seen" - @Izno, please add your comments re interaction issues with RevisionTimeline to that ticket (and anything else that seems to be relevant).

Etonkovidova added a comment.EditedApr 21 2020, 12:41 AM

Checked in betalabs the failing cases mentioned in the comments above. Also checked the overall Watchlist functionality - all seems to be normal. I'll follow up with testing in production after the deployment.

Filed a minor bug - T250775: Watchlist: While loading the page displays solid (black) bullet points

Note: Additional case to check

  • wikidata edits on Watchlist still displayed "Unseen" after visiting the wikidata diff page (clicking on diff)
Etonkovidova closed this task as Resolved.May 6 2020, 10:58 PM
Etonkovidova claimed this task.

Checked in frwiki and enwiki wmf.30 - did not see the issue happening (my watchlist is rather small, so there might be some issues with bigger watchlists).

Two cases when seen status is still not applied

  • when a diff is viewed in incognito window
  • when wikidata update is seen, but not the wiki page where the wikidata item is used

It happened yesterday for the last time.

... And today.

It happened yesterday for the last time.

So, the ticket should should be re-open?

@IKhitron - since my testing did not show any obvious problems, it seems that I am missing something. Could you please add some info on the following?

  • how big the watchlist is
  • do you have filters in place when you reload watchlist
  • and do filters, like 'unseen', also display some items with solid circles/bold titles
  • About 4,000 pages.
  • A categorization filter only.
  • Sorry, I do not understand. How can one see filters?
  • About 4,000 pages.
  • A categorization filter only.
  • Sorry, I do not understand. How can one see filters?

@IKhitron can you please post detailed steps of what you did, what you expected to see, and what you saw?

@IKhitron can you please post detailed steps of what you did, what you expected to see, and what you saw?

Sure. But there will be a lot of irrelevant details there.

  1. Open special:watchlist, grouped by days, let's say that now the hour is T.
  2. Choose some line (page A).
  3. Somebody else edits this page since T.
  4. Click on the golden key gadget button. It takes the page name from this line, finds the bottom entry of this page in the shown watchlist, gets the last seen version from diff URL, and opens in another browser tab a diff between this version and cur. So, the user can see a diff of all unseen version at once, those that are in watchlist "7 days, 1000 edits", and those after the watchlist page reload.
  5. Open some page (B) in a third tab that is watched by me, but was not edited from the last time I visited it.
  6. Edit this page.
  7. Return to the watchlist tab.
  8. Click on "What's new?" gadget button. It makes api calls to print all the pages in watchlist edited from T, with links to diffs befween the last seen version and cur.
  9. Expected always and got most of the time: page A and page B are not in the list shown, beacause I just saw the current versions of both.
  10. Got sometimes: page A is shown with a link to the diff from the paragraph 2 above as unseen. Page B is shown with a link to my last edit as unseen.
JJMC89 added a comment.EditedMay 8 2020, 3:45 AM

I made this edit at 2020-05-08 03:08. As of 2020-05-08 03:25, it was shown as unseen on my watchlist, but it shouldn't be since I've clearly seen the page because I made the edit.

Watchlist URL after navigating to https://en.wikipedia.org/wiki/Special:Watchlist: https://en.wikipedia.org/w/index.php?watchlistactivity=unseen&hidepreviousrevisions=1&limit=1000&days=30&damaging__verylikelybad_color=c5&goodfaith__verylikelybad_color=c5&userExpLevel__unregistered_color=c4&userExpLevel__newcomer_color=c4&userExpLevel__learner_color=c3&title=Special:Watchlist&urlversion=2

Taken 2020-05-08 03:20:20 with scripts enabled: F31807414

Taken 2020-05-08 03:25 with scripts disabled (&safemode=1) after a hard refresh: F31807415

Thanks, @IKhitron and @JJMC89 - I will be looking into it today.

Dvorapa reopened this task as Open.May 10 2020, 5:25 PM

I'm not sure if we can reopen this, but the issue still occurs: I have visited https://cs.wikipedia.org/w/index.php?title=Diskuse_s_wikipedistou:Martin_Urbanec&curid=1041170&diff=18497394&oldid=18497390 4 hours ago, but still see it bold in my watchlist

@IKhitron - as far as I could see the gadgets you mention are not on Special:Preferences#mw-prefsection-gadgets list?

It seems that the issue happens when the Watchlist has a large list of pages. However, I'm still struggling to reproduce to reproduce the issue.

@IKhitron - as far as I could see the gadgets you mention are not on Special:Preferences#mw-prefsection-gadgets list?

It is there. WLM, the last one in Watchlist subsection.

@IKhitron - as far as I could see the gadgets you mention are not on Special:Preferences#mw-prefsection-gadgets list?

It is there. WLM, the last one in Watchlist subsection.

Thx!

Doing it on en.wiki right now (Tue Jun 2 2020 7:53 PM PDT). My watchlist is 2,492 entries.

Doing it on en.wiki right now (Tue Jun 2 2020 7:53 PM PDT). My watchlist is 2,492 entries.

@Joeyconnick Could you please provide specific steps to about what you did, what you saw, and what you expected to see? And also let us know if you have any watchlist gadgets enabled?

What I did:
looked at pages that my watchlist indicated had been edited since last time I saw them.

What I saw:
when I returned to the watchlist after either editing said pages, or just viewing their current version, I either saw:
a) the page in its same previous location in the watchlist, but unbolded and with a hollow bullet to its left (for pages I had simply viewed) OR
b) the page at the top of my watchlist, with me listed as the last person having edited it, UNbold and with a hollow bullet

What I expected:
a) the page should not have appeared anywhere in my watchlist as it had no unseen edits OR
b) the page should not have appeared anywhere in my watchlist as I had just edited it

Apparently I have these gadgets:

  • Geonotice: display notices on your watchlist about events in your region
  • Display watchlist notices (This loads the base style for the watchlist. Please do not disable this option.)
  • Display green collapsible arrows and green bullets for changed pages in your watchlist, page history and recent changes (unavailable with the improved Watchlist user interface)
  • Display pages on your watchlist that have changed since your last visit in bold (see customizing watchlists for more options)
  • Subtle update marker: Tone down the "Changed since last visit" indicator on history pages. (By default it renders as a green-filled bar, enabling this gadget changes it to green text.)

In the last twenty minutes this was happened on many wikis at the same time, thanks @DannyS712 for his global watchlist script. And stopped at the same time, so it is not local.

Thanks @Joeyconnick. Could you please also either screenshot your active filters and/or paste the URL you see when you access Special:Watchlist? e.g. for me it's Special:Watchlist?watchlistactivity=unseen&limit=250&days=7&enhanced=1&urlversion=2

MBH removed a subscriber: MBH.Jun 6 2020, 3:00 AM

Sure:

watchlistactivity=unseen&hidebots=1&hidepreviousrevisions=1&limit=500&days=7&title=Special:Watchlist&urlversion=2