Maniphest T192198

Wikidata doesn't update recentchanges
Closed, ResolvedPublic
Actions

Description

On hewik/cawiki (and probably many others) Wikidata doesn't update recentchanges, as it appears in the DB query below:

use hewiki_p;
select rc_timestamp, now() from recentchanges where rc_source='wb' order by rc_timestamp desc limit 5;
+----------------+---------------------+
| rc_timestamp   | now()               |
+----------------+---------------------+
| 20180412143955 | 2018-04-14 05:01:39 |
| 20180412143842 | 2018-04-14 05:01:39 |
| 20180412143811 | 2018-04-14 05:01:39 |
| 20180412143802 | 2018-04-14 05:01:39 |
| 20180412143756 | 2018-04-14 05:01:39 |
+----------------+---------------------+

Likely related to: T192085

Details

Project	Branch	Lines +/-	Subject
mediawiki/services/change-propagation/jobqueue-deploy	master	+1 -1	Revert switching ChangeNotification job.
mediawiki/extensions/EventBus	wmf/1.31.0-wmf.29	+10 -3	Correctly calculate a domain based on the provided wiki id.
mediawiki/extensions/EventBus	master	+10 -3	Correctly calculate a domain based on the provided wiki id.
mediawiki/extensions/EventBus	wmf/1.31.0-wmf.29	+11 -6	Use the wiki that the queue belongs to for setting database and domain.
mediawiki/extensions/EventBus	master	+11 -6	Use the wiki that the queue belongs to for setting database and domain.

Customize query in gerrit

Related Objects
Search...

Status	Assigned	Task
Resolved	• Pchelolo	T157088 [EPIC] Develop a JobQueue backend based on EventBus
Resolved	• Pchelolo	T190327 FY17/18 Q4 Program 8 Services Goal: Complete the JobQueue transition to EventBus
Resolved	• Pchelolo	T192198 Wikidata doesn't update recentchanges

Event Timeline

eranroz created this task.Apr 14 2018, 5:10 AM

Restricted Application added a subscriber: Aklapper. · View Herald TranscriptApr 14 2018, 5:10 AM

eranroz updated the task description. (Show Details)Apr 14 2018, 5:22 AM

@Ladsgroup could you have a look maybe?

Per https://grafana.wikimedia.org/dashboard/db/jobqueue-eventbus?orgId=1&var-site=eqiad&var-type=wikibase-InjectRCRecords&from=now-7d&to=now injecting rc records is completely broken for two days now. This guarantees an unscheduled deploy, incident report, and several actionables ASAP.

Restricted Application added a project: User-Ladsgroup. · View Herald TranscriptApr 14 2018, 11:52 AM

Restricted Application added subscribers: Liuxinyu970226, TerraCodes. · View Herald Transcript

Per grafana it started in 12.04.2018 around 14:30-15:00 UTC which aligns with this in SAL:

14:46 mobrovac@tin: Synchronized wmf-config/InitialiseSettings.php: No-op: Clean up an unused global var for the EventBus-based JobQueue (duration: 01m 17s)
14:44 mobrovac@tin: Synchronized wmf-config/jobqueue.php: Switch the second bulk of low-traffic jobs for all wikis - T190327 (duration: 01m 16s)
14:43 ppchelko@tin: Finished deploy [cpjobqueue/deploy@85fbd47]: Enable second bulk of low traffic jobs for all wikis T190327 (duration: 00m 35s)
14:42 ppchelko@tin: Started deploy [cpjobqueue/deploy@85fbd47]: Enable second bulk of low traffic jobs for all wikis T190327

At first I thought it's a bug in the code but this part of code hasn't been touched for over a month now.
Pinging @mobrovac and @Pchelolo

It's definitely not related to the codebase, it's an issue in the job runner infra...

• Marostegui added a subscriber: • Marostegui.Apr 15 2018, 1:34 PM

abian added a subscriber: abian.Apr 15 2018, 3:18 PM

Change 426907 had a related patch set uploaded (by Ppchelko; owner: Ppchelko):
[mediawiki/services/change-propagation/jobqueue-deploy@master] Revert switching ChangeNotification job.

https://gerrit.wikimedia.org/r/426907

gerritbot added a project: Patch-For-Review.Apr 16 2018, 1:12 PM

Change 426969 had a related patch set uploaded (by Ppchelko; owner: Ppchelko):
[mediawiki/extensions/EventBus@master] Use the wiki that the queue belongs to for setting database and domain.

https://gerrit.wikimedia.org/r/426969

• mobrovac edited projects, added Services (doing), Event-Platform Value Stream, MediaWiki-Core-JobQueue; removed Services.Apr 16 2018, 7:13 PM

Restricted Application added a project: Analytics. · View Herald TranscriptApr 16 2018, 7:13 PM

Change 426969 merged by Mobrovac:
[mediawiki/extensions/EventBus@master] Use the wiki that the queue belongs to for setting database and domain.

https://gerrit.wikimedia.org/r/426969

Change 426975 had a related patch set uploaded (by Ppchelko; owner: Ppchelko):
[mediawiki/extensions/EventBus@wmf/1.31.0-wmf.29] Use the wiki that the queue belongs to for setting database and domain.

https://gerrit.wikimedia.org/r/426975

Change 426975 merged by Mobrovac:
[mediawiki/extensions/EventBus@wmf/1.31.0-wmf.29] Use the wiki that the queue belongs to for setting database and domain.

https://gerrit.wikimedia.org/r/426975

Mentioned in SAL (#wikimedia-operations) [2018-04-16T19:46:03Z] <mobrovac@tin> Synchronized php-1.31.0-wmf.29/extensions/EventBus/includes/EventBus.php: Use the wiki set in the JobQueue when creating the event, file 1/2 - T192198 (duration: 01m 00s)

Mentioned in SAL (#wikimedia-operations) [2018-04-16T19:47:28Z] <mobrovac@tin> Synchronized php-1.31.0-wmf.29/extensions/EventBus/includes/JobQueueEventBus.php: Use the wiki set in the JobQueue when creating the event, file 2/2 - T192198 (duration: 00m 59s)

ReleaseTaggerBot added a project: MW-1.31-release-notes (WMF-deploy-2018-04-17 (1.31.0-wmf.30)).Apr 16 2018, 8:00 PM

Change 427014 had a related patch set uploaded (by Ppchelko; owner: Ppchelko):
[mediawiki/extensions/EventBus@master] Correctly calculate a domain based on the provided wiki id.

https://gerrit.wikimedia.org/r/427014

Change 427014 merged by Mobrovac:
[mediawiki/extensions/EventBus@master] Correctly calculate a domain based on the provided wiki id.

https://gerrit.wikimedia.org/r/427014

Change 427018 had a related patch set uploaded (by Ppchelko; owner: Ppchelko):
[mediawiki/extensions/EventBus@wmf/1.31.0-wmf.29] Correctly calculate a domain based on the provided wiki id.

https://gerrit.wikimedia.org/r/427018

Change 427018 merged by Mobrovac:
[mediawiki/extensions/EventBus@wmf/1.31.0-wmf.29] Correctly calculate a domain based on the provided wiki id.

https://gerrit.wikimedia.org/r/427018

ReleaseTaggerBot edited projects, added MW-1.31-release-notes (WMF-deploy-2018-04-10 (1.31.0-wmf.29)); removed MW-1.31-release-notes (WMF-deploy-2018-04-17 (1.31.0-wmf.30)).Apr 16 2018, 9:00 PM

Mentioned in SAL (#wikimedia-operations) [2018-04-16T21:02:29Z] <mobrovac@tin> Synchronized php-1.31.0-wmf.29/extensions/EventBus/includes/EventBus.php: Use the correct way of calculating the domain from the wiki, file 1/2 - T192198 (duration: 00m 59s)

Mentioned in SAL (#wikimedia-operations) [2018-04-16T21:03:46Z] <mobrovac@tin> Synchronized php-1.31.0-wmf.29/extensions/EventBus/includes/JobQueueEventBus.php: Use the correct way of calculating the domain from the wiki, file 2/2 - T192198 (duration: 00m 58s)

It took us a while to find the root cause of this. Essentially, the problem was that the ChangeNotification jobs were being created/dispatched via a cron script. In such a case, the EventBus-based JobQueue was assigning the originating wiki (which was Wikidata) instead of the intended recipient wiki. At the same time, whether this job injects RC records or not is configurable per-wiki, and that setting was set to false for WD.

We have fixed the bug in the EventBus-based JobQueue push service and now the correct wiki name is set in the event data. Consequently, injectRCRecord jobs are now being created and executed as intended.

• mobrovac added a parent task: T190327: FY17/18 Q4 Program 8 Services Goal: Complete the JobQueue transition to EventBus.Apr 16 2018, 9:39 PM

Change 426907 abandoned by Mobrovac:
Revert switching ChangeNotification job.

Reason:
Not needed

https://gerrit.wikimedia.org/r/426907

Liuxinyu970226 removed a subscriber: Liuxinyu970226.Apr 20 2018, 1:09 PM

Michael mentioned this in T291063: Check that we have o11y for things that could go wrong with dispatching.Sep 15 2021, 9:18 AM