Kafka eqiad.mediawiki.page-delete topic is empty
Closed, ResolvedPublic

Description

No deletions are being reported to eqiad.mediawiki.page-delete topic. This seems to be due to T210013 but the fix seems to still not be deployed and also may be cause to at least some issues in T210044 with regard to page deletion.

Restricted Application added projects: Wikidata, Analytics. · View Herald TranscriptMon, Nov 26, 9:11 PM
Restricted Application added a subscriber: Aklapper. · View Herald Transcript
Smalyshev triaged this task as Unbreak Now! priority.Mon, Nov 26, 9:11 PM
Restricted Application added subscribers: Liuxinyu970226, TerraCodes. · View Herald TranscriptMon, Nov 26, 9:11 PM

Yes, the fix has not been deployed yet.

Asked to put it on Euro Mid-day SWAT

mobrovac lowered the priority of this task from Unbreak Now! to High.Mon, Nov 26, 9:34 PM
mobrovac added a subscriber: mobrovac.

Will be done in the EU SWAT window on 2018-11-27. Lowering the priority as it is a known issue and does not cause severe breakage.

It does cause pretty severe breakage - all delete updates are missing from WDQS and people are complaining (in fact, have been complaining for a while now, I just now figured out that it's because we don't have any delete events at all!)

To make the problem worse, looks like breakage started more than 30 days ago - which means we don't have a record of the old deleted entries in recentchanges, so we don't know which items to update :(

Not to be worried. We have all the failed events stored since 2018-04-18. If needed, I will fetch all the missing page deletes tomorrow.

Oo, I just did the same, or, at least I copied the relevant files. They are on stat1004:/home/otto/eventbus-validation-logs0. Stas said he might have another way so I stopped there.

I think I've extracted all I need from the DB tables for now, but I'll double-check and if anything is still missing I check the extracted data, thanks!

Change 475893 had a related patch set uploaded (by Mobrovac; owner: Ppchelko):
[mediawiki/core@wmf/1.33.0-wmf.4] Convert $archivedRevisionCount to integer.

https://gerrit.wikimedia.org/r/475893

Change 475893 merged by Mobrovac:
[mediawiki/core@wmf/1.33.0-wmf.4] Convert $archivedRevisionCount to integer.

https://gerrit.wikimedia.org/r/475893

Stashbot added a subscriber: Stashbot.

Mentioned in SAL (#wikimedia-operations) [2018-11-27T12:39:46Z] <mobrovac@deploy1001> Synchronized php-1.33.0-wmf.4/includes/page/WikiPage.php: Convert $archivedRevisionCount to integer - T210013 T210451 (duration: 00m 47s)

The fix has been deployed, delete events should start flowing again, so resolving. Let's reopen the ticket if that does not occur.

mobrovac closed this task as Resolved.Tue, Nov 27, 12:41 PM
mobrovac claimed this task.

Yep, seeing the events in grafana now, so I think it's all good now. Thanks!

Do you need the events for the last month to be replayed?

@Pchelolo No I already updated the affected items manually.