Page MenuHomePhabricator

Rebuild Recent changes for 2016-04-19 14:48 - 15:10 UTC due to Dallas data centre migration
Closed, ResolvedPublic

Description

Hello. We were read only from 14:02 to 14:47 UTC. After then, between 14:47-15:10 UTC the recent changes list did not work. We have no ability to check vandals in these 23 minutes. Can you restore the data, please?
Thank you.

Details

Event Timeline

IKhitron created this task.Apr 19 2016, 3:53 PM
Restricted Application added subscribers: TerraCodes, Aklapper. · View Herald TranscriptApr 19 2016, 3:53 PM
IKhitron triaged this task as High priority.Apr 19 2016, 3:53 PM

Well, Quarry ignores queries on the lost time in recentchanges table, but has all the data in revision table.

matmarex added a subscriber: matmarex.EditedApr 19 2016, 4:08 PM

There is a script called rebuildrecentchanges.php, but it would need some adjustments to work on a time range (right now it clears all recentchanges data and rebuilds it from scratch, which takes a long time and also would lose any patrolling information for older changes).

So, it will be OK!

Deskana renamed this task from No patrolling 23 minutes after Dallas to Special:RecentChanges contains no entries from 14:48 - 15:10 UTC on 2015-04-19 due to Dallas data centre migration.Apr 19 2016, 4:21 PM
Deskana added subscribers: mark, Deskana.

I've updated this task with some of the information from @mark's email to the engineering list.

Well, I made the list of missing edits, and they are not unmarked!

Ankry added a subscriber: Ankry.Apr 19 2016, 5:15 PM
mobrovac renamed this task from Special:RecentChanges contains no entries from 14:48 - 15:10 UTC on 2015-04-19 due to Dallas data centre migration to Special:RecentChanges contains no entries from 14:48 - 15:10 UTC on 2016-04-19 due to Dallas data centre migration.Apr 19 2016, 5:20 PM
matmarex claimed this task.Apr 19 2016, 5:49 PM

I'm going to work on the script, ops said they'll review it and run it across the wikis.

Change 284248 had a related patch set uploaded (by Bartosz Dziewoński):
[WIP] rebuildrecentchanges: Allow rebuilding specified time range only

https://gerrit.wikimedia.org/r/284248

Restricted Application added subscribers: JEumerus, Dereckson, Matanya. · View Herald TranscriptApr 19 2016, 6:27 PM
Dereckson renamed this task from Special:RecentChanges contains no entries from 14:48 - 15:10 UTC on 2016-04-19 due to Dallas data centre migration to Rebuild Recent changes for 2016-04-19 14:48 - 15:10 UTC due to Dallas data centre migration.Apr 19 2016, 6:27 PM
Dereckson moved this task from Backlog to Blocked on development on the Wikimedia-Site-requests board.

Change 284248 merged by jenkins-bot:
rebuildrecentchanges: Allow rebuilding specified time range only

https://gerrit.wikimedia.org/r/284248

matmarex reassigned this task from matmarex to aaron.Apr 19 2016, 6:45 PM
matmarex removed a project: Patch-For-Review.

Should probably find out what the exact time range is, to avoid missing anything or duplicate RC entries for some actions.

You know, @matmarex, there can be a chance that the range finished in different time on different wikis.

I don't think so, but different mw servers would've got the fix at slightly different times.

ori added a subscriber: ori.Apr 19 2016, 7:25 PM

Should probably find out what the exact time range is, to avoid missing anything or duplicate RC entries for some actions.

The full range is 2016-04-19T14:47:41Z - 2016-04-19T15:10:18Z

Change 284274 had a related patch set uploaded (by Ori.livneh):
rebuildrecentchanges: Allow rebuilding specified time range only

https://gerrit.wikimedia.org/r/284274

Change 284274 merged by Ori.livneh:
rebuildrecentchanges: Allow rebuilding specified time range only

https://gerrit.wikimedia.org/r/284274

Change 284347 had a related patch set uploaded (by Aaron Schulz):
RebuildRecentchanges improvements

https://gerrit.wikimedia.org/r/284347

ori closed this task as Resolved.Apr 19 2016, 9:47 PM

The missing entries have been backfilled, thanks to some quick and clever scripting from @matmarex and @aaron.

To actually view these 20 minutes of changes that will need reviewing, try viewing a page like this:

https://commons.wikimedia.org/w/index.php?title=Special:RecentChanges&hideliu=0&hideanons=0&hidebots=1&hideWikibase=1&hideReviewed=1&hidepatrolled=1&limit=5000&from=20160419144400

(substitute "commons.wikimedia.org" for your wiki; the additional query parameters are to hide boring changes) …scroll to the bottom, and look at the changes in the affected period (14:48 - 15:10 UTC).

Unfortunately, Special:RecentChanges can show only 5000 changes, and on very active wikis (e.g. English Wikipedia), this might already be outside the limit :( I'm not sure what to do there, they can probably be queried via the API somehow…

Pay attention that all restored changes are unmarked, even those of sysops.

(Also, for future reference: what exactly was the config problem and what was the fix? Anyone has a link handy?)

Change 284347 merged by jenkins-bot:
RebuildRecentchanges improvements

https://gerrit.wikimedia.org/r/284347