Update Elasticsearch for missing updates from outage on 20150825
Closed, ResolvedPublic
Actions

Assigned To

Authored By

	• chasemp
	Aug 25 2015, 12:48 PM

Description

https://wikitech.wikimedia.org/wiki/Incident_documentation/20150825-Redis

We have the ability to update elasticsearch for missing data for outages. We lost some jobs for sure on recovery and we need to run through the timeframes with our recovery process.

Event Timeline

• chasemp created this task.Aug 25 2015, 12:48 PM

• chasemp raised the priority of this task from to Medium.

• chasemp updated the task description. (Show Details)

• chasemp added projects: Elasticsearch, acl*sre-team.

• chasemp added a subscriber: • chasemp.

Restricted Application added a project: Discovery-ARCHIVED. · View Herald TranscriptAug 25 2015, 12:48 PM

Restricted Application added subscribers: Matanya, Aklapper. · View Herald Transcript

• chasemp added a project: Incident-20150825-Redis.Aug 25 2015, 12:56 PM

• chasemp set Security to None.

• chasemp added a subscriber: dcausse.

We could run:

mwscript extensions/CirrusSearch/maintenance/forceSearchIndex.php --wiki $wiki --from 2015-08-24T17:58:00Z --to 2015-08-25T09:00:00Z

for each wiki.

dcausse claimed this task.Aug 25 2015, 1:05 PM

dcausse added a project: Discovery-Search (Current work).

There are deletes to process as well, all documented at https://wikitech.wikimedia.org/wiki/Search#Recovering_from_an_Elasticsearch_outage.2Finterruption_in_updates

IIRC last time i tried to run that it ended up timing out on the SQL queries. If that happens again we can get around that by setting the timespan by revision id instead of timestamp.

EBernhardson claimed this task.Aug 26 2015, 6:19 PM

EBernhardson moved this task from Incoming to not in use - please delete on the Discovery-Search (Current work) board.

• ksmith added a project: Maintenance-Worktype.Aug 26 2015, 11:03 PM

EBernhardson moved this task from not in use - please delete to Needs Reporting on the Discovery-Search (Current work) board.Aug 27 2015, 4:06 PM

EBernhardson closed this task as Resolved.Sep 4 2015, 6:01 PM

Deskana moved this task from Needs Reporting to Resolved on the Discovery-Search (Current work) board.Sep 9 2015, 2:24 AM

Update Elasticsearch for missing updates from outage on 20150825Closed, ResolvedPublicActions

Description

Event Timeline

Update Elasticsearch for missing updates from outage on 20150825
Closed, ResolvedPublic
Actions