Page MenuHomePhabricator

Implement update freeze and/or delays for TTMServerMessageUpdateJob
Closed, ResolvedPublic

Description

In case we need to temporarily stop updates to the TTMServer index, it would be useful to have a way to freeze the updates for a while and have them be executed after the freeze is over.

Event Timeline

Restricted Application added projects: Discovery, Discovery-Search. · View Herald TranscriptApr 11 2016, 6:58 AM

@dcausse I think you said CirrusSearch has this feature. Do you have any suggestions how this could easily be implemented in Translate? A configuration flag? How do you delay the job is freeze is active?

Nikerabbit removed Nikerabbit as the assignee of this task.Apr 20 2016, 1:29 PM

This ticket is old. Is this still causing a problem?

It's not really causing any problems, it'd be nice to have I think.
More importantly the cross-dc feature can be convenient during elasticsearch upgrades.
Currently the solution is to copy manually the ttm index from one dc to another and run a full reindex to catch-up translations that fell into the crack.
My feeling is that is that it's not a super high priority but it requires someone aware of the problem and run some manual scripts when we do a full restart to avoid any downtime. It'd be nice to have to avoid any mistakes i.e. last time I did this I made a stupid mistake that caused a downtime, this is always a risk when running manual scripts...

To answer your question Nikerabbit (sorry I missed your ping): currently we handle freezing writes by storing a flag in a special cirrus index, when this flag is set the messages are simply resent to the jobqueue with a backoff delay. The way we detect that the cluster is frozen is done with this simple query.
I think it'd make sense to add a new config option in ttmserver to point ElasticTTMServer to this index so we can reuse it, ttm would not have all the tools to handle this index (to freeze and unfreeze) and would be dependent on cirrus for this kind of maintenance operations...

Overall it's not very difficult, ttm has already a sane foundation to support this (iirc all updates are sent to the jobqueue). Maybe we can find time to work on this early next year as part of the elastic5 upgrade?

@dcausse All that sounds good to me.

dcausse claimed this task.Jan 31 2017, 10:05 AM
dcausse moved this task from needs triage to Current work on the Discovery-Search board.

Change 337558 had a related patch set uploaded (by DCausse):
Add support for freezing writes

https://gerrit.wikimedia.org/r/337558

Change 337558 merged by jenkins-bot:
Add support for freezing writes

https://gerrit.wikimedia.org/r/337558

Deskana closed this task as Resolved.Mar 22 2017, 11:08 AM