Page MenuHomePhabricator

Decrease user visibility of lagging Change-Prop
Closed, ResolvedPublic

Description

Sometimes ChangeProp lags and it's fine - huge template updates could create significant load and concurrency limiting doesn't allow us going full speed, and it's correct, we don't want to DDOS our own services.

Although, sometimes lagging could be so long that it becomes user-visible, here's an example: T137669

To avoid this, we might wanna introduce the 'dirty' flag in RESTBase. The idea is to have a rule that performs a special request to RB marking a page dirty, but not rerendering it. This job can go crazy fast, so it will never lag. If a user request for a dirty page comes, it's treated as a no-cache request - the page is rerendered. If a no-cache request comes to a dirty page - it's rerendered and dirty flag is cleared. With this user will never see a stale page.

This job could potentially be switched on/off dynamically when lagging for certain rules reach some limit, but that's kinda a fancy feature, not a core one.

Event Timeline

Pchelolo created this task.Sep 15 2016, 5:51 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptSep 15 2016, 5:51 PM

Other, simpler ideas how to achieve that:

  • Have a separate resource_change topic for events introduced by transclusion rerender
  • Change how transclusions work - only continue after a current batch was processed.
  • Have a separate resource_change topic for events introduced by transclusion rerender

I would prefer this way for now as it relatively easy to implement and it allows us to process page edits in real time.

Yup @mobrovac I was thinking same. Will implement during this week.

Change 311594 had a related patch set uploaded (by Ppchelko):
RESTBase: Specify the topic for transclusions.

https://gerrit.wikimedia.org/r/311594

Change 311602 had a related patch set uploaded (by Ppchelko):
Added separate rules to process transcludes-derived events

https://gerrit.wikimedia.org/r/311602

Change 311602 merged by Mobrovac:
Added separate rules to process transcludes-derived events

https://gerrit.wikimedia.org/r/311602

Change 311594 merged by Giuseppe Lavagetto:
RESTBase: Specify the topic for transclusions.

https://gerrit.wikimedia.org/r/311594

mobrovac closed this task as Resolved.Sep 21 2016, 5:14 PM
mobrovac removed a project: Patch-For-Review.
mobrovac removed a subscriber: gerritbot.

We have separated the regular and transcluded topics, so real-time edits shouldn't lag behind any more because of transcludes.