Page MenuHomePhabricator

Set up update jobs for RESTBase: code done and deployed, to be configured
Closed, ResolvedPublic

Description

Until T84923 or something similar is resolved we need to have an interim way to keep RESTBase content up to date on edit, template edit, image re-upload, page move & protection.

The path of least resistance is to hack up / fork the existing Parsoid extension to schedule HTTP requests to RESTBase. This involves

  • doing some search & replace for the names,
  • some request parameter tweaking to reflect the new URL layout and update headers, and
  • deploying this to production just like the parsoid extension.

Details

Related Gerrit Patches:
mediawiki/extensions/RestBaseUpdateJobs : masterUse $wgCanonicalServer instead of $wgServer
operations/mediawiki-config : masterEnable edit updates for RESTBase on testwiki
mediawiki/extensions/RestBaseUpdateJobs : masterAdapt the extension to RESTBase's needs
mediawiki/extensions/RestBaseUpdateJobs : masterInitial repo commit
mediawiki/extensions/RestBaseUpdateJobs : masterCode clean-up

Event Timeline

GWicke claimed this task.
GWicke raised the priority of this task from to High.
GWicke updated the task description. (Show Details)
GWicke added projects: RESTBase, Services.
GWicke added a subscriber: GWicke.
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptJan 24 2015, 10:22 PM
GWicke updated the task description. (Show Details)Jan 25 2015, 9:09 PM
GWicke set Security to None.
GWicke updated the task description. (Show Details)
mobrovac claimed this task.Jan 30 2015, 7:27 PM
gerritbot added a subscriber: gerritbot.

Change 188618 had a related patch set uploaded (by Mobrovac):
T87520: Initial repo commit

https://gerrit.wikimedia.org/r/188618

Patch-For-Review

Change 188618 had a related patch set uploaded (by Mobrovac):
Initial repo commit

https://gerrit.wikimedia.org/r/188618

Patch-For-Review

Change 188677 had a related patch set uploaded (by Mobrovac):
Code clean-up

https://gerrit.wikimedia.org/r/188677

Patch-For-Review

Extra information we need beyond what's provided by the current extension in the X-Parsoid header:

  • on revision visibility change ('revision deletion / suppression'):
    • fact that it's a revision deletion (mode: 'rev_visibility'?)
    • revision id

Things we need to adjust:

  • getParsoidURL: use /{domain}/v1/page/{title}/html/{revid} (GET with no-cache and X-Parsoid header)
    • introduce / use /{domain}/v1/revision/{revision} for rev_visibility change? (GET with no-cache and X-Parsoid header)

Change 188618 merged by GWicke:
Initial repo commit

https://gerrit.wikimedia.org/r/188618

mobrovac moved this task from Backlog to In progress on the RESTBase board.Feb 5 2015, 6:01 AM

Change 188952 had a related patch set uploaded (by Mobrovac):
Adapt the extension to RESTBase's needs

https://gerrit.wikimedia.org/r/188952

Patch-For-Review

@GWicke please review the above patch so we can move to deploying the extension.

@GWicke grazie, appreciated. Now you can find my answers to your comments and a new patch set at the same address :P

GWicke renamed this task from Set up update jobs for RESTBase to Set up update jobs for RESTBase: code done, to be deployed & configured.Feb 13 2015, 7:06 PM

Change 188952 merged by GWicke:
Adapt the extension to RESTBase's needs

https://gerrit.wikimedia.org/r/188952

Change 193829 had a related patch set uploaded (by Mobrovac):
Minor fixes

https://gerrit.wikimedia.org/r/193829

mobrovac renamed this task from Set up update jobs for RESTBase: code done, to be deployed & configured to Set up update jobs for RESTBase: code done and deployed, to be configured.Mar 5 2015, 8:53 AM

Change 195348 had a related patch set uploaded (by Mobrovac):
Enable edit updates for RESTBase on testwiki

https://gerrit.wikimedia.org/r/195348

Change 195348 merged by GWicke:
Enable edit updates for RESTBase on testwiki

https://gerrit.wikimedia.org/r/195348

Change 193829 merged by GWicke:
Use $wgCanonicalServer instead of $wgServer

https://gerrit.wikimedia.org/r/193829

This is now enabled for test.wikipedia.org, full roll-out pending.

Latest version of the extension is now deployed. Lets check if updates are working tomorrow & then gradually enable this for other wikis.

We also need to get https://gerrit.wikimedia.org/r/#/c/195364/ ready, so that EditUpdate jobs are processed quickly & dependency updates don't swamp the default queue.

mobrovac added a comment.EditedMar 11 2015, 8:44 AM

I have confirmed that the page edits are propagated to RESTBase.

First:

cassandra@cqlsh:local_group_phase0_T_title__revisions> select "_domain",title,rev from data where "_domain" = 'test.wikipedia.org' and title = 'User:Mobrovac';

(0 rows)

Then went to edit my user page on test.wp, and a second later, voilà:

cassandra@cqlsh:local_group_phase0_T_title__revisions> select "_domain",title,rev from data where "_domain" = 'test.wikipedia.org' and title = 'User:Mobrovac';

 _domain            | title         | rev
--------------------+---------------+--------
 test.wikipedia.org | User:Mobrovac | 223756

(1 rows)

@mobrovac, awesome! I also just checked in the /a/mw-log/runJobs.log on fluorine, and saw quite a few successful RestbaseUpdateJob runs.

So, lets get https://gerrit.wikimedia.org/r/#/c/195364/ merged & then roll out the jobs to all wikis (maybe phase0 including mw.org today, then all tomorrow).

GWicke closed this task as Resolved.Mar 13 2015, 1:14 AM

As of now RESTBase is tracking updates to all configured wikis. Closing.