Page MenuHomePhabricator

Move category namespace to a separate blazegraph instance
Closed, ResolvedPublic

Description

Due to Blazegraph limits, it is no longer a good idea to keep categories in the same namespace as main wikidata namespace. We should create a separate instance of Blazegraph, and have categories in that instance instead.

Migration plan after we have 2 instances running:

  • Create new namespace on category instance
  • Load weeklies and dailies
  • Update cron scripts in puppet to use new endpoint
  • Update configs with CATEGORY_ENDPOINT pointing to new instance. Note that this will also switch crons to update new instance, so better to do far away from daily cron run.
  • Re-deploy scap to update vars.sh to new endpoint
  • Upload delta dailies that accumulated since initial load
  • Switch nginx to use new instance (nginx config patch)
  • Update categories alias to new namespace
  • Drop categories namespace on main instance

Event Timeline

Change 483224 had a related patch set uploaded (by Smalyshev; owner: Smalyshev):
[wikidata/query/rdf@master] Make category endpoint be configurable

https://gerrit.wikimedia.org/r/483224

Change 483310 had a related patch set uploaded (by Smalyshev; owner: Smalyshev):
[operations/puppet@production] Puppetize blazegraph config for cases where deployed one is not enough

https://gerrit.wikimedia.org/r/483310

Change 483224 merged by jenkins-bot:
[wikidata/query/rdf@master] Make category endpoint be configurable

https://gerrit.wikimedia.org/r/483224

Change 484345 had a related patch set uploaded (by Smalyshev; owner: Smalyshev):
[operations/puppet@production] Switch external category endpoint to 9990

https://gerrit.wikimedia.org/r/484345

Change 484348 had a related patch set uploaded (by Smalyshev; owner: Smalyshev):
[operations/puppet@production] Make cron endpoint configurable

https://gerrit.wikimedia.org/r/484348

Change 484344 had a related patch set uploaded (by Smalyshev; owner: Smalyshev):
[operations/puppet@production] Move categories namespace to second instance

https://gerrit.wikimedia.org/r/484344

Category data loaded up to 20190114 on all servers. We'll probably need to update to 20190115 manually since crons are still on old namespace.

Change 484348 merged by Gehel:
[operations/puppet@production] Make cron endpoint configurable

https://gerrit.wikimedia.org/r/484348

Change 484345 merged by Gehel:
[operations/puppet@production] Switch category endpoint config to 9990

https://gerrit.wikimedia.org/r/484345

Smalyshev updated the task description. (Show Details)
Smalyshev updated the task description. (Show Details)
Smalyshev updated the task description. (Show Details)

Mentioned in SAL (#wikimedia-operations) [2019-01-16T19:48:12Z] <gehel> switching wdqs categories traffic to new second instance, puppet will be disabled during the operation on all wdqs nodes - T213212

Change 484344 merged by Gehel:
[operations/puppet@production] Move categories namespace to second instance

https://gerrit.wikimedia.org/r/484344

Change 484755 had a related patch set uploaded (by Gehel; owner: Gehel):
[operations/puppet@production] wdqs: fix typo in nginx config file

https://gerrit.wikimedia.org/r/484755

Change 484755 merged by Gehel:
[operations/puppet@production] wdqs: fix typo in nginx config file

https://gerrit.wikimedia.org/r/484755

Daily reload seems to have gone fine, next step is deleting existing namespaces.

Smalyshev claimed this task.
Smalyshev updated the task description. (Show Details)