Page MenuHomePhabricator

stashbot elasticsearch migration
Closed, ResolvedPublic

Description

As part of the Toolforge elasticsearch migration process, the stashbot tool will need to be reconfigured for the new version 7 cluster before 2020-04-06.

More information on what's changed can be found at https://wikitech.wikimedia.org/wiki/News/Toolforge_Elasticsearch_migration

Event Timeline

bd808 renamed this task from staashbot elastcisearch migration to staashbot elasticsearch migration.Mar 15 2020, 9:26 PM
bd808 triaged this task as High priority.
bd808 created this task.
bd808 added a project: Stashbot.
bd808 renamed this task from staashbot elasticsearch migration to stashbot elasticsearch migration.Mar 15 2020, 9:40 PM

Reminder: the version 5 elasticsearch cluster will be shutdown on April 20th, 2020.

Change 588116 had a related patch set uploaded (by BryanDavis; owner: Bryan Davis):
[labs/tools/stashbot@master] Update for elasticsearch7

https://gerrit.wikimedia.org/r/588116

Change 588116 merged by jenkins-bot:
[labs/tools/stashbot@master] Update for elasticsearch7

https://gerrit.wikimedia.org/r/588116

Change 588118 had a related patch set uploaded (by BryanDavis; owner: Bryan Davis):
[labs/tools/stashbot@master] Fix elasticsearch indexing call for SAL entries

https://gerrit.wikimedia.org/r/588118

Change 588119 had a related patch set uploaded (by BryanDavis; owner: Bryan Davis):
[labs/tools/stashbot@master] Update !bash handling for es7

https://gerrit.wikimedia.org/r/588119

Change 588118 merged by jenkins-bot:
[labs/tools/stashbot@master] Fix elasticsearch indexing call for SAL entries

https://gerrit.wikimedia.org/r/588118

Change 588119 merged by jenkins-bot:
[labs/tools/stashbot@master] Update !bash handling for es7

https://gerrit.wikimedia.org/r/588119

Change 588120 had a related patch set uploaded (by BryanDavis; owner: Bryan Davis):
[labs/tools/stashbot@master] Update !log handling for es7

https://gerrit.wikimedia.org/r/588120

Change 588120 merged by jenkins-bot:
[labs/tools/stashbot@master] Update !log handling for es7

https://gerrit.wikimedia.org/r/588120

Mentioned in SAL (#wikimedia-cloud) [2020-04-11T23:11:22Z] <wm-bot> <bd808> Updated to f2ca4e4 925b463 Update !log handling for es7 (T247715)

  • Deleted sal index on the es7 cluster
  • Re-synced sal from the es5 cluster
  • Updated git clone on dev.toolforge.org
  • Stopped bot
  • Updated python 3.7 venv (venv-k8s-py37/bin/pip install -U -r requirements.txt)
  • Updated config to point to new cluster
  • Started bot
  • Repeat code, update, restart cycle 3x more to fix things I missed in the code

Mentioned in SAL (#wikimedia-cloud) [2020-04-11T23:17:29Z] <wm-bot> <bd808> Updated config to point to es7 cluster (T247715)

I have been fighting with getting https://tools.wmflabs.org/sal/ to work and I think I just figured it out. It appears that the newer version of ruflin/Elastica that I'm using there uses Content-Type: application/json and POST to submit some queries. This means that the SAL tool needs to authenticate to the backend. And that folks took me way, way too long to figure out. :)

This change in the PHP library didn't cause me problems with the bash tool because that tool has credentials so that it can edit and delete things from the index it uses.

Mentioned in SAL (#wikimedia-cloud) [2020-04-12T02:52:01Z] <wm-bot> <bd808> Updated to 49015bb: Manually setup Elasticsearch creds (T247715)