Page MenuHomePhabricator

Replace mjolnir venv deployment scheme in analytics
Open, MediumPublic8 Estimated Story Points

Description

As a member of search platform I want to deploy updates to mjolnir in a consistent and documented way so the whole team can deploy updates with confidence.

The current method for deploying mjolnir updates (approx):

  1. scap deploy the search/mjolnir/deploy repository to stat1007 (matching os version to analytics nodes)
  2. manually zip up the venv created by scap deployment
  3. copy the zip somewhere into hdfs (currently ebernhardson home directory)
  4. update airflow to point at new location

As of a few days ago this process is broken, the stat machines have been updated to a new debian version and the venv's it builds are no longer compatible. While we could probably find some way to prepare a fix within the existing process, it's full of manual interventions that we should optimize away to ensure more people can be involved in mjolnir. A rough idea for a replacement deployment method:

  1. Build and then zip a venv inside a docker image in CI and upload to archiva. Nothing here would be particularly new, but will require gluing together a few existing pieces.
  2. Update artifacts in airflow repo to contain the new artifact

Event Timeline

Gehel lowered the priority of this task from High to Medium.Oct 28 2020, 1:30 PM

While not captured in a ticket, I recently had to solve this to deploy an environment for airflow, in https://gerrit.wikimedia.org/r/c/wikimedia/discovery/analytics/+/666241

It seems plausible this can be resolved by building a mjolnir wheel with setup.py and placing it in the artifacts directory, along with a new environment configuration.

Change 697863 had a related patch set uploaded (by Ebernhardson; author: Ebernhardson):

[search/MjoLniR@master] Define release process for wheels, release as 1.0

https://gerrit.wikimedia.org/r/697863

Change 697865 had a related patch set uploaded (by Ebernhardson; author: Ebernhardson):

[wikimedia/discovery/analytics@master] Deploy mjolnir venv through environments

https://gerrit.wikimedia.org/r/697865

Change 697863 merged by jenkins-bot:

[search/MjoLniR@master] Define release process for wheels, release as 1.0

https://gerrit.wikimedia.org/r/697863

Change 697865 merged by jenkins-bot:

[wikimedia/discovery/analytics@master] Deploy mjolnir venv through environments

https://gerrit.wikimedia.org/r/697865

Change 698025 had a related patch set uploaded (by Ebernhardson; author: Ebernhardson):

[operations/puppet@production] Undeploy mjolnir profile from analytics

https://gerrit.wikimedia.org/r/698025

Change 698048 had a related patch set uploaded (by Ebernhardson; author: Ebernhardson):

[search/MjoLniR/deploy@master] Undeploy from analytics

https://gerrit.wikimedia.org/r/698048

Change 698048 merged by Ebernhardson:

[search/MjoLniR/deploy@master] Undeploy from analytics

https://gerrit.wikimedia.org/r/698048

Change 698025 merged by Ryan Kemper:

[operations/puppet@production] Undeploy mjolnir profile from analytics

https://gerrit.wikimedia.org/r/698025

Mentioned in SAL (#wikimedia-operations) [2021-06-10T19:31:27Z] <ryankemper> T265547 Cleanup following merge of https://gerrit.wikimedia.org/r/c/operations/puppet/+/698025: sudo -E cumin -b 5 'P:analytics::cluster::elasticsearch' 'sudo rm -rfv /etc/mjolnir /srv/deployment/search/mjolnir'