As a member of search platform I want to deploy updates to mjolnir in a consistent and documented way so the whole team can deploy updates with confidence.
The current method for deploying mjolnir updates (approx):
- scap deploy the search/mjolnir/deploy repository to stat1007 (matching os version to analytics nodes)
- manually zip up the venv created by scap deployment
- copy the zip somewhere into hdfs (currently ebernhardson home directory)
- update airflow to point at new location
As of a few days ago this process is broken, the stat machines have been updated to a new debian version and the venv's it builds are no longer compatible. While we could probably find some way to prepare a fix within the existing process, it's full of manual interventions that we should optimize away to ensure more people can be involved in mjolnir. A rough idea for a replacement deployment method:
- Build and then zip a venv inside a docker image in CI and upload to archiva. Nothing here would be particularly new, but will require gluing together a few existing pieces.
- Update artifacts in airflow repo to contain the new artifact