Page MenuHomePhabricator

Deploy revscoring 2.11.4 to ORES
Closed, ResolvedPublic


The revscoring 2.11.4 wheel should be deployed to ORES (it is already running in all Lift Wing pods).

Let's write down a procedure and then schedule a deploy :)

Event Timeline

High level plan:

  1. Check the status of ORES before deploying:

  1. Follow
  1. After the deployment of the canary node, we should test something like T302851#7963179 to verify that our fix works as expected.

Prep work:

  1. ssh ores1001.eqiad.wmnet
  2. cd /srv/deployment/ores/deploy
  3. Record the latest revision (OLDHASH) with git log -1 (in case you needed to rollback)

Deploy to canary:

  1. ssh deployment.eqiad.wmnet
  2. Update the deploy repository with:
    • cd /srv/deployment/ores/deploy
    • git log (and verify that HEAD is the hash retrieved in Prep Work on ores1001)
    • git fetch
    • git log origin (and inspect the commits between origin and local branch)
    • git pull
    • git submodule update --init
  3. Start a new session with tmux!
  4. scap deploy -v "Deploy revscoring 2.11.4 to ORES - T309536"
  5. Let it run, but when prompted to continue do not hit "y" yet!
  6. ssh ores1001.eqiad.wmnet and check the service internally: curl localhost:8081/v3/scores/wikidatawiki/1334902099/itemquality -i
  7. Monitor the logs: /srv/log/ores/main.log and app.log & Grafana/Logstash dashboards

Continue deployment to prod:
If everything works as expected, we're ready to continue.

  1. Deploy it fully by answering "y" to the scap prompt

Mentioned in SAL (#wikimedia-operations) [2022-06-01T13:40:35Z] <aikochou@deploy1002> Started deploy [ores/deploy@3d541df]: Deploy revscoring 2.11.4 to ORES - T309536

Mentioned in SAL (#wikimedia-operations) [2022-06-01T14:25:43Z] <aikochou@deploy1002> Finished deploy [ores/deploy@3d541df]: Deploy revscoring 2.11.4 to ORES - T309536 (duration: 45m 07s)

New log message is displayed correctly!

Post deployment checks are good, let's close :)