Page MenuHomePhabricator

Deploy revscoring 2.11.4 to ORES
Closed, ResolvedPublic

Description

The revscoring 2.11.4 wheel should be deployed to ORES (it is already running in all Lift Wing pods).

Let's write down a procedure and then schedule a deploy :)

Event Timeline

High level plan:

  1. Check the status of ORES before deploying:

https://grafana.wikimedia.org/d/HIRrxQ6mk/ores?orgId=1&refresh=1m
https://logstash.wikimedia.org/app/dashboards#/view/ba190230-deb8-11e8-99b8-7fba019e77c2

  1. Follow https://wikitech.wikimedia.org/wiki/ORES/Deployment#Deploy_to_production
  1. After the deployment of the canary node, we should test something like T302851#7963179 to verify that our fix works as expected.

Prep work:

  1. ssh ores1001.eqiad.wmnet
  2. cd /srv/deployment/ores/deploy
  3. Record the latest revision (OLDHASH) with git log -1 (in case you needed to rollback)

Deploy to canary:

  1. ssh deployment.eqiad.wmnet
  2. Update the deploy repository with:
    • cd /srv/deployment/ores/deploy
    • git log (and verify that HEAD is the hash retrieved in Prep Work on ores1001)
    • git fetch
    • git log origin (and inspect the commits between origin and local branch)
    • git pull
    • git submodule update --init
  3. Start a new session with tmux!
  4. scap deploy -v "Deploy revscoring 2.11.4 to ORES - T309536"
  5. Let it run, but when prompted to continue do not hit "y" yet!
  6. ssh ores1001.eqiad.wmnet and check the service internally: curl localhost:8081/v3/scores/wikidatawiki/1334902099/itemquality -i
  7. Monitor the logs: /srv/log/ores/main.log and app.log & Grafana/Logstash dashboards

Continue deployment to prod:
If everything works as expected, we're ready to continue.

  1. Deploy it fully by answering "y" to the scap prompt

Mentioned in SAL (#wikimedia-operations) [2022-06-01T13:40:35Z] <aikochou@deploy1002> Started deploy [ores/deploy@3d541df]: Deploy revscoring 2.11.4 to ORES - T309536

Mentioned in SAL (#wikimedia-operations) [2022-06-01T14:25:43Z] <aikochou@deploy1002> Finished deploy [ores/deploy@3d541df]: Deploy revscoring 2.11.4 to ORES - T309536 (duration: 45m 07s)

New log message is displayed correctly!

https://ores.wikimedia.org/v3/scores/wikidatawiki/1334902099/itemquality

Post deployment checks are good, let's close :)