Page MenuHomePhabricator

Update wikitech-static and develop procedures to keep it maintained
Closed, ResolvedPublic

Description

wikitech-static isn't updated by the deployment train. We need to develop/document an upgrade path, and then probably upgrade it by hand according to some schedule.

Also, version monitoring might be nice.

Questions:

  1. How often does it need upgrading?
  2. Who does those upgrades?
  3. How do we remember/schedule/otherwise ensure that those updates happen?

Related Objects

StatusSubtypeAssignedTask
ResolvedAndrew

Event Timeline

Restricted Application added a subscriber: Aklapper. · View Herald Transcript
Andrew renamed this task from Keep wikitech-static up to date, at least somewhat to Update wikitech-static and develop procedures to keep it maintained.Apr 26 2017, 3:54 PM
Andrew added a subtask: Restricted Task.

Maintenance tasks are just:

  • Keep packages up to date
  • Keep MW up to date

The root password to this host no longer works. Either someone fancied it up to use keys, or we need to do a rescue as per https://support.rackspace.com/how-to/reset-your-server-password/

I've updated https://wikitech.wikimedia.org/wiki/Wikitech-static with some vague maintenance instructions.

The root password now matches the one in pwstore. It's due for a rotation, but that will have to wait since the pw repo appears to be read-only today.

I updated the password so that's all good now.

I have two different sets of proposed answers:

Option A:

  1. Once a month
  2. Someone in Ops
  3. Icinga should have some kind of warning/critical test for stale wm versions

Option B:

  1. After every minor MW release
  2. Someone in Releng because they're in charge of knowing about 1)
  3. Via whatever calendering/task management Releng uses, which seems to work pretty well

Change 350920 had a related patch set uploaded (by Andrew Bogott; owner: Andrew Bogott):
[operations/puppet@production] Monitor wikitech-static mediawiki version

https://gerrit.wikimedia.org/r/350920

Change 350920 merged by Andrew Bogott:
[operations/puppet@production] Monitor wikitech-static mediawiki version

https://gerrit.wikimedia.org/r/350920

The icinga alert warned me about slippage between 1.28.1 and 1.28.2. I updated wikitech-static, and now the alert is green. So the alert works!

Change 351155 had a related patch set uploaded (by Andrew Bogott; owner: Andrew Bogott):
[operations/puppet@production] Wikitech-static: Include a doc link in the version warning.

https://gerrit.wikimedia.org/r/351155

Change 351155 merged by Andrew Bogott:
[operations/puppet@production] Wikitech-static: Include a doc link in the version warning.

https://gerrit.wikimedia.org/r/351155

We have "wikitech-static CRIT - wikitech and wikitech-static out of sync (426228s > 200000s)" alerts on silver and labtestweb2001. is it part of this? i saw we used T101803 for that before so i reopened there.

I ack'ed the alert yesterday and am working on it. It's composer, of course.

The alerting system seems to be working for this. We haven't designated a specific person in charge, but maybe this can be closed based on the assumption that alerting = ops will notice and fix :)

Bawolff closed subtask Restricted Task as Resolved.May 2 2017, 8:36 PM

We haven't designated a specific person in charge, but maybe this can be closed based on the assumption that alerting = ops will notice and fix :)

I think that's just fine. yea.

Andrew claimed this task.
Andrew closed subtask Restricted Task as Declined.Oct 3 2018, 9:38 PM