We're running very old versions of Jessie, https://tools.wmflabs.org/openstack-browser/project/ores, apparently without regular security updates. I'm not sure exactly what role our wmflabs boxes play, but there's no reason to tempt fate by skipping updates.
Description
Description
Details
Details
| Subject | Repo | Branch | Lines +/- | |
|---|---|---|---|---|
| wikilabels: Add zlib1g-dev package and cronjob to remove expired tasks | operations/puppet | production | +18 -1 |
| Status | Subtype | Assigned | Task | ||
|---|---|---|---|---|---|
| Resolved | None | T168478 Keep wmflabs scoring boxes up-to-date | |||
| Open | None | T169247 Document recommended process for installing vendor provided package upgrades in Wikimedia VPS |
Event Timeline
Comment Actions
Weird. We have been getting restarts when there's a labs-wide kernel update. What makes you think we aren't getting updates and what should we do to get them?
Comment Actions
@Halfak they update the labs machine that hosts the vm. So the machines got updated but not the vms.
Comment Actions
I'm surprised to find out that we aren't getting regular updates on these vms. Why has that happened and how do we change it?
Comment Actions
The only way to change that is to run a cron script that does apt-get update and then apt-get upgrade -y
Comment Actions
I've talked to @bd808 in #wikimedia-cloud and he doesn't think there's a best practice for this yet so I created T169247: Document recommended process for installing vendor provided package upgrades in Wikimedia VPS and set that as a blocked for this task.
Comment Actions
Mentioned in SAL (#wikimedia-cloud) [2018-08-22T11:52:34Z] <Amir1> spinning up wikilabels-02 (T168478)
Comment Actions
Change 454546 had a related patch set uploaded (by Ladsgroup; owner: Amir Sarabadani):
[operations/puppet@production] wikilabels: Add zlib1g-dev package and cronjob to remove expired tasks
Comment Actions
Mentioned in SAL (#wikimedia-cloud) [2018-08-22T18:12:55Z] <Amir1> redirecting traffic of labels.wmflabs.org from wikilabels-01.eqiad.wmflabs to wikilabels-02.eqiad.wmflabs (T168478)
Comment Actions
ores nodes are read-only while wikilabels nodes are read/write and contain sensitive information (database credentials, OAuth credentials). I made a new node and migrated the credential files and puppetized everything else so we can throw away VMs faster in future. I made a temporarily DNS proxy to expose the new node (wikilabels-02) to the outside and ran tests on it, when I was sure it writes to the database and OAuth works fine, I redirected the traffic to the new node but I keep the old node alive for a week in case anything happens.
Comment Actions
Change 454546 merged by Alexandros Kosiaris:
[operations/puppet@production] wikilabels: Add zlib1g-dev package and cronjob to remove expired tasks