Page MenuHomePhabricator
Create Task
Maniphest T98758

Upgrade beta-cluster caches to jessie
Closed, ResolvedPublic
Actions

Description

Beta cluster varnish endpoints are running Ubuntu Precise, should be Debian Jessie to match production.

This is indirectly causing some intermittent 503 issues due to prod+beta VCL changes related to gzip, which are tripping varnish bugs, which are present in the last varnish versions we built for precise (3.0.5plus~x-wm7), but fixed in the newer versions we're running on jessie (3.0.6plus-wm6).

Actions to be conducted:

  • create Jessie instances
  • prepare puppet.git patches to update Varnish config hash and DNS alias IP addresses
  • prepare mediawiki-config.git patch to change the HTCP purging entries
  • switch labs public/private IP binding in OpenStack manager (breaking change)

Then:

  • make sure purges are properly emitted
  • verify a labs instance can reach the various FQDN entries (such as en.wikipedia.beta.wmflabs.org)
  • run a few browser tests jobs that hits beta (to be determined)

Details

SubjectRepoBranchLines +/-
beta: Swap caches to deployment-cache-*04, which is jessieoperations/mediawiki-configmaster+6 -6
beta: Swap caches to deployment-cache-*04, which is jessieoperations/puppetproduction+25 -25
Customize query in gerrit

Related Objects
Search...

Event Timeline

BBlack created this task.May 11 2015, 5:23 PM
BBlack raised the priority of this task from to Needs Triage.
BBlack updated the task description. (Show Details)
BBlack added projects: acl*sre-team, Deployments, Beta-Cluster-Infrastructure, Traffic.
BBlack subscribed.
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptMay 11 2015, 5:23 PM
BBlack added a comment.May 11 2015, 5:33 PM

Paste of induced 503s related to gzip: https://phabricator.wikimedia.org/P633 , where the fetching fails with:

13 FetchError   c Junk after gzip data
13 Gzip         c u F - 20 0 80 80 90
BBlack mentioned this in T75564: Upgrade varnish automatically via puppet in Beta Cluster.May 14 2015, 10:33 PM
BBlack triaged this task as Medium priority.May 20 2015, 8:17 AM
thcipriani removed a project: Deployments.Jun 1 2015, 6:18 PM
thcipriani set Security to None.
thcipriani moved this task from To Triage to Next: Maintenance on the Beta-Cluster-Infrastructure board.Jun 15 2015, 7:41 PM
hashar subscribed.Jun 15 2015, 7:43 PM

Talked about this during the weekly Beta-Cluster-Infrastructure triage will further talk about it during our Release-Engineering-Team meeting. In short, seems a good opportunity for some team work and lot of 1/1 pairing and reviewing.

hashar updated the task description. (Show Details)Jun 16 2015, 8:47 AM
thcipriani raised the priority of this task from Medium to High.Jul 27 2015, 7:45 PM
thcipriani subscribed.

Sounds like Varnish packages won't be getting built for Trusty any longer, upping priority.

Restricted Application added subscribers: Luke081515, Matanya. · View Herald TranscriptJul 27 2015, 7:45 PM
hashar added a comment.Jul 28 2015, 1:50 PM

deployment-cache-text03 has been created with Jessie system. That is to prepare the migration of the Trusty cache deployment-cache-text02.

Krenair updated the task description. (Show Details)Jul 28 2015, 1:50 PM
• demon claimed this task.Jul 28 2015, 4:47 PM
gerritbot subscribed.Jul 29 2015, 4:27 PM

Change 227743 had a related patch set uploaded (by Chad):
beta: Swap text caches to -text04, which is jessie

https://gerrit.wikimedia.org/r/227743

gerritbot added a project: Patch-For-Review.Jul 29 2015, 4:27 PM

Change 227744 had a related patch set uploaded (by Chad):
beta: swap text caches to text04, which is jessie

https://gerrit.wikimedia.org/r/227744

• demon added a comment.Jul 30 2015, 3:05 PM

Got instances of deployment-cache-*04 running jessie, all succeeding with puppet (minus TLS stuff, which I'm skipping for now to prevent the eternal failures).

deployment-cache-mobile04 is still failing on its zerofetch.py check, but that seems more like beta's general problem...

I think we can move onto the puppet/mw-config bits now.

gerritbot added a comment.Aug 5 2015, 5:28 PM

Change 227744 merged by BBlack:
beta: Swap caches to deployment-cache-*04, which is jessie

https://gerrit.wikimedia.org/r/227744

• demon mentioned this in rOPUP48737584788f: beta: Swap caches to deployment-cache-*04, which is jessie.Aug 5 2015, 5:28 PM
gerritbot added a comment.Aug 5 2015, 5:35 PM

Change 227743 merged by BBlack:
beta: Swap caches to deployment-cache-*04, which is jessie

https://gerrit.wikimedia.org/r/227743

• demon mentioned this in rOMWCde62c4286244: beta: Swap caches to deployment-cache-*04, which is jessie.Aug 5 2015, 5:35 PM
• demon updated the task description. (Show Details)Aug 5 2015, 5:47 PM

All seems working, just need to verify purges, make sure browser tests are still ok, and then decom the old instances.

• demon updated the task description. (Show Details)Aug 13 2015, 4:47 PM

No complaint from browser testing folks, assuming that's fine. Decom'd the old instances now.

I guess technically we need to check purges are working, but I assume they are at this point.

Luke081515 awarded a token.Aug 13 2015, 8:46 PM
• demon closed this task as Resolved.Aug 19 2015, 3:34 PM
hashar reopened this task as Open.Sep 8 2015, 10:28 AM

The Parsoid cache deployment-parsoidcache02 is still on Trusty :( T103660: Migrate Parsoid cache from Trusty to Jessie

hashar added a parent task: T103660: Migrate Parsoid cache from Trusty to Jessie.Sep 8 2015, 10:29 AM
hashar mentioned this in T103660: Migrate Parsoid cache from Trusty to Jessie.
• demon added a comment.Sep 15 2015, 6:56 PM

Working on this. Failing on the usual TLS madness.

hashar added a comment.Sep 28 2015, 7:17 PM

@demon any progress on this? I guess you had more important duties. Should we pair on it Release-Engineering-Team

hashar closed this task as Resolved.Oct 28 2015, 3:40 PM

T103660: Migrate Parsoid cache from Trusty to Jessie has finally been solved. That was the last Varnish cache still using Trusty.

BBlack moved this task from Backlog to Done on the Traffic board.Nov 30 2015, 6:02 PM
hashar moved this task from Next: Maintenance to Done on the Beta-Cluster-Infrastructure board.Dec 8 2015, 12:13 PM
greg added a project: Essential-Work.Jan 11 2016, 10:50 PM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits