Page MenuHomePhabricator
Create Task
Maniphest T86648

Upgrade all HTTP frontends to Debian jessie
Closed, ResolvedPublic
Actions

Description

Our current HTTP frontend fleet (frontend & backend Varnishes for routing & caching, nginx for SSL termination) runs on Ubuntu 12.04 (precise).

Mainly motivated by HTTPS improvements (newer libssl, nginx) and IPsec rollouts, we need to start moving our fleet to a newer platform. Debian jessie is a good candidate for this, as this is the next Wikimedia OS. This work is expected to happen by the end of FY Q3 2015.

For this we'll need to:

  • Prepare infrastructure for jessie boxes (already done as part of a separate goal)
  • Upgrade one canary box to jessie (cp1008)
  • Rebuild custom-made packages for jessie, notably Varnish 3 (jessie ships with 4), varnishkafka; port upstart service files to systemd
  • Reinstall one server of each type role (text, mobile, upload, bits) in production for live testing
  • Make sure that everything works, new kernel in particular.
  • Reinstall all servers across all datacenters

Note that related to this, we'll also need the availability of jessie images for Labs so that we can perform tests and so that Beta can keep up with production. For this, the availability of jessie images in Labs, T75592, is a blocker.

Details

ProjectBranchLines +/-Subject
operations/puppetproduction+1 -1depool cp1057 for reinstall
operations/puppetproduction+1 -1depool cp1056 for reinstall
Customize query in gerrit

Related Objects
Search...

Event Timeline

faidon created this task.Jan 13 2015, 1:39 PM
faidon assigned this task to BBlack.
faidon raised the priority of this task from to High.
faidon updated the task description. (Show Details)
faidon added projects: ops-core, HTTPS-by-default.
faidon added subscribers: faidon, mark.
yuvipanda updated the task description. (Show Details)Jan 13 2015, 1:40 PM
yuvipanda set Security to None.
faidon mentioned this in T86663: Expand HTTP frontend clusters with new hardware.Jan 13 2015, 2:41 PM
faidon mentioned this in T86664: HTTPS performance & UA adoption metrics.Jan 13 2015, 2:44 PM
faidon mentioned this in T86666: HTTPS performance tuning.Jan 13 2015, 2:51 PM
faidon mentioned this in T83809: cp* boxes, pagecache issues & trying newer kernels.Jan 20 2015, 12:31 AM
• GWicke added a subscriber: • GWicke.Feb 6 2015, 6:59 PM
• GWicke added a comment.Feb 6 2015, 7:19 PM

cp1008 looking SPDY: https://spdycheck.org/#cp1008.wikimedia.org

BBlack updated the task description. (Show Details)Feb 9 2015, 4:09 PM
BBlack added a comment.Feb 9 2015, 4:12 PM

Currently we have reinstalled to the new jessie stack one of each type in eqiad (text -> cp1065, upload -> cp1064, bits -> cp1070, mobile -> cp1060) as well as amssq42 as text in esams for live testing. The only known major issue outstanding from that testing is a VM/kernel bug/tuning issue that affects the varnish backends for upload due to their network/disk i/o patterns. I'll open a separate ticket to track that. Once that's resolved, we'll be ready to reinstall all the existing servers to the new jessie-based software stack.

BBlack added a subtask: T88996: jessie kernel vm subsystem issues for upload caches.Feb 9 2015, 4:20 PM
• Gage added a subscriber: • Gage.Mar 2 2015, 4:03 PM
BBlack closed subtask T88996: jessie kernel vm subsystem issues for upload caches as Resolved.Mar 3 2015, 11:35 PM
BBlack updated the task description. (Show Details)
BBlack added a comment.Mar 3 2015, 11:38 PM

Test installs are now successful and all known issues are resolved for all cache types (e.g. systemd transitions for various daemons, varnishkafka issues, etc). Next step here is to document the reinstall process on wikitech for others to help, set up a plan for reasonable parallelism without causing performance degradation, and begin reinstalling all of the clusters. Should be done by the end of next week, approximately, although upload clusters may require more time to space them out, so as not cause performance degradation.

BBlack added a comment.EditedMar 6 2015, 8:21 PM

We're at ~15% of the cache endpoints converted and reinstalled now, and most corner-case oddities with the reinstall process are known. Next week begins mass reinstalls, expecting that almost everything with the exception of a few esams upload machines will be complete by the end of the week. esams upload is a special case that requires longer time intervals to refill disks over the network...

gerritbot added a subscriber: gerritbot.Mar 10 2015, 3:42 PM

Change 195573 had a related patch set uploaded (by Dzahn):
depool cp1056 for reinstall

https://gerrit.wikimedia.org/r/195573

gerritbot added a project: Patch-For-Review.Mar 10 2015, 3:42 PM
gerritbot added a comment.Mar 10 2015, 3:51 PM

Change 195573 merged by Dzahn:
depool cp1056 for reinstall

https://gerrit.wikimedia.org/r/195573

Dzahn mentioned this in rOPUP3df7d26fcb6c: depool cp1056 for reinstall.Mar 10 2015, 3:51 PM
gerritbot added a comment.Mar 10 2015, 7:19 PM

Change 195632 had a related patch set uploaded (by Dzahn):
depool cp1057 for reinstall

https://gerrit.wikimedia.org/r/195632

gerritbot added a comment.Mar 10 2015, 7:22 PM

Change 195632 merged by Dzahn:
depool cp1057 for reinstall

https://gerrit.wikimedia.org/r/195632

Dzahn mentioned this in rOPUP913070d32b29: depool cp1057 for reinstall.Mar 10 2015, 7:22 PM
BBlack added a subscriber: Dzahn.Mar 11 2015, 12:12 AM

Status update: ~53% of the cluster is now converted to the new config. I think we'll make the end of the week at this point, or at least very near to it. Thanks @Dzahn for helping with some of these reinstalls :)

Dzahn added a comment.Mar 11 2015, 10:58 PM

cp1053
cp1061
cp1052
cp1054
cp1057
cp1056

done. i stopped adding the bug number to commit messages because it would be too spammy

Dzahn added a comment.Mar 12 2015, 8:52 PM

text-eqiad are 100% complete

BBlack added a comment.EditedMar 12 2015, 10:09 PM

At this point, we're 100% converted globally for the text, mobile, and bits clusters. There's still a few left in the eqiad and esams upload clusters, but they should be done by sometime tomorrow.

BBlack closed this task as Resolved.EditedMar 13 2015, 7:24 PM

100% complete now for all live, pooled, public cache endpoints for text, mobile, bits, upload, and misc-web.

parsoidcache has 1/2 hosts converted (putting the other off by a week or so is the current plan, to avoid losing too much parsoidcache data at once - need to coordinate with @GWicke further to confirm - either way only incidentally related to this ticket, as parsoidcache isn't a cpuload/clientperf blocker for HTTPS-by-default ).

There are still 3x current hosts (cp1047, cp3011, cp4009) that are down for hardware issues presently, and will get installed with the new jessie config once their hardware issues are resolved (those all have separate hardware tickets noted in cache.pp depool comments).

BBlack updated the task description. (Show Details)Mar 13 2015, 7:28 PM
BBlack removed a project: Patch-For-Review.
faidon mentioned this in T35890: Support SPDY.Mar 14 2015, 12:32 PM
• GWicke added a comment.EditedMar 15 2015, 4:01 PM

@BBlack, delaying the second Parsoid cache by a week should be fine. If things go well we should have VE use RESTBase instead for all wikipedias by the end of next week (possibly even before Wednesday), at which point the issue becomes moot. See T89066: Parsoid performance: Use RESTBase from the MediaWiki Virtual Rest Service on group1/group2 wikis.

BBlack mentioned this in rOPUPfb0a522e6c10: repool cp1008, update node comments.Mar 23 2015, 3:52 PM
• MZMcBride added a subscriber: • MZMcBride.Jun 10 2015, 5:37 AM
Content licensed under Creative Commons Attribution-ShareAlike 3.0 (CC-BY-SA) unless otherwise noted; code licensed under GNU General Public License (GPL) or other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL