Page MenuHomePhabricator
Create Task
Maniphest T188163

create codfw-equivalent of bromine, make webserver_misc_static active/active in misc varnish
Closed, ResolvedPublic
Actions

Description

  • create a ganeti VM in codfw that is equivalent to bromine.eqiad.wmnet (subtask with VM request and install here)
  • use stretch and apply "misc_static"-webserver puppet role, confirm it works on stretch
  • switch backend in Varnish over to codfw temp. confirm things still work
  • reinstall bromine (currently jessie) with stretch
  • make service active/active in Varnish to serve from both DCs

This means the following sites will also be served from codfw and can still be up if we fail-over from eqiad or if the eqiad VM breaks and vice-versa. Not a single point of failure anymore for:

  1. https://annual.wikimedia.org , https://15.wikipedia.org
  2. https://static-bugzilla.wikimedia.org
  3. https://transparency.wikimedia.org , https://transparency-private.wikimedia.org
  4. https://wikiba.se (planned)
  5. https://research.wikimedia.org (new)
  6. https://design.wikimedia.org (coming soon)

Details

Related Changes in Gerrit:
Show related patches Customize query in gerrit

Related Objects
Search...

Event Timeline

Dzahn created this task.Feb 24 2018, 2:39 AM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptFeb 24 2018, 2:39 AM
gerritbot subscribed.Mar 16 2018, 5:52 PM

Change 420082 had a related patch set uploaded (by Dzahn; owner: Dzahn):
[operations/dns@master] introduce vega.codfw.wmnet (bromine equivalent)

https://gerrit.wikimedia.org/r/420082

gerritbot added a project: Patch-For-Review.Mar 16 2018, 5:52 PM
Dzahn mentioned this in T189899: VM request for bromine (misc_static_sites) equivalent.Mar 16 2018, 6:14 PM
gerritbot added a comment.Mar 16 2018, 6:29 PM

Change 420082 merged by Dzahn:
[operations/dns@master] introduce vega.codfw.wmnet (bromine equivalent)

https://gerrit.wikimedia.org/r/420082

Dzahn renamed this task from create codfw-equivalent of bromine and make webserver_misc_static active/active in misc varnish to create codfw-equivalent of bromine, make webserver_misc_static active/active in misc varnish.Mar 16 2018, 10:59 PM
Dzahn triaged this task as Medium priority.
Dzahn updated the task description. (Show Details)
Dzahn updated the task description. (Show Details)
Dzahn closed subtask T189899: VM request for bromine (misc_static_sites) equivalent as Resolved.Mar 16 2018, 11:03 PM
gerritbot added a comment.Mar 16 2018, 11:08 PM

Change 420132 had a related patch set uploaded (by Dzahn; owner: Dzahn):
[operations/puppet@production] DHCP/netboot: add vega.codfw.wmnet

https://gerrit.wikimedia.org/r/420132

gerritbot added a comment.Mar 16 2018, 11:10 PM

Change 420132 merged by Dzahn:
[operations/puppet@production] DHCP/netboot: add vega.codfw.wmnet

https://gerrit.wikimedia.org/r/420132

Stashbot subscribed.Mar 16 2018, 11:32 PM

Mentioned in SAL (#wikimedia-operations) [2018-03-16T23:32:14Z] <mutante> signing puppet cert for vega.codfw.wmnet, initial puppet run after fresh stretch install (T188163)

gerritbot added a comment.Mar 16 2018, 11:36 PM

Change 420134 had a related patch set uploaded (by Dzahn; owner: Dzahn):
[operations/puppet@production] site/webserver_misc_static: add vega as codfw node

https://gerrit.wikimedia.org/r/420134

gerritbot added a comment.Mar 16 2018, 11:39 PM

Change 420134 merged by Dzahn:
[operations/puppet@production] site/webserver_misc_static: add vega as codfw node

https://gerrit.wikimedia.org/r/420134

Dzahn updated the task description. (Show Details)Mar 16 2018, 11:49 PM
Dzahn updated the task description. (Show Details)
gerritbot added a comment.Mar 17 2018, 12:01 AM

Change 420137 had a related patch set uploaded (by Dzahn; owner: Dzahn):
[operations/puppet@production] misc:varnish: rename bromine director, clean up unused design director

https://gerrit.wikimedia.org/r/420137

gerritbot added a comment.Mar 17 2018, 12:08 AM

Change 420137 merged by Dzahn:
[operations/puppet@production] misc:varnish: rename bromine director, clean up unused design director

https://gerrit.wikimedia.org/r/420137

Stashbot added a comment.Mar 17 2018, 12:13 AM

Mentioned in SAL (#wikimedia-operations) [2018-03-17T00:13:19Z] <mutante> running puppet on all cache::misc to rename director bromine to webserver_misc_static (T188163)

Krinkle updated the task description. (Show Details)Mar 17 2018, 12:35 AM
Krinkle added a project: Sustainability.
Krinkle moved this task from Tag to Doing on the Sustainability board.
gerritbot added a comment.Mar 17 2018, 12:46 AM

Change 420142 had a related patch set uploaded (by Dzahn; owner: Dzahn):
[operations/puppet@production] cache::misc: switch webserver_misc_static to codfw backend

https://gerrit.wikimedia.org/r/420142

gerritbot added a comment.Mar 29 2018, 10:14 PM

Change 423059 had a related patch set uploaded (by Dzahn; owner: Dzahn):
[operations/dns@master] add IPv6 records for vega.codfw.wmnet

https://gerrit.wikimedia.org/r/423059

gerritbot added a comment.Mar 29 2018, 10:15 PM

Change 423059 merged by Dzahn:
[operations/dns@master] add IPv6 records for vega.codfw.wmnet

https://gerrit.wikimedia.org/r/423059

gerritbot added a comment.Mar 30 2018, 1:37 AM

Change 423079 had a related patch set uploaded (by Dzahn; owner: Dzahn):
[operations/dns@master] add IPv6 records for bromine.eqiad.wmnet

https://gerrit.wikimedia.org/r/423079

gerritbot added a comment.Mar 30 2018, 1:42 AM

Change 423079 merged by Dzahn:
[operations/dns@master] add IPv6 records for bromine.eqiad.wmnet

https://gerrit.wikimedia.org/r/423079

gerritbot added a comment.Mar 30 2018, 1:45 AM

Change 423080 had a related patch set uploaded (by Dzahn; owner: Dzahn):
[operations/puppet@production] cache::misc: add codfw backend for webserver_misc_static

https://gerrit.wikimedia.org/r/423080

gerritbot added a comment.Apr 2 2018, 10:22 PM

Change 423080 merged by Dzahn:
[operations/puppet@production] cache::misc: add codfw backend for webserver_misc_static

https://gerrit.wikimedia.org/r/423080

Dzahn updated the task description. (Show Details)Apr 2 2018, 10:39 PM
  • tested codfw backend with apache-fast-test
  • added codfw backend to make service active/active
  • have not reinstalled bromine yet, but to switch to codfw i would have had to add both first anyways.. either way
Dzahn added a comment.Apr 6 2018, 7:20 PM

https://gerrit.wikimedia.org/r/#/c/423580/

gerritbot added a comment.Apr 6 2018, 7:21 PM

Change 424657 had a related patch set uploaded (by Dzahn; owner: Dzahn):
[operations/puppet@production] rsync bugzilla-static content from bromine to vega

https://gerrit.wikimedia.org/r/424657

gerritbot added a comment.Apr 6 2018, 10:21 PM

Change 424657 merged by Dzahn:
[operations/puppet@production] rsync bugzilla-static content from bromine to vega

https://gerrit.wikimedia.org/r/424657

gerritbot added a comment.Apr 6 2018, 10:59 PM

Change 424716 had a related patch set uploaded (by Dzahn; owner: Dzahn):
[operations/puppet@production] DHCP: upgrade bromine from jessie to stretch

https://gerrit.wikimedia.org/r/424716

gerritbot added a comment.Apr 6 2018, 11:02 PM

Change 424716 merged by Dzahn:
[operations/puppet@production] DHCP: upgrade bromine from jessie to stretch

https://gerrit.wikimedia.org/r/424716

Stashbot added a comment.Apr 7 2018, 12:14 AM

Mentioned in SAL (#wikimedia-operations) [2018-04-07T00:14:55Z] <mutante> bromine - scheduled downtime, reboot for reinstall, upgrade to stretch, misc_static_services switched to codfw (T188163)

gerritbot added a comment.Apr 7 2018, 12:18 AM

Change 424727 had a related patch set uploaded (by Dzahn; owner: Dzahn):
[operations/puppet@production] static_bugzilla: reverse rsync direction after bromine reinstall

https://gerrit.wikimedia.org/r/424727

gerritbot added a comment.Apr 7 2018, 12:27 AM

Change 424727 merged by Dzahn:
[operations/puppet@production] static_bugzilla: reverse rsync direction after bromine reinstall

https://gerrit.wikimedia.org/r/424727

gerritbot added a comment.Apr 7 2018, 12:44 AM

Change 424731 had a related patch set uploaded (by Dzahn; owner: Dzahn):
[operations/puppet@production] Revert "misc_static_sites: temp disable bromine backend for reinstall"

https://gerrit.wikimedia.org/r/424731

gerritbot added a comment.Apr 7 2018, 12:45 AM

Change 424731 merged by Dzahn:
[operations/puppet@production] Revert "misc_static_sites: temp disable bromine backend for reinstall"

https://gerrit.wikimedia.org/r/424731

Dzahn closed this task as Resolved.Apr 7 2018, 12:47 AM
Dzahn updated the task description. (Show Details)

All done!

  • we are active-active
  • both eqiad and codfw are on stretch
Dzahn closed subtask T189910: upgrade bromine to stretch / reinstall as Resolved.Apr 10 2018, 1:55 AM
gerritbot added a comment.Apr 11 2018, 10:13 PM

Change 420142 abandoned by Dzahn:
cache::misc: switch webserver_misc_static to codfw backend

Reason:
duplicate. already done elsewhere. ticket is resolved

https://gerrit.wikimedia.org/r/420142

Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits