Page MenuHomePhabricator
Create Task
Maniphest T247650

replace bromine and vega with buster VMs
Closed, ResolvedPublic
Actions

Description

bromine.eqiad.wmnet and vega.codfw.wmnet are running stretch and should be replaced with new VMs on buster
and a new cluster name (miscstatic* or merged into miscweb*)

time frame: by the end of Q3 2020

Details

Show related patches Customize query in gerrit

Related Objects
Search...

Event Timeline

Dzahn created this task.Mar 13 2020, 11:59 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptMar 13 2020, 11:59 PM
Dzahn claimed this task.Mar 13 2020, 11:59 PM
Dzahn added a parent task: T247045: Migrate all of production metal and VMs to Buster or later.Mar 14 2020, 12:10 AM
gerritbot added a comment.Apr 10 2020, 12:41 PM

Change 587985 had a related patch set uploaded (by Dzahn; owner: Dzahn):
[operations/puppet@production] merge microsites into webserver_misc_apps

https://gerrit.wikimedia.org/r/587985

gerritbot added a project: Patch-For-Review.Apr 10 2020, 12:41 PM
gerritbot added a comment.Apr 20 2020, 9:44 AM

Change 587985 merged by Dzahn:
[operations/puppet@production] merge microsites into webserver_misc_apps

https://gerrit.wikimedia.org/r/587985

Maintenance_bot removed a project: Patch-For-Review.Apr 20 2020, 10:10 AM
gerritbot added a comment.Apr 20 2020, 10:41 AM

Change 591022 had a related patch set uploaded (by Dzahn; owner: Dzahn):
[operations/puppet@production] misc_apps: setup rsync to copy microsite data for migration

https://gerrit.wikimedia.org/r/591022

gerritbot added a project: Patch-For-Review.Apr 20 2020, 10:41 AM
gerritbot added a comment.Apr 20 2020, 10:57 AM

Change 591022 merged by Dzahn:
[operations/puppet@production] misc_apps: setup rsync to copy microsite data for migration

https://gerrit.wikimedia.org/r/591022

Stashbot added a comment.Apr 20 2020, 11:05 AM

Mentioned in SAL (#wikimedia-operations) [2020-04-20T11:05:57Z] <mutante> rsyncing static-bugzilla files from bromine to miscweb1002 (T247650)

Maintenance_bot removed a project: Patch-For-Review.Apr 20 2020, 11:10 AM
gerritbot added a comment.Apr 20 2020, 11:10 AM

Change 591025 had a related patch set uploaded (by Dzahn; owner: Dzahn):
[operations/puppet@production] miscapps: allow rsyncing app data to multiple destination hosts

https://gerrit.wikimedia.org/r/591025

gerritbot added a project: Patch-For-Review.Apr 20 2020, 11:10 AM
gerritbot added a comment.Apr 20 2020, 11:27 AM

Change 591025 merged by Dzahn:
[operations/puppet@production] miscapps: allow rsyncing app data to multiple destination hosts

https://gerrit.wikimedia.org/r/591025

gerritbot added a comment.Apr 20 2020, 11:32 AM

Change 591027 had a related patch set uploaded (by Dzahn; owner: Dzahn):
[operations/puppet@production] ATS: switch backend for static-bugzilla to misc-apps

https://gerrit.wikimedia.org/r/591027

gerritbot added a comment.Apr 20 2020, 11:36 AM

Change 591027 merged by Dzahn:
[operations/puppet@production] ATS: switch backend for static-bugzilla to misc-apps

https://gerrit.wikimedia.org/r/591027

Stashbot added a comment.Apr 20 2020, 11:41 AM

Mentioned in SAL (#wikimedia-operations) [2020-04-20T11:41:35Z] <mutante> puppetmaster - revoking cert for webserver-misc-apps.discovery.wmnet and recreating it with additional static microsite names (T247650)

gerritbot added a comment.Apr 20 2020, 11:47 AM

Change 591031 had a related patch set uploaded (by Dzahn; owner: Dzahn):
[operations/puppet@production] ssl: update certificate for webserver-misc-apps

https://gerrit.wikimedia.org/r/591031

gerritbot added a comment.Apr 20 2020, 11:50 AM

Change 591031 merged by Dzahn:
[operations/puppet@production] ssl: update certificate for webserver-misc-apps

https://gerrit.wikimedia.org/r/591031

Maintenance_bot removed a project: Patch-For-Review.Apr 20 2020, 12:10 PM
Dzahn triaged this task as Medium priority.Apr 20 2020, 3:41 PM
Dzahn moved this task from Incoming 🐫 to Doing 😎 on the serviceops board.
gerritbot added a comment.Apr 21 2020, 8:38 AM

Change 591052 had a related patch set uploaded (by Dzahn; owner: Dzahn):
[operations/puppet@production] ATS: switch backends for annual.wm.org and 15.wp.org

https://gerrit.wikimedia.org/r/591052

gerritbot added a project: Patch-For-Review.Apr 21 2020, 8:38 AM
gerritbot added a comment.Apr 21 2020, 8:41 AM

Change 591052 merged by Dzahn:
[operations/puppet@production] ATS: switch backends for annual.wm.org and 15.wp.org

https://gerrit.wikimedia.org/r/591052

gerritbot added a comment.Apr 21 2020, 8:51 AM

Change 591306 had a related patch set uploaded (by Dzahn; owner: Dzahn):
[operations/puppet@production] ATS: switch backends for and research and bienvenida.wm.org

https://gerrit.wikimedia.org/r/591306

gerritbot added a comment.Apr 27 2020, 8:07 AM

Change 591306 merged by Dzahn:
[operations/puppet@production] ATS: switch backends for and research and bienvenida.wm.org

https://gerrit.wikimedia.org/r/591306

Maintenance_bot removed a project: Patch-For-Review.Apr 27 2020, 8:10 AM
gerritbot added a comment.Apr 27 2020, 8:16 AM

Change 592603 had a related patch set uploaded (by Dzahn; owner: Dzahn):
[operations/puppet@production] ATS: switch backends for transparency and transparency-private.wm.org

https://gerrit.wikimedia.org/r/592603

gerritbot added a project: Patch-For-Review.Apr 27 2020, 8:16 AM
gerritbot added a comment.Apr 27 2020, 8:41 AM

Change 592610 had a related patch set uploaded (by Dzahn; owner: Dzahn):
[operations/puppet@production] ATS: switch backends for design and sitemaps static sites

https://gerrit.wikimedia.org/r/592610

gerritbot added a comment.Apr 27 2020, 8:43 AM

Change 592612 had a related patch set uploaded (by Dzahn; owner: Dzahn):
[operations/puppet@production] ATS: switch backends for wikiworkshop static sites

https://gerrit.wikimedia.org/r/592612

Stashbot mentioned this in T188362: Move "transparency.wikimedia.org/private" to "transparency-private.wikimedia.org".Apr 28 2020, 8:13 AM

Mentioned in SAL (#wikimedia-operations) [2020-04-28T08:13:09Z] <mutante> rsyncing transparency-report-private files from bromine to miscweb1002/2002. git-cloning was removed about a year ago but site still exists. need to figure out if it should be deleted (T188362 T247650)

gerritbot added a comment.Apr 28 2020, 8:50 AM

Change 592883 had a related patch set uploaded (by Dzahn; owner: Dzahn):
[operations/puppet@production] httpbb: add tests for miscweb sites

https://gerrit.wikimedia.org/r/592883

gerritbot added a comment.Apr 28 2020, 9:27 AM

Change 592603 merged by Dzahn:
[operations/puppet@production] ATS: switch backends for the 3 transparency sites

https://gerrit.wikimedia.org/r/592603

gerritbot added a comment.Apr 28 2020, 1:17 PM

Change 592610 merged by Dzahn:
[operations/puppet@production] ATS: switch backends for design and sitemaps static sites

https://gerrit.wikimedia.org/r/592610

gerritbot added a comment.Apr 28 2020, 3:21 PM

Change 592980 had a related patch set uploaded (by Dzahn; owner: Dzahn):
[design/style-guide@master] update scap-targets from bromine/vega to miscweb1002/2002

https://gerrit.wikimedia.org/r/592980

Restricted Repository Identity mentioned this in R2668:81552e636db3: build: Update scap-targets from bromine/vega to miscweb1002/2002.Apr 28 2020, 4:02 PM
gerritbot added a comment.Apr 28 2020, 4:31 PM

Change 592980 abandoned by VolkerE:
update scap-targets from bromine/vega to miscweb1002/2002

Reason:
Done in https://github.com/wikimedia/WikimediaUI-Style-Guide/pull/369 instead

https://gerrit.wikimedia.org/r/592980

Restricted Repository Identity mentioned this in R2668:335122b96f0d: build: Update scap-targets from bromine/vega to miscweb1002/2002 (#369).Apr 28 2020, 4:32 PM
gerritbot added a comment.Apr 29 2020, 11:44 AM

Change 592612 merged by Dzahn:
[operations/puppet@production] ATS: switch backends for wikiworkshop static sites

https://gerrit.wikimedia.org/r/592612

gerritbot added a comment.Apr 29 2020, 12:14 PM

Change 593218 had a related patch set uploaded (by Dzahn; owner: Dzahn):
[operations/puppet@production] remove bromine.eqiad.wmnet and vega.codfw.wmnet

https://gerrit.wikimedia.org/r/593218

gerritbot added a comment.Apr 29 2020, 12:25 PM

Change 593221 had a related patch set uploaded (by Dzahn; owner: Dzahn):
[operations/dns@master] decom bromine.eqiad.wmnet and vega.codfw.wmnet

https://gerrit.wikimedia.org/r/593221

gerritbot added a comment.May 1 2020, 11:29 AM

Change 593218 merged by Dzahn:
[operations/puppet@production] remove bromine.eqiad, vega.codfw and webserver_misc_static role

https://gerrit.wikimedia.org/r/593218

ops-monitoring-bot added a comment.May 1 2020, 11:31 AM

Icinga downtime for 1 day, 0:00:00 set by dzahn@cumin1001 on 1 host(s) and their services with reason: decom

bromine.eqiad.wmnet
ops-monitoring-bot added a comment.May 1 2020, 11:31 AM

Icinga downtime for 1 day, 0:00:00 set by dzahn@cumin1001 on 1 host(s) and their services with reason: decom

vega.codfw.wmnet
ops-monitoring-bot added a comment.May 1 2020, 11:50 AM

cookbooks.sre.hosts.decommission executed by dzahn@cumin1001 for hosts: vega.codfw.wmnet

  • vega.codfw.wmnet (PASS)
    • Downtimed host on Icinga
    • Found Ganeti VM
    • VM shutdown
    • Started forced sync of VMs in Ganeti cluster ganeti01.svc.codfw.wmnet to Netbox
    • Removed from DebMonitor
    • Removed from Puppet master and PuppetDB
    • VM removed
ops-monitoring-bot added a comment.May 1 2020, 11:52 AM

cookbooks.sre.hosts.decommission executed by dzahn@cumin1001 for hosts: bromine.eqiad.wmnet

  • bromine.eqiad.wmnet (PASS)
    • Downtimed host on Icinga
    • Found Ganeti VM
    • VM shutdown
    • Started forced sync of VMs in Ganeti cluster ganeti01.svc.eqiad.wmnet to Netbox
    • Removed from DebMonitor
    • Removed from Puppet master and PuppetDB
    • VM removed
gerritbot added a comment.May 1 2020, 11:54 AM

Change 593221 merged by Dzahn:
[operations/dns@master] decom bromine.eqiad.wmnet and vega.codfw.wmnet

https://gerrit.wikimedia.org/r/593221

Dzahn closed this task as Resolved.May 1 2020, 11:55 AM

Done! bromine and vega are gone and the services on it have been fully merged into miscweb1002/2002 which are on buster.

Dzahn removed a project: Patch-For-Review.May 1 2020, 11:55 AM
gerritbot added a comment.May 4 2020, 8:37 AM

Change 592883 merged by Dzahn:
[operations/puppet@production] httpbb: add tests for miscweb sites

https://gerrit.wikimedia.org/r/592883

Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits