Page MenuHomePhabricator

replace bromine and vega with buster VMs
Closed, ResolvedPublic

Description

bromine.eqiad.wmnet and vega.codfw.wmnet are running stretch and should be replaced with new VMs on buster
and a new cluster name (miscstatic* or merged into miscweb*)

time frame: by the end of Q3 2020

Event Timeline

Change 587985 had a related patch set uploaded (by Dzahn; owner: Dzahn):
[operations/puppet@production] merge microsites into webserver_misc_apps

https://gerrit.wikimedia.org/r/587985

Change 587985 merged by Dzahn:
[operations/puppet@production] merge microsites into webserver_misc_apps

https://gerrit.wikimedia.org/r/587985

Change 591022 had a related patch set uploaded (by Dzahn; owner: Dzahn):
[operations/puppet@production] misc_apps: setup rsync to copy microsite data for migration

https://gerrit.wikimedia.org/r/591022

Change 591022 merged by Dzahn:
[operations/puppet@production] misc_apps: setup rsync to copy microsite data for migration

https://gerrit.wikimedia.org/r/591022

Mentioned in SAL (#wikimedia-operations) [2020-04-20T11:05:57Z] <mutante> rsyncing static-bugzilla files from bromine to miscweb1002 (T247650)

Change 591025 had a related patch set uploaded (by Dzahn; owner: Dzahn):
[operations/puppet@production] miscapps: allow rsyncing app data to multiple destination hosts

https://gerrit.wikimedia.org/r/591025

Change 591025 merged by Dzahn:
[operations/puppet@production] miscapps: allow rsyncing app data to multiple destination hosts

https://gerrit.wikimedia.org/r/591025

Change 591027 had a related patch set uploaded (by Dzahn; owner: Dzahn):
[operations/puppet@production] ATS: switch backend for static-bugzilla to misc-apps

https://gerrit.wikimedia.org/r/591027

Change 591027 merged by Dzahn:
[operations/puppet@production] ATS: switch backend for static-bugzilla to misc-apps

https://gerrit.wikimedia.org/r/591027

Mentioned in SAL (#wikimedia-operations) [2020-04-20T11:41:35Z] <mutante> puppetmaster - revoking cert for webserver-misc-apps.discovery.wmnet and recreating it with additional static microsite names (T247650)

Change 591031 had a related patch set uploaded (by Dzahn; owner: Dzahn):
[operations/puppet@production] ssl: update certificate for webserver-misc-apps

https://gerrit.wikimedia.org/r/591031

Change 591031 merged by Dzahn:
[operations/puppet@production] ssl: update certificate for webserver-misc-apps

https://gerrit.wikimedia.org/r/591031

Dzahn triaged this task as Medium priority.Apr 20 2020, 3:41 PM
Dzahn moved this task from Incoming 🐫 to Doing 😎 on the serviceops board.

Change 591052 had a related patch set uploaded (by Dzahn; owner: Dzahn):
[operations/puppet@production] ATS: switch backends for annual.wm.org and 15.wp.org

https://gerrit.wikimedia.org/r/591052

Change 591052 merged by Dzahn:
[operations/puppet@production] ATS: switch backends for annual.wm.org and 15.wp.org

https://gerrit.wikimedia.org/r/591052

Change 591306 had a related patch set uploaded (by Dzahn; owner: Dzahn):
[operations/puppet@production] ATS: switch backends for and research and bienvenida.wm.org

https://gerrit.wikimedia.org/r/591306

Change 591306 merged by Dzahn:
[operations/puppet@production] ATS: switch backends for and research and bienvenida.wm.org

https://gerrit.wikimedia.org/r/591306

Change 592603 had a related patch set uploaded (by Dzahn; owner: Dzahn):
[operations/puppet@production] ATS: switch backends for transparency and transparency-private.wm.org

https://gerrit.wikimedia.org/r/592603

Change 592610 had a related patch set uploaded (by Dzahn; owner: Dzahn):
[operations/puppet@production] ATS: switch backends for design and sitemaps static sites

https://gerrit.wikimedia.org/r/592610

Change 592612 had a related patch set uploaded (by Dzahn; owner: Dzahn):
[operations/puppet@production] ATS: switch backends for wikiworkshop static sites

https://gerrit.wikimedia.org/r/592612

Mentioned in SAL (#wikimedia-operations) [2020-04-28T08:13:09Z] <mutante> rsyncing transparency-report-private files from bromine to miscweb1002/2002. git-cloning was removed about a year ago but site still exists. need to figure out if it should be deleted (T188362 T247650)

Change 592883 had a related patch set uploaded (by Dzahn; owner: Dzahn):
[operations/puppet@production] httpbb: add tests for miscweb sites

https://gerrit.wikimedia.org/r/592883

Change 592603 merged by Dzahn:
[operations/puppet@production] ATS: switch backends for the 3 transparency sites

https://gerrit.wikimedia.org/r/592603

Change 592610 merged by Dzahn:
[operations/puppet@production] ATS: switch backends for design and sitemaps static sites

https://gerrit.wikimedia.org/r/592610

Change 592980 had a related patch set uploaded (by Dzahn; owner: Dzahn):
[design/style-guide@master] update scap-targets from bromine/vega to miscweb1002/2002

https://gerrit.wikimedia.org/r/592980

Change 592980 abandoned by VolkerE:
update scap-targets from bromine/vega to miscweb1002/2002

Reason:
Done in https://github.com/wikimedia/WikimediaUI-Style-Guide/pull/369 instead

https://gerrit.wikimedia.org/r/592980

Change 592612 merged by Dzahn:
[operations/puppet@production] ATS: switch backends for wikiworkshop static sites

https://gerrit.wikimedia.org/r/592612

Change 593218 had a related patch set uploaded (by Dzahn; owner: Dzahn):
[operations/puppet@production] remove bromine.eqiad.wmnet and vega.codfw.wmnet

https://gerrit.wikimedia.org/r/593218

Change 593221 had a related patch set uploaded (by Dzahn; owner: Dzahn):
[operations/dns@master] decom bromine.eqiad.wmnet and vega.codfw.wmnet

https://gerrit.wikimedia.org/r/593221

Change 593218 merged by Dzahn:
[operations/puppet@production] remove bromine.eqiad, vega.codfw and webserver_misc_static role

https://gerrit.wikimedia.org/r/593218

Icinga downtime for 1 day, 0:00:00 set by dzahn@cumin1001 on 1 host(s) and their services with reason: decom

bromine.eqiad.wmnet

Icinga downtime for 1 day, 0:00:00 set by dzahn@cumin1001 on 1 host(s) and their services with reason: decom

vega.codfw.wmnet

cookbooks.sre.hosts.decommission executed by dzahn@cumin1001 for hosts: vega.codfw.wmnet

  • vega.codfw.wmnet (PASS)
    • Downtimed host on Icinga
    • Found Ganeti VM
    • VM shutdown
    • Started forced sync of VMs in Ganeti cluster ganeti01.svc.codfw.wmnet to Netbox
    • Removed from DebMonitor
    • Removed from Puppet master and PuppetDB
    • VM removed

cookbooks.sre.hosts.decommission executed by dzahn@cumin1001 for hosts: bromine.eqiad.wmnet

  • bromine.eqiad.wmnet (PASS)
    • Downtimed host on Icinga
    • Found Ganeti VM
    • VM shutdown
    • Started forced sync of VMs in Ganeti cluster ganeti01.svc.eqiad.wmnet to Netbox
    • Removed from DebMonitor
    • Removed from Puppet master and PuppetDB
    • VM removed

Change 593221 merged by Dzahn:
[operations/dns@master] decom bromine.eqiad.wmnet and vega.codfw.wmnet

https://gerrit.wikimedia.org/r/593221

Done! bromine and vega are gone and the services on it have been fully merged into miscweb1002/2002 which are on buster.

Change 592883 merged by Dzahn:
[operations/puppet@production] httpbb: add tests for miscweb sites

https://gerrit.wikimedia.org/r/592883