Page MenuHomePhabricator

upgrade and rename krypton & create its codfw equivalent
Closed, ResolvedPublic

Description

The server krypton.eqiad.wmnet is a "webserver_misc_apps".

Things we should probably do:

  • upgrade from jessie to stretch / switch to PHP 7
  • create an equivalent of it in codfw for failover | miscweb2001 is up
  • pick a "cluster" name for them instead of a misc host name and rename them to something1001/2001 | it is called "miscweb", created on wiki and in puppet

services hosted here that are moving to miscweb1001:

Details

Related Gerrit Patches:
operations/puppet : productionracktables: use codfw database when in codfw
operations/puppet : productioniegreview app: use codfw database when in codfw
operations/puppet : productionwikimania_scholarships app: use codfw database when in codfw
operations/puppet : productionracktables: set db host in Hiera, set to eqiad, use lookup
operations/puppet : productionwikimania_scholarships: set db host in Hiera
operations/puppet : productioniegreview: set db host in Hiera
operations/puppet : productionsite/install_server: decom krypton.eqiad.wmnet
operations/puppet : productionmisc_apps::httpd: remove jessie/php5 support
operations/puppet : productionwikimania_scholarships: remove jessie/php5 support
operations/puppet : productionATS/varnish: switch wikimania scholarships to miscweb, use TLS
operations/puppet : productionATS: configure "never-cache" for webserver-misc-apps.discovery.wmnet
operations/puppet : productioniegreview: remove jessie support
operations/puppet : productionracktables: remove jessie support
operations/puppet : productionATS/varnish: switch iegreview to miscweb backend and use TLS
operations/puppet : productionATS/varnish: replace krypton with miscweb1001, rename director
operations/puppet : productionATS/varnish: add director miscweb and switch racktables to it
operations/puppet : productioniegreview: require a mysql client to be installed
operations/puppet : productionmisc_apps::httpd: allow port 80 from deployment servers
operations/puppet : productionmisc_apps::httpd: do not load SSL httpd module
operations/puppet : productioninstall_server: switch miscweb servers to buster installer
operations/puppet : productionlogstash: replace krypton with grafana1001 in collector ferm rules
operations/puppet : productionsite: add miscweb role to miscweb1001
operations/puppet : productioninstall_server: add miscweb1001 to DHCP
operations/dns : masteradd IP for miscweb1001
operations/puppet : productionadd miscweb2001 to role webserver_misc_apps in site.pp
operations/puppet : productioncumin: add alias for new miscweb* cluster name
operations/puppet : productionnetboot: add miscweb[12]00[12] to partman

Related Objects

Event Timeline

There are a very large number of changes, so older changes are hidden. Show Older Changes

switch to PHP 7.2 (T224194)

We did it for Phabricator for some feature readded in 7.1 and for the main wikis for performance reasons, but for random misc services let's not, there's no benefit. If mediawiki moves to something post 7.2, we'd be stuck with all those misc servers
still.

pick a "cluster" name for them instead of a misc host name and rename them to something1001/2001

miscweb1001/2001?

Change 512446 had a related patch set uploaded (by Dzahn; owner: Dzahn):
[operations/dns@master] rename krypton to miscweb1001

https://gerrit.wikimedia.org/r/512446

Change 513215 had a related patch set uploaded (by Dzahn; owner: Dzahn):
[operations/puppet@production] netboot: add miscweb[12]00[12] to partman

https://gerrit.wikimedia.org/r/513215

Change 513215 merged by Dzahn:
[operations/puppet@production] netboot: add miscweb[12]00[12] to partman

https://gerrit.wikimedia.org/r/513215

Change 513227 had a related patch set uploaded (by Dzahn; owner: Dzahn):
[operations/puppet@production] cumin: add alias for new miscweb* cluster name

https://gerrit.wikimedia.org/r/513227

Change 513230 had a related patch set uploaded (by Dzahn; owner: Dzahn):
[operations/puppet@production] add miscweb2001 to role webserver_misc_apps in site.pp

https://gerrit.wikimedia.org/r/513230

Change 513227 merged by Dzahn:
[operations/puppet@production] cumin: add alias for new miscweb* cluster name

https://gerrit.wikimedia.org/r/513227

Change 513230 merged by Dzahn:
[operations/puppet@production] add miscweb2001 to role webserver_misc_apps in site.pp

https://gerrit.wikimedia.org/r/513230

Dzahn raised the priority of this task from Medium to High.Aug 27 2019, 9:41 AM

Change 512446 merged by Dzahn:
[operations/dns@master] add IP for miscweb1001

https://gerrit.wikimedia.org/r/512446

Change 532683 had a related patch set uploaded (by Dzahn; owner: Dzahn):
[operations/puppet@production] site: add miscweb role to miscweb1001

https://gerrit.wikimedia.org/r/532683

Dzahn updated the task description. (Show Details)Aug 27 2019, 10:17 AM

Mentioned in SAL (#wikimedia-operations) [2019-08-27T10:25:05Z] <mutante> ganeti eqiad - creating new VM with same specs as krypton to replace it with a stretch instance and mirror miscweb2001. krypton to be removed (T224323, T105507, T224247)

Change 532687 had a related patch set uploaded (by Dzahn; owner: Dzahn):
[operations/puppet@production] install_server: add miscweb1001 to DHCP

https://gerrit.wikimedia.org/r/532687

Change 532687 merged by Dzahn:
[operations/puppet@production] install_server: add miscweb1001 to DHCP

https://gerrit.wikimedia.org/r/532687

Change 532683 merged by Dzahn:
[operations/puppet@production] site: add miscweb role to miscweb1001

https://gerrit.wikimedia.org/r/532683

Mentioned in SAL (#wikimedia-operations) [2019-08-27T11:51:31Z] <mutante> miscweb1001 - manually remove tin.eqiad.wmnet (!) from /srv/iegreview/iegreview-cache/.config and replace with deploy1001 after first puppet run. still existing bug that tin is not fully removed (T224247, T175288, T197470)

Mentioned in SAL (#wikimedia-operations) [2019-08-27T11:54:56Z] <mutante> miscweb1001 - a2dismod mpm_event ; a2enmod php7.0 ; systemctl restart apache2 (T224247, T196968) please also see https://gerrit.wikimedia.org/r/c/operations/puppet/+/451206

Change 532695 had a related patch set uploaded (by Dzahn; owner: Dzahn):
[operations/puppet@production] trafficserver/varnish: replace krypton with miscweb1001, rename director

https://gerrit.wikimedia.org/r/532695

Change 532701 had a related patch set uploaded (by Dzahn; owner: Dzahn):
[operations/puppet@production] site/install_server: remote krypton.eqiad.wmnet

https://gerrit.wikimedia.org/r/532701

Change 532702 had a related patch set uploaded (by Dzahn; owner: Dzahn):
[operations/puppet@production] logstash: replace krypton with grafana1001 in collector ferm rules

https://gerrit.wikimedia.org/r/532702

Change 532702 merged by Dzahn:
[operations/puppet@production] logstash: replace krypton with grafana1001 in collector ferm rules

https://gerrit.wikimedia.org/r/532702

Change 532862 had a related patch set uploaded (by Dzahn; owner: Dzahn):
[operations/puppet@production] install_server: switch miscweb servers to buster installer

https://gerrit.wikimedia.org/r/532862

Change 532862 merged by Dzahn:
[operations/puppet@production] install_server: switch miscweb servers to buster installer

https://gerrit.wikimedia.org/r/532862

Change 532948 had a related patch set uploaded (by Dzahn; owner: Dzahn):
[operations/puppet@production] misc_apps::httpd: do not load SSL httpd module

https://gerrit.wikimedia.org/r/532948

Change 532948 merged by Dzahn:
[operations/puppet@production] misc_apps::httpd: do not load SSL httpd module

https://gerrit.wikimedia.org/r/532948

Change 532954 had a related patch set uploaded (by Dzahn; owner: Dzahn):
[operations/puppet@production] misc_apps::httpd: allow port 80 from deployment servers

https://gerrit.wikimedia.org/r/532954

Change 532954 merged by Dzahn:
[operations/puppet@production] misc_apps::httpd: allow port 80 from deployment servers

https://gerrit.wikimedia.org/r/532954

Change 532959 had a related patch set uploaded (by Dzahn; owner: Dzahn):
[operations/puppet@production] iegreview: require a mysql client to be installed

https://gerrit.wikimedia.org/r/532959

Change 532959 merged by Dzahn:
[operations/puppet@production] iegreview: require a mysql client to be installed

https://gerrit.wikimedia.org/r/532959

Change 533024 had a related patch set uploaded (by Dzahn; owner: Dzahn):
[operations/puppet@production] ATS/varnish: add director miscweb and switch racktables to it

https://gerrit.wikimedia.org/r/533024

Change 533024 merged by Dzahn:
[operations/puppet@production] ATS/varnish: add director miscweb and switch racktables to it

https://gerrit.wikimedia.org/r/533024

Change 533154 had a related patch set uploaded (by Dzahn; owner: Dzahn):
[operations/puppet@production] ATS/varnish: switch iegreview to miscweb backend and use TLS

https://gerrit.wikimedia.org/r/533154

Change 533155 had a related patch set uploaded (by Dzahn; owner: Dzahn):
[operations/puppet@production] racktables: remove jessie support

https://gerrit.wikimedia.org/r/533155

Change 533157 had a related patch set uploaded (by Dzahn; owner: Dzahn):
[operations/puppet@production] iegreview: remove jessie support

https://gerrit.wikimedia.org/r/533157

Change 533158 had a related patch set uploaded (by Dzahn; owner: Dzahn):
[operations/puppet@production] wikimania_scholarships: remove jessie/php5 support

https://gerrit.wikimedia.org/r/533158

Change 532695 abandoned by Dzahn:
ATS/varnish: replace krypton with miscweb1001, rename director

Reason:
doing in separate changes instead (https://gerrit.wikimedia.org/r/c/operations/puppet/ /533024, https://gerrit.wikimedia.org/r/c/operations/puppet/ /533154 )

https://gerrit.wikimedia.org/r/532695

Change 533159 had a related patch set uploaded (by Dzahn; owner: Dzahn):
[operations/puppet@production] misc_apps::httpd: remove jessie/php5 support

https://gerrit.wikimedia.org/r/533159

Change 533154 merged by Dzahn:
[operations/puppet@production] ATS/varnish: switch iegreview to miscweb backend and use TLS

https://gerrit.wikimedia.org/r/533154

Change 533155 merged by Dzahn:
[operations/puppet@production] racktables: remove jessie support

https://gerrit.wikimedia.org/r/533155

Change 533157 merged by Dzahn:
[operations/puppet@production] iegreview: remove jessie support

https://gerrit.wikimedia.org/r/533157

Change 533175 had a related patch set uploaded (by Dzahn; owner: Dzahn):
[operations/puppet@production] ATS/varnish: switch scholarschips to miscweb and use TLS

https://gerrit.wikimedia.org/r/533175

Change 533181 had a related patch set uploaded (by Dzahn; owner: Dzahn):
[operations/puppet@production] ATS: configure "never_cache" for miscweb1001 backend

https://gerrit.wikimedia.org/r/533181

Change 533181 merged by Dzahn:
[operations/puppet@production] ATS: configure "never-cache" for webserver-misc-apps.discovery.wmnet

https://gerrit.wikimedia.org/r/533181

Change 533175 merged by Dzahn:
[operations/puppet@production] ATS/varnish: switch wikimania scholarships to miscweb, use TLS

https://gerrit.wikimedia.org/r/533175

Change 533158 merged by Dzahn:
[operations/puppet@production] wikimania_scholarships: remove jessie/php5 support

https://gerrit.wikimedia.org/r/533158

Change 533159 merged by Dzahn:
[operations/puppet@production] misc_apps::httpd: remove jessie/php5 support

https://gerrit.wikimedia.org/r/533159

Change 532701 merged by Dzahn:
[operations/puppet@production] site/install_server: decom krypton.eqiad.wmnet

https://gerrit.wikimedia.org/r/532701

Dzahn updated the task description. (Show Details)Aug 29 2019, 12:22 PM
Dzahn removed a project: Patch-For-Review.
Dzahn updated the task description. (Show Details)

The following services have been moved away from krypton miscweb1001.

  • new DNS discovery record 'webserver-misc-apps.discovery.wmnet is an alias for miscweb1001.eqiad.wmnet
  • miscweb1001.eqiad.wmnet is on stretch (because we don't have the envoy package for buster just yet but coming soon)
  • using PHP7 instead of PHP5, PHP5 support in puppet code removed
  • using envoy for TLS termination on backend, https:// url in ATS config for this backend
  • equivalent of backend server in codfw exists as miscweb2001.codfw.wmnet and also the DBA connection should not be an issue anymore, though untested

TODO: DB config needs to use codfw proxy if puppet role applied on codfw node

Joe moved this task from Backlog to Doing on the serviceops board.Sep 16 2019, 3:02 PM

krypton has been decom'ed fully

Change 537761 had a related patch set uploaded (by Dzahn; owner: Dzahn):
[operations/puppet@production] racktables: set db host in Hiera, set to eqiad

https://gerrit.wikimedia.org/r/537761

Change 537762 had a related patch set uploaded (by Dzahn; owner: Dzahn):
[operations/puppet@production] iegreview: set db host in Hiera

https://gerrit.wikimedia.org/r/537762

Change 537763 had a related patch set uploaded (by Dzahn; owner: Dzahn):
[operations/puppet@production] wikimania_scholarships: set db host in Hiera

https://gerrit.wikimedia.org/r/537763

Change 537762 merged by Dzahn:
[operations/puppet@production] iegreview: set db host in Hiera

https://gerrit.wikimedia.org/r/537762

Change 537763 merged by Dzahn:
[operations/puppet@production] wikimania_scholarships: set db host in Hiera

https://gerrit.wikimedia.org/r/537763

Change 537761 merged by Dzahn:
[operations/puppet@production] racktables: set db host in Hiera, set to eqiad, use lookup

https://gerrit.wikimedia.org/r/537761

akosiaris closed this task as Resolved.Nov 20 2019, 8:31 AM
akosiaris added a subscriber: akosiaris.

krypton is no more since 7a36b4e7a94f486a400f0363c263c446c33bba80, resolving.

Dzahn reopened this task as Open.Nov 20 2019, 1:19 PM
Dzahn added a comment.Nov 22 2019, 5:51 PM

That's true, just had one last little todo here for the one in codfw. Doing that now.

Change 552551 had a related patch set uploaded (by Dzahn; owner: Dzahn):
[operations/puppet@production] wikimania_scholarships app: use codfw database when in codfw

https://gerrit.wikimedia.org/r/552551

Change 552552 had a related patch set uploaded (by Dzahn; owner: Dzahn):
[operations/puppet@production] iegreview app: use codfw database when in codfw

https://gerrit.wikimedia.org/r/552552

Change 552553 had a related patch set uploaded (by Dzahn; owner: Dzahn):
[operations/puppet@production] racktables: use codfw database when in codfw

https://gerrit.wikimedia.org/r/552553

Change 552551 merged by Dzahn:
[operations/puppet@production] wikimania_scholarships app: use codfw database when in codfw

https://gerrit.wikimedia.org/r/552551

Change 552552 merged by Dzahn:
[operations/puppet@production] iegreview app: use codfw database when in codfw

https://gerrit.wikimedia.org/r/552552

Change 552553 merged by Dzahn:
[operations/puppet@production] racktables: use codfw database when in codfw

https://gerrit.wikimedia.org/r/552553

Dzahn closed this task as Resolved.Nov 25 2019, 10:45 PM
Dzahn removed a project: Patch-For-Review.