Page MenuHomePhabricator

Setup wikitech, horizon, and striker on new labweb hardware
Closed, ResolvedPublicPRODUCTION ERROR

Description

The new labweb boxes are hardware refresh for silver and californium that are currently hosting wikitech (silver), and horizon + striker (californium). When we set up the new hardware we would like to add some redundancy to these hosts to make maintenance easier. The long term plan is to move wikitech to the production wiki cluster (T161859), but that is a bigger project with some unknowns in the roadmap.

  • Image labweb* as Debian Stretch
  • Deploy horizon to labweb* via scap3
  • Deploy striker to labweb* via scap3
  • Move wikitech's database from silver to the prod (misc) db cluster
  • Run wikitech on HHVM
  • Load balance wikitech across labweb* behind misc-varnish
  • Update wikitech-static sync process to pull from the new wikitech cluster

Details

SubjectRepoBranchLines +/-
operations/mediawiki-configmaster+13 -41
operations/puppetproduction+17 -66
operations/puppetproduction+3 -4
operations/puppetproduction+158 -3
operations/puppetproduction+0 -7
operations/puppetproduction+0 -0
operations/puppetproduction+4 -4
operations/puppetproduction+0 -2
operations/dnsmaster+2 -2
operations/puppetproduction+2 -3
operations/puppetproduction+1 -5
operations/puppetproduction+0 -1
operations/puppetproduction+19 -24
operations/puppetproduction+3 -20
operations/dnsmaster+1 -2
operations/puppetproduction+4 -2
operations/puppetproduction+3 -0
operations/mediawiki-configmaster+1 -0
operations/puppetproduction+10 -0
operations/puppetproduction+31 -45
operations/puppetproduction+4 -2
operations/dnsmaster+1 -0
operations/puppetproduction+239 -5
operations/puppetproduction+10 -0
operations/puppetproduction+11 -3
operations/puppetproduction+19 -9
operations/puppetproduction+6 -0
operations/puppetproduction+11 -1
operations/puppetproduction+17 -0
operations/puppetproduction+15 -0
operations/puppetproduction+1 K -2
operations/puppetproduction+1 K -3
Show related patches Customize query in gerrit

Related Objects

StatusSubtypeAssignedTask
Resolvedbd808
ResolvedJdforrester-WMF
ResolvedRobH
ResolvedJclark-ctr
ResolvedPRODUCTION ERRORAndrew
ResolvedAndrew
ResolvedAndrew
ResolvedAndrew
ResolvedAndrew
ResolvedAndrew
ResolvedAndrew
Resolvedbd808
ResolvedAndrew
ResolvedAndrew
ResolvedAndrew
ResolvedAndrew
ResolvedAndrew
ResolvedAndrew
ResolvedAndrew
ResolvedAndrew
ResolvedAndrew
ResolvedAndrew
ResolvedAndrew
ResolvedAndrew
ResolvedPRODUCTION ERRORAndrew

Event Timeline

There are a very large number of changes, so older changes are hidden. Show Older Changes

Change 409108 had a related patch set uploaded (by Andrew Bogott; owner: Andrew Bogott):
[operations/puppet@production] keystone: allow api access to labweb hosts

https://gerrit.wikimedia.org/r/409108

Change 409108 merged by Andrew Bogott:
[operations/puppet@production] keystone: allow api access to labweb hosts

https://gerrit.wikimedia.org/r/409108

Change 409118 had a related patch set uploaded (by Andrew Bogott; owner: Andrew Bogott):
[operations/puppet@production] keystone: allow api access to labweb hosts

https://gerrit.wikimedia.org/r/409118

Change 409118 merged by Andrew Bogott:
[operations/puppet@production] keystone: allow api access to labweb hosts

https://gerrit.wikimedia.org/r/409118

Change 409119 had a related patch set uploaded (by Andrew Bogott; owner: Andrew Bogott):
[operations/puppet@production] designate: allow labweb hosts access to designate api

https://gerrit.wikimedia.org/r/409119

Change 409119 merged by Andrew Bogott:
[operations/puppet@production] designate: allow labweb hosts access to designate api

https://gerrit.wikimedia.org/r/409119

Change 412705 had a related patch set uploaded (by Andrew Bogott; owner: Andrew Bogott):
[operations/puppet@production] create new (transitional) 'newtoolsadmin.wikimedia.org' host on labweb1001/1002

https://gerrit.wikimedia.org/r/412705

Change 412705 merged by Andrew Bogott:
[operations/puppet@production] create new (transitional) 'newtoolsadmin.wikimedia.org' host on labweb1001/1002

https://gerrit.wikimedia.org/r/412705

Change 412729 had a related patch set uploaded (by Andrew Bogott; owner: Andrew Bogott):
[operations/puppet@production] striker: alternate package requirements for Debian

https://gerrit.wikimedia.org/r/412729

Change 412729 merged by Andrew Bogott:
[operations/puppet@production] striker: alternate package requirements for Debian

https://gerrit.wikimedia.org/r/412729

Change 412964 had a related patch set uploaded (by Andrew Bogott; owner: Andrew Bogott):
[operations/puppet@production] m5: update db grants for new labweb services

https://gerrit.wikimedia.org/r/412964

Change 412970 had a related patch set uploaded (by Andrew Bogott; owner: Andrew Bogott):
[operations/puppet@production] m5: add ferm rules for new labweb hosts

https://gerrit.wikimedia.org/r/412970

Change 412964 had a related patch set uploaded (by Andrew Bogott; owner: Andrew Bogott):
[operations/puppet@production] m5: update db grants for new labweb services

https://gerrit.wikimedia.org/r/412964

Change 413748 had a related patch set uploaded (by Andrew Bogott; owner: Andrew Bogott):
[operations/puppet@production] m5: remove grants for Californium

https://gerrit.wikimedia.org/r/413748

Change 412964 merged by Andrew Bogott:
[operations/puppet@production] m5: update db grants for new labweb services

https://gerrit.wikimedia.org/r/412964

Change 412970 merged by Andrew Bogott:
[operations/puppet@production] m5: add ferm rules for new labweb hosts

https://gerrit.wikimedia.org/r/412970

Change 415019 had a related patch set uploaded (by Andrew Bogott; owner: Andrew Bogott):
[operations/puppet@production] labweb: include mediawiki profiles

https://gerrit.wikimedia.org/r/415019

Change 415019 merged by Andrew Bogott:
[operations/puppet@production] labweb: include mediawiki profiles

https://gerrit.wikimedia.org/r/415019

Change 415645 had a related patch set uploaded (by Andrew Bogott; owner: Andrew Bogott):
[operations/dns@master] add new misc-web ip for 'newwikitech.wikimedia.org'

https://gerrit.wikimedia.org/r/415645

Change 415647 had a related patch set uploaded (by Andrew Bogott; owner: Andrew Bogott):
[operations/puppet@production] labweb: set up temporary 'newwikitech.wikimedia.org' host

https://gerrit.wikimedia.org/r/415647

Change 415645 merged by Andrew Bogott:
[operations/dns@master] add new misc-web ip for 'newwikitech.wikimedia.org'

https://gerrit.wikimedia.org/r/415645

Change 415647 merged by Andrew Bogott:
[operations/puppet@production] labweb: set up temporary 'newwikitech.wikimedia.org' host

https://gerrit.wikimedia.org/r/415647

Change 415913 had a related patch set uploaded (by Andrew Bogott; owner: Andrew Bogott):
[operations/puppet@production] labweb: hhvm-enabled vhost for the new wikitech

https://gerrit.wikimedia.org/r/415913

Change 415914 had a related patch set uploaded (by Andrew Bogott; owner: Andrew Bogott):
[operations/mediawiki-config@master] multiversion: add a transitional mapping for newwikitech.wikimedia.org

https://gerrit.wikimedia.org/r/415914

Change 415913 merged by Andrew Bogott:
[operations/puppet@production] labweb: hhvm-enabled vhost for the new wikitech

https://gerrit.wikimedia.org/r/415913

Aside from the comment here T188029#4017938 we are detecting a database up and running on silver, you may want to check that, and change puppet or kill the package/datadir; https://tendril.wikimedia.org/host/view/silver.wikimedia.org/3306

Change 416235 had a related patch set uploaded (by Andrew Bogott; owner: Andrew Bogott):
[operations/puppet@production] labweb wikitech: add a few more apache confs

https://gerrit.wikimedia.org/r/416235

Change 416235 merged by Andrew Bogott:
[operations/puppet@production] labweb wikitech: add a few more apache confs

https://gerrit.wikimedia.org/r/416235

Change 415914 merged by jenkins-bot:
[operations/mediawiki-config@master] multiversion: add a transitional mapping for newwikitech.wikimedia.org

https://gerrit.wikimedia.org/r/415914

Change 417265 had a related patch set uploaded (by Andrew Bogott; owner: Andrew Bogott):
[operations/mediawiki-config@master] wikitech: on labweb, make mediawiki aware that it's behind varnishes

https://gerrit.wikimedia.org/r/417265

Change 417309 had a related patch set uploaded (by Andrew Bogott; owner: Andrew Bogott):
[operations/puppet@production] mediawiki scap: add labweb1001 and 1002 targets

https://gerrit.wikimedia.org/r/417309

Change 417309 merged by Andrew Bogott:
[operations/puppet@production] mediawiki scap: add labweb1001 and 1002 targets

https://gerrit.wikimedia.org/r/417309

Change 417925 had a related patch set uploaded (by Andrew Bogott; owner: Andrew Bogott):
[operations/puppet@production] labweb wikitech: change hostname from 'newwikitech' to 'wikitech'

https://gerrit.wikimedia.org/r/417925

Change 417926 had a related patch set uploaded (by Andrew Bogott; owner: Andrew Bogott):
[operations/dns@master] wikitech: move from silver to misc-web backed by labweb

https://gerrit.wikimedia.org/r/417926

Change 417925 merged by Andrew Bogott:
[operations/puppet@production] labweb wikitech: change hostname from 'newwikitech' to 'wikitech'

https://gerrit.wikimedia.org/r/417925

Change 417926 merged by Andrew Bogott:
[operations/dns@master] wikitech: move from silver to misc-web backed by labweb

https://gerrit.wikimedia.org/r/417926

Change 418057 had a related patch set uploaded (by Andrew Bogott; owner: Andrew Bogott):
[operations/puppet@production] wikitech: move wikitech-static backup job from silver to labweb1001

https://gerrit.wikimedia.org/r/418057

Change 418057 merged by Andrew Bogott:
[operations/puppet@production] wikitech: move wikitech-static backup job from silver to labweb1001

https://gerrit.wikimedia.org/r/418057

Change 418071 had a related patch set uploaded (by Andrew Bogott; owner: Andrew Bogott):
[operations/puppet@production] wikitech-static sync: tidy up the dump process

https://gerrit.wikimedia.org/r/418071

Change 418071 merged by Andrew Bogott:
[operations/puppet@production] wikitech-static sync: tidy up the dump process

https://gerrit.wikimedia.org/r/418071

Change 418130 had a related patch set uploaded (by Andrew Bogott; owner: Andrew Bogott):
[operations/puppet@production] wikitech-static-sync: Fix vhost for modern Apache

https://gerrit.wikimedia.org/r/418130

Change 418130 merged by Andrew Bogott:
[operations/puppet@production] wikitech-static-sync: Fix vhost for modern Apache

https://gerrit.wikimedia.org/r/418130

Change 418932 had a related patch set uploaded (by Andrew Bogott; owner: Andrew Bogott):
[operations/puppet@production] labweb: add wikitech-static monitoring to labweb hosts

https://gerrit.wikimedia.org/r/418932

Change 418932 merged by Andrew Bogott:
[operations/puppet@production] labweb: add wikitech-static monitoring to labweb hosts

https://gerrit.wikimedia.org/r/418932

Change 418941 had a related patch set uploaded (by Andrew Bogott; owner: Andrew Bogott):
[operations/puppet@production] silver: remove wikitech, mark as spare

https://gerrit.wikimedia.org/r/418941

Change 418941 merged by Andrew Bogott:
[operations/puppet@production] silver: remove wikitech, mark as spare

https://gerrit.wikimedia.org/r/418941

Change 419180 had a related patch set uploaded (by Andrew Bogott; owner: Andrew Bogott):
[operations/puppet@production] labtestweb: refactor to more closely resemble the labweb* deploy

https://gerrit.wikimedia.org/r/419180

Change 419222 had a related patch set uploaded (by Andrew Bogott; owner: Andrew Bogott):
[operations/dns@master] shorten ttl for horizon.wm.o and toolsadmin.wm.o

https://gerrit.wikimedia.org/r/419222

Change 419222 abandoned by Andrew Bogott:
shorten ttl for horizon.wm.o and toolsadmin.wm.o

Reason:
This is silly -- these boxes are staying on misc-web, it's just the backend that will change.

https://gerrit.wikimedia.org/r/419222

Change 419224 had a related patch set uploaded (by Andrew Bogott; owner: Andrew Bogott):
[operations/puppet@production] Remove misc-web config for 'newwikitech'

https://gerrit.wikimedia.org/r/419224

Change 419225 had a related patch set uploaded (by Andrew Bogott; owner: Andrew Bogott):
[operations/puppet@production] Rename newhorizon and newtoolsadmin to horizon and toolsadmin

https://gerrit.wikimedia.org/r/419225

Change 419226 had a related patch set uploaded (by Andrew Bogott; owner: Andrew Bogott):
[operations/puppet@production] Move horizon and toolsadmin to labweb backends

https://gerrit.wikimedia.org/r/419226

Change 419224 merged by Andrew Bogott:
[operations/puppet@production] Remove misc-web config for 'newwikitech'

https://gerrit.wikimedia.org/r/419224

Change 419225 merged by Andrew Bogott:
[operations/puppet@production] Rename newhorizon and newtoolsadmin to horizon and toolsadmin

https://gerrit.wikimedia.org/r/419225

Change 419420 had a related patch set uploaded (by Andrew Bogott; owner: Andrew Bogott):
[operations/puppet@production] labweb: move 'newhorizon' vhost to 'horizon'

https://gerrit.wikimedia.org/r/419420

Change 419420 merged by Andrew Bogott:
[operations/puppet@production] labweb: move 'newhorizon' vhost to 'horizon'

https://gerrit.wikimedia.org/r/419420

Change 419226 merged by Andrew Bogott:
[operations/puppet@production] Move horizon and toolsadmin to labweb backends

https://gerrit.wikimedia.org/r/419226

Change 419180 merged by Andrew Bogott:
[operations/puppet@production] labtestweb: refactor to more closely resemble the labweb* deploy

https://gerrit.wikimedia.org/r/419180

Change 419534 had a related patch set uploaded (by Andrew Bogott; owner: Andrew Bogott):
[operations/puppet@production] californium: mark as spare system

https://gerrit.wikimedia.org/r/419534

Change 419534 merged by Andrew Bogott:
[operations/puppet@production] californium: mark as spare system

https://gerrit.wikimedia.org/r/419534

Dzahn subscribed.

@Andrew Question for decom. Does this have to be updated?

hieradata/common/profile/openstack/main.yaml:profile::openstack::main::horizon_host: 'californium.wikimedia.org'

Change 420805 had a related patch set uploaded (by Andrew Bogott; owner: Andrew Bogott):
[operations/mediawiki-config@master] Post-silver cleanups

https://gerrit.wikimedia.org/r/420805

Change 420908 had a related patch set uploaded (by Andrew Bogott; owner: Andrew Bogott):
[operations/puppet@production] Horizon: remove singular horizon_host hiera setting and all uses

https://gerrit.wikimedia.org/r/420908

Can you apply the same config patch from wikitech to labtestwikitech.wikimedia.org so it doesn't try to check every single codfw s4 host replication status and fail?

Change 420908 merged by Andrew Bogott:
[operations/puppet@production] Horizon: remove singular horizon_host hiera setting and all uses

https://gerrit.wikimedia.org/r/420908

Change 420805 merged by jenkins-bot:
[operations/mediawiki-config@master] Post-silver cleanups

https://gerrit.wikimedia.org/r/420805

mmodell changed the subtype of this task from "Task" to "Production Error".Aug 28 2019, 11:10 PM