Page MenuHomePhabricator
Create Task
Maniphest T247441

move 20 new codfw parsoid servers (parse2*) into production
Closed, ResolvedPublic
Actions

Description

parse2001 through parse2020 have been racked in the parent task.

This is the ticket to get them into production.

  • - service implementer changes from 'staged' status to 'active' status in netbox'

Details

Related Changes in Gerrit:
SubjectRepoBranchLines +/-
move 20 new codfw parsoid servers into productionoperations/puppetproduction+21 -1
add new parse* servers to $wgLinterSubmitterWhitelistoperations/mediawiki-configmaster+20 -0
add new parsoid servers to conftool-dataoperations/puppetproduction+20 -0
site: add parsoid role to all new parsoid hardwareoperations/puppetproduction+2 -7
add fake keys for new parsoid hostslabs/privatemaster+0 -0
site: add parsoid role to parsoid2020.codfw.wmnetoperations/puppetproduction+6 -1
Customize query in gerrit

Related Objects
Search...

Event Timeline

Dzahn created this task.Mar 11 2020, 7:03 PM
gerritbot added a comment.Mar 11 2020, 7:04 PM

Change 579026 had a related patch set uploaded (by Dzahn; owner: Dzahn):
[operations/puppet@production] add new codfw parsoid servers into production

https://gerrit.wikimedia.org/r/579026

gerritbot added a project: Patch-For-Review.Mar 11 2020, 7:04 PM
Dzahn triaged this task as Medium priority.Jul 8 2020, 12:07 AM
Dzahn mentioned this in T243112: (Need by: TBD) codfw: rack/setup/install parse200[1-20].codfw.wmnet.Jul 15 2020, 6:40 PM
Dzahn removed a parent task: T243112: (Need by: TBD) codfw: rack/setup/install parse200[1-20].codfw.wmnet.Jul 15 2020, 6:42 PM
Dzahn added a subtask: T243112: (Need by: TBD) codfw: rack/setup/install parse200[1-20].codfw.wmnet.
Papaul updated the task description. (Show Details)Jul 15 2020, 6:57 PM
Papaul closed subtask T243112: (Need by: TBD) codfw: rack/setup/install parse200[1-20].codfw.wmnet as Resolved.
jijiki moved this task from Incoming 🐫 to cross-team active on the serviceops board.Aug 17 2020, 11:45 PM
gerritbot added a comment.Sep 10 2020, 8:46 PM

Change 626496 had a related patch set uploaded (by Dzahn; owner: Dzahn):
[operations/puppet@production] site: add parsoid role to parsoid2020.codfw.wmnet

https://gerrit.wikimedia.org/r/626496

gerritbot added a comment.Sep 10 2020, 8:49 PM

Change 626496 merged by Dzahn:
[operations/puppet@production] site: add parsoid role to parsoid2020.codfw.wmnet

https://gerrit.wikimedia.org/r/626496

gerritbot added a comment.Sep 11 2020, 12:29 AM

Change 626521 had a related patch set uploaded (by Dzahn; owner: Dzahn):
[operations/puppet@production] site: add parsoid role to all new parsoid hardware

https://gerrit.wikimedia.org/r/626521

Stashbot added a comment.Sep 11 2020, 12:38 AM

Mentioned in SAL (#wikimedia-operations) [2020-09-11T00:38:46Z] <mutante> generating mcrouter certs for parse2001 - parse2019 - mcrouter_generate_certs on puppetmaster1001 (T247441)

gerritbot added a comment.Sep 11 2020, 12:45 AM

Change 626523 had a related patch set uploaded (by Dzahn; owner: Dzahn):
[labs/private@master] add fake keys for new parsoid hosts

https://gerrit.wikimedia.org/r/626523

gerritbot added a comment.Sep 11 2020, 12:46 AM

Change 626523 merged by Dzahn:
[labs/private@master] add fake keys for new parsoid hosts

https://gerrit.wikimedia.org/r/626523

Dzahn mentioned this in rLPRI76adf72e4445: add fake keys for new parsoid hosts.Sep 11 2020, 12:56 AM
gerritbot added a comment.Sep 11 2020, 1:01 AM

Change 626521 merged by Dzahn:
[operations/puppet@production] site: add parsoid role to all new parsoid hardware

https://gerrit.wikimedia.org/r/626521

Stashbot added a comment.Sep 11 2020, 1:33 AM

Mentioned in SAL (#wikimedia-operations) [2020-09-11T01:33:20Z] <mutante> initial puppet runs on parse2001 - parse2010, staggered, not in production yet, new hardware, setup WIP (T247441)

Stashbot added a comment.Sep 11 2020, 1:53 AM

Mentioned in SAL (#wikimedia-operations) [2020-09-11T01:53:31Z] <mutante> initial puppet runs on parse2010 - parse2020, staggered, not in production yet, new hardware, setup WIP (T247441)

gerritbot added a comment.Sep 11 2020, 4:35 PM

Change 626719 had a related patch set uploaded (by Dzahn; owner: Dzahn):
[operations/mediawiki-config@master] add new parse* servers to $wgLinterSubmitterWhitelist

https://gerrit.wikimedia.org/r/626719

gerritbot added a comment.Sep 11 2020, 4:49 PM

Change 626721 had a related patch set uploaded (by Dzahn; owner: Dzahn):
[operations/puppet@production] add new parsoid servers to conftool-data

https://gerrit.wikimedia.org/r/626721

Dzahn raised the priority of this task from Medium to High.Sep 11 2020, 5:31 PM
jijiki added a comment.Sep 13 2020, 5:44 AM

The DSH group alert fired on Saturday, I acked it:)

Dzahn added a comment.Sep 13 2020, 2:38 PM

Thank you, i thought i had done 48 hour downtimes on everything but obviously it failed somehow.

gerritbot added a comment.Sep 14 2020, 3:52 PM

Change 626721 merged by Dzahn:
[operations/puppet@production] add new parsoid servers to conftool-data

https://gerrit.wikimedia.org/r/626721

Stashbot added a comment.Sep 14 2020, 4:36 PM

Mentioned in SAL (#wikimedia-operations) [2020-09-14T16:36:05Z] <mutante> pooled the first of the new parsoid servers - parse2001 (T247441)

Stashbot added a comment.Sep 14 2020, 5:51 PM

Mentioned in SAL (#wikimedia-operations) [2020-09-14T17:51:01Z] <mutante> all new parse* parsoid hardware pooled now and set to active in netbox, deploy in 10 min will add to $wgLinterSubmitterWhitelist (T247441)

gerritbot added a comment.Sep 14 2020, 6:18 PM

Change 626719 merged by jenkins-bot:
[operations/mediawiki-config@master] add new parse* servers to $wgLinterSubmitterWhitelist

https://gerrit.wikimedia.org/r/626719

Stashbot added a comment.Sep 14 2020, 6:21 PM

Mentioned in SAL (#wikimedia-operations) [2020-09-14T18:21:21Z] <urbanecm@deploy1001> Synchronized wmf-config/InitialiseSettings.php: 27ba5a1da1fb00e721cfa82dd4cd1fbac2541053: add new parse* servers to $wgLinterSubmitterWhitelist (T247441) (duration: 00m 56s)

Dzahn removed a project: Patch-For-Review.Sep 14 2020, 8:50 PM
Dzahn updated the task description. (Show Details)
Dzahn closed this task as Resolved.Sep 14 2020, 8:58 PM
  • servers had OS installed
  • servers had puppet role applied
  • icinga checks confirmed all green
  • added to conftool data
  • set weight for all to 10
  • pooled from inactive to no to add to pybal config
  • pooled first one, then 10, then all
  • set to active in netbox
  • added to $wgLinterSubmitterWhitelist in MediaWiki config and deployed to prod
  • confirmed in grafana new servers are getting traffic, CPU/memory load going down on old servers
gerritbot added a comment.Sep 16 2020, 6:20 PM

Change 579026 abandoned by Dzahn:
[operations/puppet@production] move 20 new codfw parsoid servers into production

Reason:
duplicate - already done in other patches

https://gerrit.wikimedia.org/r/579026

Papaul mentioned this in T257903: decom wtp2005 (was: wtp2005 hardware issue).Sep 23 2020, 12:10 AM
Dzahn added a comment.Oct 14 2020, 11:29 PM

There was a request to decom the wtp2* servers. There is a new ticket for that at T265558.

Dzahn mentioned this in T245888: Rename wtp* servers to parse* (Parsoid PHP servers).Oct 28 2020, 4:21 PM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits