Page MenuHomePhabricator
Create Task
Maniphest T265653

(Need By: TBD) setup/install deploy1002
Closed, ResolvedPublic
Actions

Description

This task will track the racking, setup, and OS installation of deploy1002. This host was on the budget for this year, but instead was allocated from spare pool system

Hostname / Racking / Installation Details

This is already racked, so it will only need minimal updates from on-site.

Per host setup checklist

Each host should have its own setup checklist copied and pasted into the list below.

deploy1002/wmf5177:

  • - ops-eqiad apply hostname labels to fron/back of host
  • - bios/drac/serial setup/testing/firmware updates (this was outdated with old ilom user info and firmwares)
  • - mgmt dns entries added for both asset tag and hostname
  • - network port setup (description, enable, vlan)
    • end on-site specific steps
  • - production dns entries added
  • - operations/puppet update (install_server at minimum, other files if possible)
  • - OS installation
  • - puppet accept/initial run
  • - host state in netbox set to staged

Once the system(s) above have had all checkbox steps completed, this task can be resolved.

Details

SubjectRepoBranchLines +/-
updating deploy1002 to insetupoperations/puppetproduction+5 -1
adding deploy1002 to site.ppoperations/puppetproduction+1 -1
deploy1002 prod dnsoperations/dnsmaster+4 -1
deploy1002 mac infooperations/puppetproduction+6 -0
Customize query in gerrit

Related Objects
Search...

StatusSubtypeAssignedTask
 ResolvedNoneT248925 Make MediaWiki release tarball compatible with PHP 8.0
 ResolvedJdforrester-WMFT300463 Make PHP 8.0 voting on MW master
 ResolvedJdforrester-WMFT313563 Bump lcobucci/jwt & league/uri-components for php8
 ResolvedJdforrester-WMFT313564 Bump onoi/message-reporter in vendor.git to 1.4.2 for php 8 support
 ResolvedJdforrester-WMFT247658 Make Wikimedia Production MediaWiki compatible with PHP 7.4
 Resolved• toanT243590 Fix WikibaseDataModel CI for php 7.4
 ResolvedLucas_Werkmeister_WMDET316923 Restore skipped test in ReferenceListTest.php
 ResolvedJoeT318918 Undeploy patch to use old PHP serialization in PHP 7.4
 ResolvedJdforrester-WMFT264168 Drop PHP 7.2 support from Wikibase master branch, once Wikimedia production is on 7.4
 ResolvedLadsgroupT270740 Drop hacky support for doctrine/dbal class renaming
 InvalidNoneT303505 [S] Remove Deprecated methods "serialize" and "unserialize" after php production upgrade to PHP 7.4
 ResolvedReedyT251043 Cleanup css-sanitizer when it only requires PHP >= 7.4
 OpenNoneT166010 The Great Namespaceization Effort
 ResolvedtstarlingT277618 var_dump() on various objects writes gigabytes of data and takes minutes to run
 ResolvedJdforrester-WMFT261872 Drop PHP 7.2 & 7.3 support from MediaWiki master branch, once Wikimedia production is on 7.4
 StalledNoneT302086 Set scap minimum python version to 3.7
 ResolvedNoneT247045 Migrate all of production metal and VMs to Buster or later
 ResolvedPRODUCTION ERRORLegoktmT293568 PHP Notice: Undefined offset in wikimedia/remex-html when rendering rest.php error page
 ResolvedtstarlingT297667 mysqli/mysqlnd memory leak
 ResolvedJoeT271736 Migrate WMF production from PHP 7.2 to PHP 7.4
 ResolvedKrinkleT248191 Can't reopen table in wikidb-unittest_ (from SpecialPageFatalTest)
 ResolvedhasharT278203 Migrate all CI jobs from stretch to buster or later and drop stretch testing support
 ResolvedJdforrester-WMFT252432 Drop MediaWiki testing in stretch and instead test only in buster
 ResolvedDzahnT245757 Upgrade MediaWiki clusters to Debian Buster (debian 10)
 ResolvedDzahnT265963 Replace production deployment servers and update them to Buster
Unknown Object (Task)
 Resolved• CmjohnsonT265653 (Need By: TBD) setup/install deploy1002

Event Timeline

RobH created this task.Oct 15 2020, 7:18 PM
Restricted Application added a project: SRE. · View Herald TranscriptOct 15 2020, 7:18 PM
RobH claimed this task.Oct 15 2020, 7:18 PM
RobH added a parent task: Unknown Object (Task).
RobH updated the task description. (Show Details)Oct 15 2020, 7:36 PM
gerritbot added a comment.Oct 15 2020, 8:04 PM

Change 634333 had a related patch set uploaded (by RobH; owner: RobH):
[operations/puppet@production] deploy1002 mac info

https://gerrit.wikimedia.org/r/634333

gerritbot added a project: Patch-For-Review.Oct 15 2020, 8:04 PM
RobH reassigned this task from RobH to Dzahn.Oct 15 2020, 8:05 PM
RobH moved this task from Backlog to Racking Tasks on the ops-eqiad board.
RobH added a subscriber: • Cmjohnson.

setup notes:

  • this had the old idrac login info, perhaps due to it being 'inventory' in netbox and not included in ilom user updates months ago?
  • old idrac firmware 3.30.30.30, updated to 4.22.55.53
  • bios firmware was 1.7.0, updated to 2.8.2
  • updated puppet lease file with mac address

pending question before imaging script is run:

Please note even after imaging is done, this task will need to be reassigned to @Cmjohnson so he can apply hostname labels.

RobH updated the task description. (Show Details)Oct 15 2020, 9:25 PM
Dzahn added a comment.Oct 15 2020, 11:16 PM

@RobH Please leave it on stretch for now. It relies on Mediawiki classes that are not ready for buster just yet.

Dzahn reassigned this task from Dzahn to RobH.Oct 15 2020, 11:16 PM
Dzahn subscribed.
gerritbot added a comment.Oct 16 2020, 6:44 PM

Change 634333 merged by RobH:
[operations/puppet@production] deploy1002 mac info

https://gerrit.wikimedia.org/r/634333

gerritbot added a comment.Oct 16 2020, 6:53 PM

Change 634583 had a related patch set uploaded (by RobH; owner: RobH):
[operations/dns@master] deploy1002 prod dns

https://gerrit.wikimedia.org/r/634583

gerritbot added a comment.Oct 16 2020, 6:54 PM

Change 634583 merged by RobH:
[operations/dns@master] deploy1002 prod dns

https://gerrit.wikimedia.org/r/634583

ops-monitoring-bot added a comment.Oct 16 2020, 7:24 PM

Script wmf-auto-reimage was launched by robh on cumin1001.eqiad.wmnet for hosts:

['deploy1002.eqiad.wmnet']

The log can be found in /var/log/wmf-auto-reimage/202010161924_robh_31297.log.

RobH removed a project: Patch-For-Review.Oct 16 2020, 7:28 PM
RobH updated the task description. (Show Details)
ops-monitoring-bot added a comment.Oct 16 2020, 7:35 PM

Completed auto-reimage of hosts:

['deploy1002.eqiad.wmnet']

Of which those FAILED:

['deploy1002.eqiad.wmnet']
gerritbot added a comment.Oct 16 2020, 8:09 PM

Change 634598 had a related patch set uploaded (by RobH; owner: RobH):
[operations/puppet@production] adding deploy1002 to site.pp

https://gerrit.wikimedia.org/r/634598

gerritbot added a project: Patch-For-Review.Oct 16 2020, 8:09 PM

Change 634598 merged by RobH:
[operations/puppet@production] adding deploy1002 to site.pp

https://gerrit.wikimedia.org/r/634598

ops-monitoring-bot added a comment.Oct 16 2020, 8:12 PM

Script wmf-auto-reimage was launched by robh on cumin1001.eqiad.wmnet for hosts:

['deploy1002.eqiad.wmnet']

The log can be found in /var/log/wmf-auto-reimage/202010162012_robh_7688.log.

ops-monitoring-bot added a comment.Oct 16 2020, 8:24 PM

Completed auto-reimage of hosts:

['deploy1002.eqiad.wmnet']

Of which those FAILED:

['deploy1002.eqiad.wmnet']
RobH reassigned this task from RobH to Dzahn.Oct 16 2020, 8:29 PM

This fails reimage due to the initial puppet run failing. Not sure if we should apply a different role, or if you want to take over and reimage from here.

Please note that when the image is done, this needs to be reassigned to @Cmjohnson to apply the hostname label.

Volans reassigned this task from Dzahn to RobH.Oct 17 2020, 7:19 AM
Volans raised the priority of this task from Medium to High.
Volans subscribed.

The IPs were allocated manually outside of Netbox and as such they could be allocated to a different host by Netbox in any upcoming provisioning causing conflicts.

Please always follow the procedure for provisioning as described in the wikitech pages below (all interinked):

In this specific case I think that if you manage to run successfully the reimage script it will fix Netbox too but I will need to double check the end result to make sure it's all correct.
In any case the data must be fixed ASAP and before any new provisioning in eqiad happen.

Re-assigning to @RobH

Dzahn added a comment.Oct 17 2020, 3:25 PM

@RobH Please do not assign service puppet roles on new hosts. That will almost never work. Just add new hosts in site with the "insetup" role and hand over. Service implementation will be to apply the role along with other changes in Hiera etc.

gerritbot added a comment.Oct 19 2020, 4:01 PM

Change 635011 had a related patch set uploaded (by RobH; owner: RobH):
[operations/puppet@production] updating deploy1002 to insetup

https://gerrit.wikimedia.org/r/635011

gerritbot added a comment.Oct 19 2020, 4:02 PM

Change 635011 merged by RobH:
[operations/puppet@production] updating deploy1002 to insetup

https://gerrit.wikimedia.org/r/635011

ops-monitoring-bot added a comment.Oct 19 2020, 4:03 PM

Script wmf-auto-reimage was launched by robh on cumin1001.eqiad.wmnet for hosts:

['deploy1002.eqiad.wmnet']

The log can be found in /var/log/wmf-auto-reimage/202010191603_robh_1113.log.

ops-monitoring-bot added a comment.Oct 19 2020, 4:24 PM

Completed auto-reimage of hosts:

['deploy1002.eqiad.wmnet']

Of which those FAILED:

['deploy1002.eqiad.wmnet']
ops-monitoring-bot added a comment.Oct 19 2020, 4:32 PM

Script wmf-auto-reimage was launched by robh on cumin1001.eqiad.wmnet for hosts:

['deploy1002.eqiad.wmnet']

The log can be found in /var/log/wmf-auto-reimage/202010191632_robh_29964.log.

ops-monitoring-bot added a comment.Oct 19 2020, 4:38 PM

Completed auto-reimage of hosts:

['deploy1002.eqiad.wmnet']

and were ALL successful.

RobH closed this task as Resolved.Oct 19 2020, 7:06 PM
RobH reopened this task as Open.
RobH reassigned this task from RobH to • Cmjohnson.
RobH removed a project: Patch-For-Review.
RobH updated the task description. (Show Details)

I shouldn't have resolved, hostname label has to go on.

@Cmjohnson: Once the hostname label is applied to deploy1002, this can be resolved: https://netbox.wikimedia.org/dcim/devices/2139/

C8 - U17 - WMF5177 - deploy1002

Dzahn mentioned this in T265963: Replace production deployment servers and update them to Buster.Oct 19 2020, 11:51 PM
Dzahn added a parent task: T265963: Replace production deployment servers and update them to Buster.
RobH unsubscribed.Oct 20 2020, 3:31 PM
• Cmjohnson moved this task from Racking Tasks to Backlog on the ops-eqiad board.Oct 29 2020, 8:08 PM
• Cmjohnson closed this task as Resolved.Oct 30 2020, 4:21 PM

done

RhinosF1 subscribed.Oct 30 2020, 4:26 PM
taavi mentioned this in T278689: Upgrade deployment-prep deploy* hosts to Buster.Mar 29 2021, 11:38 AM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL