Page MenuHomePhabricator
Create Task
Maniphest T245567

(Need by: 2020-03-01) rack/setup/install htmldumper1001.eqiad.wmnet.
Closed, ResolvedPublic
Actions

Description

This task will track the racking, setup, and OS installation of htmldumper1001.eqiad.wmnet.

Hostname / Racking / Installation Details

Hostnames: htmldumper1001
Racking Proposal: Can go anywhere
Networking/Subnet/VLAN/IP: Internal network, 1G is fine
Partitioning/Raid: Raid 10 for the /srv disks but we can just use francium's partman recipe

Per host setup checklist

Each host should have its own setup checklist copied and pasted into the list below.

htmldumper1001:

  • - receive in system on procurement task T242009
  • - rack system with proposed racking plan (see above) & update netbox (include all system info plus location, state of planned)
  • - bios/drac/serial setup/testing
  • - mgmt dns entries added for both asset tag and hostname
  • - network port setup (description, enable, vlan)
    • end on-site specific steps
  • - production dns entries added
  • - operations/puppet update (install_server at minimum, other files if possible)
  • - OS installation
  • - puppet accept/initial run (with role:spare)
  • - host state in netbox set to staged

Once the system(s) above have had all checkbox steps completed, this task can be resolved.

Details

SubjectRepoBranchLines +/-
dumps: switch htmldumps server from francium to htmldumper1001operations/puppetproduction+2 -2
site: update comment on htmldumper1001operations/puppetproduction+2 -1
site: add role(dumps::web::htmldumps) to htmldumper1001operations/puppetproduction+1 -1
updating dns/asset tag names to reflect correct servers htmldumper/fran1001operations/dnsmaster+4 -4
update mac address for htmldumper1001operations/puppetproduction+1 -1
use new role(insetup) on a few hosts in setupoperations/puppetproduction+3 -3
Add htmldumper1001 to site.ppoperations/puppetproduction+5 -0
Add htmldumper1001 to dhcpd file and netboot.cfgoperations/puppetproduction+6 -0
Add production dns for htmldumper1001operations/dnsmaster+2 -0
Add mgmt dns for htmldumper1001operations/dnsmaster+4 -0
Customize query in gerrit

Related Objects
Search...

StatusSubtypeAssignedTask
Unknown Object (Task)
 ResolvedArielGlennT245567 (Need by: 2020-03-01) rack/setup/install htmldumper1001.eqiad.wmnet.

Event Timeline

RobH created this task.Feb 18 2020, 10:39 PM
Restricted Application added a project: SRE. · View Herald TranscriptFeb 18 2020, 10:39 PM
RobH added a parent task: Unknown Object (Task).Feb 18 2020, 10:39 PM
RobH moved this task from Backlog to Racking Tasks on the ops-eqiad board.
wiki_willy renamed this task from (2020-03-01) rack/setup/install htmldumper1001.eqiad.wmnet. to (Need By 2020-03-01) rack/setup/install htmldumper1001.eqiad.wmnet..Feb 19 2020, 2:11 AM
wiki_willy renamed this task from (Need By 2020-03-01) rack/setup/install htmldumper1001.eqiad.wmnet. to (Need by: 2020-03-01) rack/setup/install htmldumper1001.eqiad.wmnet..Feb 24 2020, 8:26 PM
RobH unsubscribed.Feb 24 2020, 9:04 PM
gerritbot added a comment.Mar 10 2020, 7:42 PM

Change 578590 had a related patch set uploaded (by Cmjohnson; owner: Cmjohnson):
[operations/dns@master] Add mgmt dns for htmldumper1001

https://gerrit.wikimedia.org/r/578590

gerritbot added a project: Patch-For-Review.Mar 10 2020, 7:42 PM
• Cmjohnson updated the task description. (Show Details)Mar 10 2020, 7:43 PM
gerritbot added a comment.Mar 10 2020, 11:05 PM

Change 578590 merged by Cmjohnson:
[operations/dns@master] Add mgmt dns for htmldumper1001

https://gerrit.wikimedia.org/r/578590

• Cmjohnson updated the task description. (Show Details)Mar 10 2020, 11:08 PM
Maintenance_bot removed a project: Patch-For-Review.Mar 10 2020, 11:11 PM
gerritbot added a comment.Mar 10 2020, 11:17 PM

Change 578640 had a related patch set uploaded (by Cmjohnson; owner: Cmjohnson):
[operations/dns@master] Add production dns for htmldumper1001

https://gerrit.wikimedia.org/r/578640

gerritbot added a project: Patch-For-Review.Mar 10 2020, 11:17 PM

Change 578640 merged by Cmjohnson:
[operations/dns@master] Add production dns for htmldumper1001

https://gerrit.wikimedia.org/r/578640

• Cmjohnson updated the task description. (Show Details)Mar 10 2020, 11:19 PM
Maintenance_bot removed a project: Patch-For-Review.Mar 11 2020, 12:10 AM
gerritbot added a comment.Mar 11 2020, 4:52 PM

Change 578989 had a related patch set uploaded (by Cmjohnson; owner: Cmjohnson):
[operations/puppet@production] Add htmldumper1001 to dhcpd file and netboot.cfg

https://gerrit.wikimedia.org/r/578989

gerritbot added a project: Patch-For-Review.Mar 11 2020, 4:52 PM

Change 578990 had a related patch set uploaded (by Cmjohnson; owner: Cmjohnson):
[operations/puppet@production] Add htmldumper1001 to site.pp

https://gerrit.wikimedia.org/r/578990

gerritbot added a comment.Mar 11 2020, 4:54 PM

Change 578989 merged by Cmjohnson:
[operations/puppet@production] Add htmldumper1001 to dhcpd file and netboot.cfg

https://gerrit.wikimedia.org/r/578989

gerritbot added a comment.Mar 11 2020, 4:55 PM

Change 578990 merged by Cmjohnson:
[operations/puppet@production] Add htmldumper1001 to site.pp

https://gerrit.wikimedia.org/r/578990

Maintenance_bot removed a project: Patch-For-Review.Mar 11 2020, 5:10 PM
gerritbot added a comment.Mar 11 2020, 5:24 PM

Change 578996 had a related patch set uploaded (by Dzahn; owner: Dzahn):
[operations/puppet@production] use new role(insetup) on a few hosts in setup

https://gerrit.wikimedia.org/r/578996

gerritbot added a project: Patch-For-Review.Mar 11 2020, 5:24 PM

Change 578996 merged by Dzahn:
[operations/puppet@production] use new role(insetup) on a few hosts in setup

https://gerrit.wikimedia.org/r/578996

Maintenance_bot removed a project: Patch-For-Review.Mar 11 2020, 6:10 PM
• Cmjohnson updated the task description. (Show Details)Mar 11 2020, 6:57 PM
ops-monitoring-bot added a comment.Mar 11 2020, 7:28 PM

Script wmf-auto-reimage was launched by cmjohnson on cumin1001.eqiad.wmnet for hosts:

htmldumper1001.eqiad.wmnet

The log can be found in /var/log/wmf-auto-reimage/202003111928_cmjohnson_3633_htmldumper1001_eqiad_wmnet.log.

gerritbot added a comment.Mar 12 2020, 3:02 PM

Change 579286 had a related patch set uploaded (by Cmjohnson; owner: Cmjohnson):
[operations/puppet@production] update mac address for htmldumper1001

https://gerrit.wikimedia.org/r/579286

gerritbot added a project: Patch-For-Review.Mar 12 2020, 3:02 PM

Change 579286 merged by Cmjohnson:
[operations/puppet@production] update mac address for htmldumper1001

https://gerrit.wikimedia.org/r/579286

gerritbot added a comment.Mar 12 2020, 3:09 PM

Change 579288 had a related patch set uploaded (by Cmjohnson; owner: Cmjohnson):
[operations/dns@master] updating dns/asset tag names to reflect correct servers htmldumper/fran1001

https://gerrit.wikimedia.org/r/579288

gerritbot added a comment.Mar 12 2020, 3:10 PM

Change 579288 merged by Cmjohnson:
[operations/dns@master] updating dns/asset tag names to reflect correct servers htmldumper/fran1001

https://gerrit.wikimedia.org/r/579288

Maintenance_bot removed a project: Patch-For-Review.Mar 12 2020, 3:11 PM
ops-monitoring-bot added a comment.Mar 12 2020, 3:13 PM

Script wmf-auto-reimage was launched by cmjohnson on cumin1001.eqiad.wmnet for hosts:

htmldumper1001.eqiad.wmnet

The log can be found in /var/log/wmf-auto-reimage/202003121513_cmjohnson_203633_htmldumper1001_eqiad_wmnet.log.

ops-monitoring-bot added a comment.Mar 12 2020, 3:20 PM

Completed auto-reimage of hosts:

['htmldumper1001.eqiad.wmnet']

Of which those FAILED:

['htmldumper1001.eqiad.wmnet']
ops-monitoring-bot added a comment.Mar 12 2020, 3:23 PM

Script wmf-auto-reimage was launched by cmjohnson on cumin1001.eqiad.wmnet for hosts:

htmldumper1001.eqiad.wmnet

The log can be found in /var/log/wmf-auto-reimage/202003121523_cmjohnson_205338_htmldumper1001_eqiad_wmnet.log.

ops-monitoring-bot added a comment.Mar 12 2020, 4:11 PM

Script wmf-auto-reimage was launched by cmjohnson on cumin1001.eqiad.wmnet for hosts:

htmldumper1001.eqiad.wmnet

The log can be found in /var/log/wmf-auto-reimage/202003121611_cmjohnson_213880_htmldumper1001_eqiad_wmnet.log.

• Cmjohnson updated the task description. (Show Details)Mar 12 2020, 4:53 PM
• Cmjohnson subscribed.

all set, resolving

• Cmjohnson closed this task as Resolved.Mar 12 2020, 4:53 PM
Dzahn reopened this task as Open.Mar 13 2020, 12:30 AM
Dzahn subscribed.

There are a lot of Icinga alerts about all the things on htmldumper1001 for some reason:

https://icinga.wikimedia.org/cgi-bin/icinga/status.cgi?host=htmldumper1001

Dzahn added a comment.Mar 13 2020, 12:32 AM

I suspected it just needs restart of nagios-nrpe-server but i also can't SSH to it. I get asked for password.

ayounsi subscribed.EditedMar 26 2020, 7:50 AM

I don't think it got puppetized properly, https://netbox.wikimedia.org/extras/reports/puppetdb.PhysicalHosts/ alerts as missing physical device in PuppetDB: state Staged in Netbox.

Set its state back to "planned".

Dzahn reassigned this task from Jclark-ctr to • Cmjohnson.Mar 30 2020, 2:24 PM
Dzahn added a subscriber: Jclark-ctr.

Hi Chris, can this be fixed from remote? The host is in an odd state. It exists but we can't SSH to it or use install_console either.

ops-monitoring-bot added a comment.Mar 30 2020, 7:29 PM

Script wmf-auto-reimage was launched by cmjohnson on cumin1001.eqiad.wmnet for hosts:

htmldumper1001.eqiad.wmnet

The log can be found in /var/log/wmf-auto-reimage/202003301929_cmjohnson_251747_htmldumper1001_eqiad_wmnet.log.

ops-monitoring-bot added a comment.Mar 30 2020, 8:08 PM

Completed auto-reimage of hosts:

['htmldumper1001.eqiad.wmnet']

and were ALL successful.

• Cmjohnson closed this task as Resolved.Mar 30 2020, 9:58 PM

@Dzahn I reimaged and am now able to login

cmjohnson@Bolts2 ~ % ssh htmldumper1001.eqiad.wmnet
Linux htmldumper1001 4.19.0-8-amd64 #1 SMP Debian 4.19.98-1 (2020-01-26) x86_64
Debian GNU/Linux 10 (buster)
htmldumper1001 is a Host being setup for later application of a role (insetup)
The last Puppet run was at Mon Mar 30 21:06:42 UTC 2020 (20 minutes ago).
Last puppet commit: (290d2ed2a5) John Bond - cloude tlsproxy::envoy: remove ~ default
Debian GNU/Linux 10 auto-installed on Mon Mar 30 19:49:54 UTC 2020.
Last login: Mon Mar 30 20:14:26 2020 from 208.80.154.86

Dzahn added a comment.Apr 2 2020, 7:10 PM
In T245567#6012810, @Cmjohnson wrote:

@Dzahn I reimaged and am now able to login

@Cmjohnson Yes, i can login now. Thank you! :)

Dzahn added a comment.Apr 2 2020, 7:18 PM

@ArielGlenn htmldumper1001 is now usable. Any idea what kind of role we want on it?

ArielGlenn added a comment.Apr 3 2020, 6:48 AM

It should get role(dumps::web::htmldumps) like francium. It doesn't hurt to have multiple hosts with this role.

ayounsi unsubscribed.Apr 3 2020, 7:06 AM
gerritbot added a comment.Apr 3 2020, 8:06 AM

Change 585688 had a related patch set uploaded (by Dzahn; owner: Dzahn):
[operations/puppet@production] site: add role(dumps::web::htmldumps) to htmldumper1001

https://gerrit.wikimedia.org/r/585688

gerritbot added a project: Patch-For-Review.Apr 3 2020, 8:06 AM
gerritbot added a comment.Apr 3 2020, 8:10 AM

Change 585688 merged by Dzahn:
[operations/puppet@production] site: add role(dumps::web::htmldumps) to htmldumper1001

https://gerrit.wikimedia.org/r/585688

Maintenance_bot removed a project: Patch-For-Review.Apr 3 2020, 8:26 AM
Dzahn added a comment.Apr 3 2020, 8:35 AM
In T245567#6025491, @ArielGlenn wrote:

It should get role(dumps::web::htmldumps)

Done. Applied and puppet ran and no puppet failure.

ArielGlenn added a comment.Apr 3 2020, 8:36 AM

Thanks so much everyone!

Dzahn reopened this task as Open.Apr 3 2020, 8:37 AM
Dzahn reassigned this task from • Cmjohnson to ArielGlenn.
• Cmjohnson closed this task as Resolved.Apr 3 2020, 5:18 PM
Dzahn mentioned this in T273142: decommission francium.eqiad.wmnet.Jan 28 2021, 5:47 PM
gerritbot added a comment.Jan 28 2021, 5:51 PM

Change 659341 had a related patch set uploaded (by Dzahn; owner: Dzahn):
[operations/puppet@production] site: update comment on htmldumper1001

https://gerrit.wikimedia.org/r/659341

gerritbot added a project: Patch-For-Review.Jan 28 2021, 5:51 PM
gerritbot added a comment.Jan 28 2021, 5:59 PM

Change 659349 had a related patch set uploaded (by Dzahn; owner: Dzahn):
[operations/puppet@production] dumps: switch htmldumps server from francium to htmldumper1001

https://gerrit.wikimedia.org/r/659349

Dzahn added a comment.Jan 28 2021, 6:06 PM

@ArielGlenn hi! a task to decom francium has been created but I noticed it seems to still be in config. So I uploaded patches to actually switch it. But I have no idea how easy or hard it is to actually do that. If that should have a separate task to move it into production, happy to make one.

gerritbot added a comment.Jan 29 2021, 7:11 PM

Change 659341 merged by Dzahn:
[operations/puppet@production] site: update comment on htmldumper1001

https://gerrit.wikimedia.org/r/659341

gerritbot added a comment.Jan 29 2021, 7:21 PM

Change 659349 merged by Dzahn:
[operations/puppet@production] dumps: switch htmldumps server from francium to htmldumper1001

https://gerrit.wikimedia.org/r/659349

Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL