Page MenuHomePhabricator
Create Task
Maniphest T196019

setup/install phab1002(WMF4727)
Closed, ResolvedPublic
Actions

Description

xThis task will track the setup of spare pool system phab1002(WMF4727). This is a temp allocation, with the existing phab1001 being re-imaged and then failed back to.

phab1002(WMF4727):

  • - apply new hostname label (arguable if this is needed since it'll return to spares in less than a month.)
  • - bios/drac/serial setup/testing
  • - mgmt dns entries added for hostname (confirm asset tag entries exist)
  • - network port setup (description, enable, internal vlan)
  • - production dns entries added (all added, phab1002. phab1002-vcs, phab1002-aphlict, IPv6 records)
  • - operations/puppet update (install_server at minimum, other files if possible) (install_server done, added role spare, mapped IPv6)
  • - OS installation (done)
  • - puppet accept/initial run
  • - apply phabricator puppet role and confirm it works

Details

Show related patches Customize query in gerrit

Related Objects
Search...

Event Timeline

There are a very large number of changes, so older changes are hidden. Show Older Changes
gerritbot added a comment.May 30 2018, 9:08 PM

Change 436406 merged by RobH:
[operations/dns@master] phab1002 mgmt dns

https://gerrit.wikimedia.org/r/436406

Dzahn claimed this task.May 30 2018, 10:07 PM
RobH removed a project: Patch-For-Review.May 30 2018, 10:16 PM
RobH updated the task description. (Show Details)
gerritbot added a comment.May 30 2018, 10:21 PM

Change 435211 had a related patch set uploaded (by Dzahn; owner: Dzahn):
[operations/dns@master] assign wmf4727 as phab1002

https://gerrit.wikimedia.org/r/435211

gerritbot added a project: Patch-For-Review.May 30 2018, 10:21 PM
gerritbot added a comment.May 30 2018, 10:30 PM

Change 435211 abandoned by Dzahn:
assign wmf4727 as phab1002

https://gerrit.wikimedia.org/r/435211

gerritbot added a comment.May 31 2018, 8:21 PM

Change 436631 had a related patch set uploaded (by Dzahn; owner: Dzahn):
[operations/dns@master] assign 10.64.16.18 to phab1002.eqiad.wmnet

https://gerrit.wikimedia.org/r/436631

gerritbot added a comment.May 31 2018, 8:25 PM

Change 436631 merged by Dzahn:
[operations/dns@master] assign 10.64.16.18 to phab1002.eqiad.wmnet

https://gerrit.wikimedia.org/r/436631

Dzahn updated the task description. (Show Details)May 31 2018, 8:26 PM
gerritbot added a comment.May 31 2018, 8:49 PM

Change 436678 had a related patch set uploaded (by Dzahn; owner: Dzahn):
[operations/puppet@production] install_server: add phab1001 to DHCP,netboot

https://gerrit.wikimedia.org/r/436678

gerritbot added a comment.May 31 2018, 8:52 PM

Change 436678 merged by Dzahn:
[operations/puppet@production] install_server: add phab1002 to DHCP,netboot

https://gerrit.wikimedia.org/r/436678

Stashbot subscribed.May 31 2018, 9:00 PM

Mentioned in SAL (#wikimedia-operations) [2018-05-31T21:00:55Z] <mutante> dzahn@neodymium:~$ sudo wmf-auto-reimage-host --new phab1002.eqiad.wmnet (T196019)

gerritbot added a comment.May 31 2018, 9:22 PM

Change 436680 had a related patch set uploaded (by Dzahn; owner: Dzahn):
[operations/puppet@production] install_server: remove bast-test from DHCP

https://gerrit.wikimedia.org/r/436680

gerritbot added a comment.May 31 2018, 9:24 PM

Change 436680 merged by Dzahn:
[operations/puppet@production] install_server: remove bast-test from DHCP

https://gerrit.wikimedia.org/r/436680

gerritbot added a comment.May 31 2018, 9:26 PM

Change 436681 had a related patch set uploaded (by Dzahn; owner: Dzahn):
[operations/dns@master] remove bast-test.eqiad.wmnet

https://gerrit.wikimedia.org/r/436681

gerritbot added a comment.May 31 2018, 9:28 PM

Change 436681 merged by Dzahn:
[operations/dns@master] remove bast-test.eqiad.wmnet

https://gerrit.wikimedia.org/r/436681

gerritbot added a comment.May 31 2018, 9:51 PM

Change 436685 had a related patch set uploaded (by Dzahn; owner: Dzahn):
[operations/dns@master] fix IP address for phab1002, was wrong row

https://gerrit.wikimedia.org/r/436685

gerritbot added a comment.May 31 2018, 10:05 PM

Change 436685 merged by Dzahn:
[operations/dns@master] fix IP address for phab1002, was wrong row

https://gerrit.wikimedia.org/r/436685

Dzahn added a comment.May 31 2018, 11:19 PM

after wmf-auto-reimage-host said:

23:14:19 | phab1002.eqiad.wmnet | Still waiting for reboot after 45.0 minutes

i went to mgmt console to check myself and i saw:

Install the GRUB boot loader on a hard disk ├────────┐      
     │                                                                   │      
     │                     Installation step failed                      │      
     │ An installation step failed. You can try to run the failing item  │      
     │ again from the menu, or skip it and choose something else. The    │      
     │ failing step is: Install the GRUB boot loader on a hard disk
Dzahn added a comment.May 31 2018, 11:20 PM

in the installer i selected the very last step to install grub manually. next was:

[!!] Install the GRUB boot loader on a hard disk ├┐             
             │                                                    │             
  ┌──────────│         Unable to install GRUB in /dev/sda         │ ────────┐   
  │          │ Executing 'grub-install /dev/sda' failed.          │         │   
  │          │                                                    │         │   
  │          │ This is a fatal error.                             │         │   
  │ Running "│                                                    │         │   
  │          │     <Go Back>                       <Continue>     │         │   
  └──────────│                                                    │
Paladox added a comment.May 31 2018, 11:27 PM

sounds like it expected a ssd.

Dzahn added a comment.May 31 2018, 11:42 PM

See T189804 and T190093. This is the same spare machine that had this same issue before when i got it for bastion host replacement in the past. I reopened one of those.

Dzahn mentioned this in T189804: WMF4727 hardware issue - disks dont detect in installer.May 31 2018, 11:47 PM
gerritbot added a comment.Jun 1 2018, 12:09 AM

Change 436702 had a related patch set uploaded (by Dzahn; owner: Dzahn):
[operations/puppet@production] install_server: use raid1-gpt partman recipe for phab1002

https://gerrit.wikimedia.org/r/436702

gerritbot added a comment.Jun 1 2018, 12:13 AM

Change 436702 merged by Dzahn:
[operations/puppet@production] install_server: use raid1-gpt partman recipe for phab1002

https://gerrit.wikimedia.org/r/436702

Dzahn added a comment.Jun 1 2018, 12:35 AM

OS has been installed (with wmf-auto-reimage-host) after using raid1-gpt partman.

Debian GNU/Linux 9 phab1002 ttyS1




phab1002 login:



Dzahn updated the task description. (Show Details)Jun 1 2018, 12:36 AM
gerritbot added a comment.Jun 1 2018, 12:53 AM
Change 436708 had a related patch set uploaded (by Dzahn; owner: Dzahn):

[operations/puppet@production] site: add phab1002 with spare role and mapped IPv6



https://gerrit.wikimedia.org/r/436708
gerritbot added a comment.Jun 1 2018, 1:03 AM
Change 436708 merged by Dzahn:

[operations/puppet@production] site: add phab1002 with spare role and mapped IPv6



https://gerrit.wikimedia.org/r/436708
gerritbot added a comment.Jun 1 2018, 1:10 AM
Change 436709 had a related patch set uploaded (by Dzahn; owner: Dzahn):

[operations/dns@master] add IPv6 records for phab1002



https://gerrit.wikimedia.org/r/436709
gerritbot added a comment.Jun 1 2018, 1:11 AM
Change 436709 merged by Dzahn:

[operations/dns@master] add IPv6 records for phab1002



https://gerrit.wikimedia.org/r/436709
gerritbot added a comment.Jun 1 2018, 1:17 AM
Change 436710 had a related patch set uploaded (by Dzahn; owner: Dzahn):

[operations/dns@master] add aphlict and vcs service IPs for phab1002



https://gerrit.wikimedia.org/r/436710
gerritbot added a comment.Jun 1 2018, 1:24 AM
Change 436710 merged by Dzahn:

[operations/dns@master] add aphlict and vcs service IPs for phab1002



https://gerrit.wikimedia.org/r/436710
Dzahn updated the task description. (Show Details)Jun 1 2018, 1:25 AM
Dzahn updated the task description. (Show Details)
Dzahn updated the task description. (Show Details)Jun 1 2018, 1:28 AM
gerritbot added a comment.Jun 4 2018, 7:17 PM
Change 437300 had a related patch set uploaded (by Dzahn; owner: Dzahn):

[operations/puppet@production] phabricator: add role to node phab1002



https://gerrit.wikimedia.org/r/437300
• Cmjohnson moved this task from Backlog to Up next on the ops-eqiad board.Jun 5 2018, 6:16 PM
gerritbot added a comment.Jun 5 2018, 8:09 PM
Change 437558 had a related patch set uploaded (by Dzahn; owner: Dzahn):

[operations/puppet@production] dumps: add phab1002 as second phab server



https://gerrit.wikimedia.org/r/437558
ArielGlenn subscribed.Jun 5 2018, 8:19 PM
profile::dumps::distribution::datasets::fetcher  is what you want to fix. I need to go through and see which rsync confs I can kill. Later.
gerritbot added a comment.Jun 5 2018, 8:21 PM
Change 437613 had a related patch set uploaded (by Dzahn; owner: Dzahn):

[operations/puppet@production] mariadb: add phab1002 to phabricator grants



https://gerrit.wikimedia.org/r/437613
gerritbot added a comment.Jun 5 2018, 8:24 PM
Change 437615 had a related patch set uploaded (by Dzahn; owner: Dzahn):

[operations/puppet@production] hiera/phabricator: add phab1002 as phab server



https://gerrit.wikimedia.org/r/437615
gerritbot added a comment.Jun 5 2018, 8:42 PM
Change 437620 had a related patch set uploaded (by Dzahn; owner: Dzahn):

[operations/puppet@production] switch phabricator from phab1001 to phab1002



https://gerrit.wikimedia.org/r/437620
gerritbot added a comment.Jun 5 2018, 8:52 PM
Change 437625 had a related patch set uploaded (by Dzahn; owner: Dzahn):

[operations/puppet@production] mtail: replace phab1001 with phab1002?



https://gerrit.wikimedia.org/r/437625
• Cmjohnson added a comment.Jun 6 2018, 3:22 PM
@Dzahn I see phab1002 is installed and in icinga does the bios/drac/serial still need setup/testing
• Cmjohnson updated the task description. (Show Details)Jun 6 2018, 3:22 PM
• Cmjohnson moved this task from Up next to Blocked on the ops-eqiad board.
Dzahn added a comment.Jun 7 2018, 12:35 PM
@Cmjohnson i'm not sure, Rob created the task with the check boxes. I think from a template.
Dzahn added a comment.Jun 7 2018, 12:39 PM
I used wmf-auto-reimage so that was able to use the mgmt interface to install. I can also confirm i get a console. But i'm' not sure if anything else in BIOS needs to be checked.
Dzahn added a comment.Jun 7 2018, 12:42 PM
I don't have a specific problem with this instance but if this is a checkbox item that is supposed to happen on each re-assignment (like to set it back to the right BIOS setting defaults) then feel free to reboot that any time. The system isn't serving anything yet.
gerritbot added a comment.Jun 8 2018, 12:09 PM
Change 438235 had a related patch set uploaded (by Dzahn; owner: Dzahn):

[operations/puppet@production] phabricator: set service IPs for phab1002 in Hiera



https://gerrit.wikimedia.org/r/438235
gerritbot added a comment.Jun 8 2018, 12:15 PM
Change 438235 merged by Dzahn:

[operations/puppet@production] phabricator: set service IPs for phab1002 in Hiera



https://gerrit.wikimedia.org/r/438235
gerritbot added a comment.Jun 8 2018, 12:27 PM
Change 437300 merged by Dzahn:

[operations/puppet@production] phabricator: add role to node phab1002



https://gerrit.wikimedia.org/r/437300
gerritbot added a comment.Jun 8 2018, 12:36 PM
Change 437615 merged by Dzahn:

[operations/puppet@production] hiera/phabricator: add phab1002 as phab server



https://gerrit.wikimedia.org/r/437615
Dzahn added a comment.Jun 8 2018, 12:45 PM
role has been applied



these things are done:



https://gerrit.wikimedia.org/r/#/q/topic:phab1002+(status:merged)



but these are still needed:



https://gerrit.wikimedia.org/r/#/q/topic:phab1002+(status:open)



next we need to get the DB grants merged
gerritbot added a comment.Jun 8 2018, 1:41 PM
Change 437613 merged by Marostegui:

[operations/puppet@production] mariadb: add phab1002 to phabricator grants



https://gerrit.wikimedia.org/r/437613
gerritbot added a comment.Jun 11 2018, 8:24 AM
Change 437625 abandoned by Dzahn:

mtail: replace phab1001 with phab1002?



Reason:

host names for tests don't need to reflect actual machine names



https://gerrit.wikimedia.org/r/437625
Stashbot added a comment.Jun 11 2018, 10:52 AM
Mentioned in SAL (#wikimedia-operations) [2018-06-11T10:52:48Z] <mutante> phab1002 - editing cached scap config /srv/deployment/phabricator/deployment-cache/.config to replace tin.eqiad with deploy1001.eqiad deployment server, run puppet. other options: run scap with --refresh-config, delet cached .config file (T196019) (T175288)
Stashbot mentioned this in T175288: setup/install/deploy deploy1001 as deployment server.Jun 11 2018, 10:52 AM
Dzahn removed Dzahn as the assignee of this task.Jun 21 2018, 12:09 PM
for current status please see  T190568#4305040
• Vvjjkkii renamed this task from setup/install phab1002(WMF4727) to cybaaaaaaa.Jul 1 2018, 1:06 AM
• Vvjjkkii raised the priority of this task from Medium to High.
• Vvjjkkii added projects: CheckUser, Connected-Open-Heritage-Batch-uploads (RAÄ-KMB_1_2017-02), Tamil-Sites, Gamepress, Hashtags, Jade, KartoEditor, Language-2018-Apr-June, New-Editor-Experiences, Mail, TCB-Team (now WMDE-TechWish).
• Vvjjkkii updated the task description. (Show Details)
• Vvjjkkii removed subscribers: gerritbot, Aklapper.
ArielGlenn renamed this task from cybaaaaaaa to setup/install phab1002(WMF4727).Jul 1 2018, 6:59 AM
ArielGlenn lowered the priority of this task from High to Medium.
ArielGlenn removed projects: TCB-Team (now WMDE-TechWish), Mail, New-Editor-Experiences, Language-2018-Apr-June, KartoEditor, Jade, Hashtags, Gamepress, Tamil-Sites, Connected-Open-Heritage-Batch-uploads (RAÄ-KMB_1_2017-02), CheckUser.
ArielGlenn updated the task description. (Show Details)
ArielGlenn added subscribers: gerritbot, Aklapper.
Dzahn claimed this task.Jul 24 2018, 5:46 PM
Dzahn added a comment.Jan 24 2019, 9:03 PM
there has been a meeting  (https://wikitech.wikimedia.org/wiki/Phabricator/Meeting_Notes/2019-01-23)



and further progress happened at T190568#4907166 which resolved on of the follow-ups in it and unblocked testing the stretch server phab1002 installation which also stops using mod_php, has PHP 7 and so on.



T195623 is the original hardware request for this ticket and i reopened it because a blocker has been identified, which is "replacement server has 32GB but production server has 64GB RAM and we need 64GB".



So that's kind of stalled on this now.
Dzahn updated the task description. (Show Details)Jan 24 2019, 9:04 PM
I can't speak for the " - bios/drac/serial setup/testing" checkbox but the "add to operations puppet" and "make phabricator role work on stretch" part has been done. updated boxes accordingly.
Dzahn added a comment.Jan 24 2019, 9:07 PM
@RobH See above fyi and i am not sure if i should have reopened the original hardware request as i did on T195623#4903960 or if this ticket here is the most appropriate place because it is still open and about implementing the service on it. But the issue is we would a box with 64GB instead of 32GB RAM whichever way is easier. Let me know and i use the right ticket. Thank!
• Phabricator_maintenance moved this task from Backlog to Acknowledged on the SRE board.Jan 26 2019, 10:02 PM
• Cmjohnson updated the task description. (Show Details)Jan 30 2019, 10:13 PM
Dzahn mentioned this in T215335: requesting WMF7426 as phabricator system in eqiad.Feb 8 2019, 2:14 AM
Dzahn closed this task as Resolved.Mar 12 2019, 12:18 PM
Dzahn added a subtask: T215332: reimage WMF6937/mw1298.
re-closing.  replaced by new request in T215335. will be reverted in T215332
Dzahn added a subtask: T215335: requesting WMF7426 as phabricator system in eqiad.Apr 17 2019, 10:00 PM
RobH closed subtask T215335: requesting WMF7426 as phabricator system in eqiad as Resolved.Apr 18 2019, 4:28 PM
Dzahn added a subtask: T221391: decommission phab1002/WMF4727.Apr 18 2019, 7:14 PM
gerritbot added a comment.Apr 19 2019, 10:03 PM
Change 437558 had a related patch set uploaded (by Dzahn; owner: Dzahn):

[operations/puppet@production] dumps: switch phab1001->phab1003 as phab dumps source



https://gerrit.wikimedia.org/r/437558
gerritbot added a comment.Apr 23 2019, 10:00 PM
Change 437558 abandoned by Dzahn:

dumps: switch phab1001->phab1003 as phab dumps source



Reason:

merging into https://gerrit.wikimedia.org/r/c/operations/puppet/ /437620



https://gerrit.wikimedia.org/r/437558
gerritbot added a comment.May 22 2019, 5:53 PM
Change 511929 had a related patch set uploaded (by Dzahn; owner: Dzahn):

[operations/puppet@production] phabricator: activate read-only mode for maintenance



https://gerrit.wikimedia.org/r/511929
gerritbot added a comment.May 23 2019, 1:18 AM
Change 511929 merged by Dzahn:

[operations/puppet@production] phabricator: activate read-only mode for maintenance



https://gerrit.wikimedia.org/r/511929
Papaul closed subtask T221391: decommission phab1002/WMF4727 as Resolved.Oct 11 2019, 10:39 PM
• Cmjohnson closed subtask T215332: reimage WMF6937/mw1298 as Resolved.Nov 13 2019, 5:12 PM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL