Page MenuHomePhabricator
Create Task
Maniphest T180700

Rack and setup db1109 and db1110
Closed, ResolvedPublic
Actions

Description

This task will track the racking/setup/installation of 2 new databases (db1109 and db1110) for eqiad. These were ordered on T177488.
If possible rack the server in the following locations

db1109: D8
db1110: C3

db1109

  • receive in system on procurement task T177488
  • bios/drac/serial setup/testing
  • raid set up to raid 10 (standard db raid)
  • mgmt dns entries added for both asset tag and hostname
  • production dns entries added
  • network port setup
  • operations/puppet update
  • OS installation
  • puppet/salt accept/initial run
  • handoff for service implementation

db1110

  • receive in system on procurement task T177488
  • bios/drac/serial setup/testing
  • raid set up to raid 10 (standard db raid)
  • mgmt dns entries added for both asset tag and hostname
  • production dns entries added
  • network port setup
  • operations/puppet update
  • OS installation
  • puppet/salt accept/initial run
  • handoff for service implementation

Details

SubjectRepoBranchLines +/-
db-eqiad.php: Fully poll db1109,db1110operations/mediawiki-configmaster+2 -2
db-eqiad.php: Increase weight for db1109, db1110operations/mediawiki-configmaster+2 -2
db-eqiad.php: Pool db1109 and db1110operations/mediawiki-configmaster+2 -2
mariadb: Enable notifications db1109,db1110operations/puppetproduction+0 -2
db-eqiad.php: Restore original weight for db1082operations/mediawiki-configmaster+3 -3
db-eqiad,db-codfw.php: Add db1109 and db1110operations/mediawiki-configmaster+6 -0
install_server: Install db1109, db1110 as jessieoperations/puppetproduction+0 -4
s5.hosts: Add db1109 and db1110 to s5operations/softwaremaster+2 -0
mariadb: Add db1109 and db1110 to s5/s8operations/puppetproduction+7 -1
install_server: Allow install db1109 and db1110operations/puppetproduction+1 -1
Customize query in gerrit

Related Objects
Search...

StatusSubtypeAssignedTask
Unknown Object (Task)
 ResolvedMarosteguiT180700 Rack and setup db1109 and db1110

Event Timeline

Marostegui created this task.Nov 16 2017, 3:46 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptNov 16 2017, 3:47 PM
gerritbot subscribed.Nov 16 2017, 3:50 PM

Change 391845 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/puppet@production] install_server: Allow install db1109 and db1110

https://gerrit.wikimedia.org/r/391845

gerritbot added a project: Patch-For-Review.Nov 16 2017, 3:50 PM
gerritbot added a comment.Nov 16 2017, 3:59 PM

Change 391845 merged by Marostegui:
[operations/puppet@production] install_server: Allow install db1109 and db1110

https://gerrit.wikimedia.org/r/391845

• Cmjohnson updated the task description. (Show Details)Nov 16 2017, 6:58 PM
gerritbot added a comment.Nov 16 2017, 6:59 PM

Change 391879 had a related patch set uploaded (by Cmjohnson; owner: Cmjohnson):
[operations/dns@master] Adding mgmt dns for db1109 and 1110 T180700

https://gerrit.wikimedia.org/r/391879

gerritbot added a comment.Nov 16 2017, 7:00 PM

Change 391879 merged by Cmjohnson:
[operations/dns@master] Adding mgmt dns for db1109 and 1110 T180700

https://gerrit.wikimedia.org/r/391879

Marostegui moved this task from Triage to In progress on the DBA board.Nov 17 2017, 6:36 AM
gerritbot added a comment.Nov 17 2017, 1:07 PM

Change 392038 had a related patch set uploaded (by Cmjohnson; owner: Cmjohnson):
[operations/puppet@production] adding dhcpd entries for db1109-10 T180700

https://gerrit.wikimedia.org/r/392038

gerritbot added a comment.Nov 17 2017, 1:09 PM

Change 392038 merged by Cmjohnson:
[operations/puppet@production] adding dhcpd entries for db1109-10 T180700

https://gerrit.wikimedia.org/r/392038

• Cmjohnson updated the task description. (Show Details)Nov 17 2017, 1:10 PM
• Cmjohnson assigned this task to Marostegui.Nov 17 2017, 2:13 PM
• Cmjohnson updated the task description. (Show Details)

@Marostegui These are ready for you

• Cmjohnson moved this task from Backlog to Blocked on the ops-eqiad board.Nov 17 2017, 2:14 PM
Marostegui added a comment.Nov 17 2017, 2:15 PM

Thanks a lot @Cmjohnson for getting them up and running so fast! Really appreciated!

[14:15:14] marostegui@db1109:~$ uptime
 14:15:15 up 12 min,  1 user,  load average: 0.04, 0.16, 0.09

[14:15:44] marostegui@db1110:~$ uptime
 14:15:50 up 20 min,  1 user,  load average: 0.02, 0.16, 0.09
gerritbot added a comment.Nov 17 2017, 5:00 PM

Change 392068 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/puppet@production] mariadb: Add db1109 and db1110 to s5/s8

https://gerrit.wikimedia.org/r/392068

gerritbot added a comment.Nov 17 2017, 5:04 PM

Change 392068 merged by Marostegui:
[operations/puppet@production] mariadb: Add db1109 and db1110 to s5/s8

https://gerrit.wikimedia.org/r/392068

Stashbot subscribed.Nov 20 2017, 6:41 AM

Mentioned in SAL (#wikimedia-operations) [2017-11-20T06:41:02Z] <marostegui> Stop MySQL on db1082 to clone db1109 and db1110 - T180700

gerritbot added a comment.Nov 20 2017, 6:59 AM

Change 392372 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/software@master] s5.hosts: Add db1109 and db1110 to s5

https://gerrit.wikimedia.org/r/392372

gerritbot added a comment.Nov 20 2017, 7:02 AM

Change 392372 merged by jenkins-bot:
[operations/software@master] s5.hosts: Add db1109 and db1110 to s5

https://gerrit.wikimedia.org/r/392372

gerritbot added a comment.Nov 20 2017, 11:06 AM

Change 392403 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/puppet@production] install_server: Install db1109, db1110 as jessie

https://gerrit.wikimedia.org/r/392403

gerritbot added a comment.Nov 20 2017, 11:08 AM

Change 392403 merged by Marostegui:
[operations/puppet@production] install_server: Install db1109, db1110 as jessie

https://gerrit.wikimedia.org/r/392403

ops-monitoring-bot subscribed.Nov 20 2017, 11:09 AM

Script wmf-auto-reimage was launched by marostegui on neodymium.eqiad.wmnet for hosts:

db1110.eqiad.wmnet

The log can be found in /var/log/wmf-auto-reimage/201711201109_marostegui_20805_db1110_eqiad_wmnet.log.

ops-monitoring-bot added a comment.Nov 20 2017, 11:10 AM

Script wmf-auto-reimage was launched by marostegui on neodymium.eqiad.wmnet for hosts:

db1109.eqiad.wmnet

The log can be found in /var/log/wmf-auto-reimage/201711201110_marostegui_20842_db1109_eqiad_wmnet.log.

ops-monitoring-bot added a comment.Nov 20 2017, 11:32 AM

Completed auto-reimage of hosts:

['db1110.eqiad.wmnet']

and were ALL successful.

ops-monitoring-bot added a comment.Nov 20 2017, 11:33 AM

Completed auto-reimage of hosts:

['db1109.eqiad.wmnet']

and were ALL successful.

Stashbot added a comment.Nov 20 2017, 4:24 PM

Mentioned in SAL (#wikimedia-operations) [2017-11-20T16:24:27Z] <marostegui> Add db1109 and db1110 to tendril - T180700

gerritbot added a comment.Nov 20 2017, 4:34 PM

Change 392444 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/mediawiki-config@master] db-eqiad,db-codfw.php: Add db1109 and db1110

https://gerrit.wikimedia.org/r/392444

gerritbot added a comment.Nov 20 2017, 4:36 PM

Change 392444 merged by jenkins-bot:
[operations/mediawiki-config@master] db-eqiad,db-codfw.php: Add db1109 and db1110

https://gerrit.wikimedia.org/r/392444

Stashbot added a comment.Nov 20 2017, 4:38 PM

Mentioned in SAL (#wikimedia-operations) [2017-11-20T16:38:14Z] <marostegui@tin> Synchronized wmf-config/db-codfw.php: Add db1109 and db1110 to the config depooled - T180700 (duration: 00m 48s)

Stashbot added a comment.Nov 20 2017, 4:39 PM

Mentioned in SAL (#wikimedia-operations) [2017-11-20T16:39:13Z] <marostegui@tin> Synchronized wmf-config/db-eqiad.php: Add db1109 and db1110 to the config depooled - T180700 (duration: 00m 48s)

gerritbot added a comment.Nov 21 2017, 6:14 AM

Change 392579 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/mediawiki-config@master] db-eqiad.php: Restore original weight for db1082

https://gerrit.wikimedia.org/r/392579

gerritbot added a comment.Nov 21 2017, 6:18 AM

Change 392579 merged by jenkins-bot:
[operations/mediawiki-config@master] db-eqiad.php: Restore original weight for db1082

https://gerrit.wikimedia.org/r/392579

gerritbot added a comment.Nov 21 2017, 6:51 AM

Change 392584 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/mediawiki-config@master] db-eqiad.php: Pool db1109 and db1110

https://gerrit.wikimedia.org/r/392584

gerritbot added a comment.Nov 21 2017, 6:53 AM

Change 392585 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/puppet@production] mariadb: Enable notifications db1109,db1110

https://gerrit.wikimedia.org/r/392585

gerritbot added a comment.Nov 21 2017, 6:56 AM

Change 392585 merged by Marostegui:
[operations/puppet@production] mariadb: Enable notifications db1109,db1110

https://gerrit.wikimedia.org/r/392585

gerritbot added a comment.Nov 21 2017, 7:01 AM

Change 392584 merged by jenkins-bot:
[operations/mediawiki-config@master] db-eqiad.php: Pool db1109 and db1110

https://gerrit.wikimedia.org/r/392584

Stashbot added a comment.Nov 21 2017, 7:03 AM

Mentioned in SAL (#wikimedia-operations) [2017-11-21T07:03:06Z] <marostegui@tin> Synchronized wmf-config/db-eqiad.php: Pool db1109 and db1110 in s5 with small weight - T180700 (duration: 00m 48s)

gerritbot added a comment.Nov 21 2017, 7:30 AM

Change 392587 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/mediawiki-config@master] db-eqiad.php: Increase weight for db1109, db1110

https://gerrit.wikimedia.org/r/392587

gerritbot added a comment.Nov 21 2017, 7:35 AM

Change 392587 merged by jenkins-bot:
[operations/mediawiki-config@master] db-eqiad.php: Increase weight for db1109, db1110

https://gerrit.wikimedia.org/r/392587

Stashbot added a comment.Nov 21 2017, 7:36 AM

Mentioned in SAL (#wikimedia-operations) [2017-11-21T07:36:28Z] <marostegui@tin> Synchronized wmf-config/db-eqiad.php: Increase weight for db1109 and db1110 - T180700 (duration: 00m 49s)

gerritbot added a comment.Nov 21 2017, 8:06 AM

Change 392589 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/mediawiki-config@master] db-eqiad.php: Fully poll db1109,db1110

https://gerrit.wikimedia.org/r/392589

gerritbot added a comment.Nov 21 2017, 8:10 AM

Change 392589 merged by jenkins-bot:
[operations/mediawiki-config@master] db-eqiad.php: Fully poll db1109,db1110

https://gerrit.wikimedia.org/r/392589

Stashbot added a comment.Nov 21 2017, 8:11 AM

Mentioned in SAL (#wikimedia-operations) [2017-11-21T08:11:40Z] <marostegui@tin> Synchronized wmf-config/db-eqiad.php: Fully pool db1109 and db1110 - T180700 (duration: 00m 48s)

Marostegui closed this task as Resolved.Nov 21 2017, 8:11 AM

These two servers have been fully pooled in s5.

Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits