Page MenuHomePhabricator
Create Task
Maniphest T177405

rack and setup db1107 and db1108
Closed, ResolvedPublic
Actions

Description

This task will track the racking/setup/installation of 2 new databases (db1107 and db1108) for eqiad. These were ordered on T168650. @Marostegui please update task with desired racking locations.

db1107

  • receive in system on procurement task T168650
  • bios/drac/serial setup/testing
  • raid set up to raid 10 (standard db raid)
  • mgmt dns entries added for both asset tag and hostname
  • production dns entries added
  • network port setup
  • operations/puppet update
  • OS installation
  • puppet/salt accept/initial run
  • handoff for service implementation

db1108

  • receive in system on procurement task T168650
  • bios/drac/serial setup/testing
  • raid set up to raid 10 (standard db raid)
  • mgmt dns entries added for both asset tag and hostname
  • production dns entries added
  • network port setup
  • operations/puppet update
  • OS installation
  • puppet/salt accept/initial run
  • handoff for service implementation

Details

Show related patches Customize query in gerrit

Related Objects
Search...

Event Timeline

• Cmjohnson created this task.Oct 4 2017, 3:13 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptOct 4 2017, 3:13 PM
• Cmjohnson renamed this task from rack and setup db1107 and db1008 to rack and setup db1107 and db1108.Oct 4 2017, 3:14 PM
• jcrespo added a subscriber: Ottomata.Oct 4 2017, 3:20 PM

Wait, should those be named db1* ? CC @Cmjohnson @elukey @Ottomata I personally do not have a problem with that (it gets some regex configurations because it is a database), but maybe you didn't want that?

Ottomata added a comment.Oct 4 2017, 3:21 PM

I have no opinions on the naming of these boxes :)

elukey added a comment.Oct 4 2017, 3:24 PM

I have no preference, the alternative would be to explicitly mention eventlogging in their names, going to ask to my team and report back asap.

Marostegui added a comment.Oct 4 2017, 3:42 PM

I don't really have any preference on where to rack them, I would just suggest they are placed on a different rack. But as these two hosts will be mostly for Analytics, I would leave it to @elukey or @Ottomata to decide (I can help of course!)

elukey added a comment.EditedOct 4 2017, 4:01 PM

We are super fine with db1* names, no real preference, just asked to my team. If it is fine for the DBA team, we can proceed!

For the racking, I'd prefer different rows if possible..

• jcrespo added a comment.Oct 4 2017, 4:01 PM

db1* is ok, then, @Cmjohnson . I only asked because I may have thought you proposed to rename them. I am more than ok as keeping them as part of the db* family as m4 replica set. Sorry for the noise. having them on separate rows, if possible, would be indeed ideal (we have decom'ed lots of servers, so space should not a problem).

• Cmjohnson added a comment.Oct 4 2017, 5:03 PM

@jcrespo I only recommended that named based on the procurement ticket subject: "eqiad: replacements for db1046 and db1047" I have zero preference to naming.

• jcrespo added a comment.Oct 4 2017, 5:08 PM

db1* is ok

Marostegui added a comment.Oct 4 2017, 5:35 PM

As for the racking I would suggest

db1107: Take db1036 (T176311) place on B2
db1108: Take db1015 (T173570) on A2

So we would have different racks and different rows.

• jcrespo added a comment.Oct 4 2017, 5:38 PM

Note that in theory, these new servers should be 1/2 the size of the original ones, so they should fit there and even have space for extra 1U servers.

• Cmjohnson moved this task from Backlog to Up next on the ops-eqiad board.Oct 8 2017, 2:44 PM
gerritbot subscribed.Oct 10 2017, 5:27 PM

Change 383385 had a related patch set uploaded (by Cmjohnson; owner: Cmjohnson):
[operations/dns@master] Adding mgmt dns entries for db1107/1108 T177405

https://gerrit.wikimedia.org/r/383385

gerritbot added a project: Patch-For-Review.Oct 10 2017, 5:27 PM

Change 383385 merged by Cmjohnson:
[operations/dns@master] Adding mgmt dns entries for db1107/1108 T177405

https://gerrit.wikimedia.org/r/383385

• Cmjohnson updated the task description. (Show Details)Oct 10 2017, 6:03 PM
gerritbot added a comment.Oct 11 2017, 2:20 PM

Change 383583 had a related patch set uploaded (by Cmjohnson; owner: Cmjohnson):
[operations/dns@master] Adding production dns for db1107/1108 T177405

https://gerrit.wikimedia.org/r/383583

gerritbot added a comment.Oct 11 2017, 2:21 PM

Change 383583 merged by Cmjohnson:
[operations/dns@master] Adding production dns for db1107/1108 T177405

https://gerrit.wikimedia.org/r/383583

gerritbot added a comment.Oct 11 2017, 4:23 PM

Change 383609 had a related patch set uploaded (by Cmjohnson; owner: Cmjohnson):
[operations/puppet@production] Adding mac address to dhcpd file and updating netboot.cfg file for db1107 and db1108 T177405

https://gerrit.wikimedia.org/r/383609

gerritbot added a comment.Oct 11 2017, 4:34 PM

Change 383609 merged by Cmjohnson:
[operations/puppet@production] Adding macs to dhcpd file & updated netboot.cfg db1107/1108 T177405

https://gerrit.wikimedia.org/r/383609

• Cmjohnson updated the task description. (Show Details)Oct 12 2017, 6:46 PM
• Cmjohnson updated the task description. (Show Details)

@elukey these are ready for you or @Ottomata Please reassign

• jcrespo added a comment.Oct 17 2017, 5:23 PM

One of you can go ahead and apply the same roles than its replacements (IMPORTANT: with notifications disabled on hiera), once they are ready, we DBAs can copy transfer the data for you. Stretch would be suggested.

elukey triaged this task as Medium priority.Oct 18 2017, 10:26 AM
elukey edited projects, added User-Elukey, Analytics-Kanban; removed Patch-For-Review.
gerritbot added a comment.Oct 18 2017, 10:34 AM

Change 384963 had a related patch set uploaded (by Elukey; owner: Elukey):
[operations/puppet@production] Set PXE boot options and notification disabled for db110[78]

https://gerrit.wikimedia.org/r/384963

gerritbot added a project: Patch-For-Review.Oct 18 2017, 10:34 AM
gerritbot added a comment.Oct 18 2017, 11:09 AM

Change 384963 merged by Elukey:
[operations/puppet@production] Set PXE boot options and notification disabled for db110[78]

https://gerrit.wikimedia.org/r/384963

ops-monitoring-bot subscribed.Oct 18 2017, 12:54 PM

Script wmf-auto-reimage was launched by elukey on neodymium.eqiad.wmnet for hosts:

['db1107.eqiad.wmnet', 'db1108.eqiad.wmnet']

The log can be found in /var/log/wmf-auto-reimage/201710181254_elukey_13070.log.

ops-monitoring-bot added a comment.Oct 18 2017, 1:11 PM

Completed auto-reimage of hosts:

['db1108.eqiad.wmnet', 'db1107.eqiad.wmnet']

and were ALL successful.

gerritbot added a comment.Oct 18 2017, 1:31 PM

Change 384979 had a related patch set uploaded (by Elukey; owner: Elukey):
[operations/puppet@production] netboot: prevent db110[78] to be reimaged

https://gerrit.wikimedia.org/r/384979

• Cmjohnson moved this task from Up next to Blocked on the ops-eqiad board.Oct 18 2017, 5:01 PM
gerritbot added a comment.Oct 19 2017, 11:20 AM

Change 385173 had a related patch set uploaded (by Elukey; owner: Elukey):
[operations/puppet@production] Introduce mariadb eventlogging profiles for master/replica

https://gerrit.wikimedia.org/r/385173

gerritbot added a comment.Oct 19 2017, 11:46 AM

Change 384979 merged by Elukey:
[operations/puppet@production] netboot: prevent db110[78] to be reimaged

https://gerrit.wikimedia.org/r/384979

elukey claimed this task.Oct 19 2017, 3:03 PM
elukey moved this task from Next Up to In Progress on the Analytics-Kanban board.
• jcrespo added a parent task: T156844: Decommission old dbstore hosts (db1046, db1047).Oct 19 2017, 3:07 PM
elukey moved this task from Backlog to In Progress on the User-Elukey board.Oct 23 2017, 3:48 PM
gerritbot added a comment.Oct 24 2017, 1:12 PM

Change 385173 merged by Elukey:
[operations/puppet@production] role::mariadb: Introduce mariadb eventlogging profiles for master/replica

https://gerrit.wikimedia.org/r/385173

elukey added a comment.Oct 24 2017, 1:32 PM

Next steps:

  1. Create unit files and systemd config for eventlogging_sync.sh and add the guards in puppet to allow trusty/stretch to co-exist (dbstore1002 will keep running with trusty).
  2. Assign the new eventlogging replica role to db1108, make sure that everything is good
  3. Work with Manuel to move data from db1047 to db1108
  4. Start eventlogging_sync on db1108 and let it run alongside with db1047 for some days to spot anomalies
  5. Move the analytics-slave CNAME to db1108
  6. Decom db1047
gerritbot added a comment.Oct 25 2017, 9:14 AM

Change 386346 had a related patch set uploaded (by Elukey; owner: Elukey):
[operations/puppet@production] profile::mariadb::misc::eventlogging: add support for systemd

https://gerrit.wikimedia.org/r/386346

gerritbot added a comment.Oct 25 2017, 10:44 AM

Change 386359 had a related patch set uploaded (by Elukey; owner: Elukey):
[operations/puppet@production] site.pp: set db1108 as analyics db replica

https://gerrit.wikimedia.org/r/386359

gerritbot added a comment.Oct 25 2017, 10:50 AM

Change 386346 merged by Elukey:
[operations/puppet@production] profile::mariadb::misc::eventlogging: add support for systemd

https://gerrit.wikimedia.org/r/386346

gerritbot added a comment.Oct 25 2017, 11:47 AM

Change 386359 merged by Elukey:
[operations/puppet@production] site.pp: set db1108 as analyics db replica

https://gerrit.wikimedia.org/r/386359

gerritbot added a comment.Oct 25 2017, 12:06 PM

Change 386370 had a related patch set uploaded (by Elukey; owner: Elukey):
[operations/puppet@production] profile::mariadb::misc::eventlogging:repl: fix systemd template

https://gerrit.wikimedia.org/r/386370

gerritbot added a comment.Oct 25 2017, 12:06 PM

Change 386370 merged by Elukey:
[operations/puppet@production] profile::mariadb::misc::eventlogging:repl: fix systemd template

https://gerrit.wikimedia.org/r/386370

gerritbot added a comment.Oct 25 2017, 12:10 PM

Change 386371 had a related patch set uploaded (by Elukey; owner: Elukey):
[operations/puppet@production] profile::mariadb::misc::eventlogging:repl: fix typo in rsyslog config

https://gerrit.wikimedia.org/r/386371

gerritbot added a comment.Oct 25 2017, 12:11 PM

Change 386371 merged by Elukey:
[operations/puppet@production] profile::mariadb::misc::eventlogging:repl: fix typo in rsyslog config

https://gerrit.wikimedia.org/r/386371

gerritbot added a comment.Oct 25 2017, 12:36 PM

Change 386376 had a related patch set uploaded (by Elukey; owner: Elukey):
[operations/puppet@production] profile::mariadb::misc::eventlogging::database: support mariadb 10.1

https://gerrit.wikimedia.org/r/386376

gerritbot added a comment.Oct 25 2017, 12:45 PM

Change 386376 merged by Elukey:
[operations/puppet@production] profile::mariadb::misc::eventlogging::database: support mariadb 10.1

https://gerrit.wikimedia.org/r/386376

gerritbot added a comment.Oct 25 2017, 12:50 PM

Change 386379 had a related patch set uploaded (by Elukey; owner: Elukey):
[operations/puppet@production] profile::mariadb::misc::eventlogging: fix systemd unit template

https://gerrit.wikimedia.org/r/386379

gerritbot added a comment.Oct 25 2017, 12:51 PM

Change 386379 merged by Elukey:
[operations/puppet@production] profile::mariadb::misc::eventlogging: fix systemd unit template

https://gerrit.wikimedia.org/r/386379

Stashbot subscribed.Oct 26 2017, 7:43 AM

Mentioned in SAL (#wikimedia-operations) [2017-10-26T07:43:37Z] <marostegui> Stop MySQL on db1047 to copy data over db1108 - T177405 T156844

Stashbot mentioned this in T156844: Decommission old dbstore hosts (db1046, db1047).Oct 26 2017, 7:43 AM
Stashbot added a comment.Oct 26 2017, 7:57 AM

Mentioned in SAL (#wikimedia-operations) [2017-10-26T07:57:41Z] <marostegui> Drop databases in s1 and s2 from db1047 and unconfigure replication - T177405

gerritbot added a comment.Oct 26 2017, 8:28 AM

Change 386586 had a related patch set uploaded (by Elukey; owner: Elukey):
[operations/puppet@production] profile::mariadb::misc::eventlogging::database: set correct mysql params

https://gerrit.wikimedia.org/r/386586

gerritbot added a comment.Oct 26 2017, 9:04 AM

Change 386586 merged by Elukey:
[operations/puppet@production] profile::mariadb::misc::eventlogging::database: set correct mysql params

https://gerrit.wikimedia.org/r/386586

elukey mentioned this in T177101: PageContentSaveComplete. Stop collecting .Oct 26 2017, 2:02 PM
Marostegui added a comment.Oct 31 2017, 8:38 AM

db1047's data has been migrated and imported to db1108

gerritbot added a comment.Nov 2 2017, 10:59 AM

Change 388021 had a related patch set uploaded (by Elukey; owner: Elukey):
[operations/puppet@production] Enable icinga notifications for db1108

https://gerrit.wikimedia.org/r/388021

gerritbot added a comment.Nov 2 2017, 10:59 AM

Change 388021 merged by Elukey:
[operations/puppet@production] Enable icinga notifications for db1108

https://gerrit.wikimedia.org/r/388021

Stashbot added a comment.Nov 7 2017, 10:24 AM

Mentioned in SAL (#wikimedia-operations) [2017-11-07T10:24:48Z] <elukey> create staging database on db1108 (researchers scratch pad) - T177405

gerritbot added a comment.Nov 8 2017, 8:07 AM

Change 389922 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/software@master] s1,s2.hosts: Remove db1047

https://gerrit.wikimedia.org/r/389922

gerritbot added a comment.Nov 8 2017, 8:08 AM

Change 389922 merged by jenkins-bot:
[operations/software@master] s1,s2.hosts: Remove db1047

https://gerrit.wikimedia.org/r/389922

gerritbot added a comment.Nov 14 2017, 10:41 AM

Change 391182 had a related patch set uploaded (by Jcrespo; owner: Jcrespo):
[operations/puppet@production] Refactor evenlogging database hosts definition

https://gerrit.wikimedia.org/r/391182

gerritbot added a comment.Nov 14 2017, 2:38 PM

Change 391182 merged by Jcrespo:
[operations/puppet@production] Refactor evenlogging database hosts definition

https://gerrit.wikimedia.org/r/391182

Stashbot added a comment.Nov 15 2017, 9:15 AM

Mentioned in SAL (#wikimedia-operations) [2017-11-15T09:15:19Z] <marostegui> Stop mysql on db1046 to transfer its content to db1107 - T177405

gerritbot added a comment.Nov 15 2017, 9:23 AM

Change 391519 had a related patch set uploaded (by Elukey; owner: Elukey):
[operations/puppet@production] site.pp: add role to db1107 (new eventlogging master db)

https://gerrit.wikimedia.org/r/391519

gerritbot added a comment.Nov 15 2017, 11:02 AM

Change 391519 merged by Elukey:
[operations/puppet@production] site.pp: add role to db1107 (new eventlogging master db)

https://gerrit.wikimedia.org/r/391519

gerritbot added a comment.Nov 15 2017, 11:21 AM

Change 391537 had a related patch set uploaded (by Elukey; owner: Elukey):
[operations/puppet@production] profile::mariadb::misc::eventlogging::database: fix hearthbeat settings

https://gerrit.wikimedia.org/r/391537

gerritbot added a comment.Nov 15 2017, 11:23 AM

Change 391537 merged by Elukey:
[operations/puppet@production] profile::mariadb::misc::eventlogging::database: fix hearthbeat settings

https://gerrit.wikimedia.org/r/391537

elukey moved this task from In Progress to Done on the Analytics-Kanban board.Nov 15 2017, 4:03 PM
elukey closed this task as Resolved.Nov 15 2017, 5:56 PM

All the work has been completed, closing the task!

gerritbot added a comment.Nov 24 2017, 12:31 PM

Change 393220 had a related patch set uploaded (by Elukey; owner: Elukey):
[operations/puppet@production] Fix prometheus target for the Eventlogging mysql master db

https://gerrit.wikimedia.org/r/393220

gerritbot added a comment.Nov 28 2017, 9:14 AM

Change 393220 merged by Elukey:
[operations/puppet@production] Fix prometheus target for the Eventlogging mysql master db

https://gerrit.wikimedia.org/r/393220

faidon mentioned this in rOHPUa8c5a4c16a90: Adding new mysql IP in analytics-in4.Oct 3 2019, 11:10 AM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits