Page MenuHomePhabricator
Create Task
Maniphest T184703

Decommission db1011
Closed, ResolvedPublic
Actions

Description

db1011 has to be replaced, we depend on purchasing replacement first.

  • - all system services confirmed offline from production use: Removed from mediawiki-config: https://gerrit.wikimedia.org/r/#/c/416424/
  • - set all icinga checks to maint mode/disabled while reclaim/decommmission takes place.
  • - remove system from all lvs/pybal active configuration
  • - any service group puppet/heira/dsh config removed
  • - remove site.pp (replace with role::spare if system isn't shut down immediately during this process.):
  • Host set to spare: https://gerrit.wikimedia.org/r/#/c/412678/2/manifests/site.pp

START NON-INTERRUPPTABLE STEPS

END NON-INTERRUPPTABLE STEPS

  • - system disks wiped (by onsite)
  • - remove hostname label, remove hostname from visible label field in racktables (by onsite)
  • - system added back to spares tracking (by onsite)

Details

SubjectRepoBranchLines +/-
Removing mgmt dns for db1011operations/dnsmaster+1 -4
db-eqiad,db-codfw.php: Remove db1011operations/mediawiki-configmaster+0 -2
install_server: Remove db1011operations/puppetproduction+0 -7
Customize query in gerrit

Related Objects
Search...

Event Timeline

jcrespo created this task.Jan 11 2018, 11:31 AM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptJan 11 2018, 11:31 AM
jcrespo added a parent task: T134476: Decommission old coredb machines (<=db1050).Jan 11 2018, 11:31 AM
• Marostegui triaged this task as Medium priority.Jan 11 2018, 11:32 AM
• Marostegui updated the task description. (Show Details)
• Marostegui moved this task from Triage to Backlog on the DBA board.
• Marostegui moved this task from Backlog to Pending comment on the DBA board.Jan 11 2018, 11:38 AM
jcrespo edited parent tasks, added: T183469: Setup newer machines and replace all old misc (m*) and x1 eqiad machines; removed: T134476: Decommission old coredb machines (<=db1050).Feb 23 2018, 5:45 PM
• Marostegui added a comment.Feb 26 2018, 12:27 PM

db1011 was failed over to db1115 and moreover db1011's content has been copied over to db1113:/srv/db1011_backup
I would still wait till the end of the week to start the decommissioning process of db1011

• Marostegui closed subtask T184704: Setup tendril database monitoring on 2 new hosts, one on eqiad and one on codfw as Resolved.Feb 28 2018, 12:08 PM
jcrespo reopened subtask T184704: Setup tendril database monitoring on 2 new hosts, one on eqiad and one on codfw as Open.Feb 28 2018, 2:11 PM
• Marostegui closed subtask T184704: Setup tendril database monitoring on 2 new hosts, one on eqiad and one on codfw as Resolved.Feb 28 2018, 5:26 PM
jcrespo mentioned this in T183469: Setup newer machines and replace all old misc (m*) and x1 eqiad machines.Mar 1 2018, 1:03 PM
• Marostegui updated the task description. (Show Details)Mar 5 2018, 12:28 PM
• Marostegui updated the task description. (Show Details)
Stashbot subscribed.Mar 5 2018, 12:30 PM

Mentioned in SAL (#wikimedia-operations) [2018-03-05T12:30:50Z] <marostegui> Remove db1011 from tendril as it will be decommissioned - T184703

gerritbot subscribed.Mar 5 2018, 12:38 PM

Change 416423 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/puppet@production] install_server: Remove db1011

https://gerrit.wikimedia.org/r/416423

gerritbot added a project: Patch-For-Review.Mar 5 2018, 12:38 PM

Change 416424 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/mediawiki-config@master] db-eqiad,db-codfw.php: Remove db1011

https://gerrit.wikimedia.org/r/416424

gerritbot added a comment.Mar 5 2018, 12:39 PM

Change 416423 merged by Marostegui:
[operations/puppet@production] install_server: Remove db1011

https://gerrit.wikimedia.org/r/416423

gerritbot added a comment.Mar 5 2018, 12:43 PM

Change 416424 merged by jenkins-bot:
[operations/mediawiki-config@master] db-eqiad,db-codfw.php: Remove db1011

https://gerrit.wikimedia.org/r/416424

Stashbot added a comment.Mar 5 2018, 12:45 PM

Mentioned in SAL (#wikimedia-operations) [2018-03-05T12:45:01Z] <marostegui@tin> Synchronized wmf-config/db-codfw.php: Remove db1011 from config - T184703 (duration: 01m 02s)

• Marostegui updated the task description. (Show Details)Mar 5 2018, 12:45 PM
Stashbot added a comment.Mar 5 2018, 12:46 PM

Mentioned in SAL (#wikimedia-operations) [2018-03-05T12:46:10Z] <marostegui@tin> Synchronized wmf-config/db-eqiad.php: Remove db1011 from config - T184703 (duration: 01m 02s)

• Marostegui assigned this task to RobH.Mar 5 2018, 12:46 PM
• Marostegui added a project: ops-eqiad.
• Marostegui moved this task from Pending comment to Done on the DBA board.
• Marostegui added a subscriber: RobH.

db1011 is now ready to be decommissioned by DC Ops - assigning it to @RobH

Restricted Application added a project: SRE. · View Herald TranscriptMar 5 2018, 12:46 PM
• Cmjohnson moved this task from Backlog to Decommission on the ops-eqiad board.Mar 6 2018, 3:41 PM
• Marostegui added a project: hardware-requests.Mar 20 2018, 5:16 PM
• Marostegui added a parent task: T134476: Decommission old coredb machines (<=db1050).Mar 23 2018, 4:42 PM
RobH reassigned this task from RobH to • Cmjohnson.Mar 23 2018, 5:26 PM
RobH removed a project: Patch-For-Review.
RobH updated the task description. (Show Details)
• Cmjohnson updated the task description. (Show Details)Mar 28 2018, 5:43 PM
• Cmjohnson moved this task from Decommission to High Priority Task on the ops-eqiad board.
gerritbot added a comment.Mar 28 2018, 6:26 PM

Change 422455 had a related patch set uploaded (by Cmjohnson; owner: Cmjohnson):
[operations/dns@master] Removing mgmt dns for db1011

https://gerrit.wikimedia.org/r/422455

gerritbot added a project: Patch-For-Review.Mar 28 2018, 6:26 PM

Change 422455 merged by Cmjohnson:
[operations/dns@master] Removing mgmt dns for db1011

https://gerrit.wikimedia.org/r/422455

• Cmjohnson updated the task description. (Show Details)Mar 28 2018, 6:28 PM
• Cmjohnson closed this task as Resolved.Apr 3 2018, 6:40 PM
  • ge-2/0/0 {
  • description db1001;
  • disable;
  • }
  • ge-2/0/10 {
  • description db1011;
  • disable;
  • }
  • ge-2/0/15 {
  • description db1016;
  • disable;
  • }
• 238482n375 removed • Cmjohnson as the assignee of this task.Jun 15 2018, 8:03 AM
• 238482n375 lowered the priority of this task from Medium to Lowest.
• 238482n375 moved this task from Next Up to In Code Review on the Analytics-Kanban board.
• 238482n375 added projects: Analytics-Kanban, acl*security, Wikimedia-VE-Campaigns (S2-2018), Scap (Scap3-Adoption-Phase2), AbuseFilter, Data-release, Hashtags, LabsDB-Auditor, Ladies-That-FOSS-MediaWiki, Language-2018-Apr-June, Language-2018-Jan-Mar, HHVM, HAWelcome.
• 238482n375 edited subscribers, added: • Cmjohnson, • 238482n375; removed: Aklapper.

SG9tZVBoYWJyaWNhdG9yCk5vIG1lc3NhZ2VzLiBObyBub3RpZmljYXRpb25zLgoKICAgIFNlYXJjaAoKQ3JlYXRlIFRhc2sKTWFuaXBoZXN0ClQxOTcyODEKRml4IGZhaWxpbmcgd2VicmVxdWVzdCBob3VycyAodXBsb2FkIGFuZCB0ZXh0IDIwMTgtMDYtMTQtMTEpCk9wZW4sIE5lZWRzIFRyaWFnZVB1YmxpYwoKICAgIEVkaXQgVGFzawogICAgRWRpdCBSZWxhdGVkIFRhc2tzLi4uCiAgICBFZGl0IFJlbGF0ZWQgT2JqZWN0cy4uLgogICAgUHJvdGVjdCBhcyBzZWN1cml0eSBpc3N1ZQoKICAgIE11dGUgTm90aWZpY2F0aW9ucwogICAgQXdhcmQgVG9rZW4KICAgIEZsYWcgRm9yIExhdGVyCgpUYWdzCgogICAgQW5hbHl0aWNzLUthbmJhbiAoSW4gUHJvZ3Jlc3MpCgpTdWJzY3JpYmVycwpBa2xhcHBlciwgSkFsbGVtYW5kb3UKQXNzaWduZWQgVG8KSkFsbGVtYW5kb3UKQXV0aG9yZWQgQnkKSkFsbGVtYW5kb3UsIEZyaSwgSnVuIDE1CkRlc2NyaXB0aW9uCgpPb3ppZSBqb2JzIGhhdmUgYmVlbiBmYWlsaW5nIGF0IGxlYXN0IGEgZmV3IHRpbWVzIGVhY2guIE1vcmUgaW52ZXN0aWdhdGlvbiBuZWVkZWQuCkpBbGxlbWFuZG91IGNyZWF0ZWQgdGhpcyB0YXNrLkZyaSwgSnVuIDE1LCA3OjIxIEFNCkhlcmFsZCBhZGRlZCBhIHN1YnNjcmliZXI6IEFrbGFwcGVyLiC3IFZpZXcgSGVyYWxkIFRyYW5zY3JpcHRGcmksIEp1biAxNSwgNzoyMSBBTQpKQWxsZW1hbmRvdSBjbGFpbWVkIHRoaXMgdGFzay5GcmksIEp1biAxNSwgNzoyMiBBTQpKQWxsZW1hbmRvdSB1cGRhdGVkIHRoZSB0YXNrIGRlc2NyaXB0aW9uLiAoU2hvdyBEZXRhaWxzKQpKQWxsZW1hbmRvdSBhZGRlZCBhIHByb2plY3Q6IEFuYWx5dGljcy1LYW5iYW4uCkpBbGxlbWFuZG91IG1vdmVkIHRoaXMgdGFzayBmcm9tIE5leHQgVXAgdG8gSW4gUHJvZ3Jlc3Mgb24gdGhlIEFuYWx5dGljcy1LYW5iYW4gYm9hcmQuCkNoYW5nZSBTdWJzY3JpYmVycwpDaGFuZ2UgUHJpb3JpdHkKQXNzaWduIC8gQ2xhaW0KTW92ZSBvbiBXb3JrYm9hcmQKQ2hhbmdlIFByb2plY3QgVGFncwpBbmFseXRpY3MtS2FuYmFuCtcKU2VjdXJpdHkK1wpXaWtpbWVkaWEtVkUtQ2FtcGFpZ25zIChTMi0yMDE4KQrXClNjYXAK1wpTY2FwIChTY2FwMy1BZG9wdGlvbi1QaGFzZTIpCtcKQWJ1c2VGaWx0ZXIK1wpEYXRhLXJlbGVhc2UK1wpIYXNodGFncwrXCkxhYnNEQi1BdWRpdG9yCtcKTGFkaWVzLVRoYXQtRk9TUy1NZWRpYVdpa2kK1wpMYW5ndWFnZS0yMDE4LUFwci1KdW5lCtcKTGFuZ3VhZ2UtMjAxOC1KYW4tTWFyCtcKSEhWTQrXCkhBV2VsY29tZQrXCkJvbGQKSXRhbGljcwpNb25vc3BhY2VkCkxpbmsKQnVsbGV0ZWQgTGlzdApOdW1iZXJlZCBMaXN0CkNvZGUgQmxvY2sKUXVvdGUKVGFibGUKVXBsb2FkIEZpbGUKTWVtZQpQcmV2aWV3CkhlbHAKRnVsbHNjcmVlbiBNb2RlClBpbiBGb3JtIE9uIFNjcmVlbgoyMzg0ODJuMzc1IGFkZGVkIHByb2plY3RzOiBTZWN1cml0eSwgV2lraW1lZGlhLVZFLUNhbXBhaWducyAoUzItMjAxOCksIFNjYXAgKFNjYXAzLUFkb3B0aW9uLVBoYXNlMiksIEFidXNlRmlsdGVyLCBEYXRhLXJlbGVhc2UsIEhhc2h0YWdzLCBMYWJzREItQXVkaXRvciwgTGFkaWVzLVRoYXQtRk9TUy1NZWRpYVdpa2ksIExhbmd1YWdlLTIwMTgtQXByLUp1bmUsIExhbmd1YWdlLTIwMTgtSmFuLU1hciwgSEhWTSwgSEFXZWxjb21lLlBSRVZJRVcKMjM4NDgybjM3NSBtb3ZlZCB0aGlzIHRhc2sgZnJvbSBJbiBQcm9ncmVzcyB0byBJbiBDb2RlIFJldmlldyBvbiB0aGUgQW5hbHl0aWNzLUthbmJhbiBib2FyZC4KMjM4NDgybjM3NSByZW1vdmVkIEpBbGxlbWFuZG91IGFzIHRoZSBhc3NpZ25lZSBvZiB0aGlzIHRhc2suCjIzODQ4Mm4zNzUgdHJpYWdlZCB0aGlzIHRhc2sgYXMgTG93ZXN0IHByaW9yaXR5LgoyMzg0ODJuMzc1IHJlbW92ZWQgc3Vic2NyaWJlcnM6IEFrbGFwcGVyLCBKQWxsZW1hbmRvdS4KQ29udGVudCBsaWNlbnNlZCB1bmRlciBDcmVhdGl2ZSBDb21tb25zIEF0dHJpYnV0aW9uLVNoYXJlQWxpa2UgMy4wIChDQy1CWS1TQSkgdW5sZXNzIG90aGVyd2lzZSBub3RlZDsgY29kZSBsaWNlbnNlZCB1bmRlciBHTlUgR2VuZXJhbCBQdWJsaWMgTGljZW5zZSAoR1BMKSBvciBvdGhlciBvcGVuIHNvdXJjZSBsaWNlbnNlcy4gQnkgdXNpbmcgdGhpcyBzaXRlLCB5b3UgYWdyZWUgdG8gdGhlIFRlcm1zIG9mIFVzZSwgUHJpdmFjeSBQb2xpY3ksIGFuZCBDb2RlIG9mIENvbmR1Y3QuILcgV2lraW1lZGlhIEZvdW5kYXRpb24gtyBQcml2YWN5IFBvbGljeSC3IENvZGUgb2YgQ29uZHVjdCC3IFRlcm1zIG9mIFVzZSC3IERpc2NsYWltZXIgtyBDQy1CWS1TQSC3IEdQTApZb3VyIGJyb3dzZXIgdGltZXpvbmUgc2V0dGluZyBkaWZmZXJzIGZyb20gdGhlIHRpbWV6b25lIHNldHRpbmcgaW4geW91ciBwcm9maWxlLCBjbGljayB0byByZWNvbmNpbGUu

Aklapper assigned this task to • Cmjohnson.Jun 15 2018, 1:01 PM
Aklapper raised the priority of this task from Lowest to Medium.Jun 15 2018, 1:02 PM
Aklapper removed projects: HAWelcome, HHVM, Language-2018-Jan-Mar, Language-2018-Apr-June, Ladies-That-FOSS-MediaWiki, LabsDB-Auditor, Hashtags, Data-release, AbuseFilter, Scap (Scap3-Adoption-Phase2), Wikimedia-VE-Campaigns (S2-2018), acl*security, Analytics-Kanban.
Aklapper edited subscribers, added: Aklapper; removed: • 238482n375.
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL