Page MenuHomePhabricator
Create Task
Maniphest T307802

Bootstrap new Cassandra nodes (eqiad)
Closed, ResolvedPublic
Actions

Description

Add aqs1016-aqs1021 to the AQS Cassandra cluster.

  • aqs1016-a
  • aqs1016-b
  • aqs1017-a
  • aqs1017-b
  • aqs1018-a
  • aqs1018-b
  • aqs1019-a
  • aqs1019-b
  • aqs1020-a
  • aqs1020-b
  • aqs1021-a
  • aqs1021-b

After the bootstraps are complete, each instance will need it's stream throughput reset to 200Mb/s, and a cleanup operation initiated:

  • aqs1010-{a,b}
  • aqs1013-{a,b}
  • aqs1016-{a,b}
  • aqs1019-{a,b}
  • aqs1011-{a,b}
  • aqs1014-{a,b}
  • aqs1017-{a,b}
  • aqs1020-{a,b}
  • aqs1012-{a,b}
  • aqs1015-{a,b}
  • aqs1018-{a,b}
  • aqs1021-{a,b}

See: T305570

Details

SubjectRepoBranchLines +/-
Bootstrap new AQS Cassandra nodes (eqiad)operations/puppetproduction+61 -7
Customize query in gerrit

Related Objects
Search...

StatusSubtypeAssignedTask
 ResolvedEevansT307641 AQS multi-datacenter cluster expansion
 ResolvedEevansT307802 Bootstrap new Cassandra nodes (eqiad)

Event Timeline

Eevans created this task.May 6 2022, 4:02 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptMay 6 2022, 4:02 PM
Eevans triaged this task as Medium priority.May 6 2022, 4:03 PM
Eevans added projects: Cassandra, Data-Engineering, Generated Data Platform.
Eevans added a parent task: T307641: AQS multi-datacenter cluster expansion.
JArguello-WMF added a project: Data-Engineering-Radar.May 11 2022, 6:16 PM
Restricted Application removed a project: Data-Engineering. · View Herald TranscriptMay 11 2022, 6:16 PM
Eevans mentioned this in T307641: AQS multi-datacenter cluster expansion.Jun 2 2022, 1:52 PM
Eevans moved this task from Backlog to Next on the Cassandra board.Jun 23 2022, 9:34 PM
Eevans moved this task from Next to In-Progress on the Cassandra board.Jun 27 2022, 6:36 PM
Eevans mentioned this in T307035: Relocate hosts: aqs10[3-5] .Jul 7 2022, 8:54 PM
gerritbot added a comment.Jul 9 2022, 12:47 AM

Change 812426 had a related patch set uploaded (by Eevans; author: Eevans):

[operations/puppet@production] [DRAFT]: Bootstrap new AQS Cassandra nodes (eqiad)

https://gerrit.wikimedia.org/r/812426

gerritbot added a project: Patch-For-Review.Jul 9 2022, 12:47 AM
Eevans moved this task from In-Progress to Next on the Cassandra board.Aug 12 2022, 9:33 PM
lbowmaker removed a project: Generated Data Platform.Aug 26 2022, 2:08 PM
gerritbot added a comment.Nov 9 2022, 2:58 PM

Change 812426 merged by Eevans:

[operations/puppet@production] Bootstrap new AQS Cassandra nodes (eqiad)

https://gerrit.wikimedia.org/r/812426

Maintenance_bot removed a project: Patch-For-Review.Nov 9 2022, 3:30 PM
Stashbot added a comment.Nov 10 2022, 5:26 PM

Mentioned in SAL (#wikimedia-operations) [2022-11-10T17:26:49Z] <urandom> increasing stream throughput to 400mbit, aqs1011-{a,b} & aqs1013-{a,b} -- T307802

Stashbot added a comment.Nov 10 2022, 5:28 PM

Mentioned in SAL (#wikimedia-operations) [2022-11-10T17:28:19Z] <urandom> restarting bootstrap of aqs1016-a -- T307802

Stashbot added a comment.Nov 12 2022, 10:46 PM

Mentioned in SAL (#wikimedia-operations) [2022-11-12T22:46:58Z] <urandom> initiating bootstrap, aqs1016-b -- T307802

Eevans updated the task description. (Show Details)Nov 12 2022, 10:50 PM
Stashbot added a comment.Nov 14 2022, 3:13 PM

Mentioned in SAL (#wikimedia-operations) [2022-11-14T15:13:01Z] <urandom> initiating Cassandra bootstrap, aqs1019-a -- T307802

Eevans updated the task description. (Show Details)Nov 14 2022, 5:23 PM
Eevans mentioned this in T322392: Store rejection reasons as a set in Cassandra.Nov 14 2022, 9:20 PM
Eevans updated the task description. (Show Details)Nov 15 2022, 4:14 PM
Stashbot added a comment.Nov 15 2022, 4:14 PM

Mentioned in SAL (#wikimedia-operations) [2022-11-15T16:14:40Z] <urandom> initiating Cassandra bootstrap, aqs1019-b -- T307802

Eevans updated the task description. (Show Details)Nov 16 2022, 3:38 PM
Stashbot added a comment.Nov 16 2022, 3:39 PM

Mentioned in SAL (#wikimedia-operations) [2022-11-16T15:39:33Z] <urandom> initiating Cassandra bootstrap, aqs1017-a -- T307802

Eevans updated the task description. (Show Details)Nov 18 2022, 2:29 PM
Stashbot added a comment.Nov 18 2022, 2:30 PM

Mentioned in SAL (#wikimedia-operations) [2022-11-18T14:30:15Z] <urandom> initiating Cassandra bootstrap, aqs1017-b -- T307802

Eevans updated the task description. (Show Details)Nov 19 2022, 9:40 PM
Stashbot added a comment.Nov 19 2022, 9:41 PM

Mentioned in SAL (#wikimedia-operations) [2022-11-19T21:41:09Z] <urandom> initiating Cassandra bootstrap, aqs1020-a -- T307802

Stashbot added a comment.Nov 20 2022, 8:29 PM

Mentioned in SAL (#wikimedia-operations) [2022-11-20T20:29:00Z] <urandom> initiating Cassandra bootstrap, aqs1020-b -- T307802

Eevans updated the task description. (Show Details)Nov 20 2022, 8:29 PM
Eevans updated the task description. (Show Details)Nov 21 2022, 3:15 PM
Stashbot added a comment.Nov 21 2022, 3:16 PM

Mentioned in SAL (#wikimedia-operations) [2022-11-21T15:16:06Z] <urandom> initiating Cassandra bootstrap, aqs1018-a -- T307802

Eevans updated the task description. (Show Details)Nov 23 2022, 5:39 PM
Stashbot added a comment.Nov 23 2022, 5:39 PM

Mentioned in SAL (#wikimedia-operations) [2022-11-23T17:39:57Z] <urandom> initiating Cassandra bootstrap, aqs1018-a -- T307802

Eevans updated the task description. (Show Details)Nov 25 2022, 5:37 PM
Stashbot added a comment.Nov 25 2022, 5:38 PM

Mentioned in SAL (#wikimedia-operations) [2022-11-25T17:38:35Z] <urandom> initiating Cassandra bootstrap, aqs1021-a -- T307802

Stashbot added a comment.Nov 26 2022, 9:34 PM

Mentioned in SAL (#wikimedia-operations) [2022-11-26T21:34:04Z] <urandom> initiating Cassandra bootstrap, aqs1021-b -- T307802

Eevans updated the task description. (Show Details)Nov 26 2022, 9:34 PM
Eevans updated the task description. (Show Details)Nov 27 2022, 10:06 PM
Eevans closed this task as Resolved.Nov 30 2022, 1:48 AM
Eevans claimed this task.
Eevans updated the task description. (Show Details)

Complete.

Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL