Page MenuHomePhabricator
Create Task
Maniphest T224553

Migrate remaining Restbase servers to Stretch
Closed, ResolvedPublic
Actions

Description

When the current round of decommisions is done, there will be four servers in codfw and two in eqiad left running Debian jessie. We should migrate those for consistency (and it would also allow us to stop maintaining our custom openjdk-8 backports for jessie (which were originally done for improved Cassandra/GC performance with Java 8 compared to 7).

  • restbase2009.codfw.wmnet
  • restbase2010.codfw.wmnet
  • restbase2011.codfw.wmnet
  • restbase2012.codfw.wmnet

Details

SubjectRepoBranchLines +/-
hieradata: specify restbase2009 jbod devicesoperations/puppetproduction+1 -0
Add missing JBOD config for restbase1018operations/puppetproduction+1 -0
Customize query in gerrit

Related Objects
Search...

Event Timeline

MoritzMuehlenhoff created this task.May 29 2019, 10:31 AM
MoritzMuehlenhoff mentioned this in T224549: Track remaining jessie systems in production.
• mobrovac added projects: Platform Team Legacy (Next), Platform Engineering (Needs Cleaning - Security, stability, performance, and scalability (TEC1)), Services (next), RESTBase, Cassandra, RESTBase-Cassandra.May 29 2019, 12:06 PM

Agreed. We can start coordinating once T208087: Replace remaining Samsung SSDs is done. That includes T223976: Decommission restbase10(0[7-9]|1[0-5]) but also T222960: Fix restbase1017's physical rack.

• mobrovac merged a task: T224623: Upgrade RESTBase cluster to Stretch.May 29 2019, 8:22 PM
Eevans updated the task description. (Show Details)May 29 2019, 8:23 PM
Eevans merged a task: T224623: Upgrade RESTBase cluster to Stretch.
ArielGlenn triaged this task as Medium priority.Jun 11 2019, 7:59 AM
MoritzMuehlenhoff updated the task description. (Show Details)Jul 12 2019, 8:08 AM
• Pchelolo edited projects, added Platform Engineering (Needs Cleaning - Services Operations); removed Services (next), Platform Engineering (Needs Cleaning - Security, stability, performance, and scalability (TEC1)), Platform Team Legacy (Next).Jul 17 2019, 4:52 PM
MoritzMuehlenhoff added a comment.Aug 22 2019, 3:06 PM

T208087, T223976 and T222960 are fixed. Could we get restbase2009-restbase2012, restbase1018 (and the two remaining -dev servers) migrated to Stretch in the next 1-2 months?

cassandra/restbase is the only remaining use case for our custom OpenJDK 8 backports in jessie-wikimedia and it would be fantastic not to spend more time on this when the October Java security release gets released.

Eevans added a comment.Aug 22 2019, 4:03 PM
In T224553#5431770, @MoritzMuehlenhoff wrote:

T208087, T223976 and T222960 are fixed. Could we get restbase2009-restbase2012, restbase1018 (and the two remaining -dev servers) migrated to Stretch in the next 1-2 months?

cassandra/restbase is the only remaining use case for our custom OpenJDK 8 backports in jessie-wikimedia and it would be fantastic not to spend more time on this when the October Java security release gets released.

Assuming we're adhering to the process of decommissioning, re-imaging, and bootstrapping, then I suspect the only blocker to doing so would be SRE resources. :)

Eevans edited projects, added Platform Team Workboards (Clinic Duty Team); removed Platform Engineering (Needs Cleaning - Services Operations).Aug 23 2019, 8:47 PM
In T224553#5431986, @Eevans wrote:
In T224553#5431770, @MoritzMuehlenhoff wrote:

T208087, T223976 and T222960 are fixed. Could we get restbase2009-restbase2012, restbase1018 (and the two remaining -dev servers) migrated to Stretch in the next 1-2 months?

cassandra/restbase is the only remaining use case for our custom OpenJDK 8 backports in jessie-wikimedia and it would be fantastic not to spend more time on this when the October Java security release gets released.

Assuming we're adhering to the process of decommissioning, re-imaging, and bootstrapping, then I suspect the only blocker to doing so would be SRE resources. :)

After decommissioning the instances, I need someone to do the re-image to Stretch. Once the first successful Puppet run has occurred, I can take over again until the next host is ready for a re-image (lather, rinse, repeat, etc). @MoritzMuehlenhoff, I currently have some time to work on this, if you (or someone else on the SRE-side) does too I'm happy to get started, otherwise we can run it up the chain to have it scheduled.

Eevans moved this task from Inbox to Blocked Externally on the Platform Team Workboards (Clinic Duty Team) board.Aug 23 2019, 8:49 PM
Eevans added a subtask: T224554: Migrate Restbase-dev cluster to Stretch.Aug 26 2019, 9:26 PM
Eevans added a project: User-Eevans.Sep 4 2019, 8:12 PM
Eevans moved this task from Backlog to In-Progress on the User-Eevans board.
Stashbot added a comment.Sep 11 2019, 4:39 PM

Mentioned in SAL (#wikimedia-operations) [2019-09-11T16:39:28Z] <urandom> decommissioning Cassandra, restbase1018-a -- T224553

Stashbot added a comment.Sep 11 2019, 6:43 PM

Mentioned in SAL (#wikimedia-operations) [2019-09-11T18:43:47Z] <urandom> decommissioning Cassandra, restbase1018-b -- T224553

Stashbot added a comment.Sep 11 2019, 10:30 PM

Mentioned in SAL (#wikimedia-operations) [2019-09-11T22:30:26Z] <urandom> decommissioning Cassandra, restbase1018-c -- T224553

• ayounsi mentioned this in T232007: Restbase: significant increase of outbound dropped packets.Sep 11 2019, 11:19 PM
Eevans added a comment.Sep 12 2019, 1:12 AM

restbase1018 is decommissioned and ready to be reimaged.

MoritzMuehlenhoff closed subtask T224554: Migrate Restbase-dev cluster to Stretch as Resolved.Sep 12 2019, 12:27 PM
gerritbot added a comment.Sep 12 2019, 2:27 PM

Change 536201 had a related patch set uploaded (by Muehlenhoff; owner: Muehlenhoff):
[operations/puppet@production] Add missing JBOD config for restbase1018

https://gerrit.wikimedia.org/r/536201

gerritbot added a project: Patch-For-Review.Sep 12 2019, 2:27 PM
gerritbot added a comment.Sep 12 2019, 3:00 PM

Change 536201 merged by Muehlenhoff:
[operations/puppet@production] Add missing JBOD config for restbase1018

https://gerrit.wikimedia.org/r/536201

MoritzMuehlenhoff added a comment.Sep 12 2019, 3:09 PM

restbase1018 is reimaged and ready for Cassandra bootstrap.

Maintenance_bot removed a project: Patch-For-Review.Sep 12 2019, 3:10 PM
Stashbot added a comment.Sep 12 2019, 4:09 PM

Mentioned in SAL (#wikimedia-operations) [2019-09-12T16:09:48Z] <urandom> bootstrapping Cassandra, restbase1018-a -- T224553

Stashbot added a comment.Sep 12 2019, 5:46 PM

Mentioned in SAL (#wikimedia-operations) [2019-09-12T17:46:45Z] <urandom> bootstrapping Cassandra, restbase1018-b -- T224553

Stashbot added a comment.Sep 12 2019, 7:34 PM

Mentioned in SAL (#wikimedia-operations) [2019-09-12T19:34:13Z] <urandom> bootstrapping Cassandra, restbase1018-c -- T224553

Eevans updated the task description. (Show Details)Sep 12 2019, 8:51 PM
Stashbot added a comment.Sep 12 2019, 9:00 PM

Mentioned in SAL (#wikimedia-operations) [2019-09-12T21:00:02Z] <urandom> decommissioning Cassandra, restbase2009 -- T224553

gerritbot added a comment.Sep 12 2019, 10:25 PM

Change 536367 had a related patch set uploaded (by Eevans; owner: Eevans):
[operations/puppet@production] hieradata: specify restbase2009 jbod devices

https://gerrit.wikimedia.org/r/536367

gerritbot added a project: Patch-For-Review.Sep 12 2019, 10:25 PM
Stashbot added a comment.Sep 12 2019, 11:21 PM

Mentioned in SAL (#wikimedia-operations) [2019-09-12T23:21:16Z] <urandom> decommissioning Cassandra, restbase2009-b -- T224553

Eevans added a comment.Sep 13 2019, 10:21 AM

restbase2009 is fully decommissioned and ready to be reimaged.

Stashbot added a comment.Sep 13 2019, 10:41 AM

Mentioned in SAL (#wikimedia-operations) [2019-09-13T10:41:52Z] <moritzm> reimage restbase2009 to stretch T224553

gerritbot added a comment.Sep 13 2019, 10:58 AM

Change 536367 merged by Muehlenhoff:
[operations/puppet@production] hieradata: specify restbase2009 jbod devices

https://gerrit.wikimedia.org/r/536367

Maintenance_bot removed a project: Patch-For-Review.Sep 13 2019, 11:10 AM
MoritzMuehlenhoff updated the task description. (Show Details)Sep 13 2019, 11:39 AM

restbase2009 has been reimaged and is ready to be bootstrapped in Cassandra.

Stashbot added a comment.Sep 13 2019, 3:47 PM

Mentioned in SAL (#wikimedia-operations) [2019-09-13T15:47:28Z] <urandom> bootstrapping Cassandra, restbase2009-a -- T224553

Stashbot added a comment.Sep 13 2019, 5:24 PM

Mentioned in SAL (#wikimedia-operations) [2019-09-13T17:24:45Z] <urandom> bootstrapping Cassandra, restbase2009-b -- T224553

Stashbot added a comment.Sep 13 2019, 7:54 PM

Mentioned in SAL (#wikimedia-operations) [2019-09-13T19:54:02Z] <urandom> bootstrapping Cassandra, restbase2009-c -- T224553

Stashbot added a comment.Sep 16 2019, 12:58 PM

Mentioned in SAL (#wikimedia-operations) [2019-09-16T12:58:52Z] <urandom> decommissioning Cassandra, restbase2010-a -- T224553

Stashbot added a comment.Sep 16 2019, 2:54 PM

Mentioned in SAL (#wikimedia-operations) [2019-09-16T14:54:26Z] <urandom> decommissioning Cassandra, restbase2010-b -- T224553

Stashbot added a comment.Sep 16 2019, 5:42 PM

Mentioned in SAL (#wikimedia-operations) [2019-09-16T17:42:56Z] <urandom> decommissioning Cassandra, restbase2010-c -- T224553

Eevans added a comment.Sep 16 2019, 11:07 PM

restbase2010 is ready to be reimaged.

Stashbot added a comment.Sep 17 2019, 6:49 AM

Mentioned in SAL (#wikimedia-operations) [2019-09-17T06:49:33Z] <moritzm> reimage restbase2010 to Stretch T224553

MoritzMuehlenhoff updated the task description. (Show Details)Sep 17 2019, 8:28 AM

restbase2010 has been reimaged and is ready to be bootstrapped in Cassandra.

Stashbot added a comment.Sep 17 2019, 9:16 AM

Mentioned in SAL (#wikimedia-operations) [2019-09-17T09:16:25Z] <mobrovac> bootstrap restbase2010-a - T224553

Stashbot added a comment.Sep 17 2019, 10:58 AM

Mentioned in SAL (#wikimedia-operations) [2019-09-17T10:58:24Z] <mobrovac> bootstrap restbase2010-b - T224553

Stashbot added a comment.Sep 17 2019, 12:30 PM

Mentioned in SAL (#wikimedia-operations) [2019-09-17T12:30:41Z] <mobrovac> bootstrap restbase2010-c - T224553

Stashbot added a comment.Sep 17 2019, 3:38 PM

Mentioned in SAL (#wikimedia-operations) [2019-09-17T15:38:05Z] <urandom> decommissioning Cassandra, restbase2011-a -- T224553

Stashbot added a comment.Sep 17 2019, 5:09 PM

Mentioned in SAL (#wikimedia-operations) [2019-09-17T17:09:05Z] <urandom> decommissioning Cassandra, restbase2011-b -- T224553

Stashbot added a comment.Sep 17 2019, 7:05 PM

Mentioned in SAL (#wikimedia-operations) [2019-09-17T19:05:56Z] <urandom> decommissioning Cassandra, restbase2011-c -- T224553

Eevans added a comment.Sep 17 2019, 9:14 PM

restbase2011 is fully decommissioned and ready to be reimaged.

Stashbot added a comment.Sep 18 2019, 6:43 AM

Mentioned in SAL (#wikimedia-operations) [2019-09-18T06:43:51Z] <moritzm> reimaging restbase2011 to stretch T224553

MoritzMuehlenhoff updated the task description. (Show Details)Sep 18 2019, 8:04 AM
Stashbot added a comment.Sep 18 2019, 8:22 AM

Mentioned in SAL (#wikimedia-operations) [2019-09-18T08:22:11Z] <mobrovac> bootstrap restbase2011-a -- T224553

Stashbot added a comment.Sep 18 2019, 10:00 AM

Mentioned in SAL (#wikimedia-operations) [2019-09-18T10:00:39Z] <mobrovac> bootstrap restbase2011-b -- T224553

Stashbot added a comment.Sep 18 2019, 11:29 AM

Mentioned in SAL (#wikimedia-operations) [2019-09-18T11:29:57Z] <mobrovac> bootstrap restbase2011-c -- T224553

Stashbot added a comment.Sep 18 2019, 3:53 PM

Mentioned in SAL (#wikimedia-operations) [2019-09-18T15:53:08Z] <urandom> decommissioning Cassandra, restbase2012-a -- T224553

Stashbot added a comment.Sep 18 2019, 5:50 PM

Mentioned in SAL (#wikimedia-operations) [2019-09-18T17:50:51Z] <urandom> decommissioning Cassandra, restbase2012-b -- T224553

Stashbot added a comment.Sep 18 2019, 7:57 PM

Mentioned in SAL (#wikimedia-operations) [2019-09-18T19:57:06Z] <urandom> decommissioning Cassandra, restbase2012-c -- T224553

Eevans added a comment.Sep 18 2019, 9:25 PM

restbase2012 is decommissioned and can be reimaged at any time.

Stashbot added a comment.Sep 19 2019, 7:01 AM

Mentioned in SAL (#wikimedia-operations) [2019-09-19T07:01:39Z] <moritzm> reimaging restbase2012 to stretch T224553

Stashbot added a comment.Sep 19 2019, 11:48 AM

Mentioned in SAL (#wikimedia-operations) [2019-09-19T11:48:29Z] <mobrovac> bootstrap restbase2012-a -- T224553

MoritzMuehlenhoff added a comment.Sep 19 2019, 12:00 PM

restbase2012 has been reimaged and is ready to be bootstrapped in Cassandra. All jessie Cassandra instances gone!

Stashbot added a comment.Sep 19 2019, 12:39 PM

Mentioned in SAL (#wikimedia-operations) [2019-09-19T12:39:44Z] <mobrovac@deploy1001> Started deploy [restbase/deploy@7f4b7f7]: Start using RESTBase built on Stretch - T224553

Stashbot added a comment.Sep 19 2019, 1:01 PM

Mentioned in SAL (#wikimedia-operations) [2019-09-19T13:01:22Z] <mobrovac@deploy1001> Finished deploy [restbase/deploy@7f4b7f7]: Start using RESTBase built on Stretch - T224553 (duration: 21m 38s)

Stashbot added a comment.Sep 19 2019, 1:12 PM

Mentioned in SAL (#wikimedia-operations) [2019-09-19T13:12:48Z] <mobrovac> bootstrap restbase2012-b -- T224553

Stashbot added a comment.Sep 19 2019, 2:31 PM

Mentioned in SAL (#wikimedia-operations) [2019-09-19T14:31:02Z] <mobrovac> bootstrap restbase2012-c -- T224553

Eevans awarded a token.Sep 19 2019, 2:35 PM
• mobrovac closed this task as Resolved.Sep 19 2019, 3:21 PM
• mobrovac claimed this task.
• mobrovac moved this task from Blocked Externally to Done on the Platform Team Workboards (Clinic Duty Team) board.

This has now been completed. Thank you @MoritzMuehlenhoff for assisting and promptly re-imaging the servers!

• mobrovac updated the task description. (Show Details)Sep 19 2019, 3:21 PM
Stashbot added a comment.Sep 19 2019, 3:26 PM

Mentioned in SAL (#wikimedia-operations) [2019-09-19T15:26:01Z] <moritzm> repooling restbase2012 after completed Cassandra bootstrap T224553

Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits