Page MenuHomePhabricator
Create Task
Maniphest T420506

Setup backup[12]01[456789] & backup[12]020 and migrate data to them; prepare for decommission backup[12]00[34567]
Closed, ResolvedPublic
Actions

Description

Setup:

  • backup1014
  • backup1015
  • backup1016
  • backup1017
  • backup1018
  • backup1019
  • backup1020
  • backup2014
  • backup2015
  • backup2016
  • backup2017
  • backup2018
  • backup2019
  • backup2020

Decommission:

  • backup1003
  • backup1004
  • backup1005
  • backup1006
  • backup1007
  • backup2003
  • backup2004
  • backup2005
  • backup2006
  • backup2007

Details

Related Changes in Gerrit:
SubjectAuthorRepoBranchLines +/-
bacula: Remove last references to backup1003 & backup2003Jcrespooperations/puppetproduction+0 -10
backup: Set backup1003 & backup2003 as insetup for decom.Jcrespooperations/puppetproduction+4 -36
bacula: Setup backup1014 & backup2014 also as ES storage nodesJcrespooperations/puppetproduction+41 -8
mediabackup: Set backup[12]00[4-7] as insetupJcrespooperations/puppetproduction+8 -4
mediabackup: Update s3cmd client configuration to eqiad/codfw sitesJcrespooperations/puppetproduction+1 -1
installserver: Treat any attempt to reimage backup hosts as an errorJcrespooperations/puppetproduction+2 -4
mediabackup: Add resharding script using rcloneJcrespooperations/puppetproduction+62 -0
mediabackup: Add region parameter for rcloneJcrespooperations/puppetproduction+1 -0
mediabackup: Make the recovery account obsolete (but not remove it yet)Jcrespooperations/puppetproduction+5 -2
mediabackup: Followup bug fix to b3afe9d, fix syntax errorJcrespooperations/puppetproduction+1 -1
mediabackup: Add missing parameter VGW_REGION corresponding to siteJcrespooperations/puppetproduction+1 -0
mediabackups: Open s3 storage port on storage hosts from working hostsJcrespooperations/puppetproduction+11 -2
mediabackups: Clarify iam directoryJcrespooperations/puppetproduction+3 -0
mediabackup: Deploy new 24 shards (6 hosts) for mediabackups@codfwJcrespooperations/puppetproduction+34 -11
mediabackup: Apply final role for eqiad mediabackup new storagesJcrespooperations/puppetproduction+2 -2
mediabackup: Deploy new 24 shards (6 hosts) for mediabackups@eqiadJcrespooperations/puppetproduction+33 -14
Show related patches Customize query in gerrit

Related Objects
Search...

Event Timeline

• jcrespo created this task.Mar 18 2026, 5:58 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptMar 18 2026, 5:58 PM
• jcrespo triaged this task as High priority.Mar 18 2026, 5:58 PM
gerritbot added a comment.Mar 18 2026, 6:02 PM

Change #1254999 had a related patch set uploaded (by Jcrespo; author: Jcrespo):

[operations/puppet@production] mediabackup: Deploy new 24 shards (6 hosts) for mediabackups@eqiad

https://gerrit.wikimedia.org/r/1254999

gerritbot added a project: Patch-For-Review.Mar 18 2026, 6:02 PM
• jcrespo added a parent task: T410028: Unexpected media growth led to low disk resources on several media backup hosts.Mar 18 2026, 6:03 PM
gerritbot added a comment.Mar 18 2026, 6:13 PM

Change #1255003 had a related patch set uploaded (by Jcrespo; author: Jcrespo):

[operations/puppet@production] mediabackups: Open s3 storage port on storage hosts from working hosts

https://gerrit.wikimedia.org/r/1255003

gerritbot added a comment.Mar 18 2026, 6:14 PM

Change #1255000 had a related patch set uploaded (by Jcrespo; author: Jcrespo):

[operations/puppet@production] mediabackup: Deploy new 24 shards (6 hosts) for mediabackups@codfw

https://gerrit.wikimedia.org/r/1255000

gerritbot added a comment.Mar 19 2026, 6:15 PM

Change #1254999 merged by Jcrespo:

[operations/puppet@production] mediabackup: Deploy new 24 shards (6 hosts) for mediabackups@eqiad

https://gerrit.wikimedia.org/r/1254999

gerritbot added a comment.Mar 19 2026, 6:25 PM

Change #1255828 had a related patch set uploaded (by Jcrespo; author: Jcrespo):

[operations/puppet@production] mediabackup: Apply final role for eqiad mediabackup new storages

https://gerrit.wikimedia.org/r/1255828

gerritbot added a comment.Mar 19 2026, 6:34 PM

Change #1255828 merged by Jcrespo:

[operations/puppet@production] mediabackup: Apply final role for eqiad mediabackup new storages

https://gerrit.wikimedia.org/r/1255828

gerritbot added a comment.Mar 19 2026, 7:44 PM

Change #1255000 merged by Jcrespo:

[operations/puppet@production] mediabackup: Deploy new 24 shards (6 hosts) for mediabackups@codfw

https://gerrit.wikimedia.org/r/1255000

gerritbot added a comment.Mar 19 2026, 8:48 PM

Change #1255877 had a related patch set uploaded (by Jcrespo; author: Jcrespo):

[operations/puppet@production] mediabackups: Clarify iam directory

https://gerrit.wikimedia.org/r/1255877

gerritbot added a comment.Mar 19 2026, 9:03 PM

Change #1255877 merged by Jcrespo:

[operations/puppet@production] mediabackups: Clarify iam directory

https://gerrit.wikimedia.org/r/1255877

gerritbot added a comment.Mar 19 2026, 9:27 PM

Change #1255003 merged by Jcrespo:

[operations/puppet@production] mediabackups: Open s3 storage port on storage hosts from working hosts

https://gerrit.wikimedia.org/r/1255003

Maintenance_bot removed a project: Patch-For-Review.Mar 19 2026, 9:30 PM
gerritbot added a comment.Mar 20 2026, 7:47 AM

Change #1256082 had a related patch set uploaded (by Jcrespo; author: Jcrespo):

[operations/puppet@production] mediabackup: Add missing parameter VGW_REGION corresponding to site

https://gerrit.wikimedia.org/r/1256082

gerritbot added a project: Patch-For-Review.Mar 20 2026, 7:47 AM
gerritbot added a comment.Mar 20 2026, 7:58 AM

Change #1256082 merged by Jcrespo:

[operations/puppet@production] mediabackup: Add missing parameter VGW_REGION corresponding to site

https://gerrit.wikimedia.org/r/1256082

gerritbot added a comment.Mar 20 2026, 8:06 AM

Change #1256100 had a related patch set uploaded (by Jcrespo; author: Jcrespo):

[operations/puppet@production] mediabackup: Followup bug fix to b3afe9d, fix syntax error

https://gerrit.wikimedia.org/r/1256100

gerritbot added a comment.Mar 20 2026, 8:11 AM

Change #1256100 merged by Jcrespo:

[operations/puppet@production] mediabackup: Followup bug fix to b3afe9d, fix syntax error

https://gerrit.wikimedia.org/r/1256100

Maintenance_bot removed a project: Patch-For-Review.Mar 20 2026, 8:30 AM
• jcrespo updated the task description. (Show Details)Mar 20 2026, 9:20 AM

These old hosts cannot be decommissioned yet, they need to migrate its data to the new hosts first- they are still in production, just read only!

gerritbot added a comment.Mar 24 2026, 10:24 AM

Change #1259901 had a related patch set uploaded (by Jcrespo; author: Jcrespo):

[operations/puppet@production] mediabackup: Make the recovery account obsolete (but not remove it yet)

https://gerrit.wikimedia.org/r/1259901

gerritbot added a project: Patch-For-Review.Mar 24 2026, 10:24 AM
gerritbot added a comment.Mar 24 2026, 10:32 AM

Change #1259901 merged by Jcrespo:

[operations/puppet@production] mediabackup: Make the recovery account obsolete (but not remove it yet)

https://gerrit.wikimedia.org/r/1259901

Maintenance_bot removed a project: Patch-For-Review.Mar 24 2026, 11:32 AM
gerritbot added a comment.Mar 26 2026, 10:39 AM

Change #1261379 had a related patch set uploaded (by Jcrespo; author: Jcrespo):

[operations/puppet@production] mediabackup: Add region parameter for rclone

https://gerrit.wikimedia.org/r/1261379

gerritbot added a project: Patch-For-Review.Mar 26 2026, 10:39 AM
gerritbot added a comment.Mar 26 2026, 10:52 AM

Change #1261379 merged by Jcrespo:

[operations/puppet@production] mediabackup: Add region parameter for rclone

https://gerrit.wikimedia.org/r/1261379

gerritbot added a comment.Mar 26 2026, 11:17 AM

Change #1261387 had a related patch set uploaded (by Jcrespo; author: Jcrespo):

[operations/puppet@production] mediabackup: Add resharding script using rclone

https://gerrit.wikimedia.org/r/1261387

• jcrespo added a comment.Mar 26 2026, 11:36 AM

I've started the data migration to the new hosts @ eqiad, emptying backup1004. Emptying a single host would take around ~16 days, maybe more. Other hosts can be done in parallel, to some extent, but ramping up the transfer speed very slowly.

gerritbot added a comment.Mar 26 2026, 1:33 PM

Change #1261387 merged by Jcrespo:

[operations/puppet@production] mediabackup: Add resharding script using rclone

https://gerrit.wikimedia.org/r/1261387

Maintenance_bot removed a project: Patch-For-Review.Mar 26 2026, 2:32 PM
gerritbot added a comment.Apr 1 2026, 8:30 AM

Change #1266148 had a related patch set uploaded (by Jcrespo; author: Jcrespo):

[operations/puppet@production] installserver: Treat any attempt to reimage backup hosts as an error

https://gerrit.wikimedia.org/r/1266148

gerritbot added a project: Patch-For-Review.Apr 1 2026, 8:30 AM
gerritbot added a comment.Apr 1 2026, 9:33 AM

Change #1266148 merged by Jcrespo:

[operations/puppet@production] installserver: Treat any attempt to reimage backup hosts as an error

https://gerrit.wikimedia.org/r/1266148

Maintenance_bot removed a project: Patch-For-Review.Apr 1 2026, 10:32 AM
• jcrespo added a subtask: T424661: Migrate ES backups from backup[12]003 to backup[12]014 and archive the es read only ES clusters in ro-es bacula section.Apr 28 2026, 1:57 PM
gerritbot added a comment.May 22 2026, 9:30 AM

Change #1291904 had a related patch set uploaded (by Jcrespo; author: Jcrespo):

[operations/puppet@production] mediabackup: Update s3cmd client configuration to eqiad/codfw sites

https://gerrit.wikimedia.org/r/1291904

gerritbot added a project: Patch-For-Review.May 22 2026, 9:30 AM
gerritbot added a comment.May 22 2026, 9:38 AM

Change #1291904 merged by Jcrespo:

[operations/puppet@production] mediabackup: Update s3cmd client configuration to eqiad/codfw sites

https://gerrit.wikimedia.org/r/1291904

Maintenance_bot removed a project: Patch-For-Review.May 22 2026, 10:30 AM
• jcrespo added a comment.May 28 2026, 8:12 AM

Verification has finished, all files transferred ok. I found some metadata issues while doing the checking, but those do not block the decommissioning of the old hosts.

gerritbot added a comment.May 28 2026, 8:33 AM

Change #1294929 had a related patch set uploaded (by Jcrespo; author: Jcrespo):

[operations/puppet@production] mediabackup: Set backup[12]00[4-7] as insetup

https://gerrit.wikimedia.org/r/1294929

gerritbot added a project: Patch-For-Review.May 28 2026, 8:33 AM
gerritbot added a comment.May 28 2026, 8:38 AM

Change #1294929 merged by Jcrespo:

[operations/puppet@production] mediabackup: Set backup[12]00[4-7] as insetup

https://gerrit.wikimedia.org/r/1294929

Maintenance_bot removed a project: Patch-For-Review.May 28 2026, 9:32 AM
gerritbot added a comment.May 28 2026, 10:05 AM

Change #1294941 had a related patch set uploaded (by Jcrespo; author: Jcrespo):

[operations/puppet@production] bacula: Setup backup1014 & backup2014 also as ES storage nodes

https://gerrit.wikimedia.org/r/1294941

gerritbot added a project: Patch-For-Review.May 28 2026, 10:05 AM
gerritbot added a comment.May 28 2026, 10:34 AM

Change #1294941 merged by Jcrespo:

[operations/puppet@production] bacula: Setup backup1014 & backup2014 also as ES storage nodes

https://gerrit.wikimedia.org/r/1294941

Maintenance_bot removed a project: Patch-For-Review.May 28 2026, 11:31 AM
• jcrespo closed subtask T424661: Migrate ES backups from backup[12]003 to backup[12]014 and archive the es read only ES clusters in ro-es bacula section as Resolved.Jun 10 2026, 2:09 PM
• jcrespo added a subtask: T431098: Decommission backup2004, backup2005, backup2006 & backup2007.Jul 3 2026, 9:32 AM
• jcrespo added a subtask: T431097: Decommission backup1004, backup1005, backup1006 & backup1007.
• jcrespo updated the task description. (Show Details)Jul 3 2026, 12:21 PM
• jcrespo added a subscriber: Marostegui.

Waiting until mid-August to decom 1003/2003, to ensure 90 day retention of ES db backups is never challenged CC @Marostegui .

VRiley-WMF closed subtask T431097: Decommission backup1004, backup1005, backup1006 & backup1007 as Resolved.Jul 6 2026, 5:13 PM
Jhancock.wm closed subtask T431098: Decommission backup2004, backup2005, backup2006 & backup2007 as Resolved.Aug 3 2026, 4:31 PM
gerritbot added a comment.Aug 4 2026, 9:28 AM

Change #1320768 had a related patch set uploaded (by Jcrespo; author: Jcrespo):

[operations/puppet@production] backup: Set backup1003 & backup2003 as insetup for decom.

https://gerrit.wikimedia.org/r/1320768

gerritbot added a project: Patch-For-Review.Aug 4 2026, 9:28 AM
gerritbot added a comment.Aug 4 2026, 1:12 PM

Change #1320768 merged by Jcrespo:

[operations/puppet@production] backup: Set backup1003 & backup2003 as insetup for decom.

https://gerrit.wikimedia.org/r/1320768

gerritbot added a comment.Aug 4 2026, 1:18 PM

Change #1320928 had a related patch set uploaded (by Jcrespo; author: Jcrespo):

[operations/puppet@production] bacula: Remove last references to backup1003 & backup2003

https://gerrit.wikimedia.org/r/1320928

• jcrespo added a subtask: T433971: decommission backup2003.codfw.wmnet.Aug 4 2026, 1:21 PM
• jcrespo added a subtask: T433970: decommission backup1003.eqiad.wmnet.
gerritbot added a comment.Aug 6 2026, 8:59 AM

Change #1320928 merged by Jcrespo:

[operations/puppet@production] bacula: Remove last references to backup1003 & backup2003

https://gerrit.wikimedia.org/r/1320928

• jcrespo closed this task as Resolved.Aug 6 2026, 9:00 AM
• jcrespo updated the task description. (Show Details)
• jcrespo added a comment.Aug 6 2026, 9:08 AM

Last two hosts sent to decommissioning after reaching mid August buffer. Resolved.

Maintenance_bot removed a project: Patch-For-Review.Aug 6 2026, 9:32 AM
VRiley-WMF closed subtask T433970: decommission backup1003.eqiad.wmnet as Resolved.Aug 6 2026, 5:13 PM
Jhancock.wm closed subtask T433971: decommission backup2003.codfw.wmnet as Resolved.Aug 13 2026, 3:30 PM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits