Page MenuHomePhabricator
Create Task
Maniphest T254491

Puppet labs/private.git data loss incident affecting some projects
Closed, ResolvedPublic
Actions

Description

At 2020-06-04T11:12 UTC a change was merged to the operations/puppet.git repository which resulted in data loss for Cloud VPS projects using a local Puppetmaster (role::puppetmaster::standalone). The specific data loss is removal of any local to the Puppetmaster instance commits overlaid on the upstream labs/private.git repository. These patches would have contained passwords, ssh keys, TLS certificates, and similar authentication information for Puppet managed configuration.

Incident doc: https://wikitech.wikimedia.org/wiki/Incidents/2020-06-04_cloud-private-repo

Details

ProjectBranchLines +/-Subject
operations/puppetproduction+3 -0labtestpuppet: puppet::servers
Customize query in gerrit

Related Objects
Search...

Event Timeline

bd808 created this task.Jun 4 2020, 3:12 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptJun 4 2020, 3:12 PM
bd808 triaged this task as Unbreak Now! priority.Jun 4 2020, 3:12 PM
bd808 moved this task from Inbox to Doing on the cloud-services-team (Kanban) board.
Restricted Application added a subscriber: Liuxinyu970226. · View Herald TranscriptJun 4 2020, 3:12 PM
RhinosF1 added a subscriber: RhinosF1.Jun 4 2020, 3:13 PM
taavi added a subscriber: taavi.Jun 4 2020, 3:16 PM
bd808 added a comment.Jun 4 2020, 3:19 PM

Announced to community on cloud-announce: https://lists.wikimedia.org/pipermail/cloud-announce/2020-June/000291.html

bd808 claimed this task.Jun 4 2020, 3:20 PM
• Mholloway added a subscriber: • Mholloway.Jun 4 2020, 3:20 PM
bd808 added a subtask: T254473: Prevents labs/private to disappear from integration and deployment-prep puppet master.Jun 4 2020, 3:20 PM
bd808 added subscribers: Andrew, aborrero, jbond.
PeterBowman added a subscriber: PeterBowman.Jun 4 2020, 3:25 PM
bd808 added a comment.Jun 4 2020, 7:34 PM
[19:21]  <andrewbogott> One investigative tool is: 1) edit crontab to comment out the periodic puppet run 2) enable puppet 3) puppet agent --noop -tv

The crontab is /etc/cron.d/puppet

Stashbot added a comment.Jun 4 2020, 8:07 PM

Mentioned in SAL (#wikimedia-cloud) [2020-06-04T20:07:49Z] <bd808> Checked all hosts and found no missing secrets; puppet re-enabled (T254491)

Stashbot added a comment.Jun 4 2020, 8:57 PM

Mentioned in SAL (#wikimedia-cloud) [2020-06-04T20:57:25Z] <bd808> Checked all hosts and found no missing secrets; puppet re-enabled (T254491)

hashar added a subscriber: hashar.Jun 4 2020, 9:05 PM
bd808 added a subtask: Restricted Task.Jun 4 2020, 9:40 PM
bd808 lowered the priority of this task from Unbreak Now! to High.Jun 4 2020, 9:47 PM
bd808 added a subscriber: Krenair.

Lowering priority from UBN to High.

@Andrew, @aborrero, @jbond, @hashar, @Krenair, and @bd808 have worked through all of the projects where we felt that secret loss was likely. We believe that we have managed to recover almost all lost secrets. There are a few things left to do before we re-enable Puppet everywhere, but that list is short and non-urgent. Out of an abundance of caution we are not going to do these last steps until more of us are fresh and simultaneously available. We will also work on a proper incident report at that time.

bd808 removed bd808 as the assignee of this task.Jun 4 2020, 10:56 PM
jbond added a comment.Jun 5 2020, 1:03 PM

i have made a first pass at the incident report. The main sections which could use some more input are

  • impact
  • detection
  • documentation
  • actionable

The conclusion and timeline are likely also missing some points

Andrew added a comment.Jun 5 2020, 2:51 PM

Thank you @jbond!

Andrew mentioned this in T254589: Add new cloud-cumin keys.Jun 5 2020, 3:05 PM
Amorymeltzer added a subscriber: Amorymeltzer.Jun 5 2020, 5:37 PM
bd808 added a subtask: T254589: Add new cloud-cumin keys.Jun 5 2020, 7:31 PM
Paladox added a subscriber: Paladox.Jun 5 2020, 7:33 PM
Andrew closed subtask T254589: Add new cloud-cumin keys as Resolved.Jun 6 2020, 4:34 AM
TheDJ added a subscriber: TheDJ.Jun 7 2020, 6:12 PM
hashar added a comment.Jun 8 2020, 7:29 AM
In T254491#6196244, @jbond wrote:

i have made a first pass at the incident report. The main sections which could use some more input are

I have added a few bits:

  • fixed date (was May 11)
  • details how it got handled for deployment-prep / integration
  • added reference links to the IRC logs for #wikimedia-cloud #wikimedia-operations and #wikimedia-releng

The incident page <code>20200605-cloud-private-repo</code> has the date the page has been created which is a day after the incident occurred (June 4th). That caused me a bit of confusion, I would recommend to move the page <code>Incident_documentation/20200604-cloud-private-repo</code>.

bd808 added a comment.Jun 8 2020, 6:11 PM
In T254491#6200618, @hashar wrote:

The incident page <code>20200605-cloud-private-repo</code> has the date the page has been created which is a day after the incident occurred (June 4th). That caused me a bit of confusion, I would recommend to move the page <code>Incident_documentation/20200604-cloud-private-repo</code>.

{{Done}} Page moved with redirect from original title to new title which matches the incident date.

• Mholloway removed a subscriber: • Mholloway.Jun 8 2020, 6:20 PM
MoritzMuehlenhoff added a subscriber: MoritzMuehlenhoff.Jun 9 2020, 3:51 PM
Andrew mentioned this in T254931: missing maps postgres passwords in clouddb-services.Jun 9 2020, 7:07 PM
Stashbot added a comment.Jun 9 2020, 9:24 PM

Mentioned in SAL (#wikimedia-cloud) [2020-06-09T21:24:50Z] <bd808> Credentials missing from es7 cluster for tool access. Fallout from T254491

bd808 added a comment.Jun 9 2020, 9:36 PM
In T254491#6207860, @Stashbot wrote:

Mentioned in SAL (#wikimedia-cloud) [2020-06-09T21:24:50Z] <bd808> Credentials missing from es7 cluster for tool access. Fallout from T254491

This was the bash tool. Its password did not make it into the recovery done by @aborrero. I went ahead and generated a new password for it, added that to the local commit on the puppetmaster, and updated the credentials in the tool.

bd808 updated the task description. (Show Details)Jun 9 2020, 9:37 PM
bd808 added a subtask: T254931: missing maps postgres passwords in clouddb-services.Jun 9 2020, 10:42 PM
Bstorm closed subtask T254931: missing maps postgres passwords in clouddb-services as Resolved.Jun 9 2020, 11:25 PM
gerritbot added a comment.Jun 11 2020, 1:00 PM

Change 604696 had a related patch set uploaded (by Jbond; owner: John Bond):
[operations/puppet@production] labtestpuppet: puppet::servers

https://gerrit.wikimedia.org/r/604696

gerritbot added a project: Patch-For-Review.Jun 11 2020, 1:00 PM
bd808 added a comment.Jun 11 2020, 11:39 PM

I found this lurking in the deep backlog: T115177: Make sure that the 'secret' repo in self hosted puppetmasters is back-upable. Still not a bad idea.

gerritbot added a comment.Jun 12 2020, 9:24 AM

Change 604696 merged by Jbond:
[operations/puppet@production] labtestpuppet: puppet::servers

https://gerrit.wikimedia.org/r/604696

Maintenance_bot removed a project: Patch-For-Review.Jun 12 2020, 10:10 AM
aborrero closed this task as Resolved.May 11 2021, 4:31 PM
aborrero claimed this task.
Krinkle added a project: Sustainability (Incident Followup).Aug 19 2022, 3:11 PM
Krinkle updated the task description. (Show Details)
Restricted Application added a project: Infrastructure-Foundations. · View Herald TranscriptAug 19 2022, 3:11 PM
Content licensed under Creative Commons Attribution-ShareAlike 3.0 (CC-BY-SA) unless otherwise noted; code licensed under GNU General Public License (GPL) or other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL