Page MenuHomePhabricator
Create Task
Maniphest T83531

Implement Data Retention Guidelines
Closed, ResolvedPublic
Actions

Description

Parent ticket for all tasks related to the Data Retention Policy.
--
Mark Bergsma <mark at wikimedia>
Lead Operations Architect
Wikimedia Foundation

Details

Reference
rt6858
SubjectRepoBranchLines +/-
purge webrequest logs after 90 daysoperations/puppetproduction+7 -0
toss mw-logs after 90 days, not 180operations/puppetproduction+2 -4
Customize query in gerrit

Related Objects
Search...

StatusSubtypeAssignedTask
 ResolvedNoneT83531 Implement Data Retention Guidelines
 ResolvedNoneT83527 Implement MOTD warning for handling private data for shell users on (all?) systems
 InvalidNoneT83522 Make inventory of (private) data backups on all systems
 ResolvedNoneT83525 Update Server Access Responsibilities document for Data Retention policy
 DuplicateNoneT87246 automate deletion of old logs in stat1002:/a/squid/archive/edits
Restricted Task
Restricted Task
 ResolvedOttomataT92329 Delete stat1002:/a/squid/archive/arabic-banner
 ResolvedOttomataT92330 Delete stat1002:/a/squid/archive/bannerImpressions
 ResolvedOttomataT92331 Delete stat1002:/a/squid/archive/blog
 ResolvedOttomataT92332 Delete stat1002:/a/squid/archive/edits-geocoded
 ResolvedOttomataT92333 Delete stat1002:/a/squid/archive/mobile-geocoded
 ResolvedOttomataT92334 Delete stat1002:/a/squid/archive/sampled-geocoded
 ResolvedOttomataT92335 Delete stat1002:/a/squid/archive/teahouse
 ResolvedakosiarisT92336 Delete gadolinium:/a/log/fundraising/
Restricted Task
Restricted Task
Restricted Task
Restricted Task
Restricted Task
Restricted Task
Restricted Task
Restricted Task
Restricted Task
 ResolvedJgreenT110592 package udp-filter for Trusty, for use on fundraising banner_logger
 ResolvedJgreenT110740 build libcidr package for Trusty
 ResolvedJgreenT110739 build libanon package for trusty
 ResolvedJgreenT112139 build librdkafka for Trusty
 ResolvedJgreenT110591 reformulate kafkatee package to work with Trusty
Restricted Task
 ResolvedawightT97676 Verify kafkatee use for fundraising logs on erbium
 ResolvedcorenT97860 Add andyrussg to udp2log-users group to allow him to verify kafkatee generated fundraising log files on erbium
 ResolvedawightT116800 Impression log parsers should get sample rate from filenames
 ResolvedOttomataT92337 Delete gadolinium:/a/log/nginx/
 ResolvedMilimetricT92338 Purge > 90 days stat1002:/a/squid/archive/api
 ResolvedOttomataT92339 Purge > 90 days stat1002:/a/squid/archive/edits
 ResolvedMilimetricT92340 Purge > 90 days stat1002:/a/squid/archive/glam_nara
 ResolvedMilimetricT92341 Purge > 90 days stat1002:/a/squid/archive/mobile
 ResolvedMilimetricT92342 Purge > 90 days stat1002:/a/squid/archive/sampled
 ResolvedMilimetricT92343 Purge > 90 days stat1002:/a/squid/archive/zero
 ResolvedOttomataT92344 Delete stat1002:/a/squid/archive/sopa
 ResolvedOttomataT75084 Delete vanadium:/srv/eventlogging
 ResolvedMilimetricT130729 Check if we can deprecate legacy TSVs production (same time as pagecounts?)
 ResolvedNoneT92839 fix up log retention on log collection/storage hosts
 ResolvedNoneT110066 finish and automate data retention scripts
 ResolvedNoneT111020 Retention auditing: clean up rules db contents and use
 ResolvedNoneT111021 Data retention: revise audit bash scripts

Event Timeline

• rtimport raised the priority of this task from to Medium.Dec 18 2014, 1:49 AM
• rtimport added a project: ops-core.
• rtimport set Reference to rt6858.
mark created this task.Feb 17 2014, 2:39 PM
mark added a comment.Feb 17 2014, 2:40 PM

Member ticket #6855 added by mark

mark added a comment.Feb 17 2014, 2:40 PM

Member ticket #6856 added by mark

mark added a comment.Feb 17 2014, 2:40 PM

Member ticket #6857 added by mark

mark added a comment.Feb 17 2014, 2:43 PM

Subject changed from 'Implement Data Retention Policy' to 'Implement Data Retention Guidelines' by mark

ArielGlenn added a comment.Apr 14 2014, 3:27 PM

Jeff and I will be taking a first whack at this; notes on
https://office.wikimedia.org/wiki/User:ArielGlenn/Data_Retention and adding
Jeff on cc

• rtimport added a comment.Apr 14 2014, 3:27 PM

Status changed from 'new' to 'open' by RT_System

Dzahn added a subtask: T87246: automate deletion of old logs in stat1002:/a/squid/archive/edits.Jan 21 2015, 4:52 AM
• kevinator reopened this task as Open.Jan 30 2015, 12:06 AM
• kevinator closed this task as a duplicate of Restricted Task.
• kevinator set Security to None.
• kevinator subscribed.Jan 30 2015, 12:10 AM
This comment was removed by • kevinator.
faidon added a subtask: Restricted Task.Feb 12 2015, 8:57 PM
Dzahn added subscribers: Ottomata, Dzahn.Apr 29 2015, 6:46 PM

also see this patch for "purge webrequest logs after 90 days", just added a bug

https://gerrit.wikimedia.org/r/#/c/197081/

see comment from @Ottomata though -> "please wait on this, or sync up with analytics :)"

Dzahn mentioned this in rOPUPf84021509c36: toss mw-logs after 90 days, not 180.Jul 30 2015, 10:34 PM
Dzahn added a comment.Jul 30 2015, 10:35 PM

mw logs are now deleted after 90 days, not 180 anymore:

https://gerrit.wikimedia.org/r/#/c/195917/

the webrequests log change in the last comment above is still open

Restricted Application added a subscriber: Matanya. · View Herald TranscriptJul 30 2015, 10:35 PM
ArielGlenn added a subtask: T110066: finish and automate data retention scripts.Aug 24 2015, 5:24 PM
Krenair subscribed.Aug 25 2015, 12:13 AM
Dzahn added a comment.Sep 22 2015, 5:54 AM

https://gerrit.wikimedia.org/r/#/c/197081/ is stalled because of the comments on Gerrit from Kevinator

"We need to keep some data for legal purposes. Also there also needs to be some assessments done to some of the subdirectories. Not all have been cleared for deletion:"

and links to T84618

Dzahn added a subtask: Restricted Task.Sep 22 2015, 5:54 AM
Dzahn added a project: audits-data-retention.Feb 12 2016, 9:40 PM
Dzahn moved this task from Backlog to tracking on the audits-data-retention board.Feb 12 2016, 9:55 PM
Dzahn moved this task from tracking to Backlog on the audits-data-retention board.Feb 12 2016, 10:09 PM
ArielGlenn moved this task from Backlog to tracking on the audits-data-retention board.Feb 12 2016, 10:48 PM
fgiunchedi subscribed.Apr 28 2016, 2:11 PM

objections to making this task public?

• Nuria closed subtask Restricted Task as Resolved.Dec 6 2016, 6:06 PM
fgiunchedi changed the visibility from "WMF-NDA (Project)" to "Public (No Login Required)".Apr 15 2020, 2:54 PM
fgiunchedi changed the edit policy from "WMF-NDA (Project)" to "acl*sre-team (Project)".
fgiunchedi closed subtask Restricted Task as Resolved.Apr 29 2022, 9:43 AM
fgiunchedi closed subtask T83522: Make inventory of (private) data backups on all systems as Invalid.Apr 29 2022, 9:54 AM
LSobanski added subscribers: MoritzMuehlenhoff, ArielGlenn, LSobanski.Nov 2 2022, 12:36 PM

Resolving after a conversation with @MoritzMuehlenhoff and @ArielGlenn. This task is old enough to not be actionable in its current state. If you feel like this work still needs to happen please reach out to me.

LSobanski closed this task as Resolved.Nov 2 2022, 12:38 PM
LSobanski closed subtask T110066: finish and automate data retention scripts as Resolved.
LSobanski closed subtask T83525: Update Server Access Responsibilities document for Data Retention policy as Resolved.
LSobanski closed subtask T83527: Implement MOTD warning for handling private data for shell users on (all?) systems as Resolved.
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL