Page MenuHomePhabricator
Create Task
Maniphest T207670

Popups schema is keeping page/session id longer than 90 days
Closed, ResolvedPublic
Actions

Description

Per our data retention guidelines we should not retain long term articles read by users + sessionId. Please see: https://meta.wikimedia.org/wiki/Data_retention_guidelines#How_long_do_we_retain_non-public_data?

I am not sure if this schema is used but if so its whitelist settings should be adjusted.

Popups:

event:
    action: keep
    api: keep
    checkin: keep
    duration: keep
    editCountBucket: keep
    hovercardsSuppressedByGadget: keep
    isAnon: keep
    linkInteractionToken: keep
    namespaceIdHover: keep
    namespaceIdSource: keep
    pageIdSource: keep
    pageTitleHover: keep -> should be removed 
    pageTitleSource: keep -> shoudl be removed 
    pageToken: keep
    perceivedWait: keep
    popupDelay: keep
    popupEnabled: keep
    previewCountBucket: keep
    previewType: keep
    sessionID: keep -> should be removed
    sessionToken: keep
    totalInteractionTime: keep
    version: keep
webHost: keep
wiki: keep

Details

SubjectRepoBranchLines +/-
Remove page IDs and titles from Popups whitelistanalytics/refinerymaster+0 -3
Customize query in gerrit

Related Objects

Event Timeline

• Nuria created this task.Oct 22 2018, 4:35 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptOct 22 2018, 4:35 PM
• Nuria assigned this task to Jdlrobson.Oct 22 2018, 4:35 PM
• Tbayer claimed this task.Oct 22 2018, 9:11 PM
• Tbayer added a project: Product-Analytics.
• Tbayer added a subscriber: Jdlrobson.
• Tbayer added subscribers: ovasileva, phuedx.Oct 22 2018, 9:14 PM

Thanks for flagging this. I will check with @ovasileva about which of the two are more important from a product perspective, but I guess we will want to drop the page titles and keep the session-related data. (CC @phuedx )

Jdlrobson moved this task from Incoming to Needs Prioritization on the Web-Team-Backlog board.Oct 22 2018, 10:44 PM
• fdans moved this task from Incoming to Radar on the Analytics board.Oct 29 2018, 3:45 PM
kzimmerman moved this task from Triage to Backlog on the Product-Analytics board.Nov 1 2018, 6:46 PM
• Nuria added a comment.EditedNov 1 2018, 8:37 PM

Ping @phuedx can we have someone submit a fix for the issue? See similar work being done by "new contributors" team: https://gerrit.wikimedia.org/r/#/c/analytics/refinery/+/471038/

phuedx added a comment.Nov 2 2018, 10:21 AM
In T207670#4687248, @Tbayer wrote:

Thanks for flagging this. I will check with @ovasileva about which of the two are more important from a product perspective, but I guess we will want to drop the page titles and keep the session-related data. (CC @phuedx )

@Tbayer: Did anything come of this conversation?

• Tbayer mentioned this in T205458: Remove sessionId, pageId pairs from whitelist .Nov 2 2018, 8:38 PM
In T207670#4715295, @phuedx wrote:
In T207670#4687248, @Tbayer wrote:

Thanks for flagging this. I will check with @ovasileva about which of the two are more important from a product perspective, but I guess we will want to drop the page titles and keep the session-related data. (CC @phuedx )

@Tbayer: Did anything come of this conversation?

I put it on the agenda for our regular check-in meeting, which hasn't yet happened since T207670#4687248 but will on Monday.

ovasileva triaged this task as Medium priority.Nov 5 2018, 7:33 PM
In T207670#4687248, @Tbayer wrote:

Thanks for flagging this. I will check with @ovasileva about which of the two are more important from a product perspective, but I guess we will want to drop the page titles and keep the session-related data. (CC @phuedx )

discussed this with @Tbayer and we decided to keep session id and stop collecting page id

phuedx added a comment.EditedNov 6 2018, 11:01 AM
In T207670#4721637, @ovasileva wrote:

discussed this with @Tbayer and we decided to keep session id and stop collecting page id

Did you mean "keeping page ID" here? If you meant "collecting page ID", then we'll need to queue up a change to the Page Previews codebase to remove that field.

gerritbot subscribed.Nov 6 2018, 3:04 PM

Change 471980 had a related patch set uploaded (by HaeB; owner: HaeB):
[analytics/refinery@master] Remove page IDs and titles from Popups whitelist

https://gerrit.wikimedia.org/r/471980

gerritbot added a project: Patch-For-Review.Nov 6 2018, 3:05 PM
• Tbayer added a comment.EditedNov 6 2018, 3:06 PM
In T207670#4723926, @phuedx wrote:
In T207670#4721637, @ovasileva wrote:

discussed this with @Tbayer and we decided to keep session id and stop collecting page id

Did you mean "keeping page ID" here? If you meant "collecting page ID", then we'll need to queue up a change to the Page Previews codebase to remove that field.

Indeed, as mentioned in the task's name and description, this is about retaining data beyond 90 days, see https://wikitech.wikimedia.org/wiki/Analytics/Systems/EventLogging/Data_retention_and_auto-purging. I.e. we decided to stop retaining page IDs (beyond 90 days). I have submitted a patch to remove pageIdSource, pageTitleHover, and pageTitleSource from the whitelist.

• Tbayer added a comment.Nov 7 2018, 1:00 AM
This comment was removed by • Tbayer.
• Tbayer moved this task from Backlog to Doing on the Product-Analytics board.Nov 7 2018, 6:13 PM
Jdlrobson moved this task from Needs Prioritization to Product Owner Backlog on the Web-Team-Backlog board.Nov 7 2018, 9:55 PM

not sure what to do with this one in the needs analysis column @ovasileva + @phuedx - is this something the team needs to estimate and work on or is is this work @Tbayer is carrying out?

gerritbot added a comment.Nov 7 2018, 10:54 PM

Change 471980 merged by Nuria:
[analytics/refinery@master] Remove page IDs and titles from Popups whitelist

https://gerrit.wikimedia.org/r/471980

Jdlrobson removed a project: Patch-For-Review.Nov 16 2018, 12:34 AM
kzimmerman subscribed.Feb 12 2019, 11:48 PM

@Nuria can we close out this task? Is there any other work that needs to be done on the Product Analytics side?

• Nuria closed this task as Resolved.Feb 12 2019, 11:49 PM
Aklapper edited projects, added Analytics-Radar; removed Analytics.Jun 10 2020, 6:44 AM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits