Page MenuHomePhabricator
Create Task
Maniphest T192305

Index and store page preview agreggates on Druid so they are visible in pivot/superset
Closed, ResolvedPublic5 Estimated Story Points
Actions

Description

The data set is up to date in druid and is being indexed every day (with hourly granularity).
Still the monthly segment compaction is missing, because of a bug, whose fix will be deployed in short, see patch above.

Details

SubjectRepoBranchLines +/-
Fix bug in virtualpageview druid monthly indexationanalytics/refinerymaster+1 -1
Add jobs for druid indexing of virtualpageviewsanalytics/refinerymaster+1 K -7
Customize query in gerrit

Related Objects
Search...

StatusSubtypeAssignedTask
 ResolvedDerecksonT68374 Enable Hovercards on se.wikimedia.org (Swedish chapter wiki)
 ResolvedJdlrobsonT70860 [GOAL] Graduate Page Previews feature (Popups extension) out of Beta Feature
 ResolvedovasilevaT154635 [EPIC] Deploy page previews to English and German Wikipedia
 DuplicateNoneT184801 Remove A/B testing code and EventLogging instrumentation
 OpenNoneT193524 Publish data on seen page previews
 ResolvedmfornsT186728 Record and aggregate page previews
 ResolvedovasilevaT192622 [EPIC] Page previews post-deploy cleanup
 ResolvedJdlrobsonT173952 Remove A/B testing instrumentation code
 DuplicateNoneT167433 Switch all projects to the new (and yet to be built) summary-html endpoint for page previews
 DuplicateNoneT167429 Make enwiki and dewiki fetch previews from the summary-html RESTBase endpoint
 ResolvedovasilevaT165018 Page previews can consume new summary-HTML endpoint
 ResolvedovasilevaT168332 HTML previews' layout breaks text multi-line text truncation
 ResolvedovasilevaT113094 [EPIC] The Page Summary API needs to provide useful content for the majority of articles
 ResolvedphuedxT113633 Spike: Alternative to TextExtracts for Popups, Gather, Read more
 ResolvedphuedxT109867 Infobox template parameters showing in Hovercard
 DuplicateNoneT109869 Hovercards should not show <noinclude> content in the extract
 ResolvedJdlrobsonT74629 Reference lists appearing in extracts for some articles
 ResolvedJdlrobsonT73023 Invalid HTML markup standalone LI element when using references on main page
 DuplicateNoneT112137 Hovercards loses manual superscript formatting by requesting plain text
 DuplicateovasilevaT152414 Previews must display useful text for the majority of articles
 ResolvedovasilevaT159388 Text Extracts used in Page Previews shows strange characters from template
 DuplicateNoneT162219 Parentheticals: Words incorrectly concatenated due to too simple removing of spaces
 DeclinedNoneT159065 Fix formulas in HTML extracts
 DuplicateJdlrobsonT163442 Math formulas are not rendering in preview page dialogs
 ResolvedMhurdT155573 [Regression] exclude pronunciation guides from article extracts
 Declined• JMinorT164100 Consider excluding pronunciation guides from TextExtracts
 ResolvedphuedxT112226 Text extracts unable to summarise content contains {{dts}} templates
 Resolved• PcheloloT165017 Setup RESTBase HTML extract endpoint
 ResolvedJdlrobsonT165619 Spike: Specify changes to page-summary endpoint
 Declined• PcheloloT166163 Create a summary content migration filter
 ResolvedphuedxT167852 "<translate>" and <tvar|*> tag visible in Page Previews: HtmlFormatter only flattens spans
 DeclinedJdlrobsonT168329 exsentences does not work correctly when HTML output used
 DuplicateNoneT168625 Make the Page Summary API return an "intro" for a page
 Resolved• FjalapenoT167022 Inventory requirements for summary 2.0 endpoint across Reading teams
 ResolvedphuedxT168400 [EPIC] Add notion of emptiness to page summary API
 DeclinedpmiazgaT168328 Extracts for Wikimedia List articles display partial previews
 DeclinedNoneT168391 Page previews should display generic preview for disambiguation pages
 ResolvedphuedxT114418 Disable TextExtracts on file pages
 DeclinedNoneT168418 Page Previews should accept the service's notion of emptiness
 Resolved• PcheloloT164291 Make title-related properties consistent
 Resolved• bearNDT169761 Review Summary 2.0 Spec
 Resolved• MhollowayT170692 Return common URLs in summary API so clients do not have to perform bug prone string manipulation
ResolvedMSantosT177619 Return variant URLs and titles in the metadata response
 Resolved• MhollowayT178446 Expose display titles for a page in all available language variants through the action API
 InvalidNoneT69232 Hovercards: Redirect titles should be bolded in the extract
 ResolvedJdlrobsonT170617 Adjust expectations for API consumers when using the TextExtracts API
 InvalidNoneT185472 API problems with double spaces in wiki sections
 DuplicateNoneT171053 Complete entire page previews test plan for page summary api
 ResolvedJdlrobsonT171052 Add disambiguation page handling in Page Summary API
 ResolvedJdlrobsonT168848 Bootstrap an initial version of the Page Summary API in MCS
 ResolvedJdlrobsonT172021 It shouldn't be possible for coordinates to be the lead paragraph
 ResolvedJdlrobsonT174698 Parenthetical stripping is too aggressive
 ResolvedphuedxT171065 Expose disambiguation property (ppprop=disambiguation) in mobileview api
 Resolved• bearNDT175286 Do side by side comparison of old summary endpoint against new summary endpoint
 Resolved• MhollowayT176974 Mime vulnerability blocking merges to MCS
 DuplicateNoneT176063 Page preview shows only the first two words for a specific article
 ResolvedJdlrobsonT176517 Pages that are redirects give empty summary objects
 ResolvedJdlrobsonT176519 Old templates can lead to sup elements inside summary
 ResolvedphuedxT176521 data-mw attributes should be stripped from summary before scrubbing parentheticals
ResolvedJdlrobsonT176522 Intro property incorrectly identified in formatted endpoint
 ResolvedJdlrobsonT176525 Parenthetical: New edge case with nested brackets
 Resolved• bearNDT177007 Should we flatten spans in summary output
 ResolvedJdlrobsonT178125 Metadata (coordinates) is sometimes surfaced as lead intro/summary
 Resolved• MhollowayT178333 Move RESTBase page summary logic to MCS
 Resolved• MhollowayT178420 What is a "content namespace" for purposes of the summary 2.0 endpoint?
 ResolvedJdlrobsonT183833 [Bug report] Removing parentheses breaks chemical formulas
 ResolvedJdlrobsonT185050 Run comparison of html extracts again
 Resolved• bearNDT185161 Stripping parentheticals can result in trailing punctuation
 Resolved• TbayerT182315 Write and publish report on enwiki and dewiki a/b test analysis for page previews
 ResolvedNoneT184793 [EPIC] Instrument page interactions
 ResolvedmfornsT192305 Index and store page preview agreggates on Druid so they are visible in pivot/superset

Event Timeline

• Nuria triaged this task as High priority.Apr 16 2018, 4:44 PM
• Nuria created this task.
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptApr 16 2018, 4:44 PM
• Nuria assigned this task to mforns.Apr 16 2018, 4:44 PM
• Nuria moved this task from Incoming to Smart Tools for Better Data on the Analytics board.Apr 16 2018, 5:44 PM
mforns moved this task from Next Up to In Progress on the Analytics-Kanban board.Apr 17 2018, 6:28 PM
• Tbayer awarded a token.Apr 19 2018, 3:11 AM
• Tbayer subscribed.
gerritbot subscribed.Apr 19 2018, 4:06 PM

Change 427696 had a related patch set uploaded (by Mforns; owner: Mforns):
[analytics/refinery@master] Add jobs for druid indexing of virtualpageviews

https://gerrit.wikimedia.org/r/427696

gerritbot added a project: Patch-For-Review.Apr 19 2018, 4:06 PM
mforns moved this task from In Progress to In Code Review on the Analytics-Kanban board.Apr 23 2018, 3:20 PM
gerritbot added a comment.May 8 2018, 8:06 PM

Change 427696 merged by Mforns:
[analytics/refinery@master] Add jobs for druid indexing of virtualpageviews

https://gerrit.wikimedia.org/r/427696

mforns moved this task from In Code Review to Ready to Deploy on the Analytics-Kanban board.May 8 2018, 8:07 PM
gerritbot added a comment.May 11 2018, 7:58 PM

Change 432643 had a related patch set uploaded (by Mforns; owner: Mforns):
[analytics/refinery@master] Fix bug in virtualpageview druid monthly indexation

https://gerrit.wikimedia.org/r/432643

mforns updated the task description. (Show Details)May 11 2018, 8:06 PM
gerritbot added a comment.May 11 2018, 11:09 PM

Change 432643 merged by Nuria:
[analytics/refinery@master] Fix bug in virtualpageview druid monthly indexation

https://gerrit.wikimedia.org/r/432643

mforns added a comment.May 15 2018, 5:25 PM

Virtualpageviews_hourly is in druid!
https://pivot.wikimedia.org/#virtualpageviews-hourly/line-chart/2/EQUQLgxg9AqgKgYWAGgN7APYAdgC5gQAWAhgJYB2KwApgB5YBO1Azs6RpbutnsEwGZVyxALbVeAfQlhSY4AF9kwYhBkc86FWs7AKVOoxZt1XTDnwAbUmGoNiFqgDd7AV3GnmYYgzC8ATAAMAIwAHAC0AQDMYUEALHABAbiJyQEAdIkBAFr65AAm/sHhAQCsMSUJSSmJGYk5isBgAJ5Y7sBwAJIAsiASAEoAggByAOIgCvKKANroBkys7Jzc5nzUgkrCcvhSMnINAEYuEADW1GADqosaypcmjbLUAEJHp75KeS522rwACkEAIgolMwMD4Lt9TForvgQT59PR5sYlmZeAIhKI2jsHkDgHlSExbjpiMwINR8hQAOYTAC6yHILgsFiUjlI1AA7hIIBgXOQ3lNgCz2ZzubzgLSprT6YygA===

mforns moved this task from Ready to Deploy to Done on the Analytics-Kanban board.May 15 2018, 5:27 PM
• Tbayer added a comment.May 16 2018, 11:12 AM

Thanks! Already looked at it in Superset with the web team yesterday.

• Nuria closed this task as Resolved.May 29 2018, 11:20 AM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL