Page MenuHomePhabricator
Create Task
Maniphest T177426

Develop structured JSON APIs for general consumption
Closed, ResolvedPublic
Actions

Description

This task is for tracking the JSON APIs within the General (middle) layer of the PCS.

The structured data in this layer is broken into 4 distinct APIs:

  1. A Summary API for getting high level data about a page. Most useful for display within lists/previews
  2. A Gallery API for getting info in images contained in a page
  3. A References API for getting all references of a page
  4. A Metadata API for getting "all" other structured data about a page (basically anything that can be extracted like ToC, geo data, page issues, etc…)

These APIs can be used independently or along with the Content HTML API in order to create a native or JS client UI for pages.

Please see the subtasks for the specific API implementations that make up this layer and see the parent tickets for details at the system level.

Related Objects
Search...

StatusSubtypeAssignedTask
 ResolvedNoneT169242 Develop Page Content Service for Reading Clients
 ResolvedNoneT177425 Develop General Layer of PCS
 Resolved• JhernandezT177426 Develop structured JSON APIs for general consumption
 Resolved• bearNDT170690 Extract a References JSON API
 Resolved• bearNDT146398 Create Mobile content service endpoint for references
 DeclinedNoneT146396 Improve performance of mobile references API
 Resolved• FjalapenoT170294 Question: Do clients prefer to have references as structured data in order to build more useful UIs (rather than just showing a bunch of references at the bottom of the page)
 Resolved• bearNDT174810 Add swagger spec for references endpoint
 Resolved• bearNDT182647 Backlinks should be objects with href and the link text
 Resolved• bearNDT182652 Citations field becomes type with just a single value
 Resolved• bearNDT184327 Add section header to references output
 Resolved• bearNDT206467 Provide all reference lists
 Resolved• bearNDT184328 Add revision + tid to references output
 Resolved• bearNDT185133 Update references wrapper lookup
 Resolved• bearNDT185138 Update unit tests to account for Parsoid ref change from span to sup
 Resolved• bearNDT190219 Make section_heading part of the reference_list
 Resolved• MhollowayT177428 Develop Metadata JSON API
 Resolved• MhollowayT172005 Extract categories as a structured array in the metadata endpoint
 Resolved• MhollowayT172004 Extract Hatnotes as a structured array
 Resolved• MhollowayT186838 Handle dewiki hatnotes
 Resolved• MhollowayT186834 Selfref template produces nested div.hatnote elements, causing duplicate hatnote output in MCS
 Resolved• MhollowayT172002 Extract Page Issues as a structured array
 Resolved• MhollowayT179947 Improve geo coordinate reporting in MCS
 Declined• MhollowayT187841 Decide on protection output format for the metadata endpoint
 Resolved• MhollowayT188563 Document the metadata endpoint in an on-wiki spec
 Resolved• MhollowayT177430 Develop a Media JSON API
 Resolved• MhollowayT179876 Expose media endpoint in RESTBase when ready
 DuplicateNoneT179878 Security review of the MCS media endpoint
 Resolved• MhollowayT86954 Service: Image gallery must correspond one-to-one with images displayed on page (in order and visibility).
 DeclinedNoneT182329 Media: handle items that link to documents other than a File page
 Resolved• MhollowayT182330 Media: handle galleries
 Resolved• MhollowayT188562 Document the media endpoint in an on-wiki spec
 Resolved• MhollowayT177431 Develop a Summary JSON API
 ResolvedovasilevaT113094 [EPIC] The Page Summary API needs to provide useful content for the majority of articles
 ResolvedphuedxT113633 Spike: Alternative to TextExtracts for Popups, Gather, Read more
 ResolvedphuedxT109867 Infobox template parameters showing in Hovercard
 DuplicateNoneT109869 Hovercards should not show <noinclude> content in the extract
 ResolvedJdlrobsonT74629 Reference lists appearing in extracts for some articles
 ResolvedJdlrobsonT73023 Invalid HTML markup standalone LI element when using references on main page
 DuplicateNoneT112137 Hovercards loses manual superscript formatting by requesting plain text
 DuplicateovasilevaT152414 Previews must display useful text for the majority of articles
 ResolvedovasilevaT159388 Text Extracts used in Page Previews shows strange characters from template
 DuplicateNoneT162219 Parentheticals: Words incorrectly concatenated due to too simple removing of spaces
 DeclinedNoneT159065 Fix formulas in HTML extracts
 DuplicateJdlrobsonT163442 Math formulas are not rendering in preview page dialogs
 ResolvedMhurdT155573 [Regression] exclude pronunciation guides from article extracts
 Declined• JMinorT164100 Consider excluding pronunciation guides from TextExtracts
 ResolvedphuedxT112226 Text extracts unable to summarise content contains {{dts}} templates
 Resolved• PcheloloT165017 Setup RESTBase HTML extract endpoint
 ResolvedJdlrobsonT165619 Spike: Specify changes to page-summary endpoint
 Declined• PcheloloT166163 Create a summary content migration filter
 ResolvedphuedxT167852 "<translate>" and <tvar|*> tag visible in Page Previews: HtmlFormatter only flattens spans
 DeclinedJdlrobsonT168329 exsentences does not work correctly when HTML output used
 DuplicateNoneT168625 Make the Page Summary API return an "intro" for a page
 Resolved• FjalapenoT167022 Inventory requirements for summary 2.0 endpoint across Reading teams
 ResolvedphuedxT168400 [EPIC] Add notion of emptiness to page summary API
 DeclinedpmiazgaT168328 Extracts for Wikimedia List articles display partial previews
 DeclinedNoneT168391 Page previews should display generic preview for disambiguation pages
 ResolvedphuedxT114418 Disable TextExtracts on file pages
 DeclinedNoneT168418 Page Previews should accept the service's notion of emptiness
 Resolved• PcheloloT164291 Make title-related properties consistent
 Resolved• bearNDT169761 Review Summary 2.0 Spec
 Resolved• MhollowayT170692 Return common URLs in summary API so clients do not have to perform bug prone string manipulation
ResolvedMSantosT177619 Return variant URLs and titles in the metadata response
 Resolved• MhollowayT178446 Expose display titles for a page in all available language variants through the action API
 InvalidNoneT69232 Hovercards: Redirect titles should be bolded in the extract
 ResolvedJdlrobsonT170617 Adjust expectations for API consumers when using the TextExtracts API
 InvalidNoneT185472 API problems with double spaces in wiki sections
 DuplicateNoneT171053 Complete entire page previews test plan for page summary api
 ResolvedJdlrobsonT171052 Add disambiguation page handling in Page Summary API
 ResolvedJdlrobsonT168848 Bootstrap an initial version of the Page Summary API in MCS
 ResolvedJdlrobsonT172021 It shouldn't be possible for coordinates to be the lead paragraph
 ResolvedJdlrobsonT174698 Parenthetical stripping is too aggressive
 ResolvedphuedxT171065 Expose disambiguation property (ppprop=disambiguation) in mobileview api
 Resolved• bearNDT175286 Do side by side comparison of old summary endpoint against new summary endpoint
 Resolved• MhollowayT176974 Mime vulnerability blocking merges to MCS
 DuplicateNoneT176063 Page preview shows only the first two words for a specific article
 ResolvedJdlrobsonT176517 Pages that are redirects give empty summary objects
 ResolvedJdlrobsonT176519 Old templates can lead to sup elements inside summary
 ResolvedphuedxT176521 data-mw attributes should be stripped from summary before scrubbing parentheticals
ResolvedJdlrobsonT176522 Intro property incorrectly identified in formatted endpoint
 ResolvedJdlrobsonT176525 Parenthetical: New edge case with nested brackets
 Resolved• bearNDT177007 Should we flatten spans in summary output
 ResolvedJdlrobsonT178125 Metadata (coordinates) is sometimes surfaced as lead intro/summary
 Resolved• MhollowayT178333 Move RESTBase page summary logic to MCS
 Resolved• MhollowayT178420 What is a "content namespace" for purposes of the summary 2.0 endpoint?
 ResolvedJdlrobsonT183833 [Bug report] Removing parentheses breaks chemical formulas
 ResolvedJdlrobsonT185050 Run comparison of html extracts again
 Resolved• bearNDT185161 Stripping parentheticals can result in trailing punctuation
 Resolved• mobrovacT179875 Update RESTBase to get summary content from MCS Summary 1.3 endpoint when development is complete
 ResolvedABorbaWMFT182399 QA html previews on beta cluster
 ResolvedABorbaWMFT182639 Page previews cannot handle empty responses gracefully - throws error in beta cluster and doesn't display preview (browser tests were failing)
 Resolved• MhollowayT186847 Initial transcluded <p> elements on dewiki are fooling extractLeadIntroduction
 Resolved• MhollowayT186933 204 summary responses cause Android app to crash
 Resolved• MhollowayT187091 Summary: stop responding with 204s
 Resolved• MhollowayT184429 Ensure "timestamp" refers to the requested revision
 DeclinedNoneT184484 Bump summary revision to 2.0
 Resolved• MhollowayT184430 Flag project main pages with type=mainpage
Restricted Task
Restricted Task
Restricted Task
Restricted Task
Restricted Task
Restricted Task
 Resolved• MhollowayT184751 Benchmark the new page summary API
 DeclinedNoneT184753 Use stored page leads when creating page summaries to reduce MCS load
 Resolved• PcheloloT186720 Etag is not returned for mobile-sections-lead or mobile-sections-remaining if a revision parameter is present
 Resolved• MhollowayT187098 Extra span at beginning of mobile-sections-lead
 InvalidNoneT162185 Extract JSON API from MCS Page Content API
 Resolved• PcheloloT189750 Expose new PCS JSON endpoints via RESTBase
 Resolved• bearNDT189751 Expose /page/media endpoint via RESTBase
 DeclinedNoneT189728 Design strategy for updating MCS/PCS content
 Resolved• bearNDT190198 Expose /page/metadata endpoint via RESTBase
 Resolved• bearNDT190337 Expose /page/references endpoint via RESTBase

Event Timeline

• Fjalapeno created this task.Oct 4 2017, 5:39 PM
• Fjalapeno added a subtask: T170690: Extract a References JSON API.Oct 4 2017, 5:43 PM
• Fjalapeno created subtask T177428: Develop Metadata JSON API.
• Fjalapeno created subtask T177430: Develop a Media JSON API.Oct 4 2017, 5:53 PM
• Fjalapeno created subtask T177431: Develop a Summary JSON API.
• Fjalapeno added a subtask: T162185: Extract JSON API from MCS Page Content API.Oct 4 2017, 6:06 PM
• Fjalapeno added a project: Page Content Service.Oct 5 2017, 3:04 PM
• Fjalapeno updated the task description. (Show Details)
• Fjalapeno moved this task from Needs triage to Epics on the Product-Infrastructure-Team-Backlog-Deprecated board.
• Fjalapeno updated the task description. (Show Details)Oct 5 2017, 3:51 PM
• Mholloway closed subtask T177431: Develop a Summary JSON API as Resolved.Feb 21 2018, 8:04 PM
• Mholloway closed subtask T177430: Develop a Media JSON API as Resolved.Mar 15 2018, 4:51 PM
• Pchelolo closed subtask T189750: Expose new PCS JSON endpoints via RESTBase as Resolved.Mar 22 2018, 6:14 PM
• Mholloway closed subtask T177428: Develop Metadata JSON API as Resolved.Mar 22 2018, 6:40 PM
• bearND closed subtask T170690: Extract a References JSON API as Resolved.Jun 29 2018, 4:08 PM
• Jhernandez updated the task description. (Show Details)Jul 6 2018, 12:25 PM
• Jhernandez triaged this task as High priority.Jul 6 2018, 12:30 PM
• Jhernandez subscribed.

Depending to the answer on T162185: Extract JSON API from MCS Page Content API this may already be done like that one.

• Jhernandez mentioned this in T55018: Parsoid HTML/RDF for mobile devices.Aug 10 2018, 1:05 PM
LGoto closed subtask T162185: Extract JSON API from MCS Page Content API as Invalid.May 15 2019, 3:57 PM
MSantos moved this task from Needs Triage to Icebox on the Page Content Service board.Sep 2 2019, 4:47 PM
• Jhernandez closed this task as Resolved.Sep 11 2019, 12:32 PM
• Jhernandez claimed this task.

This has generally been accomplished. Other related tasks that don't block the epic's completion have been orphaned and kept in the backlog with the appropriate priority.

Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL