Page MenuHomePhabricator
Create Task
Maniphest T260568

[EPIC] Productionize WCQS
Closed, ResolvedPublic
Actions

Description

As a user of WCQS I want to have some guarantee on uptime and availability of WCQS so that I can build workflows and tools around the service.

As an operator of WCQS I want to have access to proper monitoring, metrics, alerting so that I can properly run and improve the service.

Project outline from kickoff: https://docs.google.com/document/d/10oKth0QUrWS2HrvkiXVOH9WZtqq1R2KB9xPSbeDEwwM/edit?usp=sharing

To do:

  • get new hardware
  • service implementation: puppet work / initial configuration
  • UI
  • Flink-based streaming updater for WCQS
  • authentication
  • monitoring, metrics, alerting, dashboard
  • redirect old URL
  • community communications
  • promote MediaInfo RDF format to stable

Out of scope:

  • UI updates/changes
  • authentication for WDQS <-> WCQS federation
  • WDQS UI features that are missing on WCQS

Related Objects
Search...

StatusSubtypeAssignedTask
 ResolvedLadsgroupT271851 Clean up gui from the wdqs deploy repo and puppet
 ResolvedNoneT260568 [EPIC] Productionize WCQS
 OpenNoneT258627 Autocompletion for MediaInfo items on WMCS
 DuplicateNoneT258769 ImageGrid for WCQS
 ResolvedEBernhardsonT262265 Provide real-time updates for WCQS
 DeclinedNoneT279691 Add query service lag of WCQS to Commons maxlag
 ResolvedGehelT280485 Additional capacity on the k8s Flink cluster for WCQS updater
 ResolveddcausseT293195 Add MCR slot information to revision-create events
 ResolvedEBernhardsonT293638 Create and update the process for separate WCQS mutation topic
 ResolvedEBernhardsonT298622 Adapt EntityRevisionMapGenerator for wcqs
 ResolvedGehelT254232 New clusters for SPARQL Endpoint for Commons
Unknown Object (Task)
 ResolvedRobHT276644 (Need By: 2021-04-30) rack/setup/install wcqs100[123]
Unknown Object (Task)
 ResolvedPapaulT276647 (Need By: 2021-04-30) rack/setup/install wcqs200[123]
 ResolvedMPhamWMFT279672 Community communications for productionizing WCQS
 ResolvedRKemperT280001 Set up puppet configuration for new WCQS cluster
 ResolvedEBernhardsonT280006 Set up the application authentication for WCQS on commons-query.wikimedia.org
 ResolvedEBernhardsonT290299 Replace token store in MW OAuth WCQS proxy with JWT
DuplicateNoneT290300 Serve WCQS Sparql endpoint through api.wikimedia.org with OAuth 2
 ResolvedEBernhardsonT280008 Set up WCQS monitoring
 ResolvedEBernhardsonT280247 Create WCQS UI microsite deployment
 ResolvedEBernhardsonT280487 Redirect requests from wcqs-beta.wmflabs.org to the final URL for WCQS
 ResolvedtaaviT303202 Redirect wcqs-beta.wmflabs.org to commons-query.wikimedia.org
 ResolvedEBernhardsonT282117 WCQS needs to be exposed through a wikimedia.org domain
 Resolved• ZbyszkoT284040 Enable blank node skolemization on wcqs
 ResolvedGehelT288230 Promote MediaInfo RDF format to stable
 ResolvedNoneT290671 [M] Create a stable interface policy for Commons MediaInfo RDF
 ResolvedEBernhardsonT293462 Add user blocking in WCQS
 DuplicateNoneT294865 wcqs1002 and wcqs2001 unresponsive
 ResolvedGehelT294961 Resolve kernel hang on wcqs* instances
 Resolved• ZbyszkoT296466 Add MCR filtering to Streaming Updater for WCQS
 ResolvedEBernhardsonT296468 Modify flink-job.py script to handle WCQS streaming updater deployment
 ResolvedGehelT296470 Initialize WCQS production servers
 DuplicateNoneT300159 Check WCQS usage after adding auth

Event Timeline

Gehel created this task.Aug 17 2020, 3:42 PM
Restricted Application added a project: Wikidata. · View Herald TranscriptAug 17 2020, 3:42 PM
Restricted Application added a subscriber: Aklapper. · View Herald Transcript
Gehel moved this task from Incoming to Epics on the Wikidata-Query-Service board.Aug 17 2020, 3:42 PM
CBogen renamed this task from Productionize WCQS to [EPIC] Productionize WCQS.Aug 17 2020, 3:44 PM
CBogen updated the task description. (Show Details)
CBogen merged a task: T260800: [epic] Productionize Wikimedia Commons Query Service.Aug 19 2020, 2:31 PM
CBogen added a subtask: T258627: Autocompletion for MediaInfo items on WMCS.
CBogen added a subscriber: CBogen.
CBogen added a subtask: T258769: ImageGrid for WCQS.Aug 24 2020, 12:54 PM
• Zbyszko added a subtask: T262265: Provide real-time updates for WCQS.Sep 8 2020, 12:18 PM
Gehel triaged this task as High priority.Sep 15 2020, 7:41 AM
Gehel added a subtask: Unknown Object (Task).Feb 10 2021, 4:39 PM
Gehel added a subtask: T254232: New clusters for SPARQL Endpoint for Commons.Feb 11 2021, 8:40 AM
Gehel mentioned this in Unknown Object (Task).Feb 12 2021, 1:53 PM
Gehel mentioned this in Unknown Object (Task).Feb 12 2021, 1:58 PM
Gehel mentioned this in T273579: cloudvirt-wdqs1001 getting out of space due to huge VM.Feb 12 2021, 4:04 PM
ttaylor added a subscriber: ttaylor.Feb 23 2021, 3:47 PM
MPhamWMF updated the task description. (Show Details)Apr 8 2021, 2:36 PM
• Zbyszko added a subtask: T280001: Set up puppet configuration for new WCQS cluster.Apr 13 2021, 10:31 AM
CBogen mentioned this in T280271: Update WCQS dump.Apr 15 2021, 4:58 PM
Jclark-ctr closed subtask Unknown Object (Task) as Resolved.Apr 16 2021, 6:12 PM
Zache added a subscriber: Zache.May 20 2021, 10:54 AM
MPhamWMF added a subtask: T284040: Enable blank node skolemization on wcqs.Jun 1 2021, 1:57 PM
Fuzheado added a subscriber: Fuzheado.Jun 3 2021, 5:33 PM
Gehel added a parent task: T271851: Clean up gui from the wdqs deploy repo and puppet.Jun 7 2021, 12:31 PM
Gehel closed subtask T284040: Enable blank node skolemization on wcqs as Resolved.Jun 21 2021, 11:43 AM
• Zbyszko added a project: Discovery-Search (Current work).Aug 5 2021, 12:00 PM
• Zbyszko moved this task from Incoming to Epics on the Discovery-Search (Current work) board.
So9q added a subscriber: So9q.Aug 9 2021, 1:03 PM
Gehel closed subtask T254232: New clusters for SPARQL Endpoint for Commons as Resolved.Aug 16 2021, 3:43 PM
GFontenelle_WMF added a subscriber: GFontenelle_WMF.Sep 3 2021, 8:01 PM
MPhamWMF moved this task from Epics to Current work on the Wikidata-Query-Service board.Oct 18 2021, 3:38 PM
EBernhardson added a subtask: T293462: Add user blocking in WCQS.Oct 26 2021, 6:15 PM
Gehel closed subtask T280006: Set up the application authentication for WCQS on commons-query.wikimedia.org as Resolved.Oct 28 2021, 8:04 AM
Gehel closed subtask T280001: Set up puppet configuration for new WCQS cluster as Resolved.
Gehel closed subtask T280247: Create WCQS UI microsite deployment as Resolved.Oct 28 2021, 8:08 AM
RKemper added a subtask: T294865: wcqs1002 and wcqs2001 unresponsive.Nov 3 2021, 9:05 PM
Gehel closed subtask T294961: Resolve kernel hang on wcqs* instances as Resolved.Dec 13 2021, 1:44 PM
MPhamWMF closed subtask T279672: Community communications for productionizing WCQS as Resolved.Dec 17 2021, 12:23 PM
Gehel closed subtask T280008: Set up WCQS monitoring as Resolved.Jan 10 2022, 1:43 PM
Gehel closed subtask T296468: Modify flink-job.py script to handle WCQS streaming updater deployment as Resolved.Jan 17 2022, 1:43 PM
Gehel closed subtask T262265: Provide real-time updates for WCQS as Resolved.Jan 24 2022, 3:09 PM
Gehel closed subtask T296466: Add MCR filtering to Streaming Updater for WCQS as Resolved.Jan 24 2022, 3:49 PM
MPhamWMF added a subtask: T300159: Check WCQS usage after adding auth.Jan 31 2022, 3:53 PM
Gehel closed subtask T296470: Initialize WCQS production servers as Resolved.Jan 31 2022, 3:54 PM
Gehel closed subtask T288230: Promote MediaInfo RDF format to stable as Resolved.Feb 16 2022, 8:53 PM
Gehel closed subtask T282117: WCQS needs to be exposed through a wikimedia.org domain as Resolved.Feb 21 2022, 2:23 PM
Gehel closed subtask T293462: Add user blocking in WCQS as Resolved.Mar 14 2022, 2:51 PM
Gehel closed subtask T280487: Redirect requests from wcqs-beta.wmflabs.org to the final URL for WCQS as Resolved.
MPhamWMF closed this task as Resolved.Apr 11 2022, 4:09 PM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL