Page MenuHomePhabricator
Create Task
Maniphest T251488

[epic] Create minimal SPARQL Endpoint for Commons on WMCS
Closed, ResolvedPublic
Actions

Description

This minimal endpoint has the following constraints:

  • running on WMCS
  • data loaded automatically each week from Commons TTL dumps
  • access restricted by password or oauth

Related Objects
Search...

StatusSubtypeAssignedTask
 OpenNoneT115858 Design improvements for mw.ForeignStructuredUpload.BookletLayout
 OpenNoneT115865 Insert image in content immediately after it's uploaded, skipping the "General settings" step
 DuplicateNoneT115864 Figure out if the description of the image can be used as the caption on-wiki
 OpenFeatureNoneT53032 When inserting an image, set its caption by default to be the Commons image description
 OpenFeatureNoneT39534 Wikimedia Commons should support searching by color
 DuplicateNoneT39535 Wikimedia Commons should support filtering by color
 ResolvedNoneT19503 Provide metadata support on Wikimedia Commons
 ResolvedNoneT51662 VisualEditor: Use Multimedia/Wikidata's proposed rich structured meta-data in the image insertion dialog
 ResolvedNoneT68108 [Epic] Store media information for files on Wikimedia Commons as structured data
 DuplicateNoneT141602 [Objective Fiscal 19-20/Q4] (9) Provide a Proof of Concept SPARQL endpoint in support of SDoC project
 ResolvedGehelT251488 [epic] Create minimal SPARQL Endpoint for Commons on WMCS
 ResolvedEBernhardsonT237089 Create CQS puppet configs by applying query_service module
 ResolvedEBernhardsonT251489 Validate that we have enough resources on WMCS for a SPARQL Endpoint for Commons
 ResolvedGehelT251490 Load data into the SPARQL Endpoint for Commons
 ResolvedGehelT251496 Validate and fix TTL dumps of SDoC
 ResolvedGehelT251497 Adapt munging process for SDoC
 Resolved• ZbyszkoT243292 Fix the munger to support commons RDF dump
 Resolved• ZbyszkoT251491 Enable federation between SPARQL Endpoint for Commons and WDQS
 Resolved• ZbyszkoT251498 Access restriction for SPARQL Endpoint for Commons
 Resolved• ZbyszkoT251499 Minimal authentication for SPARQL Endpoint for Commons
 Resolved• ZbyszkoT251500 oAuth authentication for SPARQL Endpoint for Commons
 ResolvedGehelT251514 UI for SPARQL Endpoint for Commons
 Resolved• ZbyszkoT258625 Querying WCQS should allow me to use prefixes for MediaInfo items
 OpenNoneT258627 Autocompletion for MediaInfo items on WMCS
 Resolved• ZbyszkoT251515 Automate data reload for SPARQL Endpoint for Commons
 Resolvedbd808T257336 Request increased quota for wikidata-query Cloud VPS project
 ResolvedGehelT258489 Create a Examples Page for WCQS
 DuplicateNoneT259884 Maintenance page during data reloads
 ResolvedGehelT259543 Notify users that wcqs is reloading its data
 Resolved• ZbyszkoT262828 Near zero downtime Data reload for WCQS

Event Timeline

Gehel created this task.Apr 30 2020, 8:01 AM
Restricted Application added a project: Wikidata. · View Herald TranscriptApr 30 2020, 8:01 AM
Restricted Application added a subscriber: Aklapper. · View Herald Transcript
Gehel added a subtask: T237089: Create CQS puppet configs by applying query_service module.Apr 30 2020, 8:01 AM
Lea_Lacroix_WMDE subscribed.Apr 30 2020, 9:39 AM
Gehel moved this task from Incoming to SDAW on the Wikidata-Query-Service board.Apr 30 2020, 11:04 AM
• Zbyszko closed subtask T251491: Enable federation between SPARQL Endpoint for Commons and WDQS as Resolved.May 27 2020, 8:51 AM
Gehel renamed this task from Create minimal SPARQL Endpoint for Commons on WMCS to [epic] Create minimal SPARQL Endpoint for Commons on WMCS.Jul 1 2020, 12:39 PM
Gehel added a project: Epic.
Gehel moved this task from SDAW to Epics on the Wikidata-Query-Service board.
Gehel closed subtask T251489: Validate that we have enough resources on WMCS for a SPARQL Endpoint for Commons as Resolved.
Gehel merged a task: T221921: Provision search endpoint for SDC. Requirements from Product Team..Jul 1 2020, 12:46 PM
Gehel added subscribers: • Nuria, • David_Haskiya_WMSE, CBogen and 22 others.
dcausse mentioned this in T257336: Request increased quota for wikidata-query Cloud VPS project.Jul 8 2020, 1:32 PM
dcausse added a subtask: T257336: Request increased quota for wikidata-query Cloud VPS project.
bd808 closed subtask T257336: Request increased quota for wikidata-query Cloud VPS project as Resolved.Jul 8 2020, 7:21 PM
EBernhardson added a subscriber: • Zbyszko.Jul 8 2020, 9:58 PM

@Zbyszko @dcausse

I stood up the new significantly larger instance (120G mem, 16 cores) for sdoc today as wcqs-beta-01.wikidata-query.eqiad.wmflabs.

As far as I can tell blazegraph came up correctly (on :9999). oauth on the other hand is not working. I did copy over the per-host hiera configuration on wdqspuppetmaster.wikidata-query.eqiad.wmflabs, but this doesn't seem to be sufficient for getting it going. Once we figure out how this is supposed to be configured I'll put together the appropriate patch for operations/puppet.

I did *not* shut down the old instance yet, in case we want to compare things between the two while figuring out why oauth isn't working yet.

• Zbyszko added a comment.Jul 9 2020, 6:29 AM

OAuth requires recent changes that are not yet published and external domain - I'm guessing that the aren't issues with puppet (I'm might be proven wrong, though).

Gehel closed subtask T251490: Load data into the SPARQL Endpoint for Commons as Resolved.Jul 13 2020, 12:46 PM
Gehel closed subtask T237089: Create CQS puppet configs by applying query_service module as Resolved.Jul 13 2020, 12:50 PM
• Zbyszko added a subtask: T258489: Create a Examples Page for WCQS.Jul 21 2020, 1:14 PM
Nintendofan885 subscribed.Jul 27 2020, 7:26 AM
Mstyles closed subtask T251515: Automate data reload for SPARQL Endpoint for Commons as Resolved.Jul 27 2020, 5:11 PM
dcausse reopened subtask T251515: Automate data reload for SPARQL Endpoint for Commons as Open.Jul 29 2020, 7:45 AM
• Zbyszko added a subtask: T259884: Maintenance page during data reloads.Aug 7 2020, 1:00 PM
• Zbyszko added a subtask: T259543: Notify users that wcqs is reloading its data.Aug 7 2020, 1:28 PM
CBogen merged a task: T141602: [Objective Fiscal 19-20/Q4] (9) Provide a Proof of Concept SPARQL endpoint in support of SDoC project.Aug 10 2020, 2:07 PM
CBogen moved this task from Epics to Current work on the Wikidata-Query-Service board.
CBogen added a project: Discovery-Search (Current work).
CBogen added subscribers: danshick-wmde, • PDrouin-WMF, Nicolas_Raoul and 16 others.
CBogen moved this task from Incoming to Epics on the Discovery-Search (Current work) board.Aug 10 2020, 2:14 PM
Tagishsimon unsubscribed.Aug 12 2020, 8:27 PM
Gehel closed subtask T258489: Create a Examples Page for WCQS as Resolved.Aug 17 2020, 12:40 PM
Gehel closed subtask T251498: Access restriction for SPARQL Endpoint for Commons as Resolved.Aug 17 2020, 12:45 PM
Gehel closed subtask T251514: UI for SPARQL Endpoint for Commons as Resolved.
Gehel updated the task description. (Show Details)Aug 19 2020, 2:24 PM
Gehel closed subtask T259543: Notify users that wcqs is reloading its data as Resolved.Aug 24 2020, 8:57 AM
Gehel closed subtask T251515: Automate data reload for SPARQL Endpoint for Commons as Resolved.Sep 1 2020, 12:24 PM
BeatEstermann subscribed.Sep 23 2020, 6:15 AM
Gehel closed this task as Resolved.Oct 6 2020, 2:25 PM
Gehel claimed this task.

WCQS has been available on WMCS for some time with no major issues. Let's close this!

Gehel added a subtask: T262828: Near zero downtime Data reload for WCQS.Oct 30 2020, 1:47 PM
Gehel closed subtask T262828: Near zero downtime Data reload for WCQS as Resolved.Nov 9 2020, 1:02 PM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits