[epic] Create minimal SPARQL Endpoint for Commons on WMCS
Closed, ResolvedPublic
Actions

Assigned To

Authored By

	Gehel
	Apr 30 2020, 8:01 AM

Description

This minimal endpoint has the following constraints:

running on WMCS
data loaded automatically each week from Commons TTL dumps
access restricted by password or oauth

Related Objects
Search...

Status	Subtype	Assigned	Task
Open		None	T115858 Design improvements for mw.ForeignStructuredUpload.BookletLayout
Open		None	T115865 Insert image in content immediately after it's uploaded, skipping the "General settings" step
Duplicate		None	T115864 Figure out if the description of the image can be used as the caption on-wiki
Open	Feature	None	T53032 When inserting an image, set its caption by default to be the Commons image description
Open	Feature	None	T39534 Wikimedia Commons should support searching by color
Duplicate		None	T39535 Wikimedia Commons should support filtering by color
Resolved		None	T19503 Provide metadata support on Wikimedia Commons
Resolved		None	T51662 VisualEditor: Use Multimedia/Wikidata's proposed rich structured meta-data in the image insertion dialog
Resolved		None	T68108 [Epic] Store media information for files on Wikimedia Commons as structured data
Duplicate		None	T141602 [Objective Fiscal 19-20/Q4] (9) Provide a Proof of Concept SPARQL endpoint in support of SDoC project
Resolved		Gehel	T251488 [epic] Create minimal SPARQL Endpoint for Commons on WMCS
Resolved		EBernhardson	T237089 Create CQS puppet configs by applying query_service module
Resolved		EBernhardson	T251489 Validate that we have enough resources on WMCS for a SPARQL Endpoint for Commons
Resolved		Gehel	T251490 Load data into the SPARQL Endpoint for Commons
Resolved		Gehel	T251496 Validate and fix TTL dumps of SDoC
Resolved		Gehel	T251497 Adapt munging process for SDoC
Resolved		• Zbyszko	T243292 Fix the munger to support commons RDF dump
Resolved		• Zbyszko	T251491 Enable federation between SPARQL Endpoint for Commons and WDQS
Resolved		• Zbyszko	T251498 Access restriction for SPARQL Endpoint for Commons
Resolved		• Zbyszko	T251499 Minimal authentication for SPARQL Endpoint for Commons
Resolved		• Zbyszko	T251500 oAuth authentication for SPARQL Endpoint for Commons
Resolved		Gehel	T251514 UI for SPARQL Endpoint for Commons
Resolved		• Zbyszko	T258625 Querying WCQS should allow me to use prefixes for MediaInfo items
Open		None	T258627 Autocompletion for MediaInfo items on WMCS
Resolved		• Zbyszko	T251515 Automate data reload for SPARQL Endpoint for Commons
Resolved		• bd808	T257336 Request increased quota for wikidata-query Cloud VPS project
Resolved		Gehel	T258489 Create a Examples Page for WCQS
Duplicate		None	T259884 Maintenance page during data reloads
Resolved		Gehel	T259543 Notify users that wcqs is reloading its data
Resolved		• Zbyszko	T262828 Near zero downtime Data reload for WCQS

Event Timeline

Gehel created this task.Apr 30 2020, 8:01 AM

Restricted Application added a project: Wikidata. · View Herald TranscriptApr 30 2020, 8:01 AM

Restricted Application added a subscriber: Aklapper. · View Herald Transcript

Gehel added a subtask: T237089: Create CQS puppet configs by applying query_service module.Apr 30 2020, 8:01 AM

Lea_Lacroix_WMDE subscribed.Apr 30 2020, 9:39 AM

Gehel moved this task from Incoming to SDAW on the Wikidata-Query-Service board.Apr 30 2020, 11:04 AM

• Zbyszko closed subtask T251491: Enable federation between SPARQL Endpoint for Commons and WDQS as Resolved.May 27 2020, 8:51 AM

Gehel renamed this task from Create minimal SPARQL Endpoint for Commons on WMCS to [epic] Create minimal SPARQL Endpoint for Commons on WMCS.Jul 1 2020, 12:39 PM

Gehel added a project: Epic.

Gehel moved this task from SDAW to Epics on the Wikidata-Query-Service board.

Gehel closed subtask T251489: Validate that we have enough resources on WMCS for a SPARQL Endpoint for Commons as Resolved.

Gehel merged a task: T221921: Provision search endpoint for SDC. Requirements from Product Team..Jul 1 2020, 12:46 PM

Gehel added subscribers: • Nuria, • David_Haskiya_WMSE, CBogen and 22 others.

• dcausse mentioned this in T257336: Request increased quota for wikidata-query Cloud VPS project.Jul 8 2020, 1:32 PM

• dcausse added a subtask: T257336: Request increased quota for wikidata-query Cloud VPS project.

• bd808 closed subtask T257336: Request increased quota for wikidata-query Cloud VPS project as Resolved.Jul 8 2020, 7:21 PM

@Zbyszko @dcausse

I stood up the new significantly larger instance (120G mem, 16 cores) for sdoc today as wcqs-beta-01.wikidata-query.eqiad.wmflabs.

As far as I can tell blazegraph came up correctly (on :9999). oauth on the other hand is not working. I did copy over the per-host hiera configuration on wdqspuppetmaster.wikidata-query.eqiad.wmflabs, but this doesn't seem to be sufficient for getting it going. Once we figure out how this is supposed to be configured I'll put together the appropriate patch for operations/puppet.

I did *not* shut down the old instance yet, in case we want to compare things between the two while figuring out why oauth isn't working yet.

OAuth requires recent changes that are not yet published and external domain - I'm guessing that the aren't issues with puppet (I'm might be proven wrong, though).

Gehel closed subtask T251490: Load data into the SPARQL Endpoint for Commons as Resolved.Jul 13 2020, 12:46 PM

Gehel closed subtask T237089: Create CQS puppet configs by applying query_service module as Resolved.Jul 13 2020, 12:50 PM

• Zbyszko added a subtask: T258489: Create a Examples Page for WCQS.Jul 21 2020, 1:14 PM

Nintendofan885 subscribed.Jul 27 2020, 7:26 AM

Mstyles closed subtask T251515: Automate data reload for SPARQL Endpoint for Commons as Resolved.Jul 27 2020, 5:11 PM

• dcausse reopened subtask T251515: Automate data reload for SPARQL Endpoint for Commons as Open.Jul 29 2020, 7:45 AM

• Zbyszko added a subtask: T259884: Maintenance page during data reloads.Aug 7 2020, 1:00 PM

• Zbyszko added a subtask: T259543: Notify users that wcqs is reloading its data.Aug 7 2020, 1:28 PM

CBogen merged a task: T141602: [Objective Fiscal 19-20/Q4] (9) Provide a Proof of Concept SPARQL endpoint in support of SDoC project.Aug 10 2020, 2:07 PM

CBogen moved this task from Epics to Current work on the Wikidata-Query-Service board.

CBogen added a project: Discovery-Search (Current work).

CBogen added subscribers: danshick-wmde, • PDrouin-WMF, Nicolas_Raoul and 16 others.