Page MenuHomePhabricator
Create Task
Maniphest T85159

[EPIC] Deploy a Wikidata Query Service into production
Closed, ResolvedPublic
Actions

Description

Create a service which supports complex queries on Wikidata:

  • external requests return within a few seconds, use reasonable resources
  • internal requests are allowed to use more resources & time

See https://www.mediawiki.org/wiki/Wikibase/Indexing.
See: https://www.mediawiki.org/wiki/WDQS_Production

Related Objects
Search...

StatusSubtypeAssignedTask
 Resolved• DeskanaT96579 EPIC: Deploy Wikidata Query 0.1 Release in labs as official service
 OpenNoneT67626 [Epic] Support for queries on-wiki (automated list generation)
 ResolvedSmalyshevT85159 [EPIC] Deploy a Wikidata Query Service into production
 ResolvedJoeT86561 Wikidata Query Service hardware
 ResolvedSmalyshevT104879 Define the details of the hardware we need to run WDQS
 Resolved• CmjohnsonT105904 relocate wmf3544 from row d into any other row
 Resolved• CmjohnsonT106320 wmf3543 can't install from PXE
 Resolved• CmjohnsonT107152 Change hardware RAID controller on wmf3543, wmf3544
 Resolved• csteippT90115 BlazeGraph Security Review
 ResolvedSmalyshevT91812 Wikidata Query: Secure BlazeGraph's SPARQL
 ResolvedSmalyshevT91813 Add whitelist of RDF datasets that queries can operate on
 ResolvedSmalyshevT91817 Wikidata Query: Create readonly SPARQL
ResolvedSmalyshevT91824 Wikidata Query: Secure BlazeGraph's SPARQL: Investigate HistoryServiceFactory
 Resolved• DeskanaT98030 Do whatever is necessary to hook up production Wikidata Query Service to HDFS and other log collection systems
 ResolvedMoritzMuehlenhoffT109357 Grant SMalyshev access to stat1002 to query hive
 ResolvedmpopovT109360 Create a script to extract request logs for query.wikidata.org for dashboards
 ResolvedakosiarisT110217 Need access for smalyshev to hive queries on stat1002
 ResolvedJoeT95679 Make a puppet role that sets up a query service and loads it
 ResolvedJoeT105185 Need deploy rights for Wikidata Query Service
 ResolvedNoneT103904 Create debian service structure for WDQS
 ResolvedNoneT101140 Create monitoring for WDQS update tool
 ResolvedSmalyshevT101139 Create monitoring for Blazegraph
 Declined• DeskanaT102312 Figure out monitoring/alerting setup for labs
 ResolvedSmalyshevT103897 Create production package for WDQS
 ResolvedSmalyshevT104930 Make the build package that uses single .war for Blazegraph
 Resolved• DeskanaT103906 Determine backup/disaster recovery scenario for WDQS
 ResolvedSmalyshevT103907 Develop external hardening rules for WDQS
 ResolvedSmalyshevT96700 Lock down public Blazegraph instance
 ResolvedSmalyshevT103908 Ensure WDQS service can not be DoSed by a single query
 ResolvedJoeT107819 Need sudo to blazegraph on wdqs1001/1002
 ResolvedSmalyshevT103911 Figure out monitoring/alerting setup for WDQS production
 ResolvedSmalyshevT103932 Create user-space documentation describing how to use WDQS
 DeclinedJoeT107601 Assign an LVS service to the wikidata query service
 ResolvedJoeT107602 Set up a public interface to the wikidata query service
 ResolvedNoneT108101 Isolate wikidata.org cookies and CORS policies
 ResolvedBeneT112087 [Bug] m.wikidata.org is not CORS whitelisted
 Resolved• csteippT100413 "You are centrally logged in." toast on every page view on commons
 OpenNoneT109538 Vagrant wikis should have mobile site setup when MobileFrontend role is enabled
 Resolved• GillesT54302 Varnish vagrant role

Event Timeline

• RobLa-WMF created this task.Dec 22 2014, 6:40 PM
• RobLa-WMF raised the priority of this task from to Needs Triage.
• RobLa-WMF updated the task description. (Show Details)
• RobLa-WMF added projects: Wikidata-Query-Service, Epic, MediaWiki-Core-Team.
• RobLa-WMF changed Security from none to None.
• RobLa-WMF subscribed.
• RobLa-WMF added subtasks: T76373: Evaluate Titan as graph storage/query engine for Wikidata Query service, T76372: Evaluate TinkerPop GraphSail frontend, T1028: Experiment with Wikidata Query Engine, T75813: Investigate OrientDB for WDQ.Dec 22 2014, 6:48 PM
Krenair subscribed.Dec 22 2014, 8:34 PM
bd808 closed subtask T75813: Investigate OrientDB for WDQ as Resolved.Dec 22 2014, 10:47 PM
Lydia_Pintscher added a project: Wikidata.Dec 23 2014, 3:50 PM
Lydia_Pintscher added subscribers: daniel, aude, JanZerebecki, JeroenDeDauw.Dec 27 2014, 12:05 PM
MrStradivarius subscribed.Jan 4 2015, 7:28 PM
waldyrious subscribed.Jan 7 2015, 6:12 AM
Lydia_Pintscher moved this task from incoming to hold on the Wikidata board.Jan 7 2015, 10:36 AM
• RobLa-WMF mentioned this in T85151: Preparing for a multi-device world plenary at 2015 MediaWiki Developer Summit.Jan 8 2015, 6:46 AM
• GWicke added a subtask: T86561: Wikidata Query Service hardware.Jan 12 2015, 7:17 PM
• GWicke removed subtasks: T75813: Investigate OrientDB for WDQ, T1028: Experiment with Wikidata Query Engine, T76372: Evaluate TinkerPop GraphSail frontend.
• GWicke added a subtask: T85181: Investigate & design public API, possibly using MQL.Jan 12 2015, 7:32 PM
Smalyshev closed subtask T76373: Evaluate Titan as graph storage/query engine for Wikidata Query service as Resolved.Jan 13 2015, 8:49 PM
MBlissett subscribed.Jan 30 2015, 10:47 AM
bd808 triaged this task as Medium priority.Feb 2 2015, 8:44 PM
bd808 moved this task from Backlog to In Dev/Progress on the MediaWiki-Core-Team board.
bd808 subscribed.
• Manybubbles moved this task from In Dev/Progress to Backlog on the MediaWiki-Core-Team board.Feb 4 2015, 5:08 PM
faidon changed the status of subtask T86561: Wikidata Query Service hardware from Open to Stalled.Feb 9 2015, 10:59 AM
Bugreporter added a parent task: T67626: [Epic] Support for queries on-wiki (automated list generation).Feb 11 2015, 10:37 AM
bd808 edited projects, added Discovery-ARCHIVED; removed MediaWiki-Core-Team.Apr 8 2015, 10:12 PM
Laddo subscribed.Apr 17 2015, 12:04 PM
• Manybubbles moved this task from Needs triage to WDQS on the Discovery-ARCHIVED board.May 7 2015, 7:55 PM
Smalyshev removed subtasks: T85181: Investigate & design public API, possibly using MQL, T76373: Evaluate Titan as graph storage/query engine for Wikidata Query service.Jun 2 2015, 5:49 PM
Smalyshev added a subtask: T90115: BlazeGraph Security Review.
Smalyshev added a subtask: T101013: Log Wikidata Query Service queries to the event gate infrastructure.
Smalyshev added a subtask: T96700: Lock down public Blazegraph instance.Jun 2 2015, 5:52 PM
Smalyshev added a subtask: T91813: Add whitelist of RDF datasets that queries can operate on.
Smalyshev edited subtasks, added: T91812: Wikidata Query: Secure BlazeGraph's SPARQL; removed: T91813: Add whitelist of RDF datasets that queries can operate on.
Smalyshev added subtasks: T98030: Do whatever is necessary to hook up production Wikidata Query Service to HDFS and other log collection systems, T95679: Make a puppet role that sets up a query service and loads it.Jun 2 2015, 5:55 PM
Smalyshev added subtasks: T101140: Create monitoring for WDQS update tool, T101139: Create monitoring for Blazegraph.Jun 2 2015, 6:12 PM
• Deskana closed subtask T101013: Log Wikidata Query Service queries to the event gate infrastructure as Declined.Jun 8 2015, 7:42 PM
Smalyshev closed subtask T91812: Wikidata Query: Secure BlazeGraph's SPARQL as Resolved.Jun 8 2015, 7:57 PM
Addshore subscribed.Jun 8 2015, 8:22 PM
Smalyshev changed the status of subtask T95679: Make a puppet role that sets up a query service and loads it from Open to Stalled.Jun 11 2015, 6:07 PM
Smalyshev closed subtask T101140: Create monitoring for WDQS update tool as Resolved.Jun 17 2015, 6:14 AM
Smalyshev closed subtask T101139: Create monitoring for Blazegraph as Resolved.Jun 23 2015, 5:04 PM
Smalyshev renamed this task from Deploy a Wikidata complex query service into production to [epic] Deploy a Wikidata Query Service into production.Jun 25 2015, 7:29 PM
Smalyshev renamed this task from [epic] Deploy a Wikidata Query Service into production to [EPIC] Deploy a Wikidata Query Service into production.Jun 25 2015, 7:35 PM
Smalyshev added a subtask: T103897: Create production package for WDQS.Jun 25 2015, 7:44 PM
Smalyshev added a subtask: T103906: Determine backup/disaster recovery scenario for WDQS.Jun 25 2015, 8:13 PM
Smalyshev added a subtask: T103907: Develop external hardening rules for WDQS.Jun 25 2015, 8:17 PM
Smalyshev added a subtask: T103908: Ensure WDQS service can not be DoSed by a single query.Jun 25 2015, 8:21 PM
Smalyshev added a subtask: T103911: Figure out monitoring/alerting setup for WDQS production.Jun 25 2015, 8:25 PM
Smalyshev added subtasks: T103922: Develop set of internal metrics that we want to monitor on WDQS services, T103904: Create debian service structure for WDQS.Jun 25 2015, 11:24 PM
Smalyshev added a subtask: T103932: Create user-space documentation describing how to use WDQS.Jun 25 2015, 11:31 PM
Smalyshev updated the task description. (Show Details)
Smalyshev changed the status of subtask T86561: Wikidata Query Service hardware from Stalled to Open.Jul 6 2015, 5:15 PM
Smalyshev changed the status of subtask T95679: Make a puppet role that sets up a query service and loads it from Stalled to Open.Jul 7 2015, 8:14 PM
Smalyshev closed subtask T103897: Create production package for WDQS as Resolved.Jul 9 2015, 5:12 PM
Smalyshev closed subtask T103904: Create debian service structure for WDQS as Resolved.Jul 21 2015, 5:44 PM
Smalyshev closed subtask T95679: Make a puppet role that sets up a query service and loads it as Resolved.
Joe closed subtask T86561: Wikidata Query Service hardware as Resolved.Jul 31 2015, 4:54 PM
Joe closed subtask T107601: Assign an LVS service to the wikidata query service as Declined.Aug 5 2015, 3:15 PM
Smalyshev closed subtask T96700: Lock down public Blazegraph instance as Resolved.Aug 5 2015, 10:40 PM
MrStradivarius unsubscribed.Aug 6 2015, 4:03 PM
Smalyshev closed subtask T103907: Develop external hardening rules for WDQS as Resolved.Aug 6 2015, 4:47 PM
• csteipp closed subtask T90115: BlazeGraph Security Review as Resolved.Aug 13 2015, 4:23 PM
Joe closed subtask T107602: Set up a public interface to the wikidata query service as Resolved.Aug 13 2015, 8:27 PM
• Deskana mentioned this in T73813: Provide SPARQL endpoint for Wikidata data.Aug 24 2015, 10:21 PM
• Deskana merged a task: T73813: Provide SPARQL endpoint for Wikidata data.
• Deskana added subscribers: • Deskana, Smalyshev, Lydia_Pintscher and 2 others.
• Deskana closed subtask T98030: Do whatever is necessary to hook up production Wikidata Query Service to HDFS and other log collection systems as Resolved.Aug 25 2015, 8:16 PM
Smalyshev closed subtask T103908: Ensure WDQS service can not be DoSed by a single query as Resolved.Aug 26 2015, 12:42 AM
Smalyshev closed subtask T103932: Create user-space documentation describing how to use WDQS as Resolved.Aug 27 2015, 12:45 AM
Dzahn closed subtask T103911: Figure out monitoring/alerting setup for WDQS production as Resolved.Sep 4 2015, 9:59 PM
• Deskana closed subtask T103906: Determine backup/disaster recovery scenario for WDQS as Resolved.Sep 10 2015, 5:07 PM
Smalyshev closed this task as Resolved.Sep 10 2015, 5:08 PM
Smalyshev claimed this task.
Smalyshev removed a subtask: T103922: Develop set of internal metrics that we want to monitor on WDQS services.

woohoo!

• Deskana added a comment.Sep 10 2015, 5:08 PM

\o/

Lydia_Pintscher added a comment.Sep 10 2015, 5:15 PM

\o/

dcausse reopened subtask T101013: Log Wikidata Query Service queries to the event gate infrastructure as Open.Oct 2 2019, 8:12 PM
dcausse removed a subtask: T101013: Log Wikidata Query Service queries to the event gate infrastructure.Oct 9 2019, 2:14 PM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL