Page MenuHomePhabricator
Create Task
Maniphest T335325

Restore loading of graphs from URL for Wikimedia allowed domains
Closed, DeclinedPublic
Actions

Description

As part of the upgrades to the Graph extension, we temporarily removed the ability to load graphs from a URL with the goal of restoring support to some graphs on Wikimedia servers. This task encompasses all work to restore graphs. This work will be prioritized based on number of graphs impacted (e.g. error rate in production)

In Wikimedia we only support geoshape (from maps.wikimedia.org), wikidatasparql (from query.wikidata.org) and wikirawupload (from upload.wikimedia.org)

These were originally defined here: https://github.com/nyurik/mw-graph-shared

TODO

  • Restore loading of URLs from trusted domains. An attempt to render a graph from an untrusted domain will render an error.
  • Explicitly disallow http by default
  • Restore loading of graphs from the wikidatasparql:// protocol
  • Restore loading of graphs from the geoshape:// protocol
  • Restore loading of graphs from the wikiraw:// protocol

Sign off steps

The Graph extension previously supported other protocols. Understand the usage of these and how and if to restore these for 3rd parties if needed.

  • Restore loading of graphs from the wikiapi:// protocol
  • Restore loading of graphs from the wikirest:// protocol
  • Restore loading of graphs from the tabular:// protocol
  • Restore loading of graphs from the map:// protocol
  • Restore loading of graphs from the wikifile:// protocol
  • Restore loading of graphs from the geoline// protocol
  • Restore loading of graphs from the mapsnapshot:// protocol

References

  • @colt_browning: "...it would be great to eventually enable map: and tabular://. We use them in ru:Template:SkyMap (8000+ transclusions)." | source
  • @RobinLeicester: "OSM Location map is working fine on the Beta site by using direct file addressing at the moment, but if that is not suitable for final release, it will be looking for access to maps.wikimedia.org/img/ and to commons images (currently accessed using filepath..." | source
  • @Snaevar: "On protocols, there are 4 modules I am aware of that are going or have gone trough vega2to5 changes. Module:Statistical on ruwiki did not use protocols prior to the change and does not do so now. The other three Module:Graph, Template:Graph:Lines and "Template:OSM Location map" are all waiting on the protocols. Not sure how much clearer that can be." | source

Details

SubjectRepoBranchLines +/-
Add $wgGraphAllowHttpmediawiki/extensions/Graphmaster+18 -3
Restore custom protocols, mapping to https equivalentmediawiki/extensions/Graphmaster+60 -8
Allow URLs that match the allowed domainsmediawiki/extensions/Graphmaster+97 -5
WIP: Restore wikidatasparql supportmediawiki/extensions/Graphmaster+352 -5
WIP: Restore Wikiraw schemamediawiki/extensions/Graphmaster+142 -0
Subdomains should be trustedmediawiki/extensions/Graphmaster+40 -1
Fix URL sanitizationmediawiki/extensions/Graphmaster+28 -26
Customize query in gerrit

Related Objects
Search...

StatusSubtypeAssignedTask
 OpenBUG REPORTCCiufo-WMFT334940 All Graphs broken on Wikimedia wikis (due to security issue T336556)
 ResolvedSecurityJdlrobsonT334895 XSS via Graph extension
 DuplicateNoneT292341 [Epic] Graphs extension maintenance
 OpenNoneT335128 Update documentation and examples of Extension:Graph after deployment of Vega 5
 StalledJdlrobsonT335078 Drop automated support for Vega 2.6
 ResolvedTheDJT214984 PHP7's stricter JSON parsing breaks some wiki content
 StalledNoneT220469 404 error for some geoshape maps
 StalledBUG REPORTNoneT251710 global property "padding" is rendered incorrectly
 StalledNoneT145944 Error: URL hostname is not whitelisted: wikiraw:///Fertility-csv
 DeclinedNoneT346291 Re-enable the Graph Extension for use at all Wikimedia Wikis
 OpenFeatureJdlrobsonT165118 Support Vega 5.0+
 ResolvedBawolffT172938 Security review new version of the Vega lib
 DeclinedNoneT182536 Fix security issues found in Graphs extension during review of vega 2
 DeclinedNoneT195627 Support Vega 3.0 in Graphoid
 DeclinedNoneT195628 Support Vega Lite 2.0 in Graphoid
 Resolved• JseddonT226250 Graph not displayed if linked to a wikidata query
 ResolvedJdlrobsonT260542 Drop support for Vega 1.5
 ResolvedJdlrobsonT223026 Add Vega 5+ support to MediaWiki Graph extension
 ResolvedsbassettT335051 Application Security Review Request : Vega 5 and related dependencies for ext:Graph
 ResolvedFeatureJdlrobsonT335068 Update Vega 5+ to the latest build (version 5.24)
 DeclinedFeatureNoneT335048 Rewrite wiki graph modules written in older schemas for compatibility with Vega 5
 DeclinedJdlrobsonT335325 Restore loading of graphs from URL for Wikimedia allowed domains
OpenNoneT120319 Migrate graphs from Vega 2 to Vega 5 syntax on edit with VE

Event Timeline

Jdlrobson created this task.Apr 24 2023, 11:39 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptApr 24 2023, 11:39 PM
Jdlrobson triaged this task as High priority.Apr 25 2023, 12:05 AM
Jdlrobson moved this task from Vega Update Phase 1 to Vega Update Phase 2 on the MediaWiki-extensions-Graph board.
gerritbot added a comment.Apr 25 2023, 12:14 AM

Change 910613 had a related patch set uploaded (by Jdlrobson; author: Jdlrobson):

[mediawiki/extensions/Graph@master] Allow URLs that match the allowed domains

https://gerrit.wikimedia.org/r/910613

gerritbot added a project: Patch-For-Review.Apr 25 2023, 12:14 AM
Jdlrobson claimed this task.Apr 25 2023, 12:27 AM
Jdlrobson mentioned this in T145944: Error: URL hostname is not whitelisted: wikiraw:///Fertility-csv.Apr 25 2023, 9:51 PM
Jdlrobson added a parent task: T145944: Error: URL hostname is not whitelisted: wikiraw:///Fertility-csv.Apr 25 2023, 9:54 PM
Jdlrobson added a parent task: T220469: 404 error for some geoshape maps.
Jdlrobson mentioned this in T220469: 404 error for some geoshape maps.Apr 25 2023, 9:56 PM
Jdlrobson mentioned this in T166667: Whitelist .wmflabs.org for wikiraw:// protocol.
gerritbot added a comment.Apr 26 2023, 5:54 AM

Change 910613 merged by jenkins-bot:

[mediawiki/extensions/Graph@master] Allow URLs that match the allowed domains

https://gerrit.wikimedia.org/r/910613

Maintenance_bot removed a project: Patch-For-Review.Apr 26 2023, 6:10 AM
gerritbot added a comment.Apr 26 2023, 7:30 AM

Change 912228 had a related patch set uploaded (by Gergő Tisza; author: Gergő Tisza):

[mediawiki/extensions/Graph@master] Fix URL sanitization

https://gerrit.wikimedia.org/r/912228

gerritbot added a project: Patch-For-Review.Apr 26 2023, 7:30 AM
Base subscribed.Apr 26 2023, 7:31 AM
gerritbot added a comment.Apr 26 2023, 1:52 PM

Change 912228 merged by jenkins-bot:

[mediawiki/extensions/Graph@master] Fix URL sanitization

https://gerrit.wikimedia.org/r/912228

Maintenance_bot removed a project: Patch-For-Review.Apr 26 2023, 2:11 PM
Jdlrobson updated the task description. (Show Details)Apr 26 2023, 2:27 PM
Jdlrobson mentioned this in T334940: All Graphs broken on Wikimedia wikis (due to security issue T336556).Apr 26 2023, 9:21 PM
gerritbot added a comment.Apr 27 2023, 12:00 AM

Change 912422 had a related patch set uploaded (by Jdlrobson; author: Jdlrobson):

[mediawiki/extensions/Graph@master] WIP: Restore wikidatasparql support

https://gerrit.wikimedia.org/r/912422

gerritbot added a project: Patch-For-Review.Apr 27 2023, 12:00 AM
Jdlrobson added a comment.Apr 27 2023, 12:01 AM

Restoring wikidatasparql appears to be blocked on T335454: Compatibility layer error: No render when using from.transform

gerritbot added a comment.Apr 27 2023, 12:47 AM

Change 912428 had a related patch set uploaded (by Jdlrobson; author: Jdlrobson):

[mediawiki/extensions/Graph@master] WIP: Restore wikirest schema

https://gerrit.wikimedia.org/r/912428

gerritbot added a comment.Apr 27 2023, 12:47 AM

Change 912429 had a related patch set uploaded (by Jdlrobson; author: Jdlrobson):

[mediawiki/extensions/Graph@master] WIP: Restore Wikiraw schema

https://gerrit.wikimedia.org/r/912429

mxn subscribed.Apr 27 2023, 1:01 AM
Eragon_Shadeslayer subscribed.Apr 27 2023, 5:57 AM
Iniquity added a parent task: T335048: Rewrite wiki graph modules written in older schemas for compatibility with Vega 5.Apr 27 2023, 3:24 PM
gerritbot added a comment.Apr 27 2023, 7:59 PM

Change 912958 had a related patch set uploaded (by Jdlrobson; author: Jdlrobson):

[mediawiki/extensions/Graph@master] Subdomains should be trusted

https://gerrit.wikimedia.org/r/912958

gerritbot added a comment.Apr 28 2023, 4:42 PM

Change 912958 merged by jenkins-bot:

[mediawiki/extensions/Graph@master] Subdomains should be trusted

https://gerrit.wikimedia.org/r/912958

gerritbot added a comment.Apr 28 2023, 6:32 PM

Change 912429 abandoned by Jdlrobson:

[mediawiki/extensions/Graph@master] WIP: Restore Wikiraw schema

Reason:

Merging into a239ec3069650a118818922ae676a34acfc3ac4c

https://gerrit.wikimedia.org/r/912429

gerritbot added a comment.Apr 28 2023, 6:33 PM

Change 912422 abandoned by Jdlrobson:

[mediawiki/extensions/Graph@master] WIP: Restore wikidatasparql support

Reason:

Merging into Ia239ec3069650a118818922ae676a34acfc3ac4c

https://gerrit.wikimedia.org/r/912422

gerritbot added a comment.May 1 2023, 8:28 PM

Change 913984 had a related patch set uploaded (by Gergő Tisza; author: Gergő Tisza):

[mediawiki/extensions/Graph@master] Add $wgGraphAllowHttp

https://gerrit.wikimedia.org/r/913984

gerritbot added a comment.May 2 2023, 6:09 PM

Change 912428 abandoned by Jdlrobson:

[mediawiki/extensions/Graph@master] Restore custom protocols, mapping to https equivalent

Reason:

I will circle back to this when we have data indicating we need to prioritize this.

https://gerrit.wikimedia.org/r/912428

gerritbot added a comment.May 3 2023, 12:29 AM

Change 913984 merged by jenkins-bot:

[mediawiki/extensions/Graph@master] Add $wgGraphAllowHttp

https://gerrit.wikimedia.org/r/913984

Maintenance_bot removed a project: Patch-For-Review.May 3 2023, 12:30 AM
Jdlrobson updated the task description. (Show Details)May 3 2023, 2:57 PM
Jdlrobson lowered the priority of this task from High to Low.May 4 2023, 12:31 AM

We are making good progress with using https and action=raw.

See this as an example of migrating wikirest to https: https://www.mediawiki.org/w/index.php?title=Template:Graph:PageViews&action=history

Snaevar subscribed.May 9 2023, 5:47 PM

I am getting a CORS error on https://mediawiki.org/wiki/Template:Graph:Lines for the first graph. The graph is supposed to get datapoints from Wikimedia Commons using the https method. Browser: Firefox 113.0 on Win10. I am not seeing an error on Template:Graph:PageViews on the same website.

Cross-Origin Request Blocked: The Same Origin Policy disallows reading the remote resource at https://commons.wikimedia.org/w/index.php?title=Data:bls.gov/US%20Women%27s%20weekly%20earnings%20as%20a%20percent%20of%20men%27s%20by%20age,%20annual%20averages.tab&action=raw&ctype=application/json. (Reason: CORS header ‘Access-Control-Allow-Origin’ missing). Status code: 200.

TheDJ subscribed.EditedMay 10 2023, 9:44 AM
In T335325#8838073, @Snaevar wrote:

I am getting a CORS error on https://mediawiki.org/wiki/Template:Graph:Lines for the first graph. The graph is supposed to get datapoints from Wikimedia Commons using the https method. Browser: Firefox 113.0 on Win10. I am not seeing an error on Template:Graph:PageViews on the same website.

Cross-Origin Request Blocked: The Same Origin Policy disallows reading the remote resource at https://commons.wikimedia.org/w/index.php?title=Data:bls.gov/US%20Women%27s%20weekly%20earnings%20as%20a%20percent%20of%20men%27s%20by%20age,%20annual%20averages.tab&action=raw&ctype=application/json. (Reason: CORS header ‘Access-Control-Allow-Origin’ missing). Status code: 200.

Yeah, I think we forgot that you can't do cross domain requests using action=raw... See also origin= param on api.php

sbassett mentioned this in T335051: Application Security Review Request : Vega 5 and related dependencies for ext:Graph.May 10 2023, 3:48 PM
Jdlrobson added a comment.May 11 2023, 3:54 AM

That's annoying.

A work colleague suggested https://commons.wikimedia.org/w/api.php?action=jsondata&formatversion=2&format=jsonfm&title=Bls.gov/US_Women%27s_weekly_earnings_as_a_percent_of_men%27s_by_age,_annual_averages.tab&format=json&origin=* - would that work?

ReleaseTaggerBot added a project: MW-1.41-notes (1.41.0-wmf.9; 2023-05-15).May 15 2023, 9:02 PM
ReleaseTaggerBot edited projects, added MW-1.41-notes (1.41.0-wmf.10; 2023-05-23); removed MW-1.41-notes (1.41.0-wmf.9; 2023-05-15).May 16 2023, 5:02 AM
Rasputin1493 subscribed.Jun 17 2023, 3:31 AM
Pamputt subscribed.Jun 19 2023, 8:08 AM
Pietrasagh subscribed.Oct 28 2023, 9:03 AM

Pseudo-protocols and CORS workaround doesn't work on beta.wmflabs for Wikimedia REST API.

Error: The host wikimedia.org is not in the list of trusted domains for the protocol https:

please add wikimedia.org to "GraphAllowedDomains" to "https"

• ppelberg added a parent task: T346291: Re-enable the Graph Extension for use at all Wikimedia Wikis.Nov 9 2023, 9:32 PM
Ladsgroup subscribed.Nov 16 2023, 11:49 PM
Jdlrobson updated the task description. (Show Details)Nov 20 2023, 10:39 PM
• ppelberg updated the task description. (Show Details)Nov 22 2023, 7:03 PM
• ppelberg added subscribers: colt_browning, RobinLeicester.
Bugreporter closed this task as Declined.Thu, Apr 11, 1:47 PM
Bugreporter subscribed.

Per T334940#9705555

Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL