Page MenuHomePhabricator

Add support for KML/KMZ filetype
Open, NormalPublic

Description

When I try to upload a valid KML file I get "File extension does not match MIME type."

See Also:

This card tracks a proposal from the 2015 Community Wishlist Survey: https://meta.wikimedia.org/wiki/2015_Community_Wishlist_Survey

This proposal received 27 support votes, and was ranked #34 out of 107 proposals. https://meta.wikimedia.org/wiki/Community_Tech_project_ideas#Support_KML_files_in_Commons

Details

Reference
bz26059

Event Timeline

There are a very large number of changes, so older changes are hidden. Show Older Changes

Can I get a hip hip hurrah for google not checking mime types... </sarcasm>.


By a brief look at template:Attached_KML, it seems that the templates only use a small portion of the KML standard. It may perhaps be less work to do a custom tag (easytimeline style) where we generate a safe kml file from a simpler language for specifying coordinates to highlight on the map.

The downside is obviously that in the future people might want more features from their kml.


But what if there were no <heading> tags, or none in the first 1KB?

That would take care of the IE6 issue, but as Chris mentions there are other concerns, in particular allowing third parties to track the ip's of our users.

(In reply to comment #28)

Can I get a hip hip hurrah for google not checking mime types... </sarcasm>.

As an aside, according to google maps docs, "HTML content is allowed but is sanitized to protect from cross-browser attacks", which makes google maps not checking mime types (and hence the Wikipedian's inline kml file hack) much less scary... :D

richardg_uk wrote:

Isn't the new ContentHandler designed to handle non-wikitext article "paradata" such as KML/KMZ?

Though discussion here had been dormant for a while, I had assumed that it was exactly the kind of case which the new handler would enable.

(In reply to comment #28)

By a brief look at template:Attached_KML, it seems that the templates only use
a small portion of the KML standard. It may perhaps be less work to do a custom
tag (easytimeline style) where we generate a safe kml file from a simpler
language for specifying coordinates to highlight on the map.
The downside is obviously that in the future people might want more features
from their kml.

A sanitised subset is exactly what is sought and required. Wikitext and SVG are already subject to tag whitelisting, which is what KML needs.

But what if there were no <heading> tags, or none in the first 1KB?

That would take care of the IE6 issue, but as Chris mentions there are other
concerns, in particular allowing third parties to track the ip's of our users.

Once external resource requests are filtered (as with SVG files), there is no more privacy leakage than there would be with a plain external URL in an article's wikitext. Google Maps just downloads the raw content of the specified subpage if a reader clicks the Attached KML link.

Qgil added a comment.Mar 25 2013, 4:27 AM

(As suggested by Bawolff at http://www.mediawiki.org/wiki/Talk:Mentorship_programs/Possible_projects#GSOC_2013_candidates_missing_one_thing_or_two_25493 )

Do you think the development of this feature is suitable for a Google Summer of Code project? If you think this make sense then we would need a short description of the project published at http://www.mediawiki.org/wiki/Mentorship_programs/Possible_projects and at least one mentor.

From there we would publish it at https://www.mediawiki.org/wiki/Summer_of_Code_2013#Project_ideas

Moved to Normal, because we do not view this as high priority at this time.

Well, Erik Moller has just sent out an email saying that JS on IE6 will be disabled completely with 1.24wm17.

Does this mean that something could be done with this bug? Or would it be better to shift efforts to Wikidata? (or both, and just have Wikidata link to files on Commons?)

Well something could have always been done, just noone has been willing to spend the time to do it.

The js announcement doesnt affect the security issues mentioned above.

The use cases for which I opened the bug are not helped in any way by Wikidata. They also have nothing to do with the WMF. So having Wikidata does not help.

Joeroen De Dauw: see bug 55549, which would solve the big-picture problem with Wikidata.

Qgil added a comment.Feb 11 2015, 1:44 PM

Wikimedia will apply to Google Summer of Code and Outreachy on Tuesday, February 17. If you want this task to become a featured project idea, please follow these instructions.

Qgil added a comment.Feb 16 2015, 11:52 PM

If there is no interest / critical mentoring mass to promote this project for GSoC / Outreachy, then maybe the current "Normal" priority should be lower?

It is hard to say, since this is (according to my understanding) a security issue, and because it is not clear how Wikidata will affect this.

TheDJ added a comment.Feb 17 2015, 7:43 AM

@Qgil, I note this is not an easy task, since there are some security aspects to it... That's the reason why it stalled. The task itself is easy, but the background is complex and a solution will require cooperation with and validation by @csteipp

Qgil added a comment.Feb 17 2015, 8:54 AM

Thank you for the quick feedback. Ok, I will remove the Possible-Tech-Projects tag. Feel free to bring it back if/when this task is a good candidate for GSoC/Outreachy.

Qgil set Security to None.

The security issues don't sound all the complicated. They basically involve building a parser, checking the file is valid according to some subset of the spec (For our usecase, it might not even have to be a very big subset).

Possibly maybe also adding 64kb of whitespace to the beginning of the file to trick the IE parser.

Not a walk in the park, but possibly within the realm of what a gsoc student could do. Of course it would still require a mentor (I am not volunteering)

Khushbuparakh removed Khushbuparakh as the assignee of this task.
Khushbuparakh added a subscriber: Khushbuparakh.
Qgil updated the task description. (Show Details)Jun 9 2015, 8:31 AM
Qgil added a project: Commons.

Not a walk in the park, but possibly within the realm of what a gsoc student could do. Of course it would still require a mentor (I am not volunteering)

I'm bringing back the Possible-Tech-Projects tag, also encouraged by the interest at https://meta.wikimedia.org/wiki/Community_Tech_project_ideas#Support_KML_files_in_Commons

Jdforrester-WMF moved this task from Untriaged to Backlog on the Multimedia board.Sep 4 2015, 6:33 PM
Restricted Application added subscribers: Steinsplitter, Matanya, Aklapper. · View Herald TranscriptSep 4 2015, 6:33 PM
Qgil added a comment.Sep 23 2015, 9:13 AM

This is a message posted to all tasks under "Backlog" at Possible-Tech-Projects. Outreachy-Round-11 is around the corner. If you want to propose this task as a featured project idea, we need a clear plan with community support, and two mentors willing to support it.

Qgil added a comment.Sep 23 2015, 9:36 AM

This is a message sent to all Possible-Tech-Projects. The new round of Wikimedia Individual Engagement Grants is open until 29 Sep. For the first time, technical projects are within scope, thanks to the feedback received at Wikimania 2015, before, and after (T105414). If someone is interested in obtaining funds to push this task, this might be a good way.

TasneemLo added a subscriber: TasneemLo.

Adding this to the "Missing Mentors" column.
@Bawolff are you still interested in mentoring this project ?

This is the last call for Possible-Tech-Projects missing mentors. The application deadline for Outreachy-Round-11 is 2015-11-02. If this proposal doesn't have two mentors assigned by the end of Thursday, October 22, it will be moved as a candidate for the next round.

Interested in mentoring? Check the documentation for possible mentors.

As previously mentioned, this task is moved to 'Recheck in February 2016' as it doesn't have two mentors assigned to it as of today, October 23 - 2015. The project will be included in the discussion of next iteration of GSoC/Outreachy, and is excluded from #Outreachy-11. Potential candidates are discouraged from submitting proposals to this task for #Outreachy-11 as it lacks mentors in this round.

I am contributing in Outreachy11 for this Project whom I can contact

I am contributing in Outreachy11 for this Project whom I can contact

@Khushbuparakh Thank you for your interest. The project lacks mentors as of now. Feel free to apply to some other project in https://phabricator.wikimedia.org/tag/possible-tech-projects/ - 'Featured for GSoC and Outreachy' and start by submitting a proposal, following https://www.mediawiki.org/wiki/Outreach_programs/Life_of_a_successful_project#Submitting_your_proposal

putnik added a subscriber: putnik.Jan 5 2016, 10:09 AM
IMPORTANT: If you are a community developer interested in working on this task: The Wikimedia Hackathon 2016 (Jerusalem, March 31 - April 3) focuses on #Community-Wishlist-Survey projects. There is some budget for sponsoring volunteer developers. THE DEADLINE TO REQUEST TRAVEL SPONSORSHIP IS TODAY, JANUARY 21. Exceptions can be made for developers focusing on Community Wishlist projects until the end of Sunday 24, but not beyond. If you or someone you know is interested, please REGISTER NOW.
Restricted Application added a subscriber: JEumerus. · View Herald TranscriptJan 21 2016, 2:53 PM
DannyH updated the task description. (Show Details)Feb 6 2016, 12:01 AM
Meno25 removed a subscriber: Meno25.Feb 8 2016, 7:53 PM
Sumit added a subscriber: Sumit.Feb 18 2016, 1:33 PM
NOTE: This task is a proposed project for Google-Summer-of-Code (2016) and Outreachy-Round-12 : GSoC 2016 and Outreachy round 12 is around the corner, and this task is listed as a Possible-Tech-Projects for the same. Projects listed for the internship programs should have a well-defined scope within the timeline of the event, minimum of two mentors, and should take about 2 weeks for a senior developer to complete. Interested in mentoring? Please add your details to the task description, if not done yet. Prospective interns should go through Life of a successful project doc to find out how to come up with a strong proposal for the same.

I started to work with this problem and hope to finish within this month, probably at Hackathon.

Qgil added a comment.Apr 18 2016, 1:29 PM

Hi @putnik, did you work on this task at the Hackathon?

Restricted Application added a subscriber: Poyekhali. · View Herald TranscriptApr 18 2016, 1:29 PM

@putnik: Any news to share? What's the status? :)

Jklamo added a subscriber: Jklamo.Aug 30 2016, 10:36 AM
Sumit added a comment.Sep 13 2016, 7:13 AM

This is a community wishlist task. Regarding the progress on this, could this task use help from an Outreachy intern( Dec 6 to March 6 )? Please note that applications are open until Oct - 17.

Let us know if possible at the earliest.
Ideally it should take about 2-3 weeks for an experienced developer to complete the task, in order to qualify as an intern project.
If the scope is wide, it could be worked out as per the internship needs :)

Qgil added a comment.Sep 13 2016, 7:27 AM

In any case, it looks like this task is beyond "Needing discussion" and into "Missing mentors"?

Evad37 added a subscriber: Evad37.Sep 15 2016, 11:20 AM
Yurik added a subscriber: Yurik.Oct 18 2016, 7:48 PM

I think we should consider a few things when implementing (or not implementing) this task. There are several reasons to store KML/KMZ data on wiki:

  • store a well known geographical object feature, such as a city/country/park/building outline, or a line (i.e. a road/river/...). I believe most of the existing KML data in wiki falls into this category.
  • store an article specific map information, such as the area of an animal habitat, or some transient feature like the army positions during an armed conflict.

I think wiki is not well suited to solve the first use case - OpenStreetMap has the vast number of tools and the needed community's process to keep this information accurate and up to date, and we already have the way to show that data in wiki articles - see examples. The only requirement is that OSM's objects are tagged with the Wikidata IDs.
The second usecase is much more valid for on-wiki storage since we don't support (yet) any external geo data sources. Yet, I think KML/KMZ format is a dead end - it is much harder to process, and the industry has moved towards formats like GeoJSON/TopoJSON, which are better for machine processing at the servers and browsers.
That said, I understand there are many instances where the legacy KML format is still in use. So I think we could implement 3 "levels of support":

  • require users to use numerous online KML to GeoJSON converters, and only store GeoJSON
  • same as above, but also allow a way to download data as KML - doing an on-the-fly conversion
  • same as first two, plus offer a way to upload KML, which will be dynamically converted and stored as GeoJSON.

Per above, I think native KML support (storing data as KML) is counterproductive, and will increase our tech debt without providing much benefit due to the first use case being solved via OSM, and the second one being solved with the native geojson storage.

Yurik awarded a token.Oct 18 2016, 7:50 PM

Any updates on this task? Could one of the subscribers be interested in mentoring this task in our upcoming programs such as Outreachy, GSOC, and RGSOC (railsgirlssummerofcode.org) or if it's beyond your scope, could you help suggest some potential mentors? Thank you very much!

Yurik added a comment.Jan 26 2017, 5:37 AM

@srishakatux, what do you think about the comment I wrote above in T28059#2726773?

TheDJ added a comment.EditedJan 26 2017, 9:50 AM

Simple KML and GPX to GeoJSON converter:
https://github.com/mapbox/togeojson

This could be used like similar to write a JS feature to add an 'import' feature to Tabular .map pages, like i did with CSV and Excel files with tabularImportExport.js

Qgil removed a subscriber: Qgil.Jan 27 2017, 3:36 PM
Base added a subscriber: Base.Nov 25 2017, 7:31 AM

Removing the Possible-Tech-Projects tag as we are planning to kill it soon! This project does not seem to fit in the Outreach-Programs-Projects category in its current state, so I am not adding that tag right now!

gpaumier removed a subscriber: gpaumier.Jul 18 2018, 5:57 PM