Page MenuHomePhabricator

Refine Commons Template:Map with Lua modules to populate the template with Wikidata
Open, Needs TriagePublic

Description

Hello, please forgive my mistakes in advance, I am a first-time Hackathon attendee and this is my first post in Phabricator.
I am trying to build a scalable workflow for uploading images of geographic maps/map editions to Wikimedia Commons that already have Wikidata items. Ideally, I could pre-fill file information with their Wikidata items' statements. I know that several templates already do this (Artwork, Book, Photograph, and Art photo,) but Map currently does not. I do not know how to edit Lua to accomplish this, but I would like to learn if anyone has time to help me.
The discussion with more details is on the template talk page, in this section: https://commons.wikimedia.org/wiki/Template_talk:Map#Can_this_template_pull_info_from_Wikidata?
Thanks in advance!
Lena

Event Timeline

Restricted Application added subscribers: Liuxinyu970226, Aklapper. · View Herald TranscriptMay 8 2020, 11:25 PM
TiagoLubiana rescinded a token.
TiagoLubiana awarded a token.
TiagoLubiana added a subscriber: TiagoLubiana.

I'm up for working with you if @Jarekt can give advise!

Abbe98 added a subscriber: Abbe98.May 9 2020, 10:02 AM

Worth noting is that this would make the template usages that pull data from Wikidata incompatible with many of the map tools including Wikimaps Warper.

That is most likely also fixable. Although I understand that the gain may be less than the effort.

What other usages do you envision? Also, it is not necessary to complete it here and now, but to set it on a trajectory that will keep it up-to-date.

Jarekt added a comment.May 9 2020, 2:09 PM

I'm up for working with you if @Jarekt can give advise!

Susanna, Stage #1 is to just reproduce the capabilities of the current template in Lua. I can get it started and maybe hand over to you. I noticed I began to work on Module:Map in 2013 but stopped, so U will take over that name.

Hi everyone! Thank you for the attention to this and thanks for letting me pull you in, @Susannaanas and @Jarekt. To answer the question about usages, what I'm hoping for is a way to set up a workflow whereby Wikidata fields that I spend a lot of time working on when I create map items, such as the coordinates, scale, etc. that are map format-specific, can get filled quite easily by someone who has an image of the map they want to upload to Commons but may not have all the cataloging information (which I have put into Wikidata) on hand. My library, like so many in these days, is trying to come up with ways that people with varying levels of technical expertise and subject matter knowledge can get involved with helping to increase access to materials through Wikimedia platforms. Therefore, I'd want people who are not thinking about metadata as constantly as I am still to feel that they could contribute, if they just want to upload images and add some categories and be on their way. Does this make sense? I don't want to bore you all but I can elaborate as needed.

Some of my specific dream features:

  • A map template that allows for the flexibility of mapmakers' roles. It's usually incorrect to describe them as {{Artist}}, but {{Creator}} is only useful with qualifiers, such as cartographer, engraver, publisher, etc. but I don't know how feasible qualifiers are.
  • "Depicted" statements so that geographic descriptions remain standardized according to the cataloging shown in Wikidata

I also want to note that I don't think the Wikidata parts of the template need to be so numerous. The "item-level" descriptions afforded to artworks, for example, are often not appropriate for historic maps because of how they were published in such variable forms even in their first editions. So, thinking about this in the context of the Historical map properties, I would want the scale, the creator, the geographic information, etc., to come from Wikidata, but I would be less interested in things like the height or width. People could fill that part in manually for specific individual copies as needed.

Please let me know how to get started with helping, if there's a way for me to collaborate on the Lua template or the map module. I am also happy to revise this and answer more questions.

Yupik added a subscriber: Yupik.May 9 2020, 6:53 PM

Some Lua basics [[T227557]] from Wikimania.

Jarekt added a comment.May 9 2020, 7:27 PM

I have very early stage of Module:Map done. At this stage the goal is to recreate Template:Map in Lua. Template Map has so many fields that at this point I added about 50-60% of fields which were shared with Artwork template. I will switch to other tasks now, but it would be great if others would help by adding the rest of the fields. Especially fields which are only used by this template and which might need translations set up at Module:I18n/map. Ping me here or on my commons talk page if there are any questions.

Thank you @Jarekt ! There will be challenges ahead as @Abbe98 mentioned, apart from unique properties. But we can work first on the properties.

  • Workflow between Wikimaps Warper and the Map template. The Map template has a value for the warp status of the image in Wikimaps Warper. Warper writes to the Map template the changed value when it alters the map image.
  • The Pattypan upload flow may be affected
  • Are there other dependencies?

BTW Cool ascii art!

Can we make a temporary template for this module, like MapWD?

@Librarian_lena You are providing really valuable information about the division between the work and the copy, and how this information should be shared between Wikidata and Commons. I wonder if you could even annotate the Historical map properties page with a column marking the "Wikidata" properties and the "Commons" properties. I hope to look at this again tomorrow.

Librarian_lena added a comment.EditedMay 9 2020, 9:41 PM

@Susannaanas I'm glad you think so! This is Wikidata-to-Commons-properties proposal I did recently for a work project, although I've removed institution-specific identifiers. Is it helpful in this context?

Also I'd be happy to work with you on this tomorrow if I can help. I'm at UTC-4. I'll keep checking this page and you can also find me on Telegram or my talk page, etc.

Jarekt added a comment.May 9 2020, 9:45 PM

Can we make a temporary template for this module, like MapWD?

I linked c:Template:Map/sandbox to the new module. I was also using Module_talk:Map/testcases.

BTW Cool ascii art!

Thanks. I use http://patorjk.com/software/taag/#p=display&f=Standard&t=Type%20Something%20 generator for all the modules I start.

As for changing workflows, that might change, I am not sure what you mean. Once a new module is written and {{Map}} start using it, if it is done right there is no difference to the user unless they start linking the data with wikidata and than the only difference is that most metadata goes to Wikidata and the template is mostly empty.

@Jarekt : This is what I thought might be possible. That the template is crafted so that it can still manage the text-based functions it has had previously. So maybe we can first see that we can safeguard warp status: If set, it will create a link to Wikimaps Warper. Wikimaps Warper in turn writes to Map template when the map image has been georeferenced in Warper. All the allowed values are: skip: (The map is not suited for the warper), help (An invitation to import the map into the warper), unwarped (The map is in the warper but not rectified), warped (The map is in the warper where it has been rectified), and external (A rectified map exists in an external georeferencer).

Workflow between Wikimaps Warper and the Map template. The Map template has a value for the warp status of the image in Wikimaps Warper. Warper writes to the Map template the changed value when it alters the map image.

This I believe should still work. The problem would be when a map is imported to Warper because it parses and reads values from the template, values that would not be available in a template powered by Wikidata.

This I believe should still work. The problem would be when a map is imported to Warper because it parses and reads values from the template, values that would not be available in a template powered by Wikidata.

I am not 100% sure to say but in my mind the link to the Commons file is the only important one, others have been there for convenience. If the other values cannot be read, the process might break, but this I think should not stop the development. Can you list the values that are read, or point to the code?

@Chippyy I wonder if you are around and interested in providing some insight to the Map template revamp?

Susannaanas added a comment.EditedMay 10 2020, 9:11 AM

The codebase is here, need to locate the point where maps are read https://github.com/wikimaps-dev/mapwarper. Branch wikimaps_new

The Map template links to Warper via the warp status property and button. So this is the sending end

skip (nothing)
unwarped = {{Clickable button|target=warper.wmflabs.org/wikimaps/new?pageid=<includeonly>{{PAGEID}}</includeonly>
warped = {{Clickable button|target=
warper.wmflabs.org/wikimaps/new?pageid=<includeonly>{{PAGEID}}</includeonly>
external = {{Clickable button 2|link={{{warp url|{{{Warp url|{{{warp_url|{{{Warp_url|}}} }}} }}} }}}|text={{ucfirst:{{I18n/map|external_warp}}}}|color=blue2}}
#default = {{Clickable button|target=//warper.wmflabs.org/wikimaps/new?pageid=<includeonly>{{PAGEID}}</includeonly>

I have worked on this page to display a bit of the FRBR division as well as mark properties that are available in the Map template https://www.wikidata.org/wiki/Wikidata:WikiProject_Maps/Historical_map_properties

I have not been able to locate the code that reads and writes to Commons yet.

I have worked on this page to display a bit of the FRBR division as well as mark properties that are available in the Map template https://www.wikidata.org/wiki/Wikidata:WikiProject_Maps/Historical_map_properties

This looks amazing, thank you!

mxn added a subscriber: mxn.May 10 2020, 3:35 PM

Hi @mxn and welcome onboard! You might have interest in thinking how Warper control point data tables should be stored and utilized by the warping process. It will not be the first thing to do here though.

I have not been able to locate the code that reads and writes to Commons yet.

Hello! Can anyone point to an example or some instructions for how to do this? I've been trying to figure it out by comparing it to other modules but nothing I've done has worked.

Lena, I am happy to help, I just cant commit to do all the work as I am stretch quite thin with maintenance of existing modules. There are no instructions other than https://www.mediawiki.org/wiki/Extension:Scribunto/Lua_reference_manual and https://www.mediawiki.org/wiki/Extension:Wikibase_Client/Lua, so the best way is to learn by example. [[Module:Artwork]] has a lot of overlap but it does a lot of other stuff (like allow transfer of data from commons to wikidata) which can be confusing. [[Module:Information]] or Module:Creator also do similar things.

The way I usually work on those is to write everything in Notepad++ and than cut and paste to Module page. I create some test page and use "Preview page with this template" to see if current code works or not, without saving

@ Jarekt: I use the same method also With Notepad++ (it's strange that the source editor for modules in the wiki does not even show the line numbers!)

Note that in the text console, the line numbers for errors located in the console input is always 6 lines larger than the actual text... That's because the console submits our input text by prepadding 6 "sandboxing" lines of code (probably ending by a "do "), and certainly appending other lines of code (most probably starting by a a newline or space followed by an "end", that we don't care about):
These 6 extra lines should be packed on the same line (replace all newlines by spaces in these 6 lines) and prepadded to the input text without any intermediate newline (just an ending space separator if needed by the syntax used), before being submitted to Lua evaluation. This would help correlate line numbers shown in error messages with what we really input in the console.

For now we still need to substract 6 mentally from the displayed error objects (This is a small bug of the Scribunto console only, not a bug of Lua itself, and not existing for errors thrown inside loaded modules), or we can copy-paste the module code into Notepad++, and move the cursor 6 lines above the indicated line number. This number 6 has sometimes changed in past versions of Scribunto.

That is strange, my system behaves differently. After opening a Module file originally there are no number lines, but than a different skin (or something ) kicks in and I can see code lines. Also if I see an error line number that is the correct line number I see in Notepad++ and in the on-wiki editor.

The effect is not the same depending on wikis. I assume they don't have the same version of Scribunto deployed.

For example, if I click on the error link shown in pages having script errors in Commons, the wikitext of the linked Module opens without line numbers (the top navbar above the editable module is not always present, so there's no "<>" tool at the start to change the editor mode, as if it was disabled and we were editing a wiki page or a talk).
In this case, testing anything in the console will then show the increment of 6 lines for lines numbers in error objects (for all lines relative to the code typed in the console and submitted to execution, not lines inside functions of external loaded modules, which show correct line numbers in these module sources)...

Thanks @Jarekt, I've been trying to figure it out based on the other modules you mentioned, especially [[Module:Artwork]], but nothing works even when I copy and paste all the lines that your comments say are about harvesting Wikidata to populate the rows. I just get empty rows. I've been editing in [[Module:Map/sandbox]] and testing it in a page called [[Template:Map/sandbox/infopull]] but I'm not getting anywhere. The files I test just end up with a blank template no matter what I do. If you (or anyone) get a chance to look and see what I'm doing wrong, I would appreciate it. I'll keep trying next week when I can devote more hours to it again.

Lydia_Pintscher moved this task from incoming to monitoring on the Wikidata board.Oct 4 2020, 3:28 PM
Jheald added a subscriber: Jheald.Nov 26 2020, 12:33 PM

A couple of days ago I have switched Tempate:Map to use the lua Module:Map rather than wikitext. Thanks to everyone on this thread who put so much work in to get Module:Map to this stage!

The old version is available as Template:Map.old, so you can go to your favourite maps and switch back and forth between the two, just to make sure everything all still works. There are a few cosmetic changes, in particular for some of the fields the source for their labels and internationalisations has been changed (eg to use labels drawn from wikidata); but with luck overall functionality should be pretty much the same.

If there are any problems, do report them on the template talk page or here. So far there doesn't seem to be an army there with pitchforks and flaming torches, so I am hoping there haven't been too many bumps.