Page MenuHomePhabricator

Technology to transclude git content into wiki pages
Closed, DeclinedPublic

Description

We have content in git repositories that we want to show in wiki pages:

Documentation in git

Sample code from git

Minimum viable product

  1. Develop an extension such that existing wiki page can transclude latest content from git using some syntax, probably like Git2Page's {{snippet}} parser function, with similar functionality.
  2. Using this editors can
    • transclude wikitext into the page
    • It can provide source code to the <source> tag or parser function to transclude highlighted source code into the page.
  3. The extension is deployed on the production wiki for mediawiki.org:
    • caching strategy
    • continuous integration
    • security review
    • etc.

Possible/desirable features

After the MVP!

  • Transcluded content is available to Lua modules to operate on, e.g. transform hooks.txt into something useful.
  • It should be possible to invoke this functionality from a wikitext template, so that e.g. a wiki editor can write a Template:Show_extension_README or `Template:PHP_code_snippet
  • Render Markdown files (e.g. using PHP Markdown)
  • Render .txt files with some minimal formatting (like the <poem> tag?)
  • Render other formats if agreed (e.g. pandoc).

For sample code, editors need to transclude parts of Git files into a wiki page (see example). Possible approaches:

  • In the parser tag, specify a range of lines from the file, like Git2pages' {{snippet}} parser function:
    • {{#snippet:repository=mediawiki/extensions/examples | filename=examples/ContentAction/ContentAction.php |startline=24 |endline=36}}
  • Could also/instead identify sections of code with starting and ending markers. Developers would annotate files in git with something like // begin name=simple-text-field and a matching // end name=simple-text-field , and then in the parser tag wiki editors specify {{#snippet: ... |section=simple-text-field` }}.
    • Obviously developers might mistype, delete, or duplicate these section marker names, but it's developer error; the extension can set a hidden category "Pages with garbled Git section transclusion", or warn, or try to do the right thing.
  • Could leave it up to a Lua module to get sections of the file using either technique.
Unlikely/non features
  • Create sets of wiki pages from git files (e.g. doc/hooks.txt)
    • if you want that, probably publish HTML to http://doc.wikimedia.org; or develop a Lua module that invokes this task's functionality and grabs the right part of the file.
  • Push git content to wiki page using some kind of src->destWikiPage mapping file
    • instead just have a wiki bot insert the parser function into pages
  • Explore alternative syntax?: develop a special page e.g. Special:FromGit, and transclude that in other pages, e.g. {{[[Special:FromGit/project/git/path/path/to/file]]}} (This is a similar approach to {{Special:ApiHelp/query+categoryinfo}} which transcludes action API documentation.)
    • it's hard to specify all the parameters. What's the actual use case for this syntax?

Prior art

Information for Possible-Tech-Projects

  • Primary mentor:
  • Co-mentor: @Tgr
  • Other mentors:

(optional, Phabricator username)

See also

Related Objects

Event Timeline

There are a very large number of changes, so older changes are hidden. Show Older Changes

@Tgr, I want to apply for this project for the next round of outreachy, kindly specify which microtask I should work on. Should I work on this T115388: Convert hooks.txt to YAML [Outreachy microtask] ?

@Tgr, I want to apply for this project for the next round of outreachy, kindly specify which microtask I should work on. Should I work on this T115388: Convert hooks.txt to YAML [Outreachy microtask] ?

@Shrutika719, since it's listed as a microtask and is yet unresolved, feel free to work on it.

That task is already assigned; I'm not sure it is a good idea to have multiple people working on the same microtask in parallel as the time of one of them will be wasted. In this case the task is flexible enough that they would probably come up with different solutions and both can be accepted, but it I would prefer to give people real tasks.

That said, I can't really think of an easy task related to this project. I'll ask around...

@Tgr, Did you look for any other microtask?

Yes; you could change maintenance/findHooks.php to work with a YAML file instead of the current textfile. That kind of presupposes T115388 but you could just convert a few hook definitions by hand, delete the rest, and use that for testing.

@bd808 can help with figuring out which YAML parser to use.

01tonythomas added a subscriber: 01tonythomas.

I am shifting this to Outreachy-Round-11 as the project description has two mentors, micro-tasks and looks ready for the 11th edition of Outreachy ( Dec 2015 - Mar 2016 ) . Potential candidates should start by submitting their proposals as a blocker for this task, by November 02.

Feel free to revert it back, if this task has some relevant issues which might block its completion in this term of Outreachy.

In T91626#1733600, @Tgr wrote:

Yes; you could change maintenance/findHooks.php to work with a YAML file instead of the current textfile. That kind of presupposes T115388 but you could just convert a few hook definitions by hand, delete the rest, and use that for testing.

Turned it into a proper task: T115959

Hello! I am interested to work on this project for the upcoming Outreachy Round 11. I have decent knowledge in php , git and database fundamentals.Hope that's enough to get started! Can anybody point out any microtasks so that i can I can start working on it. Thanks.

Hello Everyone! I am interested in this project to work on the 11th round of Outreachy internship. I have reasonable knowledge in php and Git. Can someone direct me on a possible path to get started?

@Tgr I am almost done with the microtask assigned to me. Is there any other microtask that needs to be done before submitting the proposal? If yes, kindly specify one.

@Devirk, @Hansika11: sorry for the slow answer. There are already 3 people applying to this task; unless you have a very strong preference for it, I would suggest finding another one less competition.

@Shrutika719 no more microtask required (you are of course always welcome to do more and it might help us in judging your coding skills), but you need to create a proposal as described in https://www.mediawiki.org/wiki/Outreachy/Round_11#Candidates.

It can provide source code to the <source> tag or parser function to transclude highlighted source code into the page.

Does this project demand to have all the functionalities of SyntaxHighlight_GeSHi ?

It can provide source code to the <source> tag or parser function to transclude highlighted source code into the page.

Does this project demand to have all the functionalities of SyntaxHighlight_GeSHi ?

It can require that extension and take advantage of it and even propose changes to it. We certainly don't want to reinvent the wheel.
It seems there are two ways to get syntax highlighting.

  1. Add parameters to this extension's parser function (or parser tag) to specify syntax highlighting, lang, etc., then somehow invoke SyntaxHighlight_GeSHi's processing.
  1. Let wiki editors embed this extension's parser function (or parser tag) inside the source tag.

For 2, I played around with Git2Pages on my local wiki and with some changes got this invocation to work:

{{#tag:source|{{#snippet:repository=https://gerrit.wikimedia.org/r/mediawiki/extensions/BoilerPlate |filename=BoilerPlate.php}} |lang=php}}

I.e. I invoked <source> as a parser function using {{#tag:}} (Manual:Tag extensions says "All tag extensions can also be called as a parser function using {{#tag:tagname|input|attribute_name=value}} which will have pre save transform applied). For its input text to highlight I invoked Git2Page's {{#snippet:...}} parser tag which pulled in code from a repo. Git2Pages output has to change when supplying plain text for SyntaxHighlight_GeSHi, the changes are in P2252.

This comment was removed by Akangupt.

@Akangupt and others,
This task's description, under Possible/desirable features, mentioned:

  • Transcluded content appears in a tag or template that identifies source, so that users can edit the text around it.
    • for simple transclusion MVP, simply invoke it from a wiki template similar to Template:Api_help).

I think the first bullet is a holdover from when this task was titled " "technology to push or pull remote text content into wiki pages"". If a system could push git content into a wiki page then editors would obviously want to know where it came from, e.g.

blah blah blah
<!-- the following wikitext came from extensions/SemanticResultFormats/README.wiki, inserted by MyMagicPushContent on 2015-11-06 -->
Semantic Result Formats (a.k.a. SRF) is an extension to MediaWiki that ...
The individual formats can be added 
...
<!-- end of wikitext from extensions/SemanticResultFormats/README.md -->
blah blah blah

Now that this task has been refined and narrowed to a pull technology, that doesn't apply. Wiki editors will see

blah blah blah
{{#MyMagicGitInclude: project=mediawiki/extensions/Wikibase| file=README.wiki}}
blah blah blah

So I will remove the mis-feature. I apologize for the confusion!

As for the sub-bullet, it should be possible to invoke this functionality from a wikitext template, so that e.g. a wiki editor can write a Template:Show_extension_README or Template:PHP_code_snippet. I assume that is doable, it probably just requires fiddling around to quote {{, |, =, etc. I will add that to the description.

NOTE: Outreachy round 12 applications are now open and GSoC 2016 is round the corner. This project was featured for Outreachy round 11 and has a well defined scope. Are you ready to mentor the project this season? If yes, then we'll feature this for Outreachy round 12 and GSoC 2016 as well. Please reply back in comments.

Hello @nikhil_yadala . If you're new to MediaWiki, start by going through https://www.mediawiki.org/wiki/How_to_become_a_MediaWiki_hacker to get acquainted with it. Then look for and try to solve bugs marked good first task here on Phabricator. After you gain a basic idea of MediaWiki, you can try to understand what the above task is about. Feel free to ask questions, where you don't follow, here or on irc. Thanks for your interest!

Hi,

I'm interested in doing this for GSoC.
The microtasks mentioned on this page are either closed or have a patch submitted. Could someone add a few more ?

For what is worth, @Spage cannot mentor this project in this round.

@Tgr, @Ankitashukla, are you volunteering as mentors for this project idea in the upcoming round?

@Qgil @nikhil_yadala Unfortunately I won't be able to mentor during this round of GSOC and Outreachy.

@Tgr would you be willing to mentor this project for this round?
@Qgil @Ankitashukla @Spage any suggestions on who might be interested in mentoring this project for the upcoming GSoC/Outreachy rounds?

@Bawolff would you be willing to mentor this project for this round of GSoC/Outreachy?

I'm keen to work on this project. Kindly brief me about where to start from. Your help is deeply appreciated. Thank-you.

Another query I have is that this "Bug fixing" is in order to facilitate a comprehensive understanding of mediawiki or a selection criteria. Because I've been working on mediawiki for a month now on my local machine and don't feel the need for a "basic understanding". Thank-you.

My gut feeling tells me that this is not a good project idea candidate for this round. It is a very low priority in the Wikimedia tech context, and I don't expect new mentors to show up and volunteer. I will move it from the Featured column to Missing mentors, but in fact I believe Check in the next round would be more accurate.

GSoC and Outreachy candidates are strongly encouraged to look somewhere else for this round.

Qgil lowered the priority of this task from Medium to Lowest.Mar 2 2016, 10:15 AM

Another query I have is that this "Bug fixing" is in order to facilitate a comprehensive understanding of mediawiki or a selection criteria.

Hi @Khannaanant262129! Please see https://www.mediawiki.org/wiki/Outreach_programs/Life_of_a_successful_project#Answering_your_questions for information on best audiences and successful questions. Thank you!

My gut feeling tells me that this is not a good project idea candidate for this round. It is a very low priority in the Wikimedia tech context, and I don't expect new mentors to show up and volunteer. I will move it from the Featured column to Missing mentors, but in fact I believe Check in the next round would be more accurate.

GSoC and Outreachy candidates are strongly encouraged to look somewhere else for this round.

@Qgil although we have a good scope here, do you think Wikimedia tech priority on this has changed since then to warrant this for an Outreachy internship project for the current round?

@Bawolff would you be willing to mentor this project for this round of GSoC/Outreachy?

Sorry, I don't want to be a mentor this round (Or I guess last round which was one you asked).

Right now we don't have a clear use case of this, but we might have it for the next round. Let's keep this task open just in case.

@Qgil is there a use case for this project now? We're looking for its feasibility for current round of GSoC/Outreachy. If no, can we decline this?

srishakatux added a subscriber: srishakatux.

Adding Outreach-Programs-Projects and removing Possible-Tech-Projects as we are planning on killing that workboard soon!

Aklapper removed projects: WMF-Legal, Web-APIs-Hub.
Aklapper added a subscriber: Qgil.

Right now we don't have a clear use case of this, but we might have it for the next round. Let's keep this task open just in case.

Declining as I do not see any use case for this in 2019.

If you have a use case, feel free to elaborate and reopen.

Right now we don't have a clear use case of this, but we might have it for the next round. Let's keep this task open just in case.

Declining as I do not see any use case for this in 2019.

If you have a use case, feel free to elaborate and reopen.

Pinging @EvanProdromou. It seems to me like we'll need this for the planned documentation portal.

There's also T155029: MediaWiki.org: Generate infoboxes from extension.json in git and similar issues for hooks etc. Not quite transclusion but being able to fetch and process git content in Lua would be nicer than the current bot-based workaround.