Possible-Tech-Projects ?

I wish to know what type of errors need to be detected. Since the extension-splitter-helper file has solved the problem of taking into account the added text, I wish to know what further work needs to be done.

A reference point could be to see if an edit caused any of these errors https://tools.wmflabs.org/checkwiki/cgi-bin/checkwiki.cgi?project=enwiki&view=project which wold be cause for some alarm, and we could raise score points in Huggle for a specific edit.

@Shrutika719 pretty much every syntax error in mediawiki page source, if someone edited the page, removing its random part, or overwriting part of it, placing nonsense there and so on, this all should be detected somehow. It very often happens that vandals break syntax of a page, but these edits are not always easy to filter out from regular "proper" edits.

This is a message posted to all tasks under "Backlog" at Possible-Tech-Projects. Outreachy-Round-11 is around the corner. If you want to propose this task as a featured project idea, we need a clear plan with community support, and two mentors willing to support it.

This is a message sent to all Possible-Tech-Projects. The new round of Wikimedia Individual Engagement Grants is open until 29 Sep. For the first time, technical projects are within scope, thanks to the feedback received at Wikimania 2015, before, and after (T105414). If someone is interested in obtaining funds to push this task, this might be a good way.

I volunteer to work on this project for the Outreach Round 2015, I wish to know do the mentors see this project as a potential project for the Outreach Program?

@Petrb are you interested in mentoring this project? Please update the description, I have added required fields.

Also, what do you mean with "This is WMF specific task"?

Hi,

yes, I can mentor this task, but I need to add 2 important things before anyone is going to take it:

• This task is NOT trivial
• This task requires knowledge of C++, it's not a PHP-only project - although php could be used as well

Given that this task is not trivial and that it covers only huggle, while this certain functionality "mediawiki syntax-checking" is something that many other projects could benefit from, I would like to discuss the design of potential solution with other developers if possible.

I believe it should be possible to create some sort of framework, or a library that would be able to check mediawiki syntax so that other tools could access this functionality as well. At some point this functionality could be implemented into mediawiki core / API instead and huggle would just retrieve the information about syntax validity through API, it could even be some sort of revision attribute, whether syntax is broken or not.

My point is, I don't want to waste someone's time just to create a complex (yet useful) feature like this, that could only be used by huggle and nothing else. If such effort was taken by someone to do this, I would prefer if the solution was sort of portable, or easily reusable by other tools.

Having said that, the scope of this task differs based on design that would be used, there are following options:

Implement this in huggle only

• Deep knowledge of C++ required, non trivial task
• Can't be easily reused by other tools

Create this feature mediawiki side, implement some interface in huggle that would access it

Mediawiki side

• Deep knowledge of PHP required, non trivial task
• Can be reused by other tools

Huggle side

• Elemental knowledge of C++ required, simple task

One more thing: I didn't do any research so far, maybe there already is similar library / tool that is able to do this. At least mediawiki itself is already able to throw syntax error (but only for fatal syntax issues, not minor ones). Maybe some of that code could be (re)used.

Another design option:

Create separate instance of mw syntax validator, similar to parsoid

Could be written in any language and would run as a service provider (running on a server allowing clients to submit mw code for validation). The basic idea is that the client would submit wiki source text to service and validator would return results of syntax check. That way it could be used by other tools, could be written in any language and wouldn't require modifications of MW core. Only issue I see here is performance, other 2 designs would have direct access to information whether syntax is broken or not, here you would need to issue extra query to 3rd server just to check the syntax.

I don't know how to handle errors in syntax of code that would be parsed by mediawiki extensions, as this service probably wouldn't know much about target mediawiki installation for which it would check the source of text. Therefore this would need to be "WMF specific" eg. designed for WMF installations of MediaWiki, which have uniform extension setup.

BTW if we really had this feature it could be implemented into MediaWiki's edit form, so that it would pop out a warning if user wanted to save a page with syntax errors, preventing users from saving broken pages by mistake.

In T94370#1692410, @Petrb wrote:

• This task is NOT trivial

Equivalent to 2-3 weeks of an experienced developer?

In T94370#1692468, @Qgil wrote:

In T94370#1692410, @Petrb wrote:

• This task is NOT trivial

Equivalent to 2-3 weeks of an experienced developer?

Yes, for it to cover at least the basic syntax errors, which for beginning should be sufficient, 2-3 weeks is enough.

i am interested in this project and i want to contribute in this. i have a good knowledge of c++ also

@Petrb, I cannot help you finding the right strategy for this project idea. I can remind you that in order to feature this project in Outreachy we need a description up to date and two mentors. :)

In T94370#1692439, @Petrb wrote:

Create separate instance of mw syntax validator, similar to parsoid

T48705: Parsoid-based wikitext "linting" tool for "buggy" / "deprecated" wikitext usage; keywords: broken wikitext information

OK

@Qgil I want to make 100% sure there isn't some redundancy (there might be already some tool that does same or similar thing) and I also want to make this feature as much re-usable as possible, given the resources invested in that. So that is why I would rather do some research before making some final decision of how it should work.

@Ricordisamoa I guess that thing is a wish, rather than working feature?

Does any MediaWiki dev knows if there is any code in core that could be used in order to validate wikitext syntax?

I will add some devs so that they can join this as well

At some point this feature might be used / provided by ORES as well

@Iamneha & @Shrutika719 I do welcome your interest in this task, but please wait a bit until the discussion on design of this feature settle down a bit :) I am sorry for the original description being so confusing, it's just that I rather quickly drafted it and didn't really expect any big interest in this task. Now that I see people want to work on it, I would like to polish the description as much as possible so that the final product can be as much useful as it possibly can be, but yet doable within the Outreachy difficulty level :)

I will try to make this task ready to take ASAP. Thanks for your patience guys.

I found this page https://en.wikipedia.org/wiki/Wikipedia:WikiProject_Check_Wikipedia which lists some of the existing and working tools that are able to check some syntax issues. I believe they are all open source and some parts and algorithms could be used from these.

In T94370#1694948, @Petrb wrote:

I found this page https://en.wikipedia.org/wiki/Wikipedia:WikiProject_Check_Wikipedia which lists some of the existing and working tools that are able to check some syntax issues. I believe they are all open source and some parts and algorithms could be used from these.

The tool database-thingy is at https://tools.wmflabs.org/checkwiki/ which is maintained by @Bgwhite who I cc'ed earlier.

In T94370#1694965, @Josve05a wrote:

In T94370#1694948, @Petrb wrote:

I found this page https://en.wikipedia.org/wiki/Wikipedia:WikiProject_Check_Wikipedia which lists some of the existing and working tools that are able to check some syntax issues. I believe they are all open source and some parts and algorithms could be used from these.

The tool database-thingy is at https://tools.wmflabs.org/checkwiki/ which is maintained by @Bgwhite who I cc'ed earlier.

The code for Check Wikipedia is GPL and written in Perl. Check Wikipedia can work on a dump file, list of articles or an article on-the-fly. Go ahead an steal any code. Check Wikipedia is used within AWB and WPCleaner. Both these tools also have ways to check and fix issues. AWB is GPL and written in .Net. WPCleaner is Apache and written in Java.

@Petrb thank you so much for your support. Waiting till project fully prepared.

OK, I have updated the task information, I think we can drop the idea of creating some widely available service that would allow for these checks, because that is probably overkill and wouldn't even fit within Outreachy scope, it's just a huge task.

So, I guess creating an extension for huggle that would use similar mechanisms as in other mentioned tools would be more than enough. In the end, if the source code was written in a sane way, it could be probably reused as well.

The task needs a co-mentor though (@Addshore maybe :)?), not sure what microtasks are needed. I suppose it needs to be approved by someone as well later (@Qgil?). Until that is done I am afraid it's can't be taken yet by any candidate, but isn't phabricator supposed to be used just for proposals? I thought candidates were assigned in melange.

In T94370#1697123, @Petrb wrote:

I think we can drop the idea of creating some widely available service that would allow for these checks

What a wonderful waste of interns' time.

@Ricordisamoa: can you please elaborate on this?

The task in not yet confirmed for anyone to take, it's still being prepared and I wanted to discuss this with other developers. Yet nobody voiced any opinions whatsoever (only silently removed themselves from this ticket to avoid any need to talk). So maybe if instead of complaining how bad idea it is you gave us some real input on how it should be actually done, that would be really helpful.

If it's preferred to be done as external service so that others can utilize I can change the description of task so that it requires it. But to be honest I didn't receive any input from anyone whether that actually would be useful or not. Would AWB, ORES or anything else benefit from having it?

I am waiting for your responses guys. Thank you

In T94370#1699034, @Ricordisamoa wrote:

What a wonderful waste of interns' time.

No intern has been assigned to this task and I'm not sure how this comment provided some help to others...

In T94370#1697123, @Petrb wrote:

not sure what microtasks are needed.

Any smaller task that helps getting prepared:

Applicants must provide a small contribution prior to applying. (Known as "microtask".)
Assessing newcomers based on a proposal and a code repository out of context is risky.
The first contribution must be related to the type of project proposed.

(quoted from Lessons learned)

I suppose it needs to be approved by someone as well later (@Qgil?).

See https://www.mediawiki.org/wiki/Outreachy/Round_11 and its links for the application and selection process.

If a task gets 'dumbed down' for the sake of Outreachy applicants, it becomes more of a void exercise for one of them than a real benefit for the movement.

Ok, I created https://phabricator.wikimedia.org/T114631 if we decide to do that one, this task should be probably changed just to a wrapper extension that would utilize that framework, which is pretty trivial task which could be probably just some Code-In task or similar.

Discussion is needed

In T94370#1701406, @Petrb wrote:

But yes, I recently figured out the scope of Outreachy and the fact that there is a big monetary award, I thought this is more or less something like Code-In. Given the significant monetary award I think this task should be much broader than what it is now.

So let's scratch this task for now, I will create a new one

Possible-Tech-Projects tag removed for now. Thank you @Petrb for your perseverance. :)

In T94370#1694617, @Ricordisamoa wrote:

In T94370#1692439, @Petrb wrote:

Create separate instance of mw syntax validator, similar to parsoid

T48705: Parsoid-based wikitext "linting" tool for "buggy" / "deprecated" wikitext usage; keywords: broken wikitext information
.....
!In T94370#1694819@Petrb wrote:

@Ricordisamoa I guess that thing is a wish, rather than working feature?

No, it is not a wish. Hardik Juneja did work on the project as part of GSoC 2014 based on plans here. Parsoid has some initial functionality to detect a bunch of broken wikitext scenarios (which you can test via the parse.js script on the commandline).

[subbu@earth tests] echo "<table>foo<td>bar</td></table> and <div>boo" | node parse --lint
...
[ { type: 'fostered',
    wiki: 'enwiki',
    page: 'Main Page',
    revision: undefined,
    wikiurl: 'https://en.wikipedia.org/',
    location: '[enwiki/Main Page]',
    dsr: [ 0, 30, 7, 8 ],
    src: 'foo' },
  { type: 'missing-end-tag',
    wiki: 'enwiki',
    page: 'Main Page',
    revision: undefined,
    wikiurl: 'https://en.wikipedia.org/',
    location: '[enwiki/Main Page]',
    dsr: [ 35, 43, 5, 0 ],
    src: '<div>boo' } ]
....

We haven't enabled this in production because we haven't done the last mile piece of how to integrate this with editing tools, bots, project wikicheck, etc. We had initial conversations as part of the GSoc internship, but we haven't had the time to follow through on it.

I'll move further conversation about this on T114631 but it would be great to take this project further.