Switch to Remex for de.wp, it.wp, and 170 smaller wikis (at least)
Closed, ResolvedPublic

Description

It has been about 3 months since the announcement about Tidy being replaced in about a year's time. We have provided wikis the Linter and parser-migration tools to help them fix pages that might break. Various wikis have been going about it. This is a task to track followup action needed from CLs. Please feel free to re-org this as a bulleted todo list or create a bunch of sub tasks.

Identifying new wikis to switch from Tidy

  • dewiki looks ready to make the switch away from Tidy. https://de.wikipedia.org/wiki/Spezial:LintErrors has zero errors in the main article namespace. There are issues in user namespaces, but those are not as critical. It would be good to make the switch sooner than later so we can identify any other lingering issues that needs addressing for large wikis that aren't already covered by the linter tool
  • We have over 300 wikis (mostly small ones) that have no linter errors in these high priority categories. How should be go about switching them over to Remex? Process assistance here is very welcome.

Identifying editors to make linter-identified fixes on other wikis

  • cebwiki needs to run a bot similar to what svwiki is doing since their deletable-table-tag linter error is a clone of what svwiki has. That needs followup.
  • There are probably others that I will update here as I find them.
ssastry created this task.Oct 23 2017, 8:23 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptOct 23 2017, 8:23 PM
Elitre claimed this task.Oct 25 2017, 11:23 AM
Elitre added a subscriber: Elitre.

Thanks, Subbu. I'm claiming this as followup from existing work on this project.

I'll think about drafting something ASAP.

Elitre added a comment.Nov 8 2017, 3:03 PM

We're already using Tech news when we think it's the case, and we don't we don't find agreement :)

Draft message shared with Subbu. Next week I'll look at the second part of the task

Sv.wp looking close, I hear?

Sv.wp looking close, I hear?

https://sv.wikipedia.org/wiki/Special:LintErrors and there are a number of false positives in https://sv.wikipedia.org/w/index.php?title=Special:LintErrors/pwrap-bug-workaround which will go away as the linter script i started y'day processes articles through all wikis. But, there are still other fixes needed .. which could happen before / after switch.

cebwiki will also be ready once the bot is done through its work since the biggest problem there is the one that is being fixed by the bot ( https://ceb.wikipedia.org/wiki/Espesyal:LintErrors )

cebwiki will also be ready once the bot is done through its work since the biggest problem there is the one that is being fixed by the bot ( https://ceb.wikipedia.org/wiki/Espesyal:LintErrors )

At the rate the bot is processing pages (100K per week), it will be another 14 weeks.

At this time, among large wikis, I am concerned about enwiki -- there has not been much work / progress on the high priority categories.

cebwiki will also be ready once the bot is done through its work since the biggest problem there is the one that is being fixed by the bot ( https://ceb.wikipedia.org/wiki/Espesyal:LintErrors )

At the rate the bot is processing pages (100K per week), it will be another 14 weeks.

Is that OK?

At this time, among large wikis, I am concerned about enwiki -- there has not been much work / progress on the high priority categories.

I know. All the attempts to contact active editors in the field haven't really gone far. Maybe it's time for Village Pump Technical. I'll ask my colleagues for ideas.

Elitre added a comment.EditedNov 16 2017, 11:01 AM

Sv.wp looking close, I hear?

https://sv.wikipedia.org/wiki/Special:LintErrors and there are a number of false positives in https://sv.wikipedia.org/w/index.php?title=Special:LintErrors/pwrap-bug-workaround which will go away as the linter script i started y'day processes articles through all wikis. But, there are still other fixes needed .. which could happen before / after switch.

We could post an encouragement note there.
@ssastry, please LMK when the script's done with them!

cebwiki will also be ready once the bot is done through its work since the biggest problem there is the one that is being fixed by the bot ( https://ceb.wikipedia.org/wiki/Espesyal:LintErrors )

Elitre added a comment.EditedNov 16 2017, 7:18 PM

De.wp message at https://de.wikipedia.org/wiki/Hilfe_Diskussion:Wikisyntax/Validierung#Thank_you.2C_and_next_steps_re:_Tidy_fixes. (FYI posted here as well, and here.)

Will have the opportunity to meet the Italian community this weekend, and will discuss the change with some of them, then will post the proposal there on Monday.

Will draft a note for sv.wp tomorrow

The drafting is late. Sorry about that. OTOH maybe we don't have to lose our hopes on en.wp. https://en.wikipedia.org/wiki/Wikipedia:Linter just got updated.

For sv.wp I just went ahead with https://sv.wikipedia.org/wiki/Wikipedia:Bybrunnen#Good_job_with_the_Tidy_fixes.21 . After all it's the Thanksgiving weekend in the US and this didn't need to wait until everyone else is back.

For it.wp the date of Dec. 5 is confirmed. If I hear otherwise from the Germans, will let you know.

For the 300+ small wikis that may be already good to go: just generate a list, please, I drafted the message for your review.

Nirmos added a subscriber: Nirmos.Thu, Nov 23, 4:31 PM

there are a number of false positives [...] which will go away as the linter script i started y'day processes articles through all wikis

Could you tell me when/if this script is done?

there are a number of false positives [...] which will go away as the linter script i started y'day processes articles through all wikis

Could you tell me when/if this script is done?

I think that one is. Not sure about the status of the one that was launched after that, but we'll certainly know more next week.

there are a number of false positives [...] which will go away as the linter script i started y'day processes articles through all wikis

Could you tell me when/if this script is done?

I think that one is. Not sure about the status of the one that was launched after that, but we'll certainly know more next week.

It got done on Wednesday.

For the 300+ small wikis that may be already good to go: just generate a list, please, I drafted the message for your review.

After I ran the linter script to crawl all pages on all wikis, that number went down to 171. The other 130+ got a small handful linter issues introduced.

But, you can get the list @ https://www.mediawiki.org/wiki/Parsing/Replacing_Tidy/Linter/Stats/2017-11-20#Wikis_with_zero_high-priority_errors

Elitre added a comment.EditedFri, Nov 24, 12:20 PM

For the 300+ small wikis that may be already good to go: just generate a list, please, I drafted the message for your review.

After I ran the linter script to crawl all pages on all wikis, that number went down to 171. The other 130+ got a small handful linter issues introduced.

But, you can get the list @ https://www.mediawiki.org/wiki/Parsing/Replacing_Tidy/Linter/Stats/2017-11-20#Wikis_with_zero_high-priority_errors

OK... I'm going to try and come up with a MassMessage list. For 0 errors, we can totally do them on Dec 5. This is gonna be... interesting.

So the list of the wikis now lives at https://meta.wikimedia.org/wiki/User:Elitre_(WMF)/Remex_switch_-_wikis_with_zero_errors . I have removed .wikimedia.org ones, because they're either chapter ones, and hence not reachable by the MassMessage bot I think, or read-only like https://strategy.wikimedia.org/wiki/Main_Page, or old sites like the Wikimania ones. Can contact those manually if you want. Furthermore, I was only able to actually find an existing village pump for some of the 150 left - I used the "blanket" Project:Community Portal page for the rest of those, and will apologize in the message for doing so. If we're lucky, the message will be moved to a more appropriate venue. I am open to suggestions for handling delivery better if someone has them.

Oh, we can probably find a way to determine whether the page exists, and if it doesn't, we can just remove it from the list. We'll use Tech News to announce this anyway.

Elitre added a comment.EditedFri, Nov 24, 4:30 PM

Draft message for Tech News:

Almost 170 wikis with zero high-priority errors in Linter categories will be switched to the Remex parsing library on XX December 2017. A few larger wikis, such as the German and the Italian Wikipedia, have also agreed to join this initial transition from Tidy. Other wikis will be progressively encouraged to also switch soon, based on their progress with the necessary page fixes; Tidy will be finally removed in mid 2018. [https://www.mediawiki.org/wiki/Parsing/Replacing_Tidy/Linter/Stats/2017-11-20#Wikis_with_zero_high-priority_errors] [https://www.mediawiki.org/wiki/Parsing/Replacing_Tidy/FAQ#What_will_editors_need_to_do.3F] [https://www.mediawiki.org/wiki/RemexHtml] [task number here]

Could the euphemism "sunset" get replaced by something clearer, like "removed"?

Elitre added a comment.EditedMon, Nov 27, 3:10 PM

Oh, we can probably find a way to determine whether the page exists, and if it doesn't, we can just remove it from the list. We'll use Tech News to announce this anyway.

Actually, we'll just use the 40 venues we found, plus Tech News. We certainly need better ways to communicate with smaller communities, but this task isn't about finding them now. I'll shoot a message to ambassadors once TN is frozen, maybe they can get a few key languages translated.
The TN message now lives at https://meta.wikimedia.org/wiki/Tech/News/2017/49 .

Elitre added a comment.EditedTue, Nov 28, 10:05 AM

Per Subbu, fi.wp is also ready-ish now! Messaging them with Dec 5 or 12 as a deadline. Let's see how it goes. https://fi.wikipedia.org/wiki/Wikipedia:Kahvihuone_(tekniikka)#Thank_you.2C_and_next_steps_re:_Tidy_fixes

Elitre added a comment.EditedWed, Nov 29, 1:05 PM

So the list of the wikis now lives at https://meta.wikimedia.org/wiki/User:Elitre_(WMF)/Remex_switch_-_wikis_with_zero_errors . I have removed .wikimedia.org ones, because they're either chapter ones, and hence not reachable by the MassMessage bot I think, or read-only like https://strategy.wikimedia.org/wiki/Main_Page, or old sites like the Wikimania ones. Can contact those manually if you want. Furthermore, I was only able to actually find an existing village pump for some of the 150 left - I used the "blanket" Project:Community Portal page for the rest of those, and will apologize in the message for doing so. If we're lucky, the message will be moved to a more appropriate venue. I am open to suggestions for handling delivery better if someone has them.

I did a first pass to see if there was any feedback left - only found one kind user had explained the change locally. I did find more closed wikis or inaccessible pages, will do something about them when I do the final check on Monday.

Elitre added a comment.Fri, Dec 1, 6:50 PM

(For the record I have changed my mind about pinging the ambassadors, I think that's overkill at this point.)

Elitre added a comment.Mon, Dec 4, 6:55 PM

Ok, so:

  1. Announcement going out in TN;
  2. de.wp and it.wp alerted about time of deployment tomorrow;
  3. their IRC channels pinged about it;
  4. list of mass-messaged wikis double-checked;

It's a Go on the community side for tomorrow's change.
After watching what happens tomorrow on the big wikis, we can decide whether we want to nudge another couple to also switch next week or not.

Elitre raised the priority of this task from Normal to High.
Elitre renamed this task from Followups needed with wikis for Tidy replacement to Switch to Remex for de.wp, it.wp, and 170 smaller wikis (at least).Wed, Dec 6, 12:05 PM

Subbu told me something about sh.wiki, to be investigated next week.

Elitre closed this task as Resolved.Tue, Dec 12, 9:55 AM

I believe this wave of switches is done, and Subbu has all he needs to continue working with the affected communities.

Qgil awarded a token.Tue, Dec 12, 1:32 PM