Page MenuHomePhabricator

Implement bulk fixes on ptwikibooks
Closed, ResolvedPublic

Description

The previous tasks T397420 and T397422 entailed building a "proposal" file of proposed bulk changes to the wiki:

  • Repair historic redirects to Flow boards for threads where there is an appropriate flow board
  • Re-enable LQT on pages which have associated threads so that they are no longer "lost"

Build a tool to execute these changes.

  • Ensure user credentials are in the environment, not the codebase
  • Be idempotent: do not modify a page whose contents are already as expected
  • Be safe: do not modify a page whose latest revision is not the expected one
  • Log pages whose latest revision was not as expected
  • Ensure flood right is active to avoid spamming Special:RecentChanges
  • Ask @Trizek-WMF to inform community
  • Regain flood and global-flow-create permissions
  • Ensure until community has been informed (see T388099: Inform ptwikibooks of LQT and Flow removal plan and timing)
  • Await Tuesday, September 9th
  • Check parent ticket
  • Execute:
    • Fix redirects
    • Enable LQT on boards where it was auto-disabled

Script: https://gitlab.wikimedia.org/zoe/lqt-porting-tools/-/blob/main/implement_fixes.py?ref_type=heads

proposed-redirects.json.bz2: F62401174
proposed-lqt-repairs.json.bz2: F65817560

$ sha256sum output/proposed-*
e3ec71e993c01828b458c08a67ab3ce2622a89da1ede85f95520a0b9df630668  output/proposed-lqt-repairs.json.bz2
2d558606eed52aba9f52c5f0733bba3470125dc2e3619ce85cbcf2234df630c7  output/proposed-redirects.json.bz2

Related Objects

StatusSubtypeAssignedTask
ResolvedNone
OpenNone
OpenNone
ResolvedTrizek-WMF
DuplicateNone
OpenNone
ResolvedTrizek-WMF
DuplicateNone
ResolvedSgs
ResolvedSgs
ResolvedTrizek-WMF
OpenNone
ResolvedMimurawil
ResolvedTchanders
In ProgressNone
OpenNone
DeclinedNone
ResolvedEsanders
ResolvedEsanders
ResolvedEsanders
OpenNone
ResolvedEsanders
ResolvedNone
ResolvedDLynch
OpenDLynch
ResolvedUrbanecm_WMF
ResolvedDLynch
ResolvedDLynch
OpenEsanders
ResolvedRyasmeen
ResolvedUrbanecm_WMF
ResolvedDLynch
ResolvedTrizek-WMF
Resolvedzoe
ResolvedRyasmeen
ResolvedBUG REPORTEtonkovidova
ResolvedTrizek-WMF
ResolvedNone
ResolvedPRODUCTION ERRORhubaishan
ResolvedTrizek-WMF
ResolvedDLynch
ResolvedTrizek-WMF
Resolvedppelberg
ResolvedQuiddity
Resolvedppelberg
Resolvedzoe
Resolvedzoe
ResolvedRyasmeen
Resolvedzoe
OpenNone
OpenNone
OpenNone
OpenNone
OpenNone
Resolvedmatmarex
OpenNone

Event Timeline

zoe changed the task status from Open to In Progress.Jun 20 2025, 3:16 PM

This shouldn't be visible in RecentChanges, but it could be visible at some LQT pages, where some (long forgotten) topics may re-surface.

Just a small update to say I've been out ill for a couple of weeks, but I'm back on this now

I've got fragments of a script – I can edit pages and ensure they're the expected revision. More work than I expected: I needed to create an oauth2 token for my app (locally) and I'll need to do it again for ptwikibooks.

Next up: Grabbing the current revision of all the pages to check whether we should attempt to edit them. That should make the script idempotent, at which point we can run in small batches to check our work, if necessary.

Nearly there! I've got code that does everything in a notebook, now it's a matter of popping it in a script and making sure it can take command line arguments

I think I'm there. Two more bits of sanity checking:

  1. Are edits fine if I use the non-normalized form of the title? Probably, but I'll go play with Special:APISandbox to make sure.
  2. Running the script against the first two entries and making sure the result looks sensible

Here's two sets of three runs. The first one, with -n, is a dry run that says what the script plans to do. The second actually does it, and the third decides the edit is complete and does not attempt to repeat it:

(venv) ptwikibooks-lqt-tools main ●$ python3 implement_fixes.py -n https://pt.wikibooks.org/w/api.php output/proposed-redirects.json.bz2
+++ Would edit: Tópico:Wikilivros:Diálogos comunitários/Proposta: Ativar a Extensão:ReaderFeedback
+++ Would edit: Tópico:Wikilivros:Diálogos comunitários/Proposta: Ativar a Extensão:ReaderFeedback/resposta
(venv) ptwikibooks-lqt-tools main ●$ python3 implement_fixes.py https://pt.wikibooks.org/w/api.php output/proposed-redirects.json.bz2
+++ Edited: Tópico:Wikilivros:Diálogos comunitários/Proposta: Ativar a Extensão:ReaderFeedback
+++ Edited: Tópico:Wikilivros:Diálogos comunitários/Proposta: Ativar a Extensão:ReaderFeedback/resposta
(venv) ptwikibooks-lqt-tools main ●$ python3 implement_fixes.py https://pt.wikibooks.org/w/api.php output/proposed-redirects.json.bz2
--- Complete: Tópico:Wikilivros:Diálogos comunitários/Proposta: Ativar a Extensão:ReaderFeedback
--- Complete: Tópico:Wikilivros:Diálogos comunitários/Proposta: Ativar a Extensão:ReaderFeedback/resposta
(venv) ptwikibooks-lqt-tools main ●$ python3 implement_fixes.py -n https://pt.wikibooks.org/w/api.php output/proposed-lqt-repairs.json.bz2
+++ Would edit: Utilizador Discussão:Hibisco-da-Ilha~ptwikibooks/Arquivo LQT 1
+++ Would edit: Predefinição Discussão:Projetos irmãos/LQT Archive 1
(venv) ptwikibooks-lqt-tools main ●$ python3 implement_fixes.py https://pt.wikibooks.org/w/api.php output/proposed-lqt-repairs.json.bz2
+++ Edited: Utilizador Discussão:Hibisco-da-Ilha~ptwikibooks/Arquivo LQT 1
+++ Edited: Predefinição Discussão:Projetos irmãos/LQT Archive 1
(venv) ptwikibooks-lqt-tools main ●$ python3 implement_fixes.py https://pt.wikibooks.org/w/api.php output/proposed-lqt-repairs.json.bz2
--- Complete: Utilizador Discussão:Hibisco-da-Ilha~ptwikibooks/Arquivo LQT 1
--- Complete: Predefinição Discussão:Projetos irmãos/LQT Archive 1

Here's those diffs:
https://pt.wikibooks.org/w/index.php?title=T%C3%B3pico:Wikilivros:Di%C3%A1logos_comunit%C3%A1rios/Proposta:_Ativar_a_Extens%C3%A3o:ReaderFeedback&diff=prev&oldid=500584
https://pt.wikibooks.org/w/index.php?title=T%C3%B3pico:Wikilivros:Di%C3%A1logos_comunit%C3%A1rios/Proposta:_Ativar_a_Extens%C3%A3o:ReaderFeedback/resposta&diff=prev&oldid=500585
https://pt.wikibooks.org/w/index.php?title=Utilizador_Discuss%C3%A3o:Hibisco-da-Ilha~ptwikibooks/Arquivo_LQT_1&diff=prev&oldid=500586
https://pt.wikibooks.org/w/index.php?title=Predefini%C3%A7%C3%A3o_Discuss%C3%A3o:Projetos_irm%C3%A3os/LQT_Archive_1&diff=prev&oldid=500587

And those pages:
https://pt.wikibooks.org/wiki/T%C3%B3pico:Wikilivros:Di%C3%A1logos_comunit%C3%A1rios/Proposta:_Ativar_a_Extens%C3%A3o:ReaderFeedback
https://pt.wikibooks.org/wiki/Predefini%C3%A7%C3%A3o_Discuss%C3%A3o:Projetos_irm%C3%A3os/LQT_Archive_1
https://pt.wikibooks.org/wiki/Utilizador_Discuss%C3%A3o:Hibisco-da-Ilha~ptwikibooks/Arquivo_LQT_1
https://pt.wikibooks.org/wiki/Predefini%C3%A7%C3%A3o_Discuss%C3%A3o:Projetos_irm%C3%A3os/LQT_Archive_1

I think this all looks correct to me – what do you think, @Pppery?

I'm glad I ran this check – my changes appear in Recent Changes so it looks like I'm perhaps not using my flood permissions by default. So that's the next thing to work out.

The edits themselves seem correct, but it looks like you don't have flood rights on the wiki in the first place which is why that's not working out.

Yup. I've followed up about that, but I suppose this is on pause until I get those. I think everything else is in alignment, at least. I'm glad I did a little trial run. I've also updated my script to make sure to fill out the summary field so that users can see what I'm up to and why.

zoe updated the task description. (Show Details)
zoe updated the task description. (Show Details)
Tchanders updated Other Assignee, added: Tchanders.

Following handover from @zoe, @Esanders and I looked into next steps for running the script to convert LQT pages to Flow on ptwiki.

A local test run of convertAllLqtPages found:

  • With $wgLqtTalkPages true, it converts all pages in talk namespaces to Flow boards. It makes the assumption that these are LQT pages, but this may not be true, e.g. for talk pages which existed since before LQT was enabled. For a non-LQT talk page, running the conversion script will move the original page to an archive page, and also create a Flow board which looks a bit strange (the contents are moved into the 'about' section). No content is actually lost though.
  • If there was a failed conversion in the past, <the talk page>/LQT_archive_1 already exists, so the conversion fails. This raises the stakes somewhat, since we don't want to do a failed run.

It might not be the best idea to start with ptwikibooks, in case something goes wrong in the conversion script. This is because ptwikibooks is the only wiki with LQT enabled by default, meaning there are many LQT pages. In contrast, the other wikis (except enwikinews) only have a few LQT pages. Searching for insource:/\{\{\#useliquidthreads\:1\}\}/ in all namespaces found:

  • huwiki c.10 (some are tests example)
  • enwiktionary: <= 45
  • svwiki <= 102
  • enwikinews: thousands+

However, of the 5 wikis, only ptwikibooks has Flow enabled. To test on a smaller wiki, we considered enabaling Flow in read-only mode, and then running the script, but out of the box, the script fails if Flow is in read-only mode.

If there was a failed conversion in the past, <the talk page>/LQT_archive_1 already exists, so the conversion fails. This raises the stakes somewhat, since we don't want to do a failed run.

That shouldn't happen. The code was designed to be idempotent and be able to finish half-runs if you use the same sourcestore file. Maybe that code is broken, but I thought I had tested it a week ago.

If there was a failed conversion in the past, <the talk page>/LQT_archive_1 already exists, so the conversion fails. This raises the stakes somewhat, since we don't want to do a failed run.

That shouldn't happen. The code was designed to be idempotent and be able to finish half-runs if you use the same sourcestore file. Maybe that code is broken, but I thought I had tested it a week ago.

We hit this problem locally while running the script with Flow in read-only mode (though using read-only mode shouldn't fail any more since https://gerrit.wikimedia.org/r/c/mediawiki/extensions/Flow/+/1188341)

Change #1203452 had a related patch set uploaded (by Esanders; author: Esanders):

[mediawiki/extensions/Flow@master] Create maintenance script to apply manual fixes

https://gerrit.wikimedia.org/r/1203452

Change #1203457 had a related patch set uploaded (by Esanders; author: Esanders):

[mediawiki/extensions/Flow@wmf/1.46.0-wmf.1] Create maintenance script to apply manual fixes

https://gerrit.wikimedia.org/r/1203457

Change #1203457 merged by jenkins-bot:

[mediawiki/extensions/Flow@wmf/1.46.0-wmf.1] Create maintenance script to apply manual fixes

https://gerrit.wikimedia.org/r/1203457

Mentioned in SAL (#wikimedia-operations) [2025-11-10T14:43:06Z] <esanders@deploy2002> Started scap sync-world: Backport for [[gerrit:1203457|Create maintenance script to apply manual fixes (T397426)]]

Mentioned in SAL (#wikimedia-operations) [2025-11-10T14:45:05Z] <esanders@deploy2002> esanders: Backport for [[gerrit:1203457|Create maintenance script to apply manual fixes (T397426)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.

Change #1203452 merged by jenkins-bot:

[mediawiki/extensions/Flow@master] Create maintenance script to apply manual fixes

https://gerrit.wikimedia.org/r/1203452

Mentioned in SAL (#wikimedia-operations) [2025-11-10T14:50:27Z] <esanders@deploy2002> Finished scap sync-world: Backport for [[gerrit:1203457|Create maintenance script to apply manual fixes (T397426)]] (duration: 07m 21s)

Using the above script, which is a maintenance script version of the python script Zoe wrote, I have run the proposed-redirects.json edits, and have started proposed-lqt-repairs.json.

Spot-checked a few of the fixes, seems to look reasonably good.

Esanders updated the task description. (Show Details)

All fixes applied.

Every page in redirects was edited, except the first 4 which were already marked as "Complete" as @zoe had run the python script against them:

--- Complete: Tópico:Wikilivros:Diálogos comunitários/Proposta: Ativar a Extensão:ReaderFeedback
--- Complete: Tópico:Wikilivros:Diálogos comunitários/Proposta: Ativar a Extensão:ReaderFeedback/resposta
--- Complete: Tópico:Wikilivros:Diálogos comunitários/Proposta: Ativar a Extensão:ReaderFeedback/resposta (2)
--- Complete: Tópico:Wikilivros:Diálogos comunitários/Proposta: Ativar a Extensão:ReaderFeedback/resposta (3)

Every page in repairs was edited except for :

!!! Conflict: Utilizador Discussão:.sEdivad/Arquivo LQT 1, expected revision: 352253, current revision: 500065

which @Pppery had already fixed by hand.

Full logs here: P85269