Page MenuHomePhabricator

purgeList.php maintenance script not doing anything
Closed, ResolvedPublicBUG REPORT

Description

Before 1.35 we've regularily used php purgeList.php --all --wiki=english to purge cache after changes in our LUA-Modules for Templates.

In 1.35 about the same is supposed to work via php purgeList.php --all-namespace --wiki=english but it actually doesn't do anything anymore. (--all has seemingly been removed)

The script stays open and running forever, without ever ending, or - when using --verbose to debug - actually never doing anything, as there is no list of pages displayed while using the command with verbose as i was used to before 1.35.
It simply does nothing.

Event Timeline

Mh, not sure. The last time i used this script and it was working it still had the --all parameter, which seemingly was not removed in that change.
When i tried to purge the pages i still tried with the --all parameter and it told me that one does no longer exist. I'm not sure if it also still had the --purge parameter.
The help page here https://www.mediawiki.org/wiki/Manual:PurgeList.php still doesn't make mention of the removal of --all and the introduction of --all--namespace, so i'm not sure in which version this was changed exactly.
I'm quite sure i used purgeList in 1.34 with the old parameter but i wouldn't testify for that, it might have been an older version - but in any case the current script is not working for me.

Tried with --db-touch (i would never have guessed --purge is equal to --db-touch), same result. Just not actually doing anything.

I guess for clarification, what do you mean by "doing anything"? What do you expect it to do? What cache are you expecting to be purged?

When doing 'php purgeList.php --all --purge --wiki=english --verbose' prior to 1.35 it showed a list of all pages on the wiki that it purged (like with ?action=purge). That was necessary when we changed Templates that are used on every page or changed content of LUA Modules we use to fill them.
When doing this today with 'php purgeList.php --all-namespace --db-touch --wiki=english --verbose' nothing happens. The pages still display outdated data or broken modules until ?action=purge is done. It also shows no list of pages it works through in verbose mode and the script never exits in the console.

Maybe the usage of this script has changed? In that case, how can i do ?action=purge on all wiki pages at once easily, as we need that regularily.

This comment was removed by D3nnis3n.

Like, it looks like this, for ages now:

mediawiki.png (47×1 px, 4 KB)

Using the script last version with the old parameters would have a list running down showing every single page until it finishes and exits the script. Then when i go to the wiki, all pages were purged and displayed recent data.

Currently it does nothing, never exits and the script and the wiki continues to display old data until i append ?action=purge to it. Unfortunately, cannot do that for several thousand pages that need purging.

* $wgUseSquid, $wgSquidServers, $wgSquidServersNoPurge, and $wgSquidMaxage —
  These, deprecated in 1.34, have been removed. Use $wgUseCdn, $wgCdnServers,
  $wgCdnServersNoPurge, or $wgCdnMaxAge instead.

Do you have any of those config variables in your LocalSettings.php?

If so, they need updating...

No, none. I've gone through the list in the release notes and updated all configs i had, but those weren't in my LocalSettings.php.

Ok, so I'm a little confused here... If you had none of those in your config, I don't see how purgeList.php actually ever did anything for you. Because purgeList.php is for puging proxy/cache/cdn

It doesn't force any of the re-rendering or anything like ?action=purge would (which changes fields in the database and such)

But that's exactly what it did pre 1.34 for me.
Cache related i only got '$wgMainCacheType = CACHE_DB;'

I actually used that as that was recommended when i searched for how to purge all pages at once.
Doing so worked without issues pre 1.35. I assumed that should work in 1.35 as well. If the script is not intended to this, but it was only a accidental side-effect that's a shame - as i still don't know of any way that allows me to reliably purge all pages at once.

Here, someone posted it on Stackoverflow, first answer: https://stackoverflow.com/questions/25597846/purging-all-pages-in-mediawiki

That's why i did it and it worked perfectly until 1.35.

Note that's a comment from 2014. And not everything you read on StackOverflow is a good idea

There is a purgePage.php maintenance script that probably does more like what you want, but it doesn't have an all/namespace option.

The only changes to that script since 1.34 is rMW3c7f29a6b922: Add small HtmlCacheUpdater service class to normalize purging code (2) and rMW55439af8f7f9: maintenance: Rename purgeList.php --purge to --db-touch

Looking again at the code... I think the problem is --db-touch only works under Purge URL coming from stdin. Which would do the other types of cache invalidation (which would have the seemingly desired effect.. maybe)... But this wasn't ever the case under 1.34 either for purging everything (or all of a namespace), so I don't see how that would've broken it.

So there is definitely a Documentation issue here;

		$this->addOption( 'db-touch', 'Update the page.page_touched database field', false, false );

is only true if namespace or all-namespaces is not being passed. But that isn't different to the REL1_34 code; which, again would only run $title->invalidateCache(); if it was reading input from stdin (so not namespace or all), and --purge was being passed

The other question is what other changes swapping from CdnCacheUpdate to HtmlCacheUpdater has actually brought

If you're having issues with pages being updated after editing modules. And purging pages seems to solve it. Is your job queue actually being run?

Because you really shouldn't have to be regularly purging all pages to get updates to propagate

And as part of debugging... --namespace 0 executes and outputs quickly... --all-namespace does seem to hang

Change 630385 had a related patch set uploaded (by Reedy; owner: Reedy):
[mediawiki/core@master] purgeList.php Fix all-namespaces option to match one used in code

https://gerrit.wikimedia.org/r/630385

Spotted a typo

And the reason it's sitting there doing nothing, is because it's waiting for input from stdin

Change 630243 had a related patch set uploaded (by Reedy; owner: Reedy):
[mediawiki/core@REL1_35] purgeList.php Fix all-namespaces option to match one used in code

https://gerrit.wikimedia.org/r/630243

Well, it's an old comment, but i've been using that method since five years now - without any issues. Now i suddenly had issues with it, so i tried to report that issue.
I'm sorry that i didn't know the intention of the script, but that really didn't bother me at that time - it fixed a problem i had for a long time and was a perfect solution.
I'm just a normal user, no developer. If it works for me, it's good ;)

purgePage.php, as you already said, does not include an all option - that doesn't help me as i'm unable to add 3000 pages whose names i mostly don't even know into that manually.

If you're having issues with pages being updated after editing modules. And purging pages seems to solve it. Is your job queue actually being run?
Because you really shouldn't have to be regularly purging all pages to get updates to propagate

I don't know. We mostly use it after doing live changes for our template boxes for the content, as that gets updated quite regularily. Just doing that doesn't update the pages, though, so we want all pages immediately being purged so the new content is visible immediately.
I'm assuming job queue only does that after specific time like a cronjob? That is not what we need - and maybe that works, though we had reports of users going to some wiki pages that havent been purged three days after change still not seeing the new content until ?action=purge.
So i don't know. But i don't really want to make a support thing out of this ticket. My intention was that i saw something i did earlier was no longer working and i assumed that might be a bug. If it was never meant to do that, well, then i'm sorry for the report!

If the script still did what it did before it would be a massive ease-of-use, otherwise i need to look for other options on how to purge pages immediately.

Just looked up when i last used the purgeList.php to purge all pages and it was working, that was on August 19, 2020 - and we definately had 1.34 installed back then. (We log actions in our discord)

I'm assuming job queue only does that after specific time like a cronjob

It's ran by calling runJobs.php constantly on loop. On a small wiki it should be fairly quick. You could set a script to run it on a loop fairly easily. There's a few set ups from cron to systemd to kafka.

Ah, yeh. I tried executing that manually as well, but it said there is no outstanding jobs to do - the pages weren't up-to-date and needed a purge nontheless, though. So that might indeed not work for some reason.

Okay, the job queue isn't actually being run and thats only my fault. When setting up the family wiki the article recommended to set $wgJobRunRate to 0. So it's not actually running.
https://www.mediawiki.org/wiki/Manual:Wiki_family#Configure - I should have looked up what it does or rather that i need to setup a cronjob manually then.
So i guess my problem should be solved with setting up the cronjob for it. And if the script here is not supposed to purge pages, its not a bug either. But i'd really like to have the option of an easy --all purge if you were ever to consider an addition to purgePage.php.

So i don't know. But i don't really want to make a support thing out of this ticket. My intention was that i saw something i did earlier was no longer working and i assumed that might be a bug. If it was never meant to do that, well, then i'm sorry for the report!

The task description of purgeList.php maintenance script not doing anything is definitely valid; and I've uploaded a fix for that

Change 630385 had a related patch set uploaded (by Reedy; owner: Reedy):
[mediawiki/core@master] purgeList.php Fix all-namespaces option to match one used in code

https://gerrit.wikimedia.org/r/630385

Whether after the fix it'll still have your desired purging effect, is a different issue... But if it fixes your issue (as in theory it should run like it did before), then it's all good too

Which seems to agree with what we said ;)

Absolutely! The replying person there also assumed that --db-touch would at least do what i wanted, though, while it's currently not :)

Which seems to agree with what we said ;)

Absolutely! The replying person there also assumed that --db-touch would at least do what i wanted, though, while it's currently not :)

Heh.

My fix attached to this bug should be reviewed/merged shortly. It will come out in the 1.35.1 point release, but you can fix it locally yourself too by replacing

		$this->addOption( 'all-namespace', 'Purge pages in all namespaces', false, false );

with

		$this->addOption( 'all-namespaces', 'Purge pages in all namespaces', false, false );

in purgeList.php. And then it should output as expected... Maybe it will fix your purging issue too.

Then T263957: purgeList.php db-touch issue filed as to what to do about db-touch not working on all code paths... Whether we fix that in documentation, or fix the code... The latter seems likely, at least, for wikis not using $wgMiserMode

That indeed makes it show a list of what it does in --verbose mode, great!
Unfortunately my Jobs Queue is working well now, so i'm not currently having something to test if it does what i want. (Okay, that's not unfortunate)
The next time we change the infobox content module i'll test it again though :)

Thank you very much for your patience with me and the fixes, very appreciated. And sorry again i haven't been clear enough on what my 'issue' is from the very beginning!

Thank you very much for your patience with me and the fixes, very appreciated. And sorry again i haven't been clear enough on what my 'issue' is from the very beginning!

No problem!

Often it's the case the reporter doesn't know what is actually wrong, or there's multiple things wrong. So it takes a little bit to tease it out and work out what's going on and then what the fix might be

Either way, you definitely found a bug with the renaming of the parameters, and also highlighted at minimum a documentation issue in another part of the code too

Change 630243 merged by jenkins-bot:
[mediawiki/core@REL1_35] purgeList.php Fix all-namespaces option to match one used in code

https://gerrit.wikimedia.org/r/630243

Change 630385 merged by jenkins-bot:
[mediawiki/core@master] purgeList.php Fix all-namespaces option to match one used in code

https://gerrit.wikimedia.org/r/630385

Reedy claimed this task.

Ok, I'm closing this as the underlying issue of "script not doing anything". Issue is fixed in master, and in MW-1.35-release, it will come out in 1.35.1

T263957: purgeList.php db-touch issue is still open and a way forward to be completely decided for that one.