Maniphest T344402

maintenance: Improve efficiency of page filtering of refreshLinks.php
Closed, ResolvedPublic
Actions

Assigned To

Authored By

	Func
	Aug 17 2023, 4:38 AM

Project Tags

Referenced Files

None

Subscribers

Description

I am running a live archive of a large wiki by importing incremental XML dumps regularly using importDump.php with the --no-updates option, since refreshing after each revision imported is not efficient.

After importing, I need to run refreshLinks.php for updating the links table etc.

I noticed that its efficiency of page filtering is poor since it's loading and checking each page via the WikiPage object. I then have to create my own version of refreshLinks.php for the task.

I would like to submit some improvements I made, I think they would also benefit T159512.

Details

Related Changes in Gerrit:

	Subject	Author	Repo	Branch	Lines +/-
	refreshLinks: Fix refreshing pages in category	Func	mediawiki/core	master	+3 -1
	refreshLinks: Introduce `--touched-only` option	Func	mediawiki/core	master	+7 -0
	refreshLinks: Skip DFN if the namespace option is given	Func	mediawiki/core	master	+8 -20
	refreshLinks: Improve efficiency of page filtering	Func	mediawiki/core	master	+83 -169

Customize query in gerrit

Related Objects

Mentioned Here: T159512: Add option to refreshLinks.php to only update pages that haven't been updated since a timestamp

Event Timeline

Func created this task.Aug 17 2023, 4:38 AM

Restricted Application added a project: Platform Engineering. · View Herald TranscriptAug 17 2023, 4:38 AM

Restricted Application added a subscriber: Aklapper. · View Herald Transcript

Change 947391 had a related patch set uploaded (by Func; author: Func):

[mediawiki/core@master] refreshLinks: Improve efficiency of page filtering

https://gerrit.wikimedia.org/r/947391

gerritbot added a project: Patch-For-Review.Aug 17 2023, 5:56 AM

Change 949218 had a related patch set uploaded (by Func; author: Func):

[mediawiki/core@master] refreshLinks: Introduce `--touched-only` option

https://gerrit.wikimedia.org/r/949218

Change 949221 had a related patch set uploaded (by Func; author: Func):

[mediawiki/core@master] refreshLinks: Skip DFN if the namespace option are given

https://gerrit.wikimedia.org/r/949221

Change 947391 merged by jenkins-bot:

[mediawiki/core@master] refreshLinks: Improve efficiency of page filtering

https://gerrit.wikimedia.org/r/947391

Change 949221 merged by jenkins-bot:

[mediawiki/core@master] refreshLinks: Skip DFN if the namespace option is given

https://gerrit.wikimedia.org/r/949221

Change 949218 merged by jenkins-bot:

[mediawiki/core@master] refreshLinks: Introduce `--touched-only` option

https://gerrit.wikimedia.org/r/949218

Maintenance_bot removed a project: Patch-For-Review.Aug 22 2023, 1:29 AM

Change 951872 had a related patch set uploaded (by Func; author: Func):

[mediawiki/core@master] refreshLinks: Fix refreshing pages in category

https://gerrit.wikimedia.org/r/951872

gerritbot added a project: Patch-For-Review.Aug 24 2023, 6:23 AM

Change 951872 merged by jenkins-bot:

[mediawiki/core@master] refreshLinks: Fix refreshing pages in category

https://gerrit.wikimedia.org/r/951872

Maintenance_bot removed a project: Patch-For-Review.Aug 25 2023, 7:30 AM

ReleaseTaggerBot added a project: MW-1.41-notes (1.41.0-wmf.25; 2023-09-05).Aug 29 2023, 11:01 PM

I suppose this is done? Thanks!