Page MenuHomePhabricator

Global usage shows file usage on deleted pages
Closed, ResolvedPublic0 Story Points

Description

Example:
https://beta.wikiversity.org/wiki/USA:s_presidenter is deleted, but See https://commons.wikimedia.org/w/index.php?title=Special%3AGlobalUsage&limit=500&target=Harry-truman.jpg says that it is "IN USE".

Pleas purge the Database.

Thank you.


Version: wmf-deployment
Severity: major

Details

Reference
bz63594

Event Timeline

bzimport raised the priority of this task from to Normal.
bzimport set Reference to bz63594.
bzimport added a subscriber: Unknown Object (MLST).

Usually this is not an issue for other link tables, since we inner join on page, so its not noticeable.

As it stands, the only way to address the issue is to recreate the page in question, and then delete it.

We should probably

  • Make the refreshlinks option of API purge kill all links if the page doesn't exist
  • Possibly make normal purge on non-existent pages also do that (Not sure, normally we do not have purge touch the link tables, but to be fair purging images does really complex (and expensive) cache clearing, so it wouldn't be that different)

This was a example. All databases are affected. This is a problem and should be fixed.

A other exaple:
https://commons.wikimedia.org/w/index.php?title=Special%3AGlobalUsage&limit=50&target=Flag+of+Veneto.png

But the page on incubator.wikimedia.org is deleted.

A lotos of files are affected...

MariaDB [commonswiki_p]> SELECT * FROM globalimagelinks WHERE gil_page_title = "USA:s_presidenter" AND gil_wiki = "betawikiversity";
I see only on this page ~50 affected files...
Global are thousands of files affected

Tgr added a comment.May 14 2014, 6:43 PM

Please see https://www.mediawiki.org/wiki/Bugzilla/Fields#Severity
If this is obstructing work in a major way, it is more helpful to give an explanation of how that happens.

How long time do we need to wait @Tisza Gergő ...?

Restored prio, CC Erik. Major loss of function in an important area.

Tgr added a comment.May 14 2014, 7:05 PM

As I said, explaining how this is hindering your work or causing you problems is more helpful than playing around with severity flags (which does not cause bugs to magically fix themselves).

It is a problem in DeletionRequests if nonexisting usage is schown, it is a problem for the GlobalBot Commons Delinker (flooding the error log)...

etc etc. A experienced wikimedian schold know this.

Change 133351 had a related patch set uploaded by Aaron Schulz:
Added a links refresh mode to fix links to deleted pages

https://gerrit.wikimedia.org/r/133351

Change 133351 merged by jenkins-bot:
Added a links refresh mode to fix links to deleted pages

https://gerrit.wikimedia.org/r/133351

The script needs to actually be run. This bug can track that.

  • Bug 72510 has been marked as a duplicate of this bug. ***
tomasz removed a project: Shell.Feb 23 2015, 8:01 PM
tomasz set Security to None.
Tgr claimed this task.

Will run this on Monday: foreachwiki extensions/GlobalUsage/refreshGlobalimagelinks.php --pages=nonexisting

Tgr added a comment.Mar 9 2015, 10:50 PM
tgr@terbium:~$ mwscript extensions/GlobalUsage/refreshGlobalimagelinks.php --wiki=testwiki --pages=nonexisting
Querying for broken links after (page_id) = (0)
Querying for broken links after (page_id) = (80125)
[e4d6d611] [no req]   MWException from line 2206 of /srv/mediawiki/php-1.25wmf20/includes/db/Database.php: DatabaseBase::makeList: empty input for field page_id
Backtrace:
#0 /srv/mediawiki/php-1.25wmf20/includes/db/Database.php(1779): DatabaseBase->makeList(array, integer)
#1 /srv/mediawiki/php-1.25wmf20/includes/db/Database.php(1727): DatabaseBase->selectSQLText(string, string, array, string, array, array)
#2 /srv/mediawiki/php-1.25wmf20/extensions/GlobalUsage/refreshGlobalimagelinks.php(126): DatabaseBase->select(string, string, array, string)
#3 /srv/mediawiki/php-1.25wmf20/maintenance/doMaintenance.php(101): RefreshGlobalImageLinks->execute()
#4 /srv/mediawiki/php-1.25wmf20/extensions/GlobalUsage/refreshGlobalimagelinks.php(149): require_once(string)
#5 /srv/mediawiki/multiversion/MWScript.php(97): require_once(string)
#6 {main}

Change 195479 had a related patch set uploaded (by Gergő Tisza):
Fix deleted page handling in update script

https://gerrit.wikimedia.org/r/195479

Change 195479 merged by jenkins-bot:
Fix deleted page handling in update script

https://gerrit.wikimedia.org/r/195479

Tgr edited a custom field.

Change 195812 had a related patch set uploaded (by Gergő Tisza):
Fix deleted page handling in update script

https://gerrit.wikimedia.org/r/195812

Change 195812 merged by jenkins-bot:
Fix deleted page handling in update script

https://gerrit.wikimedia.org/r/195812

Change 195813 had a related patch set uploaded (by Gergő Tisza):
Update GlobalUsage submodule

https://gerrit.wikimedia.org/r/195813

Tgr added a comment.Mar 11 2015, 12:48 AM

Will push the fix via Wed evening SWAT then run it.

Steinsplitter moved this task from Incoming to Backlog on the Commons board.Mar 11 2015, 12:50 PM

Change 195813 merged by jenkins-bot:
Update GlobalUsage submodule

https://gerrit.wikimedia.org/r/195813

Tgr added a comment.Mar 12 2015, 4:24 AM

Finished running the script, but the output looked suspicious - it seems most of it went to stderr (which I stupidly did not log, only stdout). @Steinsplitter, can you verify if it worked correctly?

Tgr added a comment.Mar 12 2015, 11:35 PM

I dug up a bunch of deleted pages with images from enwiki AfD and those have correct globalusage lists. Not sure if they would have been affected by this bug at all, though - GlobalUsage had an article deletion hook from the start, so in theory no maintenance script would be needed. Is there a definite period in which that hook failed? Or does anyone have concrete reproduction steps?

Tgr added a comment.Mar 13 2015, 12:40 AM

Re-run on enwiki with proper logging:

Querying for broken links after (page_id) = (0)
A database query error has occurred.
Query: SELECT  page_id  FROM `page`   WHERE page_id IN ('10','12','25','39','42',
'128','270','278','290','303','305','307','308','309','316','324','330','334',
...
...
...
'45657111','45657112','45657113','45657117','45657118','45657121','45657122',
'45657128','45657130','45657134','45657139','45657142','45657151','45657153',
'45657154','45657155','45657165','45657169','45657170','45657173','45657180',
'45657183','45657191')   
Function: RefreshGlobalImageLinks::execute
Error: 2006 MySQL server has gone away (10.64.48.28)

The problem is that for existing pages the script uses a hardcoded batch size of 500, for non-existing it uses mBatchSize but never calls setBatchSize().

Change 196501 had a related patch set uploaded (by Gergő Tisza):
Add setBatchSize() call for refreshGlobalimagelinks.php

https://gerrit.wikimedia.org/r/196501

Change 196501 merged by jenkins-bot:
Add setBatchSize() call for refreshGlobalimagelinks.php

https://gerrit.wikimedia.org/r/196501

In T65594#1112135, @Tgr wrote:
@Steinsplitter, can you verify if it worked correctly?

I checked a few files and it looks like yes :-)

Tgr added a comment.Mar 13 2015, 4:39 PM

It seems to have died halfway though. I'll put the fix for SWAT and re-run it on Monday.

Change 196991 had a related patch set uploaded (by Gergő Tisza):
Add setBatchSize() call for refreshGlobalimagelinks.php

https://gerrit.wikimedia.org/r/196991

Change 196992 had a related patch set uploaded (by Gergő Tisza):
Add setBatchSize() call for refreshGlobalimagelinks.php

https://gerrit.wikimedia.org/r/196992

Change 196991 merged by jenkins-bot:
Add setBatchSize() call for refreshGlobalimagelinks.php

https://gerrit.wikimedia.org/r/196991

Change 196992 merged by jenkins-bot:
Add setBatchSize() call for refreshGlobalimagelinks.php

https://gerrit.wikimedia.org/r/196992

Change 196993 had a related patch set uploaded (by Gergő Tisza):
Update GlobalUsage submodule

https://gerrit.wikimedia.org/r/196993

Change 196994 had a related patch set uploaded (by Gergő Tisza):
Update GlobalUsage submodule

https://gerrit.wikimedia.org/r/196994

Change 196993 merged by jenkins-bot:
Update GlobalUsage submodule

https://gerrit.wikimedia.org/r/196993

Change 196994 merged by jenkins-bot:
Update GlobalUsage submodule

https://gerrit.wikimedia.org/r/196994

The patch is now deployed on the cluster... Next item here is for it to be run again.

Tgr added a comment.Mar 17 2015, 8:53 PM

Thanks @Krenair! The script ran successfully this time.

@Bawolff, are we still looking for the original bug which caused deleted pages to become outdated in the first place, or has that been fixed earlier?

Gilles added a subscriber: Gilles.Mar 18 2015, 9:11 AM

What is there to test?

Tgr closed this task as Resolved.Mar 18 2015, 9:48 PM

What is there to test?

I don't have a good test case (apart from running the script on all wikis and observing that it does not give any error, which I did). All the examples of phantom pages on global usage lists that I knew of have been fixed by the first, partial run already. Steinsplitter said the same, so I think we can just assume that inconsistencies between the GU table and actual file usage have been fixed.

Whether there is a bug causing new inconsistencies is another matter. But at a glance the code seems good. I'll just close this and if someone sees new instances of the GU list being out of sync / finds reproduction steps, they can reopen it.