Page MenuHomePhabricator

[[special:ShortPages]] includes file pages on Commons
Closed, ResolvedPublic3 Story Points

Description

The change in T167077 to $wgContentNamespaces broke special:ShortPages in the eyes of a Commons user. That report is now incorrectly flooded by file pages. Please fix the ShortPages report in a way that it does no longer include file pages. (The discussion in the Village pump did not ask to change the ShortPages report)

Options

  • We can revert the change of T167077 however this will break Special;Nearby on mobile and Special:Random on mobile will no longer show file pages.
    • If we take this option we'll also have to run updateArticleCount.php per T169822
  • Explore ShortPages implementation and see if there is a way to configure it to not include file namespace. This would be preferable as it has the least impact on the rest of the mobile product.

Proposed solutions

a) hardcode skipping NS_FILE inside SpecialShortPages which is probably bad, but most of pages inside NS_FILE are short
b) introduce new config var that defines a list of namespaces to include on SpecialShortPages page
c) introduce new config var that defines a list of namespaces to exclude on SpecialShortPages page
d) use ShortPagesQuery hook and remove NS_FILE only on commons wiki (it may read a config variable what to not include in the report)
e) ~do nothing as NS_FILE is a content namespace in commons wiki.~ JR: not an option.
f) Update Special:ShortPages to allow filtering different namespaces. e.g. Special:ShortPages/0, Special:ShortPages/3 (where number is the namespace)

History

https://commons.wikimedia.org/wiki/Commons:Village_pump/Archive/2017/07#Should_content_pages_consist_of_galleries_only_or_also_include_File_pages.3F

Proposal

We'll do (c) since it's cheap and solves the immediate problem.
(f) would be the most useful but would need some design on how to make these accessible.

It would be nice to do the backend work for (f) so power users can use it if they know the URL, but let's talk about what that means.

Acceptance criteria

Next steps

Given we've had lots of issues with this already we should not rush to a decision until we've fully understood if there are any other problems.

We've opened T172215 to capture other changes to make this feature more useful.

Sign off steps

Details

Related Gerrit Patches:

Event Timeline

There are a very large number of changes, so older changes are hidden. Show Older Changes
phuedx added a subscriber: phuedx.Jul 14 2017, 4:23 PM

I believe T167077: Use wgContentNamespaces instead of $wgMFContentNamespace was raised to high priority when this was reported.

Jdlrobson lowered the priority of this task from High to Medium.Jul 14 2017, 4:25 PM

I dropped it to normal. Original priority was set by non team member.

@Jcb I agree doing nothing is not an option but we like to list all possible solutions to be thorough :)

@Jcb are any other special pages experiencing this problem or is it just this one?

@pmiagza I'm inclined to suggest we add a blacklist configuration variable that allows the removal of namespaces from that query. E.g. wgShortPageBlacklist - what do you think?

Looking at the page I'm not sure why file namespace would ever make sense to be included in this page's results.

@Jcb are any other special pages experiencing this problem or is it just this one?

Its very common to use content namespaces to decide what is on a special page. The config variable affects:

SpecialAncientpages
SpecialDeadendpages
SpecialFewestrevisions
SpecialLonelypages
SpecialLonelypages
SpecialMostcategories
SpecialMostinterwikis
SpecialRandompage
SpecialShortpages
SpecialUncategorizedpages
SpecialWithoutinterwiki

and numerous other things in MediaWiki

@Bawolff sure but I should rephrase - are any of these pages not proving useful in current form or is it just Special:ShortPages (since that is what the bug report complained about)?

@Bawolff sure but I should rephrase - are any of these pages not proving useful in current form or is it just Special:ShortPages (since that is what the bug report complained about)?

Its not clear how universally people at commons feel that way. 12 byte file pages also probably need to be cleaned up - a 12 byte file page is almost certainly not licensed or described properly and very likely needs to be deleted. It seems more like a behavior that would be good for some maintenance tasks and bad for others. The only comments complaining about this behavior that I saw were on phab - there are none on the commons thread.

@Bawolff thanks for the feedback and yes I think we need to stew over this some more before making any changes and more voices from Commons would be useful here. There's a lot at stake here.

pmiazga added a comment.EditedJul 14 2017, 6:02 PM

@Jdlrobson I agree, having a blacklist sounds good. In point b) :

b) introduce new config var that defines a list of namespaces to include on SpecialShortPages page

I forgot to add include/exclude. Defining a new variable would require setting up a configuration for all wikis which would require much more work. Creating a blacklist sounds more reasonable.

wgShortPageBlacklist sounds good, my concern is that it may be too narrow. First, we need to find all affected pages, and then try to come up with proper config variable name.

Probably a Special:DeadendPages is also a good candidate to not show pages from NS_FILE namespace.

@Bawolff sure but I should rephrase - are any of these pages not proving useful in current form or is it just Special:ShortPages (since that is what the bug report complained about)?

Its not clear how universally people at commons feel that way. 12 byte file pages also probably need to be cleaned up - a 12 byte file page is almost certainly not licensed or described properly and very likely needs to be deleted. It seems more like a behavior that would be good for some maintenance tasks and bad for others. The only comments complaining about this behavior that I saw were on phab - there are none on the commons thread.

Err maybe I'm wrong. All those 12 byte pages are just template including the actual info, so they're ok

Jcb added a comment.Jul 14 2017, 7:21 PM

I think Special:ShortPages is the only important report that should not include File pages.

E.g. Special:DeadEndPages is useless anyway at Commons, it lists almost every gallery, because most galleries in Commons do not link to other galleries.

Dcljr added a subscriber: Dcljr.EditedJul 15 2017, 12:07 PM

@Jcb I just wanted to clarify, since you have said (at T167077 and T168010) "The discussion in the Village pump did not ask to change the ShortPages report" and "I do not see any consensus to add file pages to Special:ShortPages":

The discussion in the Commons' Village pump did not "ask" for anything. It was opened in response to the settings change T167077: Use wgContentNamespaces instead of $wgMFContentNamespace that had already happened, and the discussion didn't actually come to any consensus about anything. (I thought that some evidence of a consensus would be necessary before Commons' "article count" was updated, but that happened without any resolution to the discussion.)

Now that a different issue has been identified related to the same settings change, it should also be discussed at Commons:Village pump#Should content pages consist of galleries only or also include File pages? . I have posted again in that thread to ty to spur that discussion. I expect you'll want to participate…

Interested sysadmins and developers here might want to also keep an eye on the Commons discussion (unfortunately, it is a high-volume page that is hard to "watch" using the MW feature: you just have to visit it periodically), and weigh in if there's something you think the community needs to know — especially any relevant "technical" issues.

Since this has not been explicitly mentioned here, I feel I should suggest that if "we" (you all) are considering making changes to the MediaWiki code to fix this problem, it is worth considering whether the code should instead be changed in some way to fix whatever problem T167077 was thought to be a solution for.

I'm just saying…

Jcb added a comment.Jul 15 2017, 9:28 PM

I think the best thing for now is to revert the change that leaded to all these problems, so that we could finally use [[Special:ShortPages]] again. It will show a lot of test-pages now the report has been out of service for several weeks. And then maybe reapply the change as soon as it is in a state in which it does not break several reports.

Don't expect more participation from me from 17 July till 5 August, I will be completely offline in that period. (Human aid in a slums area, no internet)

Zppix added a subscriber: Zppix.Jul 15 2017, 9:37 PM

Make shortpages exclude file: namespace on commons could be an option no?

I think the best thing for now is to revert the change that leaded to all these problems, so that we could finally use [[Special:ShortPages]] again. It will show a lot of test-pages now the report has been out of service for several weeks. And then maybe reapply the change as soon as it is in a state in which it does not break several reports.

I don't think this is an option to go. Not even as short term fix. I can understand your anger, but to solve something by breaking something else is never good.

Looking at pmiazga/JdIrobson's proposals I think c) use ShortPagesQuery hook and remove NS_FILE only on commons wiki (it may read a config variable what to not include in the report) could serve at least as short term fix, before the final solution is decided.

Jcb added a comment.Jul 15 2017, 10:18 PM

I can understand your anger

Where did I express anger?

I stated an opinion with which you may agree or disagree. It's very dangerous to try to read emotions from written text. Don't.

Jdlrobson updated the task description. (Show Details)Jul 17 2017, 4:33 PM
Jdlrobson updated the task description. (Show Details)

I think the best thing for now is to revert the change that leaded to all these problems, so that we could finally use [[Special:ShortPages]] again. It will show a lot of test-pages now the report has been out of service for several weeks

We won't be reverting the change hastily as that will break various other things and create more problems and work for us. It's really awesome we're getting all this input (and a good learning experience) as it helps us understand how the software is used and gives us an opportunity to make this better for everyone and find a fix that addresses all the problems we're experiencing with wgContentNamespaces.

Make shortpages exclude file: namespace on commons could be an option no?

@Zppix yup, there's lots of ways to do this. I've updated the description to provide them. Right now I'm leaning towards option f) - having a more granular way to view this page seems the most useful for everyone. If that's not solving the problem b) or c) seem like the better solutions.

What do people think? (I've asked the same question on Commons )

a) hardcode skipping NS_FILE inside SpecialShortPages which is probably bad, but most of pages inside NS_FILE are short

Yes, that's probably a bad idea.

b) introduce new config var that defines a list of namespaces to include on SpecialShortPages page
c) introduce new config var that defines a list of namespaces to exclude on SpecialShortPages page
d) use ShortPagesQuery hook and remove NS_FILE only on commons wiki (it may read a config variable what to not include in the report)

Those could work.

f) Update Special:ShortPages to allow filtering different namespaces. e.g. Special:ShortPages/0, Special:ShortPages/3 (where number is the namespace)

You'll run into issues with the caching done by QueryPage when $wgMiserMode is set. To get it to work right, instead of caching the first QueryPage::getMaxResults() short pages you'd instead have to cache that many pages per namespace. Otherwise people will still complain if they hit Special:ShortPages/0 and get only 3 pages listed because the other 9997 shortest pages are in the File namespace.

I think the best thing for now is to revert the change that leaded to all these problems, so that we could finally use [[Special:ShortPages]] again. It will show a lot of test-pages now the report has been out of service for several weeks

We won't be reverting the change hastily as that will break various other things and create more problems and work for us. It's really awesome we're getting all this input (and a good learning experience) as it helps us understand how the software is used and gives us an opportunity to make this better for everyone and find a fix that addresses all the problems we're experiencing with wgContentNamespaces.

OTOH, refusing to revert when real problems came up during an initial rollout is why VE is still hated on enwiki. Keep that in mind when you're deciding not to revert it.

Jdlrobson added a comment.EditedJul 17 2017, 5:34 PM

OTOH, refusing to revert when real problems came up during an initial rollout is why VE is still hated on enwiki. Keep that in mind when you're deciding not to revert it.

Reverting will break several other special pages and create more of a mess as well as require running updateArticleCount ''again'', so we have to be careful.

You'll run into issues with the caching done by QueryPage when $wgMiserMode is set. To get it to work right, instead of caching the first QueryPage::getMaxResults() short pages you'd instead have to cache that many pages per namespace. Otherwise people will still complain if they hit Special:ShortPages/0 and get only 3 pages listed because the other 9997 shortest pages are in the File namespace.

Thanks for this input, that's good to know. Where is wgMiserMode used?

You'll run into issues with the caching done by QueryPage when $wgMiserMode is set. To get it to work right, instead of caching the first QueryPage::getMaxResults() short pages you'd instead have to cache that many pages per namespace. Otherwise people will still complain if they hit Special:ShortPages/0 and get only 3 pages listed because the other 9997 shortest pages are in the File namespace.

Thanks for this input, that's good to know. Where is wgMiserMode used?

$wgMiserMode is set on all WMF wikis.

You'll run into issues with the caching done by QueryPage when $wgMiserMode is set. To get it to work right, instead of caching the first QueryPage::getMaxResults() short pages you'd instead have to cache that many pages per namespace. Otherwise people will still complain if they hit Special:ShortPages/0 and get only 3 pages listed because the other 9997 shortest pages are in the File namespace.

Short pages is a cheap (uncached) special page i believe, so this should not be an issue.

Jdlrobson added a subscriber: ovasileva.

@ovasileva we should talk and work on this sooner rather than later otherwise it's going to become interrupt work.

Jdlrobson updated the task description. (Show Details)Jul 24 2017, 10:14 PM
Jdlrobson updated the task description. (Show Details)Jul 25 2017, 4:37 PM
Jdlrobson updated the task description. (Show Details)Jul 25 2017, 4:43 PM
Jdlrobson set the point value for this task to 3.

We estimated fix c) and captured it in acceptance criteria. Fix is planned for next week.

Tbayer updated the task description. (Show Details)Jul 26 2017, 5:55 PM

Change 368325 had a related patch set uploaded (by Jdlrobson; owner: Jdlrobson):
[mediawiki/core@master] Allow blacklisting certain namespaces in Special:ShortPages

https://gerrit.wikimedia.org/r/368325

I pulled this in early in the sprint as I'm expecting/hoping for code review from outside team members and it's small and I had some time :)

From the monthly update:

We're considering adding a namespace filter to Special:ShortPages. Would this be useful to people?

Yes! It would be useful on many of the pages listed above in T170687#3439746

E.g.
Special:ShortPages and Special:DeadendPages could then be used with the Help: and Project: namespace to find things to cleanup or merge
Special:LongPages could then be used with the usertalk namespace to find people to nag to archive their overwhelming talkpages.
Special:AncientPages could then be used with the Help: and Mediawiki: namespace to find things to update or reconsider the defaults.

Note: Special:Random already has this functionality, similar to solution (f) but using full namespace names, per mw:Help:Random_page e.g. full list at https://en.wikipedia.org/wiki/Wikipedia:Random

@Quiddity - thanks for your input. Those are really good ideas, and I think it's worth exploring. Could you create phab tickets per each "functionality' and we will talk about it more as it requires community input. I would like to keep this task strictly only about Special:ShortPages as this is a bugfix, not a feature change.

Change 368325 merged by jenkins-bot:
[mediawiki/core@master] Allow blacklisting certain namespaces in Special:ShortPages

https://gerrit.wikimedia.org/r/368325

@pmiagza I asked for input in Chris email :) ill open bugs as necessary but you can expect more eyes on this bug and comments.

Change 369503 had a related patch set uploaded (by Jdlrobson; owner: Jdlrobson):
[operations/mediawiki-config@master] Exclude files from Special:ShortPages on commons

https://gerrit.wikimedia.org/r/369503

Jdlrobson updated the task description. (Show Details)
Jdlrobson moved this task from 2017-18 Q1 to Upcoming on the Readers-Web-Backlog board.

The remaining work for this is a SWAT deploy. I've pulled this out of the sprint and put back in Upcoming so we don't forget to do this.

Change 369503 merged by jenkins-bot:
[operations/mediawiki-config@master] Exclude files from Special:ShortPages on commons

https://gerrit.wikimedia.org/r/369503

Mentioned in SAL (#wikimedia-operations) [2017-08-07T18:38:45Z] <ebernhardson@tin> Synchronized wmf-config/InitialiseSettings.php: T170687 - Exclude files from Special:ShortPages on commons (duration: 00m 46s)

Moving back onto board. Verifying the fix is blocked until Wednesday deploy.

Change 370924 had a related patch set uploaded (by Jdlrobson; owner: Jdlrobson):
[operations/mediawiki-config@master] Correct config - commonswiki not commons

https://gerrit.wikimedia.org/r/370924

I went to sign this off and realised it wasn't working. Seems I did a bad config change :( I'll correct this tomorrow at 11am PST (1pm my time) https://wikitech.wikimedia.org/wiki/Deployments#Thursday.2C.C2.A0August.C2.A010

Change 370924 merged by jenkins-bot:
[operations/mediawiki-config@master] Correct config - commonswiki not commons

https://gerrit.wikimedia.org/r/370924

Jdlrobson closed this task as Resolved.Aug 10 2017, 6:45 PM

I've verified that https://commons.wikimedia.org/wiki/Special:ShortPages no longer shows file pages. Please open a new bug if there's any follow up work to do here.

T37758 follows this up with the idea to add a namespace filter to this page.

Jcb added a comment.Aug 11 2017, 4:16 PM

I confirm ShortPages works fine again. Thanks everybody!