I believe T167077: Use wgContentNamespaces instead of $wgMFContentNamespace was raised to high priority when this was reported.

@Jcb are any other special pages experiencing this problem or is it just this one?

@pmiagza I'm inclined to suggest we add a blacklist configuration variable that allows the removal of namespaces from that query. E.g. wgShortPageBlacklist - what do you think?

Looking at the page I'm not sure why file namespace would ever make sense to be included in this page's results.

In T170687#3439743, @Jdlrobson wrote:

@Jcb are any other special pages experiencing this problem or is it just this one?

Its very common to use content namespaces to decide what is on a special page. The config variable affects:

SpecialAncientpages
SpecialDeadendpages
SpecialFewestrevisions
SpecialLonelypages
SpecialLonelypages
SpecialMostcategories
SpecialMostinterwikis
SpecialRandompage
SpecialShortpages
SpecialUncategorizedpages
SpecialWithoutinterwiki

and numerous other things in MediaWiki

@Bawolff sure but I should rephrase - are any of these pages not proving useful in current form or is it just Special:ShortPages (since that is what the bug report complained about)?

In T170687#3439749, @Jdlrobson wrote:

@Bawolff sure but I should rephrase - are any of these pages not proving useful in current form or is it just Special:ShortPages (since that is what the bug report complained about)?

Its not clear how universally people at commons feel that way. 12 byte file pages also probably need to be cleaned up - a 12 byte file page is almost certainly not licensed or described properly and very likely needs to be deleted. It seems more like a behavior that would be good for some maintenance tasks and bad for others. The only comments complaining about this behavior that I saw were on phab - there are none on the commons thread.

@Bawolff thanks for the feedback and yes I think we need to stew over this some more before making any changes and more voices from Commons would be useful here. There's a lot at stake here.

@Jdlrobson I agree, having a blacklist sounds good. In point b) :

b) introduce new config var that defines a list of namespaces to include on SpecialShortPages page

I forgot to add include/exclude. Defining a new variable would require setting up a configuration for all wikis which would require much more work. Creating a blacklist sounds more reasonable.

wgShortPageBlacklist sounds good, my concern is that it may be too narrow. First, we need to find all affected pages, and then try to come up with proper config variable name.

Probably a Special:DeadendPages is also a good candidate to not show pages from NS_FILE namespace.

In T170687#3439756, @Bawolff wrote:

In T170687#3439749, @Jdlrobson wrote:

@Bawolff sure but I should rephrase - are any of these pages not proving useful in current form or is it just Special:ShortPages (since that is what the bug report complained about)?

Its not clear how universally people at commons feel that way. 12 byte file pages also probably need to be cleaned up - a 12 byte file page is almost certainly not licensed or described properly and very likely needs to be deleted. It seems more like a behavior that would be good for some maintenance tasks and bad for others. The only comments complaining about this behavior that I saw were on phab - there are none on the commons thread.

Err maybe I'm wrong. All those 12 byte pages are just template including the actual info, so they're ok

I think Special:ShortPages is the only important report that should not include File pages.

E.g. Special:DeadEndPages is useless anyway at Commons, it lists almost every gallery, because most galleries in Commons do not link to other galleries.

@Jcb I just wanted to clarify, since you have said (at T167077 and T168010) "The discussion in the Village pump did not ask to change the ShortPages report" and "I do not see any consensus to add file pages to Special:ShortPages":

The discussion in the Commons' Village pump did not "ask" for anything. It was opened in response to the settings change T167077: Use wgContentNamespaces instead of $wgMFContentNamespace that had already happened, and the discussion didn't actually come to any consensus about anything. (I thought that some evidence of a consensus would be necessary before Commons' "article count" was updated, but that happened without any resolution to the discussion.)

Now that a different issue has been identified related to the same settings change, it should also be discussed at Commons:Village pump#Should content pages consist of galleries only or also include File pages? . I have posted again in that thread to ty to spur that discussion. I expect you'll want to participate…

Interested sysadmins and developers here might want to also keep an eye on the Commons discussion (unfortunately, it is a high-volume page that is hard to "watch" using the MW feature: you just have to visit it periodically), and weigh in if there's something you think the community needs to know — especially any relevant "technical" issues.

Since this has not been explicitly mentioned here, I feel I should suggest that if "we" (you all) are considering making changes to the MediaWiki code to fix this problem, it is worth considering whether the code should instead be changed in some way to fix whatever problem T167077 was thought to be a solution for.

I'm just saying…

I think the best thing for now is to revert the change that leaded to all these problems, so that we could finally use [[Special:ShortPages]] again. It will show a lot of test-pages now the report has been out of service for several weeks. And then maybe reapply the change as soon as it is in a state in which it does not break several reports.

Don't expect more participation from me from 17 July till 5 August, I will be completely offline in that period. (Human aid in a slums area, no internet)

Make shortpages exclude file: namespace on commons could be an option no?

In T170687#3441602, @Jcb wrote:

I think the best thing for now is to revert the change that leaded to all these problems, so that we could finally use [[Special:ShortPages]] again. It will show a lot of test-pages now the report has been out of service for several weeks. And then maybe reapply the change as soon as it is in a state in which it does not break several reports.

I don't think this is an option to go. Not even as short term fix. I can understand your anger, but to solve something by breaking something else is never good.

Looking at pmiazga/JdIrobson's proposals I think c) use ShortPagesQuery hook and remove NS_FILE only on commons wiki (it may read a config variable what to not include in the report) could serve at least as short term fix, before the final solution is decided.

In T170687#3441608, @Vachovec1 wrote:

I can understand your anger

Where did I express anger?

I stated an opinion with which you may agree or disagree. It's very dangerous to try to read emotions from written text. Don't.

I think the best thing for now is to revert the change that leaded to all these problems, so that we could finally use [[Special:ShortPages]] again. It will show a lot of test-pages now the report has been out of service for several weeks

We won't be reverting the change hastily as that will break various other things and create more problems and work for us. It's really awesome we're getting all this input (and a good learning experience) as it helps us understand how the software is used and gives us an opportunity to make this better for everyone and find a fix that addresses all the problems we're experiencing with wgContentNamespaces.

Make shortpages exclude file: namespace on commons could be an option no?

@Zppix yup, there's lots of ways to do this. I've updated the description to provide them. Right now I'm leaning towards option f) - having a more granular way to view this page seems the most useful for everyone. If that's not solving the problem b) or c) seem like the better solutions.

What do people think? (I've asked the same question on Commons )

a) hardcode skipping NS_FILE inside SpecialShortPages which is probably bad, but most of pages inside NS_FILE are short

Yes, that's probably a bad idea.

b) introduce new config var that defines a list of namespaces to include on SpecialShortPages page
c) introduce new config var that defines a list of namespaces to exclude on SpecialShortPages page
d) use ShortPagesQuery hook and remove NS_FILE only on commons wiki (it may read a config variable what to not include in the report)

Those could work.

f) Update Special:ShortPages to allow filtering different namespaces. e.g. Special:ShortPages/0, Special:ShortPages/3 (where number is the namespace)

You'll run into issues with the caching done by QueryPage when $wgMiserMode is set. To get it to work right, instead of caching the first QueryPage::getMaxResults() short pages you'd instead have to cache that many pages per namespace. Otherwise people will still complain if they hit Special:ShortPages/0 and get only 3 pages listed because the other 9997 shortest pages are in the File namespace.

In T170687#3444612, @Jdlrobson wrote:

I think the best thing for now is to revert the change that leaded to all these problems, so that we could finally use [[Special:ShortPages]] again. It will show a lot of test-pages now the report has been out of service for several weeks

We won't be reverting the change hastily as that will break various other things and create more problems and work for us. It's really awesome we're getting all this input (and a good learning experience) as it helps us understand how the software is used and gives us an opportunity to make this better for everyone and find a fix that addresses all the problems we're experiencing with wgContentNamespaces.

OTOH, refusing to revert when real problems came up during an initial rollout is why VE is still hated on enwiki. Keep that in mind when you're deciding not to revert it.

OTOH, refusing to revert when real problems came up during an initial rollout is why VE is still hated on enwiki. Keep that in mind when you're deciding not to revert it.

Reverting will break several other special pages and create more of a mess as well as require running updateArticleCount ''again'', so we have to be careful.

You'll run into issues with the caching done by QueryPage when $wgMiserMode is set. To get it to work right, instead of caching the first QueryPage::getMaxResults() short pages you'd instead have to cache that many pages per namespace. Otherwise people will still complain if they hit Special:ShortPages/0 and get only 3 pages listed because the other 9997 shortest pages are in the File namespace.

Thanks for this input, that's good to know. Where is wgMiserMode used?

In T170687#3444846, @Jdlrobson wrote:

You'll run into issues with the caching done by QueryPage when $wgMiserMode is set. To get it to work right, instead of caching the first QueryPage::getMaxResults() short pages you'd instead have to cache that many pages per namespace. Otherwise people will still complain if they hit Special:ShortPages/0 and get only 3 pages listed because the other 9997 shortest pages are in the File namespace.

Thanks for this input, that's good to know. Where is wgMiserMode used?

$wgMiserMode is set on all WMF wikis.

You'll run into issues with the caching done by QueryPage when $wgMiserMode is set. To get it to work right, instead of caching the first QueryPage::getMaxResults() short pages you'd instead have to cache that many pages per namespace. Otherwise people will still complain if they hit Special:ShortPages/0 and get only 3 pages listed because the other 9997 shortest pages are in the File namespace.

Short pages is a cheap (uncached) special page i believe, so this should not be an issue.

@ovasileva we should talk and work on this sooner rather than later otherwise it's going to become interrupt work.

We estimated fix c) and captured it in acceptance criteria. Fix is planned for next week.

Change 368325 had a related patch set uploaded (by Jdlrobson; owner: Jdlrobson):
[mediawiki/core@master] Allow blacklisting certain namespaces in Special:ShortPages

https://gerrit.wikimedia.org/r/368325

From the monthly update:

We're considering adding a namespace filter to Special:ShortPages. Would this be useful to people?

Yes! It would be useful on many of the pages listed above in T170687#3439746

E.g.
Special:ShortPages and Special:DeadendPages could then be used with the Help: and Project: namespace to find things to cleanup or merge
Special:LongPages could then be used with the usertalk namespace to find people to nag to archive their overwhelming talkpages.
Special:AncientPages could then be used with the Help: and Mediawiki: namespace to find things to update or reconsider the defaults.

Note: Special:Random already has this functionality, similar to solution (f) but using full namespace names, per mw:Help:Random_page e.g. full list at https://en.wikipedia.org/wiki/Wikipedia:Random

@Quiddity - thanks for your input. Those are really good ideas, and I think it's worth exploring. Could you create phab tickets per each "functionality' and we will talk about it more as it requires community input. I would like to keep this task strictly only about Special:ShortPages as this is a bugfix, not a feature change.

Change 368325 merged by jenkins-bot:
[mediawiki/core@master] Allow blacklisting certain namespaces in Special:ShortPages

https://gerrit.wikimedia.org/r/368325

@pmiagza I asked for input in Chris email :) ill open bugs as necessary but you can expect more eyes on this bug and comments.

Change 369503 had a related patch set uploaded (by Jdlrobson; owner: Jdlrobson):
[operations/mediawiki-config@master] Exclude files from Special:ShortPages on commons

https://gerrit.wikimedia.org/r/369503

The remaining work for this is a SWAT deploy. I've pulled this out of the sprint and put back in Upcoming so we don't forget to do this.

Change 369503 merged by jenkins-bot:
[operations/mediawiki-config@master] Exclude files from Special:ShortPages on commons

https://gerrit.wikimedia.org/r/369503

Mentioned in SAL (#wikimedia-operations) [2017-08-07T18:38:45Z] <ebernhardson@tin> Synchronized wmf-config/InitialiseSettings.php: T170687 - Exclude files from Special:ShortPages on commons (duration: 00m 46s)

Moving back onto board. Verifying the fix is blocked until Wednesday deploy.

Change 370924 had a related patch set uploaded (by Jdlrobson; owner: Jdlrobson):
[operations/mediawiki-config@master] Correct config - commonswiki not commons

https://gerrit.wikimedia.org/r/370924

I went to sign this off and realised it wasn't working. Seems I did a bad config change :( I'll correct this tomorrow at 11am PST (1pm my time) https://wikitech.wikimedia.org/wiki/Deployments#Thursday.2C.C2.A0August.C2.A010

Change 370924 merged by jenkins-bot:
[operations/mediawiki-config@master] Correct config - commonswiki not commons

https://gerrit.wikimedia.org/r/370924

I confirm ShortPages works fine again. Thanks everybody!