I'm afraid this is not very feasible, because categories can be very deeply nested and we have no efficient way to get members of subcategories (and subsubcategories, etc) right now.

Probably the only ways we could do this performantly are:

• Maintaining a separate store for this purpose, e.g. in Redis (either for all categories or for a limited number).
• Changing the DB schema (unlikely)

In T158572#3044052, @Catrope wrote:

I'm afraid this is not very feasible, because categories can be very deeply nested and we have no efficient way to get members of subcategories (and subsubcategories, etc) right now.

That nesting issue is known. That's why most category-searching tools have a parameter to limit depth in search. After, filtering is irrelevant. Would it help?

In T158572#3046973, @Trizek-WMF wrote:

That nesting issue is known. That's why most category-searching tools have a parameter to limit depth in search. After, filtering is irrelevant. Would it help?

CategoryFinder is implemented as a post-processing step: https://phabricator.wikimedia.org/source/mediawiki/browse/master/includes/specials/SpecialRecentchanges.php;d4f51b7881e801362eac46516f0f7e312a8066d8$853

Either with that (or something similar), or with the Redis-based solution, we'll have the issue of limit mismatches, because AFAIK descendant categories can not be done in a single query.

I.E. Assume there are thousands of edits in the recentchanges table to articles nested within Category:History of the United States. You request 500 results on Special:RecentChanges.

Because the DB can not implement this in a single query, you get 500 results (including edits to articles that are not within that category at all). Then, CategoryFinder runs, and gives you a small fraction of that. You then either:

1. Get that small fraction as your final result set, or
2. Have to keep re-running the DB query until you get 500 or exhaust the table.

The same issue applies if it's implemented with Redis. (To clarify, the Redis idea is to have a Redis set for each category, containing all articles directly or indirectly within that category. This would be maintained after each edit, similar to what GettingStarted does (but GettingStarted does not handle nesting)).

The query/limit issue is not a deal-breaker, but we have to decide how to handle it.

We've had similar issues with Flow.

Also, we need to confirm with Ops about Redis persistence (from T158239: Improve GettingStarted data storage strategy, "Persistency on redis (and replication) has always been guaranteed as best-effort, which doesn't seem to be great in this case.") We may need a new instance.

Yes, I would say doable, but there are both product (500/limit stuff above, plus whether to limit nesting) and technical issues that make it kind of a big deal.

In T158572#3417252, @jmatazzoni wrote:

• When we list categories in the categories menu (T163433), would it be possible to show as well the stats for pages and subcategories next to the name, as happens all over the wiki on category pages. E.g.: Rock Music Groups (1 p, 51 c). Once users got used to this, they might start realizing that categories such as this one really don't have articles in them.

Yes, but not sure if that communicates the subcategory issue to the user (may depend if they're a power user).

• What if we just always searched the category the user specifies as well as all categories one level down. I.e., if I searched the category Rock music groups by genre‎, I'd see changes made to articles in any of the 51 genre subcategories. Still too much?

We need to figure out how many categories we can practically combine into one query (this is both relevant to this, and to "This means that when multiple categories are added to a search, each broadens the search, since all Category filters relate to one another via an OR." in general).

https://en.wikipedia.org/wiki/Category:People_by_nationality has 251 sub-categories. There are probably non-obscure categories with more.

Even if we accept a large number of categories explicitly specified by the user (perhaps they can judge for themselves what is too slow, and reduce the number), this automatic option could potentially cause unintuitive slowdown.

• Alternatively, would something like the following be possible? Imagine the user searches the broad categories Rock Music Groups and Punk Rock Groups. We return the small number of relevant article results that might appear, but also display a box at the top of the page saying:

Yes, but see above about limits (could also have UI issues with hundreds of categories).

In T158572#3417394, @Mattflaschen-WMF wrote:

We need to figure out how many categories we can practically combine into one query

They can be done in one SQL query just by adding to the cl_to clause, but I don't know when we'll start seeing issues. Also, I think this will cause the same (and need the same solution) as T168501: When querying for multiple tags, and more than one is in the edit, duplicate results.