I agree, we need to do this. We ought to call the new hook ChangeTagsRegister, ChangeTagsDefine or something similar.

However, I would keep it simple to begin with and have only 3 keys initially:

• active (bool)
• defined-by (the name of the extension defining the tag, to be used in a system message as per T20670: Defined by the $1 extension)
• defined-by-msg (a message key whose content is similar to "Defined by the edit filter")

I don't think "importance" is really necessary. Without wanting to sound harsh, I worry that in general, you would like to make the change tagging system too complex and difficult to understand. If it's too had to understand, people will be afraid of it and won't use it properly, or at all.

It indeed makes sense to have 'active' as a sub-key since we might want to still pass params for inactive tags.

In T91535#1090058, @TTO wrote:

I don't think "importance" is really necessary. Without wanting to sound harsh, I worry that in general, you would like to make the change tagging system too complex and difficult to understand. If it's too had to understand, people will be afraid of it and won't use it properly, or at all.

I'm still convinced the only way to properly implement requests like "articles expanded from redirects should appear as in Special:NewPages" is to selectively enable recent changes patrolling for some tags. It doesn't have to be an integer param, we can do with a boolean 'patrollable' and have Special:ProblemChanges use the same. Or would 'needcheck' be more understandable ?

In T91535#1090058, @TTO wrote:

However, I would keep it simple to begin with and have only 3 keys initially:

• active (bool)
• defined-by (the name of the extension defining the tag, to be used in a system message as per T20670: Defined by the $1 extension)
• defined-by-msg (a message key whose content is similar to "Defined by the edit filter")

Interesting ideas, we definitely need to get the abuse filters applying a specific tag. And we should pass the params provided by the hook to the appropriate messages at SpecialTags.php.

Couldn't the message key be simply tags-description-extension-$extensionName though ? So we don't need to pass it too, just the extension's name.

We could do like this :

• tags-source-extension uses the $extensionName (provided by the hook)
• a default system message tags-description-extension-$extensionName is placed just below the generic tag description message
• numeric parameters are passed to this message from a param $refs provided by the hook (an array of numbers)
• extensions define their default tag description message and pass the $refs in the hook

For AbuseFilter, $refs would be the ids of filters applying the tags and tags-description-extension-AbuseFilter could contain Tagged by filter $1(hist · log). These would be removed from custom tag descriptions, so the final result should render exactly as it does now on enwiki.

Back on the subject of performance, I've noticed that every time the tag filter selector is built, a check !count( self::listDefinedTags() ) is made, cf line 266 of ChangeTags.php. I wonder why this is needed ? Wikis may have no currently defined tags but still have some previously applied (a check that would make more sense theoretically is of tagUsageStatistics). Getting the list of all defined tags and counting is inefficient (and tagUsageStatistics would be even worse). We shouldn't make this check, it's too expensive for what it's worth, incomplete, and we have a config variable $wgUseTagFilter for this.

Change 201905 had a related patch set uploaded (by Cenarium):
Use single hook to register tags and performance improvements

https://gerrit.wikimedia.org/r/201905

Change 218265 had a related patch set uploaded (by Cenarium):
Overhaul caching of tag usage statistics

https://gerrit.wikimedia.org/r/218265

Change 218776 had a related patch set uploaded (by Cenarium):
Don't include never-applied defined tags in tagUsageStatistics function

https://gerrit.wikimedia.org/r/218776

Change 218776 merged by jenkins-bot:
Don't include never-applied defined tags in tagUsageStatistics function

https://gerrit.wikimedia.org/r/218776

• https://gerrit.wikimedia.org/r/218776 (branch master): MW-1.26-release, WMF-deploy-2015-06-16_(1.26wmf10)

https://gerrit.wikimedia.org/r/#/c/218265/ and https://gerrit.wikimedia.org/r/#/c/201905/ still awaiting review for this task...

@aaron might be interested in some of the patches linked above, and at Cenarium's account at Gerrit. At the very least, he can say yay or nay as to whether the patch is solving an actual problem and is taking the right approach.

Some profiling of https://en.wikipedia.org/wiki/Special:Tags?forceprofile=1 :

100.00% 7745.827      1 - main()
 99.72% 7723.827      1 - MediaWiki::run
 99.71% 7723.675      1 - MediaWiki::main
 97.81% 7576.048      1 - MediaWiki::performRequest
 97.58% 7558.297      1 - SpecialPageFactory::executePath
 97.57% 7557.402      1 - SpecialPage::run
 97.53% 7554.324      1 - SpecialTags::execute
 97.51% 7553.103      1 - SpecialTags::showTagList
 77.53% 6005.335    559 - DatabaseBase::select
 76.86% 5953.625    559 - DatabaseBase::query
 76.32% 5911.841    560 - section.DatabaseBase::query
 74.98% 5808.055    567 - DatabaseMysqli::doQuery
 74.21% 5747.997    336 - WANObjectCache::getWithSetCallback
 71.43% 5532.785    107 - ChangeTags::tagUsageStatistics
 71.22% 5516.503      1 - ChangeTags::__invoke
 71.12% 5509.073      1 - section.query: SELECT ct_tag,count(*) AS hitcount FROM `change_tag` GROUP BY ct_tag ORDER BY hitcount DESC  [TRX#f8ff7465a544]
 24.98% 1935.126     97 - SpecialTags::doTagRow
 16.19% 1253.704   1232 - Message::toString
 13.97% 1082.083    197 - Message::parse
 13.72% 1062.707    200 - Message::parseText
 13.69% 1060.455    200 - MessageCache::parse
 13.53% 1047.728    201 - Parser::parse

If the statistics table was in ObjectCache::getMainStashInstance() instead of the WAN cache, it could be updated via the jobqueue (a simple de-duplicated job).

listExplicitlyDefinedTags/listExtensionDefinedTags need a process cache, which I'll just make a patch for.

Change 248281 had a related patch set uploaded (by Aaron Schulz):
Fix slow callbacks in getWithSetCallback() using lockTSE

https://gerrit.wikimedia.org/r/248281

Change 248281 merged by jenkins-bot:
Fix slow callbacks in getWithSetCallback() using lockTSE

https://gerrit.wikimedia.org/r/248281

In T91535#1707837, @TTO wrote:

@aaron might be interested in some of the patches linked above, and at Cenarium's account at Gerrit. At the very least, he can say yay or nay as to whether the patch is solving an actual problem and is taking the right approach.

The problem as I see it: at the moment, on enwiki Special:Tags takes about 7-8 seconds to load. It's a lot already for a special page, but it's not all.
We need a list of tags for drop down menus in recent changes, contributions, etc. Those pages load almost instantly, so we really need to increase performance.

The cache is constantly invalidated by the frequent tags mobile edit, visual editor, etc. The db queries are slow and we can't do much about that.
With 218265, tags with more than a given number of hits will no longer cause a cache purge when updated.
So the cache will be used much more often (at the expense of a slight delay in updates for tags with many hits).
For drop down menus, adding another cache further reduces the probability of a db query being required.

In T91535#1707959, @aaron wrote:

listExplicitlyDefinedTags/listExtensionDefinedTags need a process cache, which I'll just make a patch for.

This is done in 201905 (in the ChangeTagsContext class).

Change 218265 abandoned by Cenarium:
Overhaul caching of tag usage statistics

Reason:
merged back into https://gerrit.wikimedia.org/r/#/c/211497

https://gerrit.wikimedia.org/r/218265

Change 334337 had a related patch set uploaded (by Cenarium):
Add change_tag_statistics table to speed up tag hitcount retrieval

https://gerrit.wikimedia.org/r/334337

It takes several minutes to load Special:Tags at wikidata, the way we're retrieving hitcounts is not efficient or sustainable.
It's also a concern for the dropdown (T27909) on cache misses.
My suggestion, implemented in the above patch set, is to create a table holding tag hitcounts, updated when adding/removing tags.

Looks like this is broken since January 31, see here.

Adding this to the Wikidata radar, since this is a scalability problem for us. It's not super urgent since it's not critical functionality, but it would certainly be nice if this would work.

Lydia linked me here. In my current draft for a living persons policy (https://www.wikidata.org/wiki/Wikidata:Living_persons_(draft)) I have a portion that suggests that for every bot edit that a bots that doesn't have a specific "living persons" flag, the bot is supposed to check whether the item is about a living person and if so check whether the property that gets added subclasses Q44597997 or Q44601380.

I don't want to write policy that produces major scalability problems. Do you think it's doable to change the software in a way that such a check isn't a performance problem or would you recommend to skip that part of the policy?

In T91535#3838530, @ChristianKl wrote:

I have a portion that suggests that for every bot edit that a bots that doesn't have a specific "living persons" flag, the bot is supposed to check whether the item is about a living person

It's not obvious that this has to do with change tagging. As far as I know, checking whether a particular edit has a certain change tag attached to it is a very fast operation, thanks to the indexes on the change_tag table. Is that what you are referring to?

I'm not sure why Lydia linked to this task. If you think that the proposed check causes no performance that would be great. If it produces other performance issues it would be good to know.

Change 334337 abandoned by Ladsgroup:
Add change_tag_statistics table to speed up tag hitcount retrieval

Reason:
Done through T185355

https://gerrit.wikimedia.org/r/334337

Change 201905 abandoned by Umherirrender:

[mediawiki/core@master] Unify hooks and performance improvements for change tags

Reason:

Old patch set, linked bug already resolved

https://gerrit.wikimedia.org/r/201905