In T362937#11457727, @SD0001 wrote:

So that the data can be used alongside other parser capabilities, like Scribunto's SVG support to generate graphs. It could also be used with Charts, which I now see has a ticket for this use-case: T393500, which would be solved by this patch

We could put the deployments in a shared table, with wiki ID columns.

In T412337#11454743, @Ladsgroup wrote:

I haven't had time to review the proposed schema in depth (I'm in an offsite until next week) but to get the discussion starting, the number of tables per-wiki is still too many[1]. I wonder if the table with one row only can be merged into something else. I don't think these tables will grow much so I'd lean towards de-normalizing as much as possible.

[1] in s3, MariaDB process needs to open many many files because of combination of many wikis and many tables and each table creating two files. Adding three tables to every wiki adds around 6000 more files to be opened by the process which impacts the performance the database.

Why get the parser involved? Why not load from the pageviews API on the client side?

It's reproducible by just putting sleep(1) after the proc_terminate().

In T410527#11399132, @Taylor wrote:

Reportedly LUA can use 50 MiO RAM, quite generous. But if 7'000 pages hog 50 MiO, then every single page consumes ca 7 KiO. Does it really need so much? I could try to call "mw.title.newBatch(...)" in batches of 25 pages, and copy the relevant info from bloated title objects into a LUA table of booleans.

In T279736#11427420, @Nardog wrote:

The "accept" part is resolved, but the "suggest" part is not.

In T355763#9484918, @wmr wrote:

I can't read PHP. ChatGPT says: In this part of the code, if $isEncoding is true and the key $k is a string containing only digits (checked using ctype_digit), it converts the string to an integer using (int)$k. This is done to handle cases where JSON decoding may result in numeric keys represented as strings. The function then checks if the converted integer key forms a sequential sequence.

Is this intended? Is this a bug? If this is intended, can you add an option to allow no conversion like mw.loadJsonData("file.json", digit_conversion=False).

This appears to be fixed.

In T344001#9084731, @Theknightwho wrote:

It's a last resort measure due to the major problems we've been having at the English Wiktionary with memory limits.

In T260838#10124651, @Dreamy_Jazz wrote:

15 of these in the last week. I think we can solve this using locks.

In T396168#11408696, @TheDJ wrote:

Shall we make a new subtickets for the JSON-LD part of this, so that this ticket is fully wrapped up ?
This part is now split off into T411108: Add VideoObject JSON-LD to video File pages

In T408894#11408083, @Samwilson wrote:

Error 1055: 'mediawiki_wiki1_46.recentchanges_actor.actor_user' isn't in GROUP BY

The sitemap has only been submitted direct to Google, it's not in robots.txt and it hasn't been submitted to Bing. SRE are afraid that adding it to robots.txt will cause too much crawler traffic.

In T279736#11414054, @Izno wrote:

This task looks like it might have adjusted the vertical spacing unintentionally; the preview line was more attached to the form below it than the form above it before today's deploy.

There are also two world-writable directories: /etc/wmerrors and /etc/php/8.3/fpm/env. They have the potential for escalation and need to be fixed. I haven't fully isolated this but it seems to be related to helmfile. The deployment chart declares files in these directories, and the directories are created with incorrect permissions.

Awesome

Wishathon project plan

There are two implementations of this feature: one for core EditPage and one for VE. Both use TitleInputWidget, which I think will need to be subclassed. Compare MWTemplateTitleInputWidget in VE. We could override getApiParams to add prop=templates, tltemplates=....

You can already use rcprop=oresscores to serve the linked use case. Example API response:

@Joe do you have any comments on the above? I think you were involved in requirements development for this task. I would like to get a more nuanced view of what kinds of namespaces should be excluded and why.

$wgExtraNamespaces has 1705 namespaces, total for all wikis. Of those, 246 are in $wgNamespaceRobotPolicies and are presumably for internal community use.

From the point of view of image size and memory usage, it might be better to merge all the languages into a single file, and combine identical values by having a numerically indexed global value array. Maybe also abbreviate the message keys by mapping them to integers. We can tolerate a little bit of indirection at runtime.

The point of the script is to report mediawiki.GrowthExperiments.tasktype_count.* metrics. I see this code was updated as recently as December 2024 by @Sgs and @Michael as part of T374050, although I can't find a dashboard in Grafana where these metrics are shown.

Still some cleanup to do

An average over the past 3 days, compared to a week earlier, suggests this intervention reduced slow queries (in terms of total duration per day) by 59% for Watchlist and 77% for RecentChangesLinked.

Watchlist slow query count -- deployment was at 23:35

I made a Grafana dashboard for the relevant metrics: https://grafana.wikimedia.org/goto/jL2bYqRDg?orgId=1

InvalidArgumentException is the wrong exception class for validating data from the database, that is for programmer errors. It should have a special subclass of RuntimeException so that it can be safely caught by all callers. I see RevisionStore is catching this exception at lines 1622 and 1699, but not at lines 1752 or 3236. An exception should either be caught or not, there's no rationale for conditionally catching it.

There is a cron job running listTaskCounts.php --topictype ores which makes a TopicDecorator with $useOresTopics=true, so TopicDecorator::loadTopics() returns the list of topics from ArticleTopicFeature::TERMS_TO_LABELS. The array access on lines 146, 147, 149 and 152 could be properly guarded, but I wonder if the script is still doing a useful thing after refactoring in this area e.g. T393872.

In T407127#11303775, @HCoplin-WMF wrote:

Could you clarify what you mean about the meta robots tag?

Did this actually do anything? The linked documentation seems to imply that max-image-preview: standard is the default, so just sending that tag on every page should have no effect.

In T404416#11298388, @Kgraessle wrote:

Translate filter:

In T407087#11312721, @Kgraessle wrote:

The first is when you have negative namespace values, I'm not seeing the results being filtered by that namespace and I'm seeing a warning returned:

This happens for both Special and Media namespaces.

On db2202 I created the cross-join table watchlist_label_member_T406843 using the definition from the task description. Then I populated it with some rows as follows:

In T407087#11303517, @Kgraessle wrote:

I was testing the API endpoint locally and I did find one minor issue where bot edits were coming back from the API even with the !bot flag set:

In T406843#11305035, @Samwilson wrote:

Oh, that makes sense. So basically it's adding a wlm_list column to watchlist?

In T406843#11304995, @Samwilson wrote:

Has there been any discussion of being able to expire labels? That seems like a new feature that no one's asking for.

For the other special pages, we can only expect a performance advantage when a table other than recentchanges is first in the join order. Most Special:RecentChanges slow queries are due to ORES, which is not suitable for placing first in the join order and thus there is no advantage. I checked the EXPLAIN for a few other RC queries, but they were all putting recentchanges first.

I tested the watchlist in the task description by calling SpecialWatchlist::getRows() from eval.php. The first time with no partitioning, the main query took 17.5 seconds. The second time, after warmup, it took 3.4 seconds. With partitioning, there were two queries for a total query time of 0.19 seconds.

In T406843#11303639, @Cparle wrote:

@tstarling just want to make sure I understand - the main argument for "copy" instead of "cross-join" is to build the label into the indices to make filtering faster, is that correct?

In T408057#11301470, @A_smart_kitten wrote:

cc @tstarling FYI, potentially related to (or is) T408040?

This is done. However, I could not reproduce any performance improvement. The database does actually optimise a not-equals condition, so the index usage doesn't change.

Now seeing

Error 1054 from MediaWiki\RecentChanges\ChangesListQuery\ChangeTagsCondition::isDenseTagFilter, Unknown column 'changetagdisplay.ct_rc_id' in 'WHERE' EXPLAIN SELECT  *  FROM `change_tag`    WHERE (changetagdisplay.ct_rc_id >= 7363925) AND changetagdisplay.ct_tag_id = 31   db1159

testwiki with days=30, limit=20:

Daniel points out that watchlist expiry is awkward if you want an expiry time to apply to all labels on a page but we_item links to wlm_id.

I added wld_count for limiting the size of a watchlist. The page move REPLACE query is a potentially tricky detail.

Thanks everyone. I guess it's resolved? Please reopen if there's something left to do.

There's no index which would allow efficient autocomplete prefix searches on pages that use a given template. But we could apply a CSS class to items in the menu depending on whether the page uses the template. If the number of pages using the template is small (<200) then there could be a separate mode where we only suggest those pages, although the mode switch might be surprising to users.

In ApiQueryRecentChanges, there is a "slot" parameter which does essentially what is requested, but it's inefficiently implemented, so I don't want to see that solution extended to RecentChanges highlighting.