If it's waiting on DBA signoff, to make it clear that it looks good on our side. Just make sure: 1- to catalog it in table catalog 2- avoid creating it on all wikis (I think it'll be only non sul wikis + one central wiki).

For tech news:

There is a new special page listing templates that trigger the largest number of lint errors. For example: [[Special:LintTemplateErrors/night-mode-unaware-background-color]]

In T170874#11592180, @Ladsgroup wrote:

This should fix it. See SpecialMostLinkedCategories for comparison of query info. I‌ want to test it locally but composer is not cooperating right now.

I was running update special pages in beta cluster for a different thing and this exploded:

Wikimedia\Rdbms\DBQueryError from line 1226 of /srv/mediawiki-staging/php-master/includes/libs/Rdbms/Database/Database.php: Error 1054: Unknown column 'file_name' in 'field list'
Function: MediaWiki\Specials\SpecialUnusedImages::reallyDoQueryInternal
Query: SELECT  6 AS `namespace`,file_name AS `title`,fr_timestamp AS `value`  FROM `image`,`page` LEFT JOIN `categorylinks` ON ((cl_from = page_id)) LEFT JOIN `imagelinks` ON ((il_to = page_title))   WHERE il_target_id IS NULL AND file_deleted = 0 AND page_namespace = 6 AND cl_from IS NULL AND (file_name = page_title)  ORDER BY value LIMIT 5000

This should fix it. See SpecialMostLinkedCategories for comparison of query info. I‌ want to test it locally but composer is not cooperating right now.

I have reverted the rate limit for "medium" browser score before the weekend to reduce disruptions to people. We have collected enough data points to know what to fix next but once we re-enable it and you see it again, make sure to get 1- The URL you're getting the error 2- The response body. Thanks!

Can you share your ldap username? I'll add you in horizon.

I‌ think this should fix it: https://gerrit.wikimedia.org/r/c/operations/mediawiki-config/+/1237517 but I‌ need to get an opinion of an expert in MediaViewer on it before merging.

This image is in a standard size and passes through our rate limit: https://upload.wikimedia.org/wikipedia/commons/thumb/8/8f/Eclectic_building_%28_1886%29._Bem_quay_side._Facade._-_15_F%C5%91_Street%2C_Budapest.JPG/1280px-Eclectic_building_%28_1886%29._Bem_quay_side._Facade._-_15_F%C5%91_Street%2C_Budapest.JPG

In T414805#11589974, @Tacsipacsi wrote:

MediaViewer got broken by this. ☹ For example, https://commons.wikimedia.org/wiki/Category:F%C5%91_Street_15_(Budapest)#/media/File:Eclectic_building_(_1886)._Bem_quay_side._Facade._-_15_F%C5%91_Street,_Budapest.JPG generates an HTTP 429 on my 1920×1088px viewport. Is there a task specifically tracking that extension?

My guess is some sort of scraper. I‌ haven't looked into it though.

Since you might want to have more target per event eventually. This looks good to me.

Two notes:

• Maybe add unsigned to give yourself more breathing room.
• If you have the user as unique key, why not making it PK directly and avoid having an extra auto_increment id? See T411433#11584647 onwards

In T411433#11585235, @Daimona wrote:

In T411433#11584647, @Ladsgroup wrote:

Hi, no concern from my side but do you want to make the index unique so it would have one row per event only? It seems nicer but not a big deal.

The uniqueness constraint would hold in the very initial version, but we have plans to allow multiple goals per event in a future version.

If that's the case, you can also simply make the event as PK. No need to set up a dedicated auto_increment field just for counter.

Generally speaking, I'm curious about this. I seemed to recall a recommendation of always having a surrogate primary key (at least for maintenance reasons, don't remember if there was more), is that actually not the case?

In T416578#11587004, @Marostegui wrote:

It is very likely a case of changing it from core to misc on the zarcillo table

Having SELECTs happening on master is not unexpected per se. Usually it's the code path that is going to do a write immediately after and wants to get the most updated data to avoid race conditions. But this is acceptable as long as 1- The volume is low 2- They are properly indexed.

Running updateSpecialPage on beta cluster consistently fatals with this:

Wikimedia\Rdbms\DBQueryError from line 1226 of /srv/mediawiki-staging/php-master/includes/libs/Rdbms/Database/Database.php: Error 1366: Incorrect integer value: 'Template:Delete' for column `aawiki`.`querycachetwo`.`qcc_namespace` at row 1
Function: MediaWiki\Linter\SpecialLintTemplateErrors::recache
Query: INSERT INTO `querycachetwo` (qcc_type,qcc_namespace,qcc_title,qcc_value,qcc_titletwo) VALUES ('LintTemplateErrors','Template:Delete','Template:Delete',1,'night-mode-unaware-background-color')

I‌ think I‌ have admin rights and I‌ think I‌ deleted the record. I‌ keep a copy here in case I‌ break things:

platform_eng/dags/querypage/querypage_most_categories_monthly_dag.py
{
  "wikis_to_run": [
    "testwiki",
    "frwiki"
  ],
  "hive_querycache_table": "wmf.querycache",
  "hive_categorylinks_table": "wmf_raw.mediawiki_categorylinks",
  "hive_linktarget_table": "wmf_raw.mediawiki_private_linktarget",
  "wmf_raw_tables_path": "hdfs://analytics-hadoop/wmf/data/raw/mediawiki/tables",
  "wmf_raw_private_tables_path": "hdfs://analytics-hadoop/wmf/data/raw/mediawiki_private/tables",
  "hdfs_destination_dir": "hdfs://analytics-hadoop/wmf/data/published/datasets/querypage/MostCategories/",
  "temporary_directory": "hdfs://analytics-hadoop//tmp/platform_eng/querypage/MostCategories/",
  "monthly_dag_start_date": "2025-12-01T00:00:00",
  "dag_hql": "hdfs://analytics-hadoop/wmf/refinery/current/hql/querypage/MostCategories.hql",
  "dag_sla": "P10D",
  "alerts_email": "platform-eng-alerts@wikimedia.org"
}

In T358266#11550578, @Ladsgroup wrote:

There are requests to vp9 on an iOS 26 UA‌ but the number is basically peanuts: https://w.wiki/HaJS i.e. 5%‌ of hits to mjpeg (of course the partial requests could be the reason for this discrepancy)‌

Hi, no concern from my side but do you want to make the index unique so it would have one row per event only? It seems nicer but not a big deal. If that's the case, you can also simply make the event as PK. No need to set up a dedicated auto_increment field just for counter.

Regarding 2 and 3, it'd be T373546

In T198628#11552043, @New_York-air wrote:

@Ladsgroup Wow, thank you so much for having a look into this!
I can't imagine that it's just a quick fix noone has been able to do for years!

That was the logging, the code I pushed last night is live (https://gerrit.wikimedia.org/r/c/mediawiki/extensions/GrowthExperiments/+/1236352)

Thanks! I think it is something else causing issues and we found this on the way that shouldn't happen regardless. It happens quite a lot when I'm debugging a production issue. I investigate why that is the case but the current state is not helping :D

I‌ confirm that the values written in the binlogs are much smaller and even when I‌ decoded a couple of large ones, nothing had zeros in it. I'm not seeing a major drop in binlog sizes which I think it's mostly because we are RBR‌ so the previous value is still written heavily on the binlogs, waiting for it to actually kick in. It's going to take a while.

And more like that:

Sending data                                                  | SELECT /* GrowthExperiments\UserImpact\ComputedUserImpactLookup::getCreatedArticleCount  */  log_pag |    0.000 |

I'm seeing this:

| 823987454 | wikiuser2023    | 10.194.182.235:41778 | idwiki       | Query       |        0 | Sending data                                                  | SELECT /* GrowthExperiments\UserImpact\ComputedUserImpactLookup::getCreatedArticleCount  */  COUNT(* |    0.000 |

@Sgs I found the bug: ExpensiveUserImpact::filterViewCounts() filters based on array of ['key' => value] e.g. dates. That works in dailyTotalViews key which is fine but it skips all zeros in dailyArticleViews because it tries to remove zeros from array of ['key' => array] which won't work. I try to make a patch plus regression testing ASAP.

Also if you check UA, most logs are simply from "MediaWiki/1.46.0-wmf.13" or "ChangePropagation/WMF". Can we sample these?

Go for it.

In T384166#11565913, @xcollazo wrote:

Since the vast majority of analytic jobs are scheduled via Airflow, we could certainly change the schedule cron definitions of each DAG to achieve this, but it would require careful planning, as today we rely heavily on the @daily alias, which makes everything run at midnight UTC.

In T415792#11565456, @Ladsgroup wrote:

Something to consider: You can get rid of 1.5x IMHO. It is removed from mediawiki from what I'm seeing. One less thing to adjust.

Hi, It would be great to priotize this. logstash is quite under pressure T390215: Logstash is overwhelmed and we are also pretty close to our limits of electricity consumption in eqiad. It is fine if something is needed and we (SRE) would accommodate (buying more power, more hardware) but if something is not needed, dropping it can help us a lot. Thank you!

Something to consider: You can get rid of 1.5x IMHO. It is removed from mediawiki from what I'm seeing. One less thing to adjust.

To double check, should it be background-size: cover;?

why not simply using https://upload.wikimedia.org/wikipedia/commons/thumb/6/61/Contribs_icon-black.svg/20px-Contribs_icon-black.svg.png Is 1px causing issues?

Thanks! I‌ will let the dev team know about this.

I can try to optimize the tables there. Is it only happening on that host in all of logstash logs?

A quite possible scenario is that a bug that got deployed during that time inadvertently brought back the 0s in the module. I also wouldn't really call it micro-optimization as checking one example, it cut the size of the module to half.

Filed a L2SC request: https://app.asana.com/1/3758245663860/project/1201320945652311/task/1212944194261410?focus=true

In T412630#11553524, @Jonesey95 wrote:

It would be great if everyone could stop referring to these Linter issue lists and tables using the word "category", since that word has a specific technical meaning on Wikipedia. Linter does not cause any sort of categorization.

It would also be great if Linter issues could actually assign pages to categories so that they would be more noticeable to editors and easier to find using Search, but that is a separate, long-standing ticket.

I am unclear what this ticket proposes. If the dark mode Linter table goes away, for example, does that mean that there would be no reliable way to find pages with the syntax that currently causes pages to be tagged with that issue on their Page information page?

I would support removal of the "large table" issue entirely; it was worth a try, but it is not something that can or should be fixed on many pages, and there have always been too many false positives.

I would also support removal or refinement of the dark mode Linter issue, since it has so many false positives. If it can be refined to highlight only issues that actually cause problems in dark mode, I think the table would be a lot smaller.

Please prioritize mitigating this. Disk write on x1 on average is now 20 times as disk writes of s1. This is quite high risk and can easily cause large-scale issues if combined with other issues (loss of redundancy, spike of write in other areas, etc.)

In T414080#11553207, @Michael wrote:

In T414080#11520193, @Ladsgroup wrote:

Oh based on grafana it was either 19-06-2025 or 19-08-2025: https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&refresh=5m&var-server=db1179&var-datasource=000000026&var-cluster=mysql&viewPanel=panel-28&from=now-1y&to=now&timezone=utc

Mh, that would point to GrowthExperimentsUserImpactUpdater: Support temporary and non-special-homepage accounts as the likely culprit, merged on Monday the 16th of June. This would likely not have increased the size of an individual user-impact blob, but it likely massively increased the number of user impact blob writes due to writing them for all those temporary accounts. (cc @kostajh)

The URL to our thumbnails is not an stable API‌ and shouldn't be treated as such. The actual APIs return URL to turn to nearest larger standard size and if the programmer really cares, they should use that instead of hard-coding the URL. For example https://commons.wikimedia.org/w/api.php?action=query&titles=File:Albert%20Einstein%20Head.jpg&prop=imageinfo&iiurlwidth=100&iiprop=url

I‌ love the idea. One thing we could also try to do is to instead of looking at views which is not directly available in our mediawiki env, we can look at whether it's been used in main namespace of any other wiki so if a video is used in an article in German Wikipedia, it would get the "high-compression" treatment. Easier to check as well.

There are requests to vp9 on an iOS 26 UA‌ but the number is basically peanuts: https://w.wiki/HaJS i.e. 5%‌ of hits to mjpeg (of course the partial requests could be the reason for this discrepancy)‌

In T413031#11549703, @bvibber wrote:

IIRC here's the breakdown you should see on Apple:

Mobile devices:

• iOS/iPadOS 17.4 or later: WebM VP9 if there is a hardware VP9 codec in the device, otherwise WebM VP8
• iOS/iPadOS before 17.4: MJPEG .mov

I think that's a bug and needs investigation.

I think this is done. The actual run for this year is happening in another ticket.

The reduction in update frequency is actually intentional: T323495 if some system is built to clean up the old data. We can increase the frequency again. It shouldn't be too hard but FR‌ is unowned

The thing is that I got my hands on a recent iPhone and tested it and it was playing and preferring VP9 so it works in some circumstances at least. I‌ couldn't check the UA but I'd be very surprised if the referrer is stripped out. It doesn't make sense.

Ah, virtually all of the requests for mjepg don't have any referrers and the UA is "AppleCoreMedia/1.0.0.xxx (iPhone; U; CPU OS 26_2 like Mac OS X; en_us)" and similar UA parsers say this is "Apple podcast app" which I‌ have no idea what it is.

Cross-referencing T413031#11547604, it's hard to see exactly what number of requests are coming to mjpeg 144px since we are sending out a lot of partial responses (HTTP‌ status 206). But it still it looks like we are serving a LOT of mjpegs and better to switch it to something slightly better?

Ah it was 206 response code:

I need to write some important code so naturally I decide to procrastinate. I‌ looked at the number of different transcodings being requested. Since the URI‌ path is different between different transcodings, the resulting table is weird but you get the idea:

spark-sql (default)> select reverse(split(reverse(uri_path), '[.]')[1]) as encoding, reverse(split(reverse(uri_path), '[.]')[2]) as transcode_size, count(*) as hitcount from wmf.webrequest where year = 2026 and month = 1 and day = 10 and uri_path like '%/transcoded/%' and webrequest_source = 'upload' and content_type like 'video%' and http_status = 200 group by encoding, transcode_size order by hitcount desc limit 50;
encoding        transcode_size  hitcount
vp9     480p    754204
vp9     1080p   91851
360p    ogv     76654
vp9     240p    62416
360p    webm    40899
vp9     720p    13485
mjpeg   144p    11446
vp9     360p    10203
360p    ogg     3230
360p    mpg     132
vp9     1440p   124
vp9     2160p   94
*more weird uris*
Time taken: 54.366 seconds, Fetched 21 row(s)

FTR, the PCS‌change is now deployed, caches need to expire but anything from PCS‌ should be serving standard sizes instead. Should we close this?

To emphasize: MediaSearch does respond with standard sizes but the js uses a config that should be used.

In T410630#11536354, @Neslihan_Turan_WMDE wrote:

French wiki has some high numbers accessing all statements: https://phabricator.wikimedia.org/P87630
This could be related with Lua investigation T403008 or this could be something else.

Blocked on T415219: Privacy review of x1 tables in preparation of adding them to wikireplicas

\o/ One thing to note: In one place in one API endpoint, the query has where condition on fr_archive_name only which is not indexed. I had a mental note to deal with it later which I guess it is now :D we have two options: 1- check if that condition is really needed and maybe just disable/remove it 2- Add an index 3- Add an index for now, then deprecate and remove it later. I leave the decision to you :P

Yeah, to me this feels a bit of x/y problem. A proper solution would be having an OLAP infra which would also include cross-wiki queries and historical queries (going back to snapshots in time) and this too.

In T348255#11538572, @Ladsgroup wrote:

Thinking about it, I'm also rethinking whether we should enable the storage of postprocess on Commons and Wikidata at all. They are usually the majority of the storage in PC hosts given that each have roughly 150M pages. I wonder if it's okay not to ever enable postprocess cache there and let them take the performance hit (as trade-off between storage and CPU) given that wikidata doesn't even use parsoid for majority of pages and commons with the multilingual setup can explode in number of caches.

Sorry I couldn't comment earlier. SRE stuff. Anyway. In principle, if the post-processed entry is "enough" on retrieval and wouldn't trigger a reparse if the main entry doesn't exist, then we don't have a depool causing issues problem. But my main worry is sneaky and hard-to-detect bugs that could trigger such reparses en masse. So as a defense in depth mechanism, I still think it's a good idea to force locality by moving the "postprocess-pcache" part of the key to after the hash stop.

I've collected and analyzed requests to non-standard thumbnail sizes: 2/3rd are medium browser scores. https://docs.google.com/document/d/1mvStefN4aNmvx7swhY7BlhMaoQI5NOZj1qjSCP7Ga0g/edit?tab=t.0 several actors account for majority of these (some universities, etc.)

As long as they are a shared table in x1 I have no objections.

Yup

Collab wiki also exists. I honestly don't think this is a good idea.

\o/ Thank you!

In T198628#11532393, @TheDJ wrote:

In T198628#11532177, @Ladsgroup wrote:

One thing to note that SRE has a vested interest to see MPEG-DASH deployed in our production (T373546) which would mess up with our stats in a different way (it will count actual video plays but it will count them multiple times depending on how long the user have watched it.) but we can probably find someone with SQL skills to group the webrequest rows. Future-me problem.

Noting that part of the reason the script was made, is explicitly because people want to know actual view time, not just play start.

This is somewhat easy to fix without needing to load js and doing tracking. Someone should have looked at the data in depth. Let me give you an example: Top video in https://mvc.toolforge.org/index.php?category=Videos_by_Terra_X&timespan=now-90&rangestart=&rangeend=&limit=100 which gets 400K views (!) is this video: https://commons.wikimedia.org/wiki/File:Taj_Mahal_-_Mausoleum_der_Liebe_(CC_BY-SA_4.0).webm

Many times when we want to remove code, we get the sentence that "it's not much code" and that can be true for one place or one extension but they add up because we have many places like that. Just a couple days ago I saw the exact same thing somewhere else: T412819#11466909 so I won't repeat myself but we have 3M line of code in production (just mediawiki) and we, the devs, cannot reasonably maintain this ocean of tech debt.

In the light of vector 2022 being rolled out, I would like to revisit this. I know some users still use non-compat version but the benefit of sorting in a nice way doesn't outweigh the cost of maintaining this extension in production (and translations, etc.). I'm sure these users can live with alphabetical sorting of the language code.

The post-processing is before hash stop :( this will cause issues

| ruwiki:pcache:2101895:|#|:idoptions                                                                         |
| thwiki:postproc-parsoid-pcache:19490:|#|:idhash:injectTOC=0!postproc=1!skin=vector-2022!useParsoid=1        |
| thwiki:postproc-parsoid-pcache:19490:|#|:idoptions                                                          |
| commonswiki:pcache:74334962:|#|:idhash:wb=3!wbMobile=0                                                      |
| commonswiki:pcache:74334962:|#|:idoptions                                                                   |
| enwiki:parsoid-pcache:24603480:|#|:idhash:useParsoid=1                                                      |
| enwiki:parsoid-pcache:24603480:|#|:idoptions

So it can't work with imported images because img_actor is set to zero. I'm going to just delete the row from the db when it's imported.

Wikis with more than 1M dark mode lint errors:

arwiki
5572177
--
cebwiki
4333563
--
commonswiki
17224290
--
dewiki
1925572
--
dewikisource
1018570
--
enwiki
13851902
--
enwiktionary
1710670
--
eswiki
3296663
--
fawiki
1245541
--
frwiki
3842180
--
hiwiki
1241293
--
huwiki
1340336
--
hywiki
1165824
--
idwiki
8181366
--
incubatorwiki
2522384
--
itwiki
17938235
--
jawiki
5026805
--
kowiki
2128614
--
mswiki
1775274
--
ptwiki
5377732
--
ruwiki
1472063
--
ruwikinews
4010311
--
shwiki
2073617
--
srwiki
2051348
--
svwiki
2996301
--
trwiki
4208663
--
ttwiki
1132662
--
ukwiki
3811945
--
viwiki
9811904
--
warwiki
1255387
--
zhwiki
10915653