I did a "bash" one-liner so bad I'm proud of it:

root@db1169:~# find /srv/sqldata/ -size +10G -exec du -h {} \; | cut -f2 |  python3 -c "import sys, re;print(re.sub('/srv/sqldata/([^/]+?)/([^/]+?)\.ibd', r'\1.\2', ''.join(sys.stdin)))" | grep '\.' | xargs -I{} bash -c 'echo {}; mysql -e "show create table {};"' | grep -v COMPRE | grep -v "Create Table"
<string>:1: SyntaxWarning: invalid escape sequence '\.'
--------------
show create table /srv/sqldata/db1052.err
--------------

That would be all of issues in small.dblist wikis. Now running medium wikis.

LGTM, I ask to drop the indexes once you're done with it. Even though it's small. Let's add them when they become needed.

If this is about RecentChangesLinked: I think the only solution we can do is to query templatelinks/imagelinks/etc from the requested page with limit of 1000 and then query rc table with condition of rc_page IN (...). Meaning for most cases it'll still continue to work as is but it'll break for more heavily used pages.

@Kgraessle do you need the ores tables? like ores_models and ores_classification?

User notice suggestion:

URLs to other wikimedia projects won't be registered as external links meaning they won't be recorded on externallinks table, won't be searchable in Special:LinkSearch and won't be checked against spam blacklist or abuse filters added_links variable. This is to reduce the undies load on the databases abs improve editing save time. Also a limited number of trusted websites that are heavily used might be added to each wiki individually. For example Creative Commons website will be added to the ignore list in Wikimedia Commons due to large number of links to their license pages.

Compared to another host that isn't optimized yet:

root@db1175:/srv/sqldata/ruwikinews# ls -Ssh
total 256G
130G externallinks.ibd
 31G categorylinks.ibd
 22G page_props.ibd
 16G logging.ibd

FTR:

root@db1157:/srv/sqldata/ruwikinews# ls -Ssh | head
total 81G
 17G externallinks.ibd
 14G logging.ibd
9.7G categorylinks.ibd

ah okay, I leave it now. FWIW it's inode:

ladsgroup@codesearch9:~$ df -i | grep -i srv
/dev/sdb       5242880 5242879       1  100% /srv

Well, I can't even ssh into the host to check what's going on 😢

In T375981#11430509, @cscott wrote:

I think having two thumbnail preferences ("normal" and "large/accessible") might solve most issues, although we'd still have the complexity in core of serving HTML with two different sizes. I haven't seen a discussion of responsive images yet: in many cases even though the screen px is fixed, our HTML mentions thumbnails in 3 or so different resolutions based on the screen magnification factor.

I'm seeing the check for archive failing sometimes with this:

Wikimedia\Assert\ParameterAssertionException from line 72 of /srv/mediawiki/php-1.46.0-wmf.4/vendor/wikimedia/assert/src/Assert.php: Bad value for parameter $page: must represent a proper page
#0 /srv/mediawiki/php-1.46.0-wmf.4/includes/Revision/RevisionArchiveRecord.php(80): Wikimedia\Assert\Assert::parameter(false, '$page', 'must represent ...')
#1 /srv/mediawiki/php-1.46.0-wmf.4/includes/Revision/RevisionStore.php(1639): MediaWiki\Revision\RevisionArchiveRecord->__construct(Object(MediaWiki\Title\Title), Object(MediaWiki\User\UserIdentityValue), Object(MediaWiki\CommentStore\CommentStoreComment), Object(stdClass), Object(MediaWiki\Revision\RevisionSlots), false)
#2 /srv/mediawiki/php-1.46.0-wmf.4/includes/Revision/RevisionStore.php(2038): MediaWiki\Revision\RevisionStore->newRevisionFromArchiveRowAndSlots(Object(stdClass), Object(MediaWiki\Revision\RevisionSlots), 0, Object(MediaWiki\Title\Title))
#3 [internal function]: MediaWiki\Revision\RevisionStore->MediaWiki\Revision\{closure}(Object(stdClass))
#4 /srv/mediawiki/php-1.46.0-wmf.4/includes/Revision/RevisionStore.php(2017): array_map(Object(Closure), Array)
#5 /srv/mediawiki/php-1.46.0-wmf.4/maintenance/findBadBlobs.php(287): MediaWiki\Revision\RevisionStore->newRevisionsFromBatch(Object(Wikimedia\Rdbms\MysqliResultWrapper), Array)
#6 /srv/mediawiki/php-1.46.0-wmf.4/maintenance/findBadBlobs.php(216): FindBadBlobs->loadArchiveByRevisionId(10451, 34930298, 1000)
#7 /srv/mediawiki/php-1.46.0-wmf.4/maintenance/findBadBlobs.php(128): FindBadBlobs->scanRevisionsByTimestamp()
#8 /srv/mediawiki/php-1.46.0-wmf.4/maintenance/includes/MaintenanceRunner.php(696): FindBadBlobs->execute()
#9 /srv/mediawiki/php-1.46.0-wmf.4/maintenance/run.php(53): MediaWiki\Maintenance\MaintenanceRunner->run()
#10 /srv/mediawiki/multiversion/MWScript.php(221): require_once('/srv/mediawiki/...')

There is only 31 pages left in that category.

\o/

I beat my usual drum 😅 : It all should be under one "depool" cookbook.

In T411497#11426803, @Marostegui wrote:

Thank you both. After @Krinkle's message above, I am not sure whether I can proceed deleting them on parsercache or I should wait until the code is cleaned up?

In T375981#11426268, @Jdlrobson-WMF wrote:

Remove all options greater than 250px.

Yes. I think it would be sensible to remove these.

One thing that could be a middle ground is to have "large thumbnail" mode and split the PC into two modes, default and large. Then all other options would simply turn into a css. PC would split into two but better than six/seven. Loading everything with 500px is a non-starter. The increase would be quite big.

So the tables are built from the structure of objectcache table. They take the schema of objectcache and use it template to create other tables (see SqlBagOStuff:createTables) and SqlBagOStuff is flexible in terms of the name of the table it needs to query. i.e. the objectcache table schema in core is a template many parts of core use to create tables with other names (including Parser cache tables, main stash tables, etc.)

In T375981#11423752, @Ladsgroup wrote:

So in terms of the load on the infra, this is not an issue anymore since mediawiki automatically turns them into one of the steps (T360589) so this is merely a css config at this point.

So in terms of the load on the infra, this is not an issue anymore since mediawiki automatically turns them into one of the steps (T360589) so this is merely a css config at this point. I had plans and did clean up some of these values though T106640: Choose a sensible set of thumbnail sizes for Special:Preferences Specially my suggestion in T106640#10761152

In T411497#11423699, @Marostegui wrote:

$ sudo db-mysql pc1011 parsercache -e "show tables like 'objec%'";
+--------------------------------+
| Tables_in_parsercache (objec%) |
+--------------------------------+
| objectcache                    |
+--------------------------------+

With that ^ only 8% left (298 classes: P85110)

It has finally stopped

Don't ask how I got this:

Nov 28 14:38:09 codesearch9 write_config.py[705745]: <class 'dict'> {'url': 'https://gerrit-replica.wikimedia.org/r/labs/countervandalism/stillalive.git', 'url-pattern': {'base-url': 'https://gerrit.wikimedia.org/g/labs/countervandalism/stillalive/+/{rev}/{path}{anchor}', 'anchor': '#{line}'}, 'ms-between-poll': 5400000}
Nov 28 14:38:09 codesearch9 write_config.py[705745]: <class 'str'> https://github.com/toolforge/paws
Nov 28 14:38:09 codesearch9 write_config.py[705745]: <class 'int'> 5400000

Write config is broken:

Nov 28 14:24:49 codesearch9 write_config.py[705103]: Skip unsupported remote URL: https://git.push-f.com/mw-code/
Nov 28 14:24:49 codesearch9 write_config.py[705103]: Traceback (most recent call last):
Nov 28 14:24:49 codesearch9 write_config.py[705103]:   File "/srv/codesearch/write_config.py", line 632, in <module>
Nov 28 14:24:49 codesearch9 write_config.py[705103]:     main()
Nov 28 14:24:49 codesearch9 write_config.py[705103]:   File "/srv/codesearch/write_config.py", line 581, in main
Nov 28 14:24:49 codesearch9 write_config.py[705103]:     make_conf('search', args,
Nov 28 14:24:49 codesearch9 write_config.py[705103]:   File "/srv/codesearch/write_config.py", line 548, in make_conf
Nov 28 14:24:49 codesearch9 write_config.py[705103]:     new = extract_urls(conf)
Nov 28 14:24:49 codesearch9 write_config.py[705103]:           ^^^^^^^^^^^^^^^^^^
Nov 28 14:24:49 codesearch9 write_config.py[705103]:   File "/srv/codesearch/write_config.py", line 568, in extract_urls
Nov 28 14:24:49 codesearch9 write_config.py[705103]:     return {repo['url'] for repo in conf['repos'].values()}
Nov 28 14:24:49 codesearch9 write_config.py[705103]:            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Nov 28 14:24:49 codesearch9 write_config.py[705103]:   File "/srv/codesearch/write_config.py", line 568, in <setcomp>
Nov 28 14:24:49 codesearch9 write_config.py[705103]:     return {repo['url'] for repo in conf['repos'].values()}
Nov 28 14:24:49 codesearch9 write_config.py[705103]:             ~~~~^^^^^^^
Nov 28 14:24:49 codesearch9 write_config.py[705103]: TypeError: string indices must be integers, not 'str'

I'm forcing a restart of the services, let's see if that makes an impact.

Apologies, we had to revert this. It was causing 701K errors in 15 minutes. In comparison, the rest of mediawiki in that period caused only 4K errors.

In T410696#11415793, @jijiki wrote:

Hello! Since 1212204 was backported, it has been producing thousands of error messages https://logstash.wikimedia.org/goto/103ab3d23a65b901740f79fc62e71e9a

oh one thing: I should check x1 dbs, if these wikis get recreated, then it'll be quite a fun time with x1.

Ah fair. Thanks for spotting it. Let me see if I can fix it easily.

Wouldn't browsers been able to do it on the fly? I'd be surprised if FF can't upscale a svg. I can try in beta cluster but it's down.

nbd. Go for it.

In T402792#11407730, @Ladsgroup wrote:

I don't know the details of ForeignAPIFile but the API endpoints actually return the correct standardized thumb urls. See for example: https://commons.wikimedia.org/w/api.php?action=query&titles=File:Albert%20Einstein%20Head.jpg&prop=imageinfo&iiurlwidth=100&iiprop=url and since this is on our side, it should be technically automatically deployed everywhere. Obviously, if InstantCommons/ForeignAPIFile uses another API endpoint, then it's a different story. I'd argue we still could rate limit and encourage people to either use standard sizes or build a cache. We are not fully disallowing it, just pushing for a more responsible use of the infra by rate limiting excessive requests.

For "page" I don't mean two different parser cache types. That's correct. I should have given more context. When we had 3 pc shard and in old times, when a cluster (pc1 for example) turned empty for various reasons. 1/3rd of parser cache entries got invalidated because "idoptions" key was missing, and 1/3rd got invalidated because "idhash" entry was missing so we ended up with 2/3rd of entries getting invalidated (if you want to be more exact, it'd be 56% because 1/9th would be in the same). Adding sister keys avoided that so when we depool a cluster, only 1/nth of the page views require a reparse. That obviously means it's fine whether legacy parser or parsoid is living in separate clusters since they don't share a code path. I don't know about post-parse processing though. I assume it'd be better to make sure they still around the same page in parsoid to avoid partial loss of parsed data. Do I make sense? or I'm just confusing you more. If you want, we can hop on a call.

In T410145#11412966, @Marostegui wrote:

In T410145#11412961, @Ladsgroup wrote:

Another point to consider: https://meta.wikimedia.org/wiki/Wikimedia_Foundation_Community_Affairs_Committee/Sister_Projects_Task_Force/Results_of_the_consultation_about_Wikispore_and_Wikinews#Wikinews it's very likely most if not all wikinews projects will be closed and archived which means number of closed wikis will reach ~170. Having a small archive cluster that we could move all of these wikis to would help s3 a lot.

That's great - however, remember that for closed projects (with no writes) this whole process is way way easier.

Another point to consider: https://meta.wikimedia.org/wiki/Wikimedia_Foundation_Community_Affairs_Committee/Sister_Projects_Task_Force/Results_of_the_consultation_about_Wikispore_and_Wikinews#Wikinews it's very likely most if not all wikinews projects will be closed and archived which means number of closed wikis will reach ~170. Having a small archive cluster that we could move all of these wikis to would help s3 a lot.

Stopping cookie licking but I will hope to take a look in Wikimedia Hackathon

LGTM. No concerns from our side. nitpick: The name of the field is quite a mouthful. Maybe a shorter name? :P

Data persistence opinion has been requested. I don't think it could cause any problems to the infra to add the extra keys to PC, but it would be extremely important that extra keys should end up in the same host using the concept of sister keys. The work I did in gerrit:1078013 and gerrit:1079677. This way, if we depool a host for maintenance, only 1/8th of the pages will be affected (status quo) and if they are not sister keys then it would be quite a problem. But if that's taken care of, I don't think we have any issues. Once it's rolled out, let us know so we can keep an eye on the infra.

I don't know the details of ForeignAPIFile but the API endpoints actually return the correct standardized thumb urls. See for example: https://commons.wikimedia.org/w/api.php?action=query&titles=File:Albert%20Einstein%20Head.jpg&prop=imageinfo&iiurlwidth=100&iiprop=url and since this is on our side, it should be technically automatically deployed everywhere. Obviously, if InstantCommons/ForeignAPIFile uses another API endpoint, then it's a different story. I'd argue we still could rate limit and encourage people to either use standard sizes or build a cache. We are not fully disallowing it, just pushing for a more responsible use of the infra by rate limiting excessive requests.

It would be great if we can get the simple patch deployed in the meantime so we can start working on rate limiting non-standard sizes

T411013: Popups should use standard thumbnail sizes for longer term solution

Deployed now, feel free to clean it up.

I made https://gerrit.wikimedia.org/r/c/mediawiki/extensions/Popups/+/1211107 as a stop-gap to take care of the biggest chunk.

I was kinda sure it was Popups and lo and behold, it's Popups: https://gerrit.wikimedia.org/g/mediawiki/extensions/Popups/+/8754132e08cedfc0ac70b2bb1dded01e55472e93/src/gateway/rest.js#119

Okay, I checked several more cases and they all seems to be coming from rest endpoint for page summary. For example, there is another one that hits https://en.wikipedia.org/api/rest_v1/page/summary/World_War_II and immediately https://uploads.wikimedia.org/wikipedia/commons/thumb/1/10/Bundesarchiv_Bild_101I-646-5188-17%2C_Flugzeuge_Junkers_Ju_87.jpg/480px-Bundesarchiv_Bild_101I-646-5188-17%2C_Flugzeuge_Junkers_Ju_87.jpg which is the same image provided in the thumbnail attribute of the page summary endpoint but when I open the link the thumbnail is a different size o.O

I picked a random path that was hit and looked the IP and basically looked at the previous and after requests at the same time but the same IP. I picked a IPv6 to reduce the chance of overlap.

The query was wrong, the like should have an extra % at the end. Let me try again.

According to the table catalog these tables only exist in wikishared in x1: https://going-merry.toolforge.org/?table=wikimedia_editor_tasks_counts

The extension is fully undeployed so this can go now.

It'll be live in 24 hours, please reopen if it doesn't.

In T408663#11387971, @Marostegui wrote:

Yeah, let's leave it running for the weekend and if all is fine I can start unifying weights across the board. The host with lower weight will have 100.

^ This would fix the flaky test if anyone is feeling like a +2

Let me debug this a bit. Very likely caused by https://gerrit.wikimedia.org/r/c/mediawiki/core/+/1205190

(honestly it wouldn't be the end of the world if we just switch indexing it off)

So I don't have a way to point to the file. Since except the .git file, nothing exists in netbox-exports endpoint:

root@netbox1003:/srv/netbox-exports# ls 
dns.git  netbox-dns  netbox-hiera

["groupLoadsBySection"]=>
array(1) {
  ["DEFAULT"]=>
  array(1) {
    [""]=>
    array(1) {
      ["WMF_MAINTENANCE_OFFLINE_placeholder"]=>
      int(0)
    }
  }
}

o.O

The root cause is very likely this https://gerrit.wikimedia.org/r/c/mediawiki/core/+/1205190. The patch applied on mw-experimental work just fine, locally works fine, beta cluster works fine. It just explodes in production on maint scripts.

I haven't found anything in gadgets, etc. https://commons.wikimedia.org/w/index.php?title=Special:Search&limit=500&offset=0&ns8=1&search=%22480px-%22 (same in enwiki, etc.) I found a lot of weird stuff in meta but still all unrelated: https://meta.wikimedia.org/w/index.php?search=%22%2F480px-%22&title=Special%3ASearch&profile=advanced&fulltext=1&ns8=1

This is really really crazy. I've been debugging this in T405087#11388580 and onwards. The error doesn't make any sense.

https://grafana.wikimedia.org/d/35WSHOjVk/application-servers-red-k8s?orgId=1&from=2025-11-21T13:37:29.200Z&to=2025-11-21T14:28:00.361Z&timezone=utc&var-site=$__all&var-deployment=mw-web&var-method=GET&var-code=200&var-handler=php&var-service=mediawiki&refresh=1m&viewPanel=panel-9