Page MenuHomePhabricator

Ladsgroup (Amir Sarabadani)
Shah of Bugs, Emir of database architecture, World-renowned rubber duckAdministrator

Projects (35)

Today

  • Clear sailing ahead.

Tomorrow

  • Clear sailing ahead.

Thursday

  • Clear sailing ahead.

User Details

User Since
Oct 6 2014, 9:53 PM (442 w, 6 h)
Roles
Administrator
Availability
Available
IRC Nick
Amir1
LDAP User
Ladsgroup
MediaWiki User
Ladsgroup [ Global Accounts ]

Staff Database Architect in SRE data persistence team in WMF. Used to be Wikidata software engineer in WMDE

I'm also open source enthusiast, mediawiki volunteer developer, and long-term Wikipedian.

All edits on tickets about databases are in my work capacity and anything else is in my volunteer capacity unless mentioned otherwise.

Babel: fa-N, en-4, de-2, tr-1, hu-1

Recent Activity

Today

Ladsgroup added a project to T333174: Machine Learning team - k8s resources access: Machine-Learning-Team.
Tue, Mar 28, 2:49 AM · Machine-Learning-Team, Patch-For-Review, SRE, Lift-Wing, SRE-Access-Requests
Ladsgroup renamed T333174: Machine Learning team - k8s resources access from Machine Learning team - k8s resources ccess to Machine Learning team - k8s resources access.
Tue, Mar 28, 2:48 AM · Machine-Learning-Team, Patch-For-Review, SRE, Lift-Wing, SRE-Access-Requests

Yesterday

Ladsgroup added a comment to T280604: Post-deployment: (partly) ramp parser cache retention back up .

Will do it once some clean ups are in place.

Mon, Mar 27, 11:41 PM · Patch-For-Review, DBA, Performance-Team (Radar), Editing-team, DiscussionTools
Ladsgroup added a comment to T328034: ParsoidCachePrewarmJob should set the useParsoid ParserOption.

repeating what I said in the meeting with Daniel. If it's using the same general backend (MySQL PC), only splitting it via even fragmentation should just work out of the box. They are distributed via consistent hashing so it'll be in a random different (or the same in some cases) physical host. That makes it homogeneous and easy to maintain instead of maintaining two separate clusters.

Mon, Mar 27, 11:40 PM · Patch-For-Review, Parsoid-Read-Views (Phase 2 - testwiki Main namespace support), Content-Transform-Team-WIP, MW-1.40-notes (1.40.0-wmf.21; 2023-01-30), Performance-Team (Radar), MediaWiki-Parser, RESTbase Sunsetting, Parsoid
Ladsgroup added a comment to T333223: Adding user_is_temp to the user table.

Yup, adding a new column is not that hard. There are some documentation on this https://wikitech.wikimedia.org/wiki/Schema_changes and https://www.mediawiki.org/wiki/Manual:Schema_changes

Mon, Mar 27, 10:50 PM · Data-Persistence, Data-Engineering, IP Masking
Ladsgroup added a comment to T330173: page_assessments table uses an inadequate VARBINARY length for pa_class/pa_importance.

I'm planning to overhaul the documentation regarding this soon. Stay tuned.

Mon, Mar 27, 10:12 PM · Data-Persistence (work done), Schema-change-in-production, Community-Tech, MediaWiki-extensions-PageAssessments
Ladsgroup added a comment to T331899: Requesting access to deployment for ItamarWMDE.

https://wikitech.wikimedia.org/wiki/Deployments/Training

Mon, Mar 27, 9:40 PM · Patch-For-Review, SRE, SRE-Access-Requests
Ladsgroup claimed T331899: Requesting access to deployment for ItamarWMDE.

I'm on clinic duty this week. Waiting for signoff by Tyler. Maybe a deployment training can be arranged (or other devs in wmde can do an informal training)

Mon, Mar 27, 9:39 PM · Patch-For-Review, SRE, SRE-Access-Requests
Ladsgroup added a comment to T332063: Requesting access to analytics-privatedata-users for Yiannis Giannelos.

superset should be automatically done via wmf ldap group. If Jgiannelos is in the ldap group, it should be done already. Correct?

Mon, Mar 27, 9:37 PM · SRE, SRE-Access-Requests
Ladsgroup closed T328501: Request to clean my watchlist from articles in namespace 0 and 1 as Resolved.

Martin was busy so I did it.

Mon, Mar 27, 9:27 PM · User-Urbanecm, Wikimedia-maintenance-script-run, Wikimedia-Site-requests
Ladsgroup closed T251191: FlaggedRevs has lots of database drifts but only in s1 and s5, a subtask of T313253: Fix Flaggedrevs database schema drifts in production, as Resolved.
Mon, Mar 27, 9:16 PM · MediaWiki-extensions-FlaggedRevs, DBA
Ladsgroup closed T251191: FlaggedRevs has lots of database drifts but only in s1 and s5 as Resolved.

This is actually done already as part of T313253: Fix Flaggedrevs database schema drifts in production

Mon, Mar 27, 9:16 PM · DBA
Ladsgroup added a comment to T332329: UpdateQueryBuilder does not support IDatabase::ALL_ROWS updates.

You’re both talking about the wrong extension / class. EntitySchema’s SqlIdGenerator works on a database table that only has one row, and doesn’t have an id_type column. The id_value is the only column in the table – there’s nothing else we even could use to address any specific row.

Mon, Mar 27, 9:15 PM · Performance-Team (Radar), MediaWiki-libs-Rdbms
Ladsgroup added a comment to T332942: Warning: SQLPlatform::isWriteQuery fallback to regex (from ApiQueryRevisions).

It got merged today, I'm planning to deploy this tomorrow and until then it shouldn't be closed, otherwise people might see the warning and create it again.

Mon, Mar 27, 8:45 PM · MediaWiki-Action-API, Performance-Team (Radar), MW-1.41-notes (1.41.0-wmf.2; 2023-03-27)
Ladsgroup added a comment to T332329: UpdateQueryBuilder does not support IDatabase::ALL_ROWS updates.

Yeah to me it really feels like it should be avoided and I see it as a good opportunity to discourage it. You can set the id_type in the where condition?

Mon, Mar 27, 9:52 AM · Performance-Team (Radar), MediaWiki-libs-Rdbms
Ladsgroup closed T332941: Warning: SQLPlatform::isWriteQuery fallback to regex (from Wikibase EntityUsageTable) as Resolved.

https://logstash.wikimedia.org/goto/b3869e1ef82447c9fd65f7abee8f1279

Mon, Mar 27, 8:42 AM · MW-1.41-notes (1.41.0-wmf.1; 2023-03-20), wdwb-tech, Wikidata, MediaWiki-extensions-WikibaseClient, Performance-Team (Radar)
Ladsgroup closed T332941: Warning: SQLPlatform::isWriteQuery fallback to regex (from Wikibase EntityUsageTable), a subtask of T210206: Deprecate raw SQL conditions for IDatabase methods (select, insert, etc.), as Resolved.
Mon, Mar 27, 8:42 AM · Patch-For-Review, Platform Engineering (Icebox), Performance-Team (Radar), MediaWiki-libs-Rdbms
Ladsgroup closed T332941: Warning: SQLPlatform::isWriteQuery fallback to regex (from Wikibase EntityUsageTable), a subtask of T326181: Stop doing regex on SQL in Database, as Resolved.
Mon, Mar 27, 8:42 AM · Performance-Team, MediaWiki-libs-Rdbms
Ladsgroup updated the task description for T332921: Requesting access to production for kamila.
Mon, Mar 27, 8:30 AM · SRE, SRE-Access-Requests
Ladsgroup moved T251191: FlaggedRevs has lots of database drifts but only in s1 and s5 from Backlog to In progress on the DBA board.

I'm already doing this as part of the main one. I just need to do switchovers in eqiad and we will be done with this too

Mon, Mar 27, 7:44 AM · DBA

Sun, Mar 26

Ladsgroup added a comment to T333073: Please add patchdemo to codesearch index.

Timo is working on devtools now: https://gerrit.wikimedia.org/r/c/labs/codesearch/+/902880

Sun, Mar 26, 12:54 PM · VPS-project-Codesearch
Ladsgroup added a comment to T332805: Decide the prefix character for temporary usernames.

FWIW these are characters that can be found in a standard Persian keyboard (as a way to check for universality of the choice): !٬٫﷼٪×،*)(ـ+÷؟|

Sun, Mar 26, 12:50 PM · IP Masking

Sat, Mar 25

Ladsgroup claimed T333073: Please add patchdemo to codesearch index.

I'll add it: https://github.com/wikimedia/labs-codesearch/blob/master/write_config.py

Sat, Mar 25, 10:41 PM · VPS-project-Codesearch
Ladsgroup removed a project from T333065: SelectQueryBuilder::fetchRowCount() always returns 1 for DISTINCT queries: DBA.

That's not really DBA stuff, I remove the tag but keep myself assigned.

Sat, Mar 25, 10:38 PM · Patch-For-Review, Regression, Performance-Team, MediaWiki-libs-Rdbms
Ladsgroup updated the task description for T275246: Populate rev_actor and rev_comment_id.
Sat, Mar 25, 3:04 PM · Patch-For-Review, MW-1.39-release, MediaWiki-Core-Revision-backend, MW-1.38-release, Platform Engineering Roadmap, Code-Health-Objective, Platform Team Initiatives (Revision Storage Schema Improvements), Technical-Debt

Fri, Mar 24

Ladsgroup added a comment to T242031: Allow multiple different 2FA devices.

The way we do other data migrations is that we keep the duplicate data around and remove it with the drop of the old tables/fields. Basically my suggestion is to simply avoid deleting the rows in the old table. Would that fix your issue?

Fri, Mar 24, 11:46 PM · Community-Wishlist-Survey-2023, MW-1.40-notes (1.40.0-wmf.22; 2023-02-06), MediaWiki-extensions-OATHAuth
Ladsgroup added a comment to T220703: Increase the max length of URL to be shortened.

For the specific case of wdqs/wcqs, I think it'd be quite easy to set up a toolforge service to become de-facto key-value store for queries would that suffice?

Fri, Mar 24, 10:56 PM · wdwb-tech, Wikidata, Patch-For-Review, Security, Wikimedia-Site-requests, MediaWiki-extensions-UrlShortener
Ladsgroup reassigned T332995: Add integration/quibble to codesearch from Ladsgroup to Jdforrester-WMF.
Fri, Mar 24, 10:30 PM · Quibble, VPS-project-Codesearch
Ladsgroup closed T332995: Add integration/quibble to codesearch as Resolved.

It should be there in a couple of hours.

Fri, Mar 24, 10:30 PM · Quibble, VPS-project-Codesearch
Ladsgroup added a comment to T332594: Drop FlaggedRevs tables in database for ptwikisource.

The former is correct, a lot of FR tables don't start with flaggedrevs

Fri, Mar 24, 11:24 AM · DBA
Ladsgroup added a comment to T332965: Kazakh wikimedia.

I believe Base is looking for https://meta.wikimedia.org/wiki/Affiliations_Committee/Resolutions/Wikimedia_wiki's_policy

Fri, Mar 24, 11:22 AM · User-Urbanecm, Wiki-Setup (Create)

Tue, Mar 21

Ladsgroup added a comment to T315510: Start maintenance script to backfill talk page comment database.

Yes and no. It is possible but currently we are running write-heavy maint scripts as well. There is comment migration happening T275246 and externallinks fields T326314 so make sure it doesn't conflict and it should be fine.

Tue, Mar 21, 8:20 PM · Goal, MW-1.40-notes (1.40.0-wmf.6; 2022-10-17), MW-1.39-notes (1.39.0-wmf.26; 2022-08-22), Editing-team (Kanban Board), TPP-Phase1, DiscussionTools

Sun, Mar 19

Ladsgroup added a comment to T331320: [SPIKE] Evaluate development of Zebra #9 prototype.

I just wanted to say I love this change <3

Sun, Mar 19, 1:11 PM · Patch-For-Review, Readers-Web-Backlog (Web Team FY2022-23 Q3 Sprint 5), Desktop Improvements (Vector 2022)

Fri, Mar 17

Ladsgroup added a comment to T330173: page_assessments table uses an inadequate VARBINARY length for pa_class/pa_importance.

I wrote this long time ago. Maybe that'd be useful? https://www.mediawiki.org/wiki/User:ASarabadani_(WMF)/Database_for_devs_toolkit/How-to/Data_migration

Fri, Mar 17, 9:57 AM · Data-Persistence (work done), Schema-change-in-production, Community-Tech, MediaWiki-extensions-PageAssessments
Ladsgroup added a comment to T331762: Remove FlaggedRevs for ptwikisource.

Yeah, RC patrolling is highly recommended. You can add auto-patrolled group and such similar to the default config.

Fri, Mar 17, 9:57 AM · Data-Persistence (work done), User-Kizule, Wikimedia-Site-requests

Wed, Mar 15

Ladsgroup added a comment to T328311: Special:AbuseLog is missing the `mw-tempuserlink` class from temporary account user links.

It might be because those temporary users are not in the actor or user tables.

Assuming that we always assign temporary user names sequentially, there would appear to be lots of temporary users accounts which are not recorded in the database on beta dewiki:

That is intentional. In order to avoid race conditions and similar issues, a user id is picked from a sharding system plus some randomness.

I did not know this either. To clarify - you are saying that it is possible to have *Unregistered 1 and *Unregistered 4 in the database with *Unregistered 2 and *Unregistered 3 never having been created?

Yes.

My follow up question then is:

  1. Can we determine and/or define how fast these numbers will grow? Going by Dom's data in T328311#8573392 it seems like the numbers are getting larger rather rapidly. At this pace we will end up with very large numbers to indicate temporary users very quickly. This can be detrimental to the end user experience.

I need to look at the code and get you some numbers. It is probably adjustable and possibly won't happen that often on production cases because users get created in short span of time (the whole reason this sharding was implemented for) so the gaps will be much smaller. I double check and get back to you soon.

  1. How do we assign user IDs to new users? Do we deploy a similar mechanism when choosing user IDs?
Wed, Mar 15, 12:24 AM · Anti-Harassment, AbuseFilter, IP Masking

Tue, Mar 14

Ladsgroup added a comment to T328667: Add --pause-after-testserver-sync option to deploy_security.py.

Oh gosh, I messed it up. I wrote both the function that calls subprocess (the whole file actually) and the change that makes it wait for stdin making everything collapse. I should have known better. In my defense, I have none. I will take care of it once I'm properly back. So far the revert should be good for now (which is done). Sorry for this.

Tue, Mar 14, 11:58 PM · SecTeam-Processed, Security-Team, Release-Engineering-Team
Ladsgroup added a comment to T331831: Create Central Kurdish Wiktionary.

BTW, a future-me problem:

Traceback (most recent call last):
  File "/home/amir/Phabricator-maintenance-bot/new_wikis_handler.py", line 564, in <module>
    main()
  File "/home/amir/Phabricator-maintenance-bot/new_wikis_handler.py", line 559, in main
    hande_task(task_details)
  File "/home/amir/Phabricator-maintenance-bot/new_wikis_handler.py", line 548, in hande_task
    handler.handle()
  File "/home/amir/Phabricator-maintenance-bot/new_wikis_handler.py", line 79, in handle
    handler()
  File "/home/amir/Phabricator-maintenance-bot/new_wikis_handler.py", line 161, in _handle_analytics
    maker.run()
  File "/home/amir/Phabricator-maintenance-bot/gerrit.py", line 109, in run
    self.changes()
  File "/home/amir/Phabricator-maintenance-bot/patch_makers.py", line 113, in changes
    with open('static_data/pageview/whitelist/whitelist.tsv', 'r') as f:
FileNotFoundError: [Errno 2] No such file or directory: 'static_data/pageview/whitelist/whitelist.tsv'
Tue, Mar 14, 11:38 PM · MW-1.40-notes (1.40.0-wmf.27; 2023-03-13), MW-1.41-notes (1.41.0-wmf.1; 2023-03-20), User-Urbanecm, Wiki-Setup (Create)
Ladsgroup updated subscribers of T299077: CI job wmf-quibble-selenium-php72-docker get a deadlock on `change_tag_def` (NameTableStore).

Context: https://gerrit.wikimedia.org/r/c/mediawiki/extensions/Wikibase/+/699194
And T281453

Tue, Mar 14, 11:34 PM · MW-1.41-notes (1.41.0-wmf.1; 2023-03-20), wdwb-tech, Wikidata, MediaWiki-extensions-WikibaseClient, Browser-Tests, MediaWiki-Core-Tests, ci-test-error
Ladsgroup closed T233666: Close wikimediameta-l mailing list as Resolved.

During the upgrade, there were around 100 mailing lists that were not obvious if they were actually closed or not. For whatever reason, some steps probably were missed. E.g. lack of [Archived] in the description.

Tue, Mar 14, 11:25 PM · Wikimedia-Mailing-lists, SRE
Ladsgroup added a comment to T328501: Request to clean my watchlist from articles in namespace 0 and 1.

Martin is a volunteer who might not have time for it. If he doesn't get to it before I'm back (March 27th), I'll take over.

Tue, Mar 14, 11:00 PM · User-Urbanecm, Wikimedia-maintenance-script-run, Wikimedia-Site-requests
Ladsgroup added a comment to T332054: Codesearch frontend still says β in title after promotion to codesearch.wmcloud.org.

For me the html field looks okay:

<title>MediaWiki Codesearch</title>

Maybe it got fixed in the mean time? *shrugs*

Tue, Mar 14, 10:55 PM · VPS-project-Codesearch

Sun, Mar 12

Ladsgroup added a watcher for ARM support: Ladsgroup.
Sun, Mar 12, 11:47 PM

Wed, Mar 8

Ladsgroup added a comment to T318605: Deploy new externallinks fields to production.

(The schema change would take a bit)

Wed, Mar 8, 5:05 PM · Schema-change-in-production, DBA
Ladsgroup updated the task description for T318605: Deploy new externallinks fields to production.
Wed, Mar 8, 5:05 PM · Schema-change-in-production, DBA
Ladsgroup added a comment to T329071: Integration of Revert Risk Scores to Recent Changes as a filter.

The problem that I see with 1) is that we are already filtering (and rightfully so) a lot of events, meanwhile researchers may want the whole stream scored.

Wed, Mar 8, 4:45 PM · Data-Engineering-Planning, Event-Platform Value Stream, Machine-Learning-Team, Edit-Review-Improvements-Integrated-Filters, Growth-Team, Research
Ladsgroup added a comment to T321882: Reduce number of files directly under includes/.

For MediaWikiVersionFetcher it's better to simply add it to the bluepsice repo and hard-deprecate it in core. It'd be great to do it before the branch cut. I'll try ASAP.

Wed, Mar 8, 4:16 PM · MW-1.40-notes (1.40.0-wmf.26; 2023-03-06), MW-1.40-release, MediaWiki-General
Ladsgroup reopened T330705: Provide a json output of confctl as "Open".

After talking to Joe, we decided to reopen it. We need something in web to build data visualizations.

Wed, Mar 8, 3:26 PM · conftool, Sustainability (Incident Followup)
Ladsgroup added a comment to T329071: Integration of Revert Risk Scores to Recent Changes as a filter.

Thanks a lot!

Regarding the jobs, the reason ores ext doesn't trigger a job is not that it can't, it's because it could 1- overwhelm the ores service 2- it could fill the mw mysql tables with crap. The biggest example is 22M edits done monthly in Wikidata that only a very small fraction of them is valuable for ores ext (edits that are not auto-patrolled by mediawiki are needed for patrollers) so the extension simply ignores edits done by auto-patrolled users (including bots) which filters out 99.9% of edits.

Just to understand - the extension does trigger async jobs in the job queue right? IIUC calling ORES and inserting in the DB is not done at edit time, but later on (forgive my ignorance but I'd like to be sure about these things, Mediawiki is not my area of expertise :)

Wed, Mar 8, 3:01 PM · Data-Engineering-Planning, Event-Platform Value Stream, Machine-Learning-Team, Edit-Review-Improvements-Integrated-Filters, Growth-Team, Research
Ladsgroup added a comment to T331264: Error 2006 from GrowthExperiments\UserImpact\ComputedUserImpactLookup::getEditData.

yeah, clearly wrong join order. The good solution here is to do join decomposition. given that there is no condition on page table, it's better to query rev_page values and feed that to a second query (programmatically, not via subquery) to a second query to get page_title and page_namesapce.

Wed, Mar 8, 2:52 PM · MW-1.40-notes (1.40.0-wmf.25; 2023-02-27), Growth-Team (Current Sprint), Wikimedia-production-error, GrowthExperiments, mariadb-optimizer-bug, DBA
Ladsgroup added a comment to T331264: Error 2006 from GrowthExperiments\UserImpact\ComputedUserImpactLookup::getEditData.

What is the wrong query plan? Very likely wrong join order. You probably (haven't checked the code) do join decomposition or add straight join, specially if the query I'd directly build in GE and not care so the impact would be limited

Wed, Mar 8, 1:25 PM · MW-1.40-notes (1.40.0-wmf.25; 2023-02-27), Growth-Team (Current Sprint), Wikimedia-production-error, GrowthExperiments, mariadb-optimizer-bug, DBA
Ladsgroup added a comment to T331138: FileBackendMultiWrite multi-dc and thumbnail handling.

My bad. Pre-genarated ones are different from user perf ones. Two sizes can be dropped from pre-gen sizes with neglible impacting on users but don't touch the user pref thumb sizes

Wed, Mar 8, 1:17 PM · SRE-swift-storage, MediaWiki-File-management
Ladsgroup added a comment to T331138: FileBackendMultiWrite multi-dc and thumbnail handling.

Also: if pre-generation of thumbs makes sense (does it? do we have any numbers on this stuff?) then it should happen in all datacenters, not just the one where swift is "master".

I have a feeling @Ladsgroup [on leave all month] did look at this when we adjusted the sizes of some of the thumbs we were pre-generating and using (to make them a bit more standard).

Wed, Mar 8, 1:11 PM · SRE-swift-storage, MediaWiki-File-management

Thu, Mar 2

Ladsgroup added a comment to T222604: Kartographer maps with "type": "Feature" and a image in the description is not shown in mobile view.

Reading through T326147, I think this is going to happen--it would eliminate the variation in parser output. I'll set this task to block on that one.

Thu, Mar 2, 6:32 PM · WMDE-GeoInfo-FocusArea, Patch-For-Review, User-TheDJ, Maps (Kartographer), Mobile
Ladsgroup edited projects for T331005: Ambigous namespace for hu:Vita:WP:AZ, added: MediaWiki-General; removed DBA.

That's not really a database issue. It's mediawiki saying this page can't exist:
https://hu.wikipedia.org/wiki/Vita:WP:AZ

The requested page title refers to a talk page that cannot exist.

Thu, Mar 2, 5:04 PM · MediaWiki-General
Ladsgroup added a comment to T314174: Expand cookie blocking to enforce existing blocks with the same duration.

My suggestion is that for the first part, at least indef block users who have indef block cookie from a previous block. That will have a massive impact.

Thu, Mar 2, 3:59 PM · MediaWiki-Blocks
Ladsgroup removed a project from T331019: Edits not saving on beta cluster (db replication error, corrupted table): DBA.

Sorry, DBAs don't maintain beta cluster DBs, I'm currently busy with this UBN T330942 otherwise I would have helped

Thu, Mar 2, 2:50 PM · Release-Engineering-Team, Beta-Cluster-Infrastructure
Ladsgroup added a comment to T330942: Latest image thumbnails aren't replaced correctly after image reupload.

foreach ( $iterator as $file ) { var_dump( $file ); }

[debug] [FileOperation] HTTP start: GET https://ms-fe.svc.eqiad.wmnet/auth/v1.0
[debug] [FileOperation] HTTP complete: GET https://ms-fe.svc.eqiad.wmnet/auth/v1.0 code=200 size=0 total=0.102922 connect=0.031767
[debug] [FileOperation] HTTP start: GET https://ms-fe.svc.eqiad.wmnet/v1/AUTH_mw/wikipedia-test-local-thumb
[debug] [FileOperation] HTTP complete: GET https://ms-fe.svc.eqiad.wmnet/v1/AUTH_mw/wikipedia-test-local-thumb code=204 size=0 total=0.062615 connect=0.000053

Thu, Mar 2, 2:23 PM · Commons, MediaWiki-File-management, Wikimedia-Site-requests, Wikimedia-Incident, Regression
Ladsgroup added a comment to T330942: Latest image thumbnails aren't replaced correctly after image reupload.

I'm sure of the root cause, the purging of old thumbnails during reupload short circuits (most likely because FileBackend is set to read only in codfw and didn't get fixed during switchover, I'm guessing, need to double check)

Thu, Mar 2, 1:14 PM · Commons, MediaWiki-File-management, Wikimedia-Site-requests, Wikimedia-Incident, Regression
Ladsgroup added a comment to T330942: Latest image thumbnails aren't replaced correctly after image reupload.

The thumbnail URL it hits to rerender is the user-facing, cached one:

Thu, Mar 2, 12:40 PM · Commons, MediaWiki-File-management, Wikimedia-Site-requests, Wikimedia-Incident, Regression
Ladsgroup added a comment to T330942: Latest image thumbnails aren't replaced correctly after image reupload.

Maybe I'm doing it wrong but the job in test setup returns true while actually not updating the thumbnail (it's quite fast, way too fast for a re-render of thumbnail)

ladsgroup@mwdebug2002:~$ mwscript eval.php --wiki=testwiki --profiler=text
> $job = new ThumbnailRenderJob(Title::newFromText('File:Wikitech-2021-blue-large-icon-ww (4th copy).svg'),[ 'transformParams' => ['width' => 640 ] ] );
Thu, Mar 2, 12:31 PM · Commons, MediaWiki-File-management, Wikimedia-Site-requests, Wikimedia-Incident, Regression
Ladsgroup added a comment to T330942: Latest image thumbnails aren't replaced correctly after image reupload.

This is special because of this:

	/**
	 * Whether to retry the job.
	 * @return bool
	 */
	public function allowRetries() {
		// ThumbnailRenderJob is a warmup for the thumbnails cache,
		// so loosing it is not a problem. Most times the job fails
		// for non-renderable or missing images which will not be fixed
		// by a retry, but will create additional load on the renderer.
		return false;
	}
Thu, Mar 2, 12:19 PM · Commons, MediaWiki-File-management, Wikimedia-Site-requests, Wikimedia-Incident, Regression
Ladsgroup added a comment to T330942: Latest image thumbnails aren't replaced correctly after image reupload.

Job run duration has flatlined since yesterday: https://grafana.wikimedia.org/d/CbmStnlGk/jobqueue-job?orgId=1&var-dc=codfw%20prometheus%2Fk8s&var-job=ThumbnailRender&viewPanel=3&from=1677597263851&to=1677757572286

grafik.png (968×1 px, 116 KB)

Thu, Mar 2, 11:47 AM · Commons, MediaWiki-File-management, Wikimedia-Site-requests, Wikimedia-Incident, Regression
Ladsgroup added a comment to T330942: Latest image thumbnails aren't replaced correctly after image reupload.

Every time someone re-uploads an image, some jobs are triggered to build the thumbnail on "default sizes". the sizes are: [ 320, 640, 800, 1024, 1280, 1920 ]. That job is clearly broken. If you take a look at https://test.wikipedia.org/wiki/File:Wikitech-2021-blue-large-icon-ww_(4th_copy).svg. All thumbnails in the sizes mentioned are wrong and everything else is correct. It can be either the job completely failing to execute (or shortening out) or it's replacing the wrong file in the swift. Debugging further.

Thu, Mar 2, 11:43 AM · Commons, MediaWiki-File-management, Wikimedia-Site-requests, Wikimedia-Incident, Regression
Ladsgroup awarded T213295: Close and delete TestCommons from production a Heartbreak token.
Thu, Mar 2, 11:27 AM · Release-Engineering-Team (Priority Backlog 📥), Structured-Data-Backlog (Current Work), Structured-Data-Team-Current-Work, Structured Data Engineering, Wiki-Setup (Delete / Redirect)
Ladsgroup added a comment to T330942: Latest image thumbnails aren't replaced correctly after image reupload.

This might be the culprit: https://gerrit.wikimedia.org/r/c/operations/mediawiki-config/+/891962/2/wmf-config/filebackend.php

Thu, Mar 2, 11:13 AM · Commons, MediaWiki-File-management, Wikimedia-Site-requests, Wikimedia-Incident, Regression
Ladsgroup added a comment to T330991: Switchover s8 master (db1109 -> db1126).

Thanks <3 I will make the script figure out the primary dc properly instead of hard-coding it.

Thu, Mar 2, 10:31 AM · DBA
Ladsgroup added a comment to T330988: Switchover s8 master (db1109 -> db1126).

Yeah but it has the check option for it and the stuff for --read-only-master in switchover, the usual difference between primary and non-primary dc.

Thu, Mar 2, 10:26 AM · DBA
Ladsgroup added a comment to T330988: Switchover s8 master (db1109 -> db1126).

The script needed to be updated after the dc switchover now (it has change for dns and such while it's not needed), I fixed it so you can fix this ticket or just abandon the patch and call switchmaster again.

Thu, Mar 2, 10:19 AM · DBA
Ladsgroup added a comment to T248418: Roll out videojs as the only video/audio player on all Wikimedia wikis.

NICE <3 %$%^#@

Thu, Mar 2, 10:05 AM · User-notice-archive, User-Ladsgroup, Patch-For-Review, Readers-Web-Backlog, Wikimedia-Site-requests, VideoJS player, Performance-Team (Radar), Multimedia, Wikimedia-Video

Wed, Mar 1

Ladsgroup added a comment to T330509: Should Echo be available to temporary users?.

Hiya, adding @Niharika to confirm but I don't think we want to add extra functionality for temp accounts over IP editors in MVP, so temp accounts should not have access to Special:Preferences.

Wed, Mar 1, 8:43 PM · Patch-For-Review, Data-Persistence, IP Masking, Notifications, Growth-Team
Ladsgroup added a comment to T329071: Integration of Revert Risk Scores to Recent Changes as a filter.

Hi, my sincere apologies for late answer, we are understaffed even more than the usual. Anyway, so in paper, ores extension should be able to handle any sort of model, it's not bound to ores or reverting/damaging. The class for building the value out of a json response is well encapsulated. So you should be able to make some changes to ores extension and get it to work with lift wing models. Renaming a deployed extension is practically impossible though. People have tried it with Flow before. It doesn't matter though, it's just internal/technical facing

Wed, Mar 1, 4:27 PM · Data-Engineering-Planning, Event-Platform Value Stream, Machine-Learning-Team, Edit-Review-Improvements-Integrated-Filters, Growth-Team, Research
Ladsgroup added a comment to T300919: It should be possible to set a default skin for existing users.

So running the maint script to set the skin would break things

Could you expand on this @Ladsgroup - what exactly would it break?

Wed, Mar 1, 3:01 PM · Readers-Web-Backlog, MediaWiki-Core-Preferences
Ladsgroup added a comment to T326147: Stop fragmenting ParserCache entries for mobile frontend .

Hit ratio has had a bump since the deployment and still going up as it's getting warmed:
https://grafana.wikimedia.org/d/000000106/parser-cache?orgId=1&from=1677070991551&to=1677675793552&viewPanel=7

grafik.png (968×1 px, 234 KB)

Wed, Mar 1, 1:05 PM · User-Ryasmeen, Readers-Web-Backlog (Web Team FY2022-23 Q3 Sprint 3), MW-1.40-notes (1.40.0-wmf.22; 2023-02-06), DBA, Patch-For-Review, MediaWiki-Parser, MobileFrontend
Ladsgroup reopened T18491: Support for Chemical Markup Language, a subtask of T19598: Chemistry support (tracking), as Open.
Wed, Mar 1, 12:22 PM · Tracking-Neverending, MediaWiki-General
Ladsgroup reopened T18491: Support for Chemical Markup Language, a subtask of T44725: Multimedia file format support (tracking), as Open.
Wed, Mar 1, 12:22 PM · Tracking-Neverending, WMF-General-or-Unknown
Ladsgroup reopened T18491: Support for Chemical Markup Language, a subtask of T56213: Infographics, graph, chart, diagram, plotting, animation support (tracking), as Open.
Wed, Mar 1, 12:22 PM · Community-Wishlist-Survey-2016, Commons, Tracking-Neverending, WMF-General-or-Unknown
Ladsgroup reopened T18491: Support for Chemical Markup Language as "Open".

Yup, we need the CML support in Wikipedia, something I hope to pick up after thumbor getting upgraded.

Wed, Mar 1, 12:22 PM · MediaWiki-extensions-MolHandler, Wikimedia-Extension-setup, Commons
Ladsgroup added a comment to T326130: Special:SpecialPages needs a table of contents.

In Norwegian wikipedia, it's deleted. The rest, I did it.

Wed, Mar 1, 12:11 PM · MW-1.40-notes (1.40.0-wmf.25; 2023-02-27), MediaWiki-Special-pages

Tue, Feb 28

Ladsgroup created P44904 (An Untitled Masterwork).
Tue, Feb 28, 5:58 PM
Ladsgroup added a comment to T290989: switchdc cache warmup should include URLs that warmup relevant Wikidata caches.

Sigh, this is not very useful:

mysql:research@s1-analytics-replica.eqiad.wmnet [enwiki]> select page_title, page_namespace, count(*) from wbc_entity_usage join page on eu_page_id = page_id where page_namespace = 0 group by eu_page_id order by count(*) desc limit 50;
+----------------------------------------------------------------+----------------+----------+
| page_title                                                     | page_namespace | count(*) |
+----------------------------------------------------------------+----------------+----------+
| List_of_lighthouses_in_China                                   |              0 |     2209 |
| List_of_dams_in_Hokkaido                                       |              0 |     2075 |
| List_of_lighthouses_in_Scotland                                |              0 |     1666 |
| List_of_Polish_mathematicians                                  |              0 |     1363 |
| List_of_crossings_of_the_River_Thames                          |              0 |     1324 |
| List_of_learned_societies                                      |              0 |     1287 |
| Ying_Fan_Reinfelder                                            |              0 |     1260 |
| List_of_lighthouses_in_Argentina                               |              0 |     1216 |
| Lidia_Morawska                                                 |              0 |     1007 |
| Listed_buildings_at_the_University_of_Leeds                    |              0 |      921 |
| Lightvessel_stations_of_Great_Britain                          |              0 |      816 |
| List_of_learned_societies_in_the_United_Kingdom                |              0 |      774 |
| List_of_dams_in_Akita_Prefecture                               |              0 |      742 |
| List_of_parks_and_open_spaces_in_the_London_Borough_of_Croydon |              0 |      731 |
| List_of_dams_in_Yamaguchi_Prefecture                           |              0 |      706 |
| List_of_dams_in_Toyama_Prefecture                              |              0 |      696 |
| List_of_dams_in_Yamagata_Prefecture                            |              0 |      673 |
| List_of_dams_in_Hyogo_Prefecture                               |              0 |      652 |
| List_of_dams_in_Gifu_Prefecture                                |              0 |      638 |
| List_of_monuments_in_Meknes                                    |              0 |      629 |
| List_of_lighthouses_in_Madagascar                              |              0 |      617 |
| Catenin_beta-1                                                 |              0 |      614 |
| List_of_monuments_in_Rabat,_Morocco                            |              0 |      591 |
| List_of_dams_in_Hiroshima_Prefecture                           |              0 |      565 |
| List_of_dams_in_Chiba_Prefecture                               |              0 |      562 |
| List_of_lightvessels_of_Great_Britain                          |              0 |      557 |
| List_of_dams_in_Nagano_Prefecture                              |              0 |      547 |
| List_of_dams_in_Kagawa_Prefecture                              |              0 |      539 |
| TGF_beta_1                                                     |              0 |      533 |
| List_of_lighthouses_in_Japan                                   |              0 |      521 |
| Bone_morphogenetic_protein_4                                   |              0 |      507 |
| List_of_mountain_peaks_by_prominence                           |              0 |      503 |
| List_of_dams_in_Fukuoka_Prefecture                             |              0 |      499 |
| Notch_1                                                        |              0 |      491 |
| Sonic_hedgehog_protein                                         |              0 |      487 |
| P53                                                            |              0 |      483 |
| Protein_Wnt-5a                                                 |              0 |      483 |
| Dopamine_receptor_D2                                           |              0 |      481 |
| List_of_dams_in_Shimane_Prefecture                             |              0 |      480 |
| AKT1                                                           |              0 |      475 |
| List_of_learned_societies_in_the_United_States                 |              0 |      468 |
| List_of_crossings_of_the_Danube                                |              0 |      467 |
| List_of_monuments_in_Marrakesh                                 |              0 |      461 |
| List_of_dams_in_Fukushima_Prefecture                           |              0 |      459 |
| Proto-oncogene_tyrosine-protein_kinase_Src                     |              0 |      457 |
| Epidermal_growth_factor_receptor                               |              0 |      454 |
| List_of_historic_places_in_the_Edmonton_Capital_Region         |              0 |      451 |
| CALM2                                                          |              0 |      450 |
| HLA-B                                                          |              0 |      439 |
| List_of_dams_in_Iwate_Prefecture                               |              0 |      437 |
+----------------------------------------------------------------+----------------+----------+
50 rows in set (2 min 59.544 sec)
Tue, Feb 28, 5:32 PM · Data-Persistence (work done), Patch-For-Review, Datacenter-Switchover, wdwb-tech, Wikidata
Ladsgroup added a comment to T318272: MultiWriteBagOStuff caches are missing DI defaults.

You asked me to take a look and I just got to look at this, yes. I agree the DI and parameterization of config for DB object creation is literally all over the place[1]. gerrit:886011 would help but we more to do, once the refactors are done, we can take a look at fixing this once it's more central. Sorry it's a vague response, I have nothing better to contribute :(

Tue, Feb 28, 4:10 PM · MediaWiki-libs-Rdbms, MW-1.40-notes (1.40.0-wmf.19; 2023-01-16), Developer Productivity, Regression, Performance-Team, MediaWiki-libs-ObjectCache
Ladsgroup added a comment to T326147: Stop fragmenting ParserCache entries for mobile frontend .

It is hard to measure the change's impact right now. Because when we deployed it, while it started to use desktop's pre-filled cache, it basically stopped using any PC cache that existed in mobile only. This cancelled a bit of the impact it had which takes some time to actually warm up and show itself (our current cache time for PC is 21 days, so in 21 days we should start measuring averages and compare them with a couple days ago).

Tue, Feb 28, 1:55 PM · User-Ryasmeen, Readers-Web-Backlog (Web Team FY2022-23 Q3 Sprint 3), MW-1.40-notes (1.40.0-wmf.22; 2023-02-06), DBA, Patch-For-Review, MediaWiki-Parser, MobileFrontend
Ladsgroup added a comment to T300919: It should be possible to set a default skin for existing users.

To be more explicit: T321527: Support dynamic defaults for user properties

Tue, Feb 28, 1:01 PM · Readers-Web-Backlog, MediaWiki-Core-Preferences
Ladsgroup closed T330382: Upgrading from 1.35.8 to 1.39.2 using PostgreSQL fails, index "tl_namespace" does not exist as Resolved.

Awesome. Closing it then.

Tue, Feb 28, 12:48 PM · MW-1.40-notes (1.40.0-wmf.26; 2023-03-06), MW-1.39-notes, MW-1.39-release, PostgreSQL, MediaWiki-Installer
Ladsgroup added a comment to T330382: Upgrading from 1.35.8 to 1.39.2 using PostgreSQL fails, index "tl_namespace" does not exist.

hi, @joernc_unibi is it possible to try it with this patch and see if it fixes your upgrade issue?

Tue, Feb 28, 12:10 PM · MW-1.40-notes (1.40.0-wmf.26; 2023-03-06), MW-1.39-notes, MW-1.39-release, PostgreSQL, MediaWiki-Installer
Ladsgroup added a comment to T138207: [Open question] Improve bot identification at scale.

@leila I totally agreed that it's a complex problem but I hope we can move forward with improving quality of our data and metric in some shape or form.

Tue, Feb 28, 11:15 AM · Data-Engineering, Research-Backlog
Ladsgroup added a comment to T286740: Maintenance bot creates duplicates for new wiki tasks.

A simple solution would be to check wiki exists (by hitting e.g. az.wikimedia.org and making sure it's not 404) before creating them, if the wiki exists, it shouldn't create it

Tue, Feb 28, 10:40 AM · phabricator maintenance bot
Ladsgroup added a comment to T330382: Upgrading from 1.35.8 to 1.39.2 using PostgreSQL fails, index "tl_namespace" does not exist.

ah okay. Good.

Tue, Feb 28, 10:34 AM · MW-1.40-notes (1.40.0-wmf.26; 2023-03-06), MW-1.39-notes, MW-1.39-release, PostgreSQL, MediaWiki-Installer
Ladsgroup added a comment to T330707: Post-creation work for azwikimedia.

@Ladsgroup - This (and its subtasks) are dupes of T317119: Post-creation work for azwikimedia. Seems the bot is creating dupe post-creation tasks after the creation?

Tue, Feb 28, 10:34 AM · Countervandalism-Network, Platform Engineering, Wiki-Setup
Ladsgroup added a comment to T330705: Provide a json output of confctl.

That is great and sorta exactly what I need but I need it publicly or half-publicly accessible in https://config-master.wikimedia.org. Something like https://config-master.wikimedia.org/pybal/codfw.json. Then I can build a gui in toolforge to show you all services with their health status as an overview, so you'd quickly be able to spot a service is mistakenly fully depooled or if depooling of all services in a rack/row has led to a service being dangerously low capacity.

Tue, Feb 28, 10:33 AM · conftool, Sustainability (Incident Followup)

Mon, Feb 27

Ladsgroup added a comment to T330704: Prepare and check storage layer for vewikimedia.

Awesome, ran puppet agent and now it's clean:

root@db1154:~# check_private_data.py -S /run/mysqld/mysqld.s5.sock
-- Non-public databases that are present:
-- Non-public tables that are present:
-- Unfiltered columns that are present:
root@db1154:~#
Mon, Feb 27, 10:22 PM · Data-Engineering, cloud-services-team, Data-Services, DBA
Ladsgroup added a comment to T330704: Prepare and check storage layer for vewikimedia.

I guess mw-config needs to be updated in db1154, we can wait for a bit: https://gerrit.wikimedia.org/g/operations/puppet/+/3311226c98544255063d0d8f10828ca85cf05154/modules/role/files/mariadb/check_private_data.py

Mon, Feb 27, 10:12 PM · Data-Engineering, cloud-services-team, Data-Services, DBA
Ladsgroup added a comment to T330704: Prepare and check storage layer for vewikimedia.

I still see it, I need to figure out where does this come from

Mon, Feb 27, 10:10 PM · Data-Engineering, cloud-services-team, Data-Services, DBA
Ladsgroup added a comment to T330704: Prepare and check storage layer for vewikimedia.

I don't know if that's the reason but it needs fixing regardless, it's in https://gerrit.wikimedia.org/g/operations/mediawiki-config/+/ba364eb6dc99aa6d81ab7255a3411608401d8400/dblists/deleted.dblist

Mon, Feb 27, 9:55 PM · Data-Engineering, cloud-services-team, Data-Services, DBA
Ladsgroup added a comment to T330390: Create Amical Wikimedia wiki.

Yes. To expand on what Zabe said, for various reasons we can't give you the exact same wiki you had in outside wikimedia infrastructure. For example that extension needs at least a security review. So you need to either have a fishbowl wiki (public but only members can edit) or private (main page public, the rest can only be viewed by members)

Mon, Feb 27, 9:50 PM · MW-1.40-notes (1.40.0-wmf.25; 2023-02-27), Patch-For-Review, affects-Miraheze, Wiki-Setup (Create), User-Urbanecm
Ladsgroup added a comment to T330704: Prepare and check storage layer for vewikimedia.

(Possibly because it's fishbowl wiki?)

Mon, Feb 27, 9:45 PM · Data-Engineering, cloud-services-team, Data-Services, DBA
Ladsgroup added a comment to T330704: Prepare and check storage layer for vewikimedia.

I'm getting this:

root@db1154:~# for i in vewikimedia; do echo $i; redact_sanitarium.sh -d $i -S /run/mysqld/mysqld.s5.sock | mysql -S /run/mysqld/mysqld.s5.sock $i ; done
vewikimedia
Mon, Feb 27, 9:44 PM · Data-Engineering, cloud-services-team, Data-Services, DBA
Ladsgroup added a comment to T330705: Provide a json output of confctl.

I wonder if it's possible to do the same for dbctl too while it's not in config-master (or I couldn't find it)

Mon, Feb 27, 9:36 PM · conftool, Sustainability (Incident Followup)
Ladsgroup created T330705: Provide a json output of confctl.
Mon, Feb 27, 9:34 PM · conftool, Sustainability (Incident Followup)