Will do it once some clean ups are in place.

repeating what I said in the meeting with Daniel. If it's using the same general backend (MySQL PC), only splitting it via even fragmentation should just work out of the box. They are distributed via consistent hashing so it'll be in a random different (or the same in some cases) physical host. That makes it homogeneous and easy to maintain instead of maintaining two separate clusters.

Yup, adding a new column is not that hard. There are some documentation on this https://wikitech.wikimedia.org/wiki/Schema_changes and https://www.mediawiki.org/wiki/Manual:Schema_changes

I'm planning to overhaul the documentation regarding this soon. Stay tuned.

https://wikitech.wikimedia.org/wiki/Deployments/Training

I'm on clinic duty this week. Waiting for signoff by Tyler. Maybe a deployment training can be arranged (or other devs in wmde can do an informal training)

superset should be automatically done via wmf ldap group. If Jgiannelos is in the ldap group, it should be done already. Correct?

Martin was busy so I did it.

This is actually done already as part of T313253: Fix Flaggedrevs database schema drifts in production

In T332329#8728312, @Lucas_Werkmeister_WMDE wrote:

You’re both talking about the wrong extension / class. EntitySchema’s SqlIdGenerator works on a database table that only has one row, and doesn’t have an id_type column. The id_value is the only column in the table – there’s nothing else we even could use to address any specific row.

It got merged today, I'm planning to deploy this tomorrow and until then it shouldn't be closed, otherwise people might see the warning and create it again.

Yeah to me it really feels like it should be avoided and I see it as a good opportunity to discourage it. You can set the id_type in the where condition?

https://logstash.wikimedia.org/goto/b3869e1ef82447c9fd65f7abee8f1279

I'm already doing this as part of the main one. I just need to do switchovers in eqiad and we will be done with this too

Timo is working on devtools now: https://gerrit.wikimedia.org/r/c/labs/codesearch/+/902880

FWIW these are characters that can be found in a standard Persian keyboard (as a way to check for universality of the choice): !٬٫﷼٪×،*)(ـ+÷؟|

I'll add it: https://github.com/wikimedia/labs-codesearch/blob/master/write_config.py

That's not really DBA stuff, I remove the tag but keep myself assigned.

The way we do other data migrations is that we keep the duplicate data around and remove it with the drop of the old tables/fields. Basically my suggestion is to simply avoid deleting the rows in the old table. Would that fix your issue?

For the specific case of wdqs/wcqs, I think it'd be quite easy to set up a toolforge service to become de-facto key-value store for queries would that suffice?

It should be there in a couple of hours.

The former is correct, a lot of FR tables don't start with flaggedrevs

I believe Base is looking for https://meta.wikimedia.org/wiki/Affiliations_Committee/Resolutions/Wikimedia_wiki's_policy

Yes and no. It is possible but currently we are running write-heavy maint scripts as well. There is comment migration happening T275246 and externallinks fields T326314 so make sure it doesn't conflict and it should be fine.

I just wanted to say I love this change <3

I wrote this long time ago. Maybe that'd be useful? https://www.mediawiki.org/wiki/User:ASarabadani_(WMF)/Database_for_devs_toolkit/How-to/Data_migration

Yeah, RC patrolling is highly recommended. You can add auto-patrolled group and such similar to the default config.

In T328311#8684512, @Niharika wrote:

In T328311#8574691, @Ladsgroup wrote:

In T328311#8573392, @dom_walden wrote:

It might be because those temporary users are not in the actor or user tables.

Assuming that we always assign temporary user names sequentially, there would appear to be lots of temporary users accounts which are not recorded in the database on beta dewiki:

That is intentional. In order to avoid race conditions and similar issues, a user id is picked from a sharding system plus some randomness.

I did not know this either. To clarify - you are saying that it is possible to have *Unregistered 1 and *Unregistered 4 in the database with *Unregistered 2 and *Unregistered 3 never having been created?

Yes.

My follow up question then is:

1. Can we determine and/or define how fast these numbers will grow? Going by Dom's data in T328311#8573392 it seems like the numbers are getting larger rather rapidly. At this pace we will end up with very large numbers to indicate temporary users very quickly. This can be detrimental to the end user experience.

I need to look at the code and get you some numbers. It is probably adjustable and possibly won't happen that often on production cases because users get created in short span of time (the whole reason this sharding was implemented for) so the gaps will be much smaller. I double check and get back to you soon.

2. How do we assign user IDs to new users? Do we deploy a similar mechanism when choosing user IDs?

Oh gosh, I messed it up. I wrote both the function that calls subprocess (the whole file actually) and the change that makes it wait for stdin making everything collapse. I should have known better. In my defense, I have none. I will take care of it once I'm properly back. So far the revert should be good for now (which is done). Sorry for this.

BTW, a future-me problem:

Traceback (most recent call last):
  File "/home/amir/Phabricator-maintenance-bot/new_wikis_handler.py", line 564, in <module>
    main()
  File "/home/amir/Phabricator-maintenance-bot/new_wikis_handler.py", line 559, in main
    hande_task(task_details)
  File "/home/amir/Phabricator-maintenance-bot/new_wikis_handler.py", line 548, in hande_task
    handler.handle()
  File "/home/amir/Phabricator-maintenance-bot/new_wikis_handler.py", line 79, in handle
    handler()
  File "/home/amir/Phabricator-maintenance-bot/new_wikis_handler.py", line 161, in _handle_analytics
    maker.run()
  File "/home/amir/Phabricator-maintenance-bot/gerrit.py", line 109, in run
    self.changes()
  File "/home/amir/Phabricator-maintenance-bot/patch_makers.py", line 113, in changes
    with open('static_data/pageview/whitelist/whitelist.tsv', 'r') as f:
FileNotFoundError: [Errno 2] No such file or directory: 'static_data/pageview/whitelist/whitelist.tsv'

Context: https://gerrit.wikimedia.org/r/c/mediawiki/extensions/Wikibase/+/699194
And T281453

During the upgrade, there were around 100 mailing lists that were not obvious if they were actually closed or not. For whatever reason, some steps probably were missed. E.g. lack of [Archived] in the description.

Martin is a volunteer who might not have time for it. If he doesn't get to it before I'm back (March 27th), I'll take over.

For me the html field looks okay:

<title>MediaWiki Codesearch</title>

Maybe it got fixed in the mean time? *shrugs*

(The schema change would take a bit)

In T329071#8676986, @elukey wrote:

The problem that I see with 1) is that we are already filtering (and rightfully so) a lot of events, meanwhile researchers may want the whole stream scored.

For MediaWikiVersionFetcher it's better to simply add it to the bluepsice repo and hard-deprecate it in core. It'd be great to do it before the branch cut. I'll try ASAP.

After talking to Joe, we decided to reopen it. We need something in web to build data visualizations.

In T329071#8662812, @elukey wrote:

Thanks a lot!

Regarding the jobs, the reason ores ext doesn't trigger a job is not that it can't, it's because it could 1- overwhelm the ores service 2- it could fill the mw mysql tables with crap. The biggest example is 22M edits done monthly in Wikidata that only a very small fraction of them is valuable for ores ext (edits that are not auto-patrolled by mediawiki are needed for patrollers) so the extension simply ignores edits done by auto-patrolled users (including bots) which filters out 99.9% of edits.

Just to understand - the extension does trigger async jobs in the job queue right? IIUC calling ORES and inserting in the DB is not done at edit time, but later on (forgive my ignorance but I'd like to be sure about these things, Mediawiki is not my area of expertise :)

yeah, clearly wrong join order. The good solution here is to do join decomposition. given that there is no condition on page table, it's better to query rev_page values and feed that to a second query (programmatically, not via subquery) to a second query to get page_title and page_namesapce.

What is the wrong query plan? Very likely wrong join order. You probably (haven't checked the code) do join decomposition or add straight join, specially if the query I'd directly build in GE and not care so the impact would be limited

My bad. Pre-genarated ones are different from user perf ones. Two sizes can be dropped from pre-gen sizes with neglible impacting on users but don't touch the user pref thumb sizes

In T331138#8675706, @MatthewVernon wrote:

In T331138#8675245, @Joe wrote:

Also: if pre-generation of thumbs makes sense (does it? do we have any numbers on this stuff?) then it should happen in all datacenters, not just the one where swift is "master".

I have a feeling @Ladsgroup [on leave all month] did look at this when we adjusted the sizes of some of the thumbs we were pre-generating and using (to make them a bit more standard).

In T222604#8622146, @awight wrote:

Reading through T326147, I think this is going to happen--it would eliminate the variation in parser output. I'll set this task to block on that one.

That's not really a database issue. It's mediawiki saying this page can't exist:
https://hu.wikipedia.org/wiki/Vita:WP:AZ

The requested page title refers to a talk page that cannot exist.

My suggestion is that for the first part, at least indef block users who have indef block cookie from a previous block. That will have a massive impact.

Sorry, DBAs don't maintain beta cluster DBs, I'm currently busy with this UBN T330942 otherwise I would have helped

foreach ( $iterator as $file ) { var_dump( $file ); }

[debug] [FileOperation] HTTP start: GET https://ms-fe.svc.eqiad.wmnet/auth/v1.0
[debug] [FileOperation] HTTP complete: GET https://ms-fe.svc.eqiad.wmnet/auth/v1.0 code=200 size=0 total=0.102922 connect=0.031767
[debug] [FileOperation] HTTP start: GET https://ms-fe.svc.eqiad.wmnet/v1/AUTH_mw/wikipedia-test-local-thumb
[debug] [FileOperation] HTTP complete: GET https://ms-fe.svc.eqiad.wmnet/v1/AUTH_mw/wikipedia-test-local-thumb code=204 size=0 total=0.062615 connect=0.000053

I'm sure of the root cause, the purging of old thumbnails during reupload short circuits (most likely because FileBackend is set to read only in codfw and didn't get fixed during switchover, I'm guessing, need to double check)

The thumbnail URL it hits to rerender is the user-facing, cached one:

Maybe I'm doing it wrong but the job in test setup returns true while actually not updating the thumbnail (it's quite fast, way too fast for a re-render of thumbnail)

ladsgroup@mwdebug2002:~$ mwscript eval.php --wiki=testwiki --profiler=text
> $job = new ThumbnailRenderJob(Title::newFromText('File:Wikitech-2021-blue-large-icon-ww (4th copy).svg'),[ 'transformParams' => ['width' => 640 ] ] );

This is special because of this:

	/**
	 * Whether to retry the job.
	 * @return bool
	 */
	public function allowRetries() {
		// ThumbnailRenderJob is a warmup for the thumbnails cache,
		// so loosing it is not a problem. Most times the job fails
		// for non-renderable or missing images which will not be fixed
		// by a retry, but will create additional load on the renderer.
		return false;
	}

Job run duration has flatlined since yesterday: https://grafana.wikimedia.org/d/CbmStnlGk/jobqueue-job?orgId=1&var-dc=codfw%20prometheus%2Fk8s&var-job=ThumbnailRender&viewPanel=3&from=1677597263851&to=1677757572286

Every time someone re-uploads an image, some jobs are triggered to build the thumbnail on "default sizes". the sizes are: [ 320, 640, 800, 1024, 1280, 1920 ]. That job is clearly broken. If you take a look at https://test.wikipedia.org/wiki/File:Wikitech-2021-blue-large-icon-ww_(4th_copy).svg. All thumbnails in the sizes mentioned are wrong and everything else is correct. It can be either the job completely failing to execute (or shortening out) or it's replacing the wrong file in the swift. Debugging further.

This might be the culprit: https://gerrit.wikimedia.org/r/c/operations/mediawiki-config/+/891962/2/wmf-config/filebackend.php

Thanks <3 I will make the script figure out the primary dc properly instead of hard-coding it.

Yeah but it has the check option for it and the stuff for --read-only-master in switchover, the usual difference between primary and non-primary dc.

The script needed to be updated after the dc switchover now (it has change for dns and such while it's not needed), I fixed it so you can fix this ticket or just abandon the patch and call switchmaster again.

NICE <3 %$%^#@

In T330509#8658191, @RHo wrote:

Hiya, adding @Niharika to confirm but I don't think we want to add extra functionality for temp accounts over IP editors in MVP, so temp accounts should not have access to Special:Preferences.

Hi, my sincere apologies for late answer, we are understaffed even more than the usual. Anyway, so in paper, ores extension should be able to handle any sort of model, it's not bound to ores or reverting/damaging. The class for building the value out of a json response is well encapsulated. So you should be able to make some changes to ores extension and get it to work with lift wing models. Renaming a deployed extension is practically impossible though. People have tried it with Flow before. It doesn't matter though, it's just internal/technical facing

In T300919#8654004, @Jdlrobson wrote:

So running the maint script to set the skin would break things

Could you expand on this @Ladsgroup - what exactly would it break?

Hit ratio has had a bump since the deployment and still going up as it's getting warmed:
https://grafana.wikimedia.org/d/000000106/parser-cache?orgId=1&from=1677070991551&to=1677675793552&viewPanel=7

Yup, we need the CML support in Wikipedia, something I hope to pick up after thumbor getting upgraded.

In Norwegian wikipedia, it's deleted. The rest, I did it.

Sigh, this is not very useful:

mysql:research@s1-analytics-replica.eqiad.wmnet [enwiki]> select page_title, page_namespace, count(*) from wbc_entity_usage join page on eu_page_id = page_id where page_namespace = 0 group by eu_page_id order by count(*) desc limit 50;
+----------------------------------------------------------------+----------------+----------+
| page_title                                                     | page_namespace | count(*) |
+----------------------------------------------------------------+----------------+----------+
| List_of_lighthouses_in_China                                   |              0 |     2209 |
| List_of_dams_in_Hokkaido                                       |              0 |     2075 |
| List_of_lighthouses_in_Scotland                                |              0 |     1666 |
| List_of_Polish_mathematicians                                  |              0 |     1363 |
| List_of_crossings_of_the_River_Thames                          |              0 |     1324 |
| List_of_learned_societies                                      |              0 |     1287 |
| Ying_Fan_Reinfelder                                            |              0 |     1260 |
| List_of_lighthouses_in_Argentina                               |              0 |     1216 |
| Lidia_Morawska                                                 |              0 |     1007 |
| Listed_buildings_at_the_University_of_Leeds                    |              0 |      921 |
| Lightvessel_stations_of_Great_Britain                          |              0 |      816 |
| List_of_learned_societies_in_the_United_Kingdom                |              0 |      774 |
| List_of_dams_in_Akita_Prefecture                               |              0 |      742 |
| List_of_parks_and_open_spaces_in_the_London_Borough_of_Croydon |              0 |      731 |
| List_of_dams_in_Yamaguchi_Prefecture                           |              0 |      706 |
| List_of_dams_in_Toyama_Prefecture                              |              0 |      696 |
| List_of_dams_in_Yamagata_Prefecture                            |              0 |      673 |
| List_of_dams_in_Hyogo_Prefecture                               |              0 |      652 |
| List_of_dams_in_Gifu_Prefecture                                |              0 |      638 |
| List_of_monuments_in_Meknes                                    |              0 |      629 |
| List_of_lighthouses_in_Madagascar                              |              0 |      617 |
| Catenin_beta-1                                                 |              0 |      614 |
| List_of_monuments_in_Rabat,_Morocco                            |              0 |      591 |
| List_of_dams_in_Hiroshima_Prefecture                           |              0 |      565 |
| List_of_dams_in_Chiba_Prefecture                               |              0 |      562 |
| List_of_lightvessels_of_Great_Britain                          |              0 |      557 |
| List_of_dams_in_Nagano_Prefecture                              |              0 |      547 |
| List_of_dams_in_Kagawa_Prefecture                              |              0 |      539 |
| TGF_beta_1                                                     |              0 |      533 |
| List_of_lighthouses_in_Japan                                   |              0 |      521 |
| Bone_morphogenetic_protein_4                                   |              0 |      507 |
| List_of_mountain_peaks_by_prominence                           |              0 |      503 |
| List_of_dams_in_Fukuoka_Prefecture                             |              0 |      499 |
| Notch_1                                                        |              0 |      491 |
| Sonic_hedgehog_protein                                         |              0 |      487 |
| P53                                                            |              0 |      483 |
| Protein_Wnt-5a                                                 |              0 |      483 |
| Dopamine_receptor_D2                                           |              0 |      481 |
| List_of_dams_in_Shimane_Prefecture                             |              0 |      480 |
| AKT1                                                           |              0 |      475 |
| List_of_learned_societies_in_the_United_States                 |              0 |      468 |
| List_of_crossings_of_the_Danube                                |              0 |      467 |
| List_of_monuments_in_Marrakesh                                 |              0 |      461 |
| List_of_dams_in_Fukushima_Prefecture                           |              0 |      459 |
| Proto-oncogene_tyrosine-protein_kinase_Src                     |              0 |      457 |
| Epidermal_growth_factor_receptor                               |              0 |      454 |
| List_of_historic_places_in_the_Edmonton_Capital_Region         |              0 |      451 |
| CALM2                                                          |              0 |      450 |
| HLA-B                                                          |              0 |      439 |
| List_of_dams_in_Iwate_Prefecture                               |              0 |      437 |
+----------------------------------------------------------------+----------------+----------+
50 rows in set (2 min 59.544 sec)

You asked me to take a look and I just got to look at this, yes. I agree the DI and parameterization of config for DB object creation is literally all over the place[1]. gerrit:886011 would help but we more to do, once the refactors are done, we can take a look at fixing this once it's more central. Sorry it's a vague response, I have nothing better to contribute :(

It is hard to measure the change's impact right now. Because when we deployed it, while it started to use desktop's pre-filled cache, it basically stopped using any PC cache that existed in mobile only. This cancelled a bit of the impact it had which takes some time to actually warm up and show itself (our current cache time for PC is 21 days, so in 21 days we should start measuring averages and compare them with a couple days ago).

To be more explicit: T321527: Support dynamic defaults for user properties

Awesome. Closing it then.

hi, @joernc_unibi is it possible to try it with this patch and see if it fixes your upgrade issue?

@leila I totally agreed that it's a complex problem but I hope we can move forward with improving quality of our data and metric in some shape or form.

A simple solution would be to check wiki exists (by hitting e.g. az.wikimedia.org and making sure it's not 404) before creating them, if the wiki exists, it shouldn't create it

ah okay. Good.

In T330707#8650991, @MarcoAurelio wrote:

@Ladsgroup - This (and its subtasks) are dupes of T317119: Post-creation work for azwikimedia. Seems the bot is creating dupe post-creation tasks after the creation?

That is great and sorta exactly what I need but I need it publicly or half-publicly accessible in https://config-master.wikimedia.org. Something like https://config-master.wikimedia.org/pybal/codfw.json. Then I can build a gui in toolforge to show you all services with their health status as an overview, so you'd quickly be able to spot a service is mistakenly fully depooled or if depooling of all services in a rack/row has led to a service being dangerously low capacity.

Awesome, ran puppet agent and now it's clean:

root@db1154:~# check_private_data.py -S /run/mysqld/mysqld.s5.sock
-- Non-public databases that are present:
-- Non-public tables that are present:
-- Unfiltered columns that are present:
root@db1154:~#

I guess mw-config needs to be updated in db1154, we can wait for a bit: https://gerrit.wikimedia.org/g/operations/puppet/+/3311226c98544255063d0d8f10828ca85cf05154/modules/role/files/mariadb/check_private_data.py

I still see it, I need to figure out where does this come from

I don't know if that's the reason but it needs fixing regardless, it's in https://gerrit.wikimedia.org/g/operations/mediawiki-config/+/ba364eb6dc99aa6d81ab7255a3411608401d8400/dblists/deleted.dblist

Yes. To expand on what Zabe said, for various reasons we can't give you the exact same wiki you had in outside wikimedia infrastructure. For example that extension needs at least a security review. So you need to either have a fishbowl wiki (public but only members can edit) or private (main page public, the rest can only be viewed by members)

(Possibly because it's fishbowl wiki?)

I'm getting this:

root@db1154:~# for i in vewikimedia; do echo $i; redact_sanitarium.sh -d $i -S /run/mysqld/mysqld.s5.sock | mysql -S /run/mysqld/mysqld.s5.sock $i ; done
vewikimedia

I wonder if it's possible to do the same for dbctl too while it's not in config-master (or I couldn't find it)