fwiw I could replicate this on the pip installed version of pywikibot, but when I cloned https://github.com/wikimedia/pywikibot to be able to run it with a debugger a bit easier, it's now running okay... are you able to try that instead?

(Hopefully that task rename makes sense)

(Also noting Slack discussion)

In T332636#8712334, @MusikAnimal wrote:

I spent several hours debugging this and finally concluded it's something with MediaWiki-extensions-OAuth itself that's failing. I've filed T332650: Frequent OAuth failures on Wikimedia wikis since eqiad was repooled due to db-mainstash replication lag.

CopyPatrol is supposed to show an error, but there's apparently an unrelated issue with flash messages not showing. I'm debugging that now.

we want new articles to be created by experienced users for example super new users cannot create one in main name space

Note added to Tech News.

In T309609#8715260, @Daimona wrote:

In T309609#8715216, @TheresNoTime wrote:

In T309609#8715209, @Daimona wrote:

Should this go in Tech News?

I had the same thought — I suppose a quick line in "Recent changes" wouldn't hurt?

Agreed, mostly so that people can update local documentation etc.

In T309609#8715209, @Daimona wrote:

Should this go in Tech News?

Nope, still happening:

Noting FTR that this was again raised at https://meta.wikimedia.org/wiki/Requests_for_comment/Make_global_abuse_filters_opt-out

Deploying this week :-)

In T331762#8710102, @Stashbot wrote:

Mentioned in SAL (#wikimedia-operations) [2023-03-20T14:05:55Z] <TheresNoTime> [samtar@mwmaint2002 ~]$ mwscript maintenance/migrateUserGroup.php --wiki ptwikisource 'autoreview' 'autopatrol' T331762

Script output:

[samtar@mwmaint2002 ~]$ mwscript maintenance/namespaceDupes.php --wiki slwiki --fix

Script output:

Unlick for now..

Glad to see you've managed to resolve the problem :-)

In T332273#8702656, @colewhite wrote:

The MediaWiki errors over time by channel visualization has enabled "Drop Partial Buckets" and "Minimum Interval: Hour" which in combination would prevent the last hour from showing. Is this setting not expected?

Patch merged

Patch merged

FYI, the bug reported in T332076 also affects this on Vector, but only on Chrome/Edge:

In T324759#8708477, @Samwilson wrote:

At the moment there's no way to switch to inline mode anyway, so it won't be seen by very many people. Or is it better to get the dropdown in place first?

In T332234#8708974, @jcrespo wrote:

I found another major regression: I cannot read the task: https://phabricator.wikimedia.org/T331695

It says "Undefined index: state" in an unhandled exception, maybe triggered by the recent upgrade (and revert?).

In T332507#8708750, @hashar wrote:

Does it mean that loosing the replica caused Wikimedia\Rdbms\LoadMonitor::computeServerStates to stale and thus prevented update.php from running?

Do we understand why it works locally but it doesn't in production or beta? If htmlCacheUpdate never runs then phonosIPAFilePersist will never be schedule since it needs the parser to run from another job or script. See https://gerrit.wikimedia.org/r/plugins/gitiles/mediawiki/extensions/Phonos/+/refs/heads/master/includes/Phonos.php#186

The "out of the box" job queue setup is very different to the job queue runners we have configured in production/beta — "By default, jobs are run at the end of a web request.". WMF uses a Kafka job queue.

Hi — it looks like this is being caused by the head template ({{head|mul|symbol|head=''M''_🜨}}). Removing that template and previewing the page shows the expected page title formatting

In discussion

In T324759#8705216, @JSengupta-WMF wrote:

@TheresNoTime tried another version with a reduced font size and kept the padding to the left and right 6px and top and bottom 4px. I am keeping the highlight box 32px tall to align with the drop down on the right. Also dropped the space between legends. wdyt?

Ideally I wanted to keep the same code font for the legends too but that may look weird when the font on the screen is different in the inline mode :-/

In T322469#8708351, @Stashbot wrote:

Mentioned in SAL (#wikimedia-releng) [2023-03-19T22:10:20Z] <TheresNoTime> (beta) GRANT SELECT, INSERT, UPDATE, DELETE, CREATE, REFERENCES, INDEX, ALTER, CREATE TEMPORARY TABLES, LOCK TABLES, EXECUTE, CREATE VIEW, SHOW VIEW, CREATE ROUTINE, ALTER ROUTINE, EVENT, TRIGGER ON mainstash.* TO 'wikiadmin'@'172.16.%'; T322469

It's caught up, and https://integration.wikimedia.org/ci/view/Beta/job/beta-update-databases-eqiad/65938/console is running successfully (so far..)

killing the stuck mysqld process on deployment-db13 and restarting the service has brought mysql back up on db13.. it's now replicating (it's very behind but catching up..)

deployment-db11

Mar 19 20:10:30 deployment-db11 mysqld[14965]: 2023-03-19 20:10:30 5858284 [Warning] Access denied for user 'wikiadmin'@'172.16.%' to database 'mainstash'
Mar 19 20:10:30 deployment-db11 mysqld[14965]: 2023-03-19 20:10:30 5858285 [Warning] Access denied for user 'wikiadmin'@'172.16.%' to database 'mainstash'
Mar 19 20:20:52 deployment-db11 mysqld[14965]: 2023-03-19 20:20:52 5858688 [Warning] Aborted connection 5858688 to db: 'apiportalwiki' user: 'wikiadmin' host: '172.16.4.233' (Got an error reading communication packets)
Mar 19 20:20:52 deployment-db11 mysqld[14965]: 2023-03-19 20:20:52 5858689 [Warning] Aborted connection 5858689 to db: 'aawiki' user: 'wikiadmin' host: '172.16.4.233' (Got an error reading communication packets)
Mar 19 20:27:52 deployment-db11 mysqld[14965]: 2023-03-19 20:27:52 5858877 [Warning] Aborted connection 5858877 to db: 'apiportalwiki' user: 'wikiadmin' host: '172.16.4.233' (Got an error reading communication packets)
Mar 19 20:27:52 deployment-db11 mysqld[14965]: 2023-03-19 20:27:52 5858878 [Warning] Aborted connection 5858878 to db: 'aawiki' user: 'wikiadmin' host: '172.16.4.233' (Got an error reading communication packets)
Mar 19 20:30:34 deployment-db11 mysqld[14965]: 2023-03-19 20:30:34 5858951 [Warning] Aborted connection 5858951 to db: 'aawiki' user: 'wikiadmin' host: '172.16.4.233' (Got an error reading communication packets)

Very much just an opinion, and a nitpick-y one at that — the legend on the page feels like it's taking up a bit "too much" room?

Just tried to replicate this — appears to be working as expected? (GIF below)

d'oh.. you're right!

In T294448#8696051, @Jdlrobson wrote:

I checked out the github for the trending service and it's still working - it would just need to be hosted in a reliable way. Previously I didn't do a good job of this as I've found CloudVPS a bit of a pain for running Node.js sites. Ive updated the README with setup instructions: https://github.com/jdlrobson/wikipedia-trending-endpoint#setting-up-server

In T332207#8702786, @DMartin-WMF wrote:

@TheresNoTime - follow-up question related to this ticket: Although UpdateImplementationsJob is now running, our extension's other job, CacheTesterResultsJob, still does not appear to be running. In your work on the config files, did you change or add something specifically related to UpdateImplementationsJob (and which maybe we need to change similarly for CacheTesterResultsJob)?

In T331761#8688633, @Hokwelum wrote:

One way this could be done might be to add the checksum for dumps created this month via a script and run the script daily for new dump files in some directory trees. Most dumps in https://dumps.wikimedia.org/other belong to other teams and are not maintained by us (Platform Engineering). They are also produced by different scripts, some on different hosts.

I am not confident in my understanding of the ramifications of doing this:

Also agreed irt phasing out enwp.org in favour of w.wiki (be nice if you could do w.wiki/en/A_page or something, but that's for another time..) however that desire isn't mutually exclusive to having the domain donated to us — I'd personally much rather we decide to break a load of enwp.org links now, than the domain expire one day and be used for something malicious..

Going to boldly assume, given this is JobQueue related, that it's allllll down to T332211: deployment-docker-changeprop01: `worker died, restarting`

Resolved per T332211: deployment-docker-changeprop01: `worker died, restarting`

Ah heck, I missed this task and created T332211: deployment-docker-changeprop01: `worker died, restarting` as a dupe — issue seems to be resolved though!

@DMartin-WMF this task (and my T325786) prompted me to go digging, find T332211: deployment-docker-changeprop01: `worker died, restarting` and after much hair-pulling, fix some config files — I now see MediaWiki\Extension\WikiLambda\Jobs\UpdateImplementationsJob::run Job being run! (e.g.)

Following https://wikitech.wikimedia.org/wiki/Changeprop#To_deployment-prep (thanks @taavi) and getting the config files updated resolved this issue — tentatively looking like jobs have been restored...

In T327708#8697048, @Samwilson wrote:

Actually, I was thinking the opposite: that it does make sense to track the usage, even though it's wrong. Because if someone's trying to use a jpg as an audio file, it's still a usage and is useful to know when e.g. deleting that file (or whatever).

In T327708#8696828, @dom_walden wrote:

@TheresNoTime Should the file usage be registered if it is not an audio file? For example, the file usage for https://en.wikipedia.beta.wmflabs.org/wiki/File:Cover_Lutung_Kasarung_Wikibook.png lists https://en.wikipedia.beta.wmflabs.org/wiki/Phonos_non-audio_file.

Perhaps it does not matter as people are unlikely to do this.

I've had this intermittently today as well — https://upload.wikimedia.org/wikipedia/commons/thumb/f/f9/Octicons-gift.svg/12px-Octicons-gift.svg.png being the most recent example

Tagging User-notice as this change is:

• a completion of a CWS proposal
• may conflict with some user scripts which already add the CentralAuth link (imho, this "conflict" will be "two links appear")