I can attest to @Soda's professionalism and due care when submitting and reviewing patches for PageTriage :-)

Beta being beta, let's close this as a transient.

Undeployed from beta

@Djackson-ctr you should now have access to deployment-prep :-)

Noting that the worker_long@ww_host worker is dying after a inconsistent amount of time — there were ~140 items in its queue to process, and I have been manually restarting the ww_celery service when I notice it has died. It has now got around ~40 items in its queue. I'll keep doing this proactively until its queue is empty.

Thanks @hnowlan, all working afaict too :-)

In T351172#9332505, @MusikAnimal wrote:

I'm a little confused what went wrong. What you did for /var/run/celery sounds like it was necessary and probably explains why things go down occasionally? Or do you believe the PID file was getting created before, but the permissions changed or something, which you've now fixed?

The graphite_host issue may be related to https://wikitech.wikimedia.org/wiki/News/2023_Cloud_VPS_metrics_changes — I'm unsure of what, if any, changes need to be made to get puppet to pass (and if that would resolve the issue)

Puppet hasn't run on deployment-restbase04, same on deployment-restbase-bullseye — cannae be good?

Putting High, as it appears to be blocking some CI (T350353)

In T279960#9329825, @M2k_dewiki wrote:

Hello,

the error is appearing again.

[...]

Added

UUID=dc441956-0a90-434a-803e-ef3ba7bcb4fd /pickles ext4 nofail,x-systemd.device-timeout=2s 0 2
UUID=7fa2581f-39cc-4eb6-94d0-952677c0f1b9 /pickles-02 ext4 nofail,x-systemd.device-timeout=2s 0 2

to /etc/fstab

Steps to resolve this incident:

• Created /var/run/celery, did chown wikiwho /var/run/celery (PID file was not being created)
• Mounted /dev/sdb to /pickles
• Mounted /dev/sdc to /pickles-02

I'm looking, but cc @MusikAnimal, I know you recently made some changes ref. the services (T344936: Systemd services rely on cron to restart)

Have tried restarting ww_celery/ww_flower

thank you for the patch, will review shortly :-)

(968397 deployed now)

TypeError: Cannot read properties of undefined (reading 'Keys') introduced by OO.ui.Keys.ENTER (1af2997)? I can't replicate this though..

There was a spike of TypeError: Cannot read properties of undefined (reading 'href') for that article (e.g. b7e3e924-ed2e-4498-8e45-de6a00dddbc0, and a bit of

+1!

Ah, T103285: HotCat doesn't load after saving a page in VisualEditor (must refresh/reload first) [AOI]

VE doesn't refresh the page (and thus reload HotCat) after saving an edit – this means when HotCat retrieves the wgCurRevisionId, it'll be referencing the revision ID prior to the VE edit, causing an edit conflict.

In T334766#9297043, @Novem_Linguae wrote:

Patch: https://github.com/wikimedia/WhoWroteThat/commit/6fe078bc7bdc24dbad9589b1662a2934d2b92a47

Available for testing on https://en.wikipedia.beta.wmflabs.org/wiki/Main_Page (see https://en.wikipedia.beta.wmflabs.org/wiki/Special:BlockedExternalDomains)

nb. also sometimes getting an error popup with Error: Project switch failed for user "Samtar".

@GMikesell-WMF ack, thank you — I'm going to ask for opinions on if this is desired behaviour, as afaik LinkRenderer (which creates the missing file upload link) returns HTML and the error popup just accepts wikitext (and thus renders the broken file link as just that)

Horrible hacky "fix" — changed host: prometheus-labmon.eqiad.wmnet to host: 127.0.0.1 in the hiera config metrics stanzas

In T349648#9277041, @TheresNoTime wrote:

[...]
Error: getaddrinfo ENOTFOUND prometheus-labmon.eqiad.wmnet sounds suspiciously related to the recently completed T344974: De-provision beta-specific Prometheus

hate to be that person, but is there a reason why y'all can't just use https://en.wikipedia.beta.wmflabs.org/wiki/Special:NewPagesFeed for testing?

Looks like it was just a transient error during deployment, closing as invalid.

Just FYI, although https://en.wiktionary.org/wiki/a now loads without errors, the following (expected) log entry is generated: Parsing 'a' was slow, took 30.63 seconds

In T348487#9274339, @MusikAnimal wrote:

In T348487#9272333, @TheresNoTime wrote:

@MusikAnimal ready for test.wiki? :-)

There's no harm I suppose! But I think T341361 should be resolved before we deploy to any non-test wikis.

Done, API key is limited to 185.15.56.0/22 — I'll leave this task here for a moment just in case anyone reports issues with the copyvios tool

As the key is apparently only being used by copyvios.toolforge.org, it should be okay to limit it to WMCS IP addresses — doing so now, and will monitor for any breakage :-)

In T349407#9274974, @Kotz wrote:

Which Monday? 😊

In T325825#9209401, @Danbloch wrote:

Why was this task closed? It's still happening.

Unstalling, work in progress