From what I've gathered:

• Apache has a RewriteRule for /wiki(/.*)? to send requests to /w/index.php
• MediaWiki then extracts the page title from that request using $wgArticlePath see documentation

Thanks, removed downtime and repooled.

Temporary braindump of most likely solution:

• 3 Ganeti VM per core DC
• 1 redis instance per VM
• No replication between redis instances
• No replication between datacentres

Depooled, downtimed, go ahead.

In T432716#12180337, @Jhancock.wm wrote:

it's magical to me. Updated the server location. I was gonna give the reimage a try but i didn't know which distro was on it.
@jijiki lmk which distro or you can take over.

Hey @Jhancock.wm you can move them whenever you want, just depool them first and repool them after, or we can coordinate to do that for you.
Will they need any special action after moving within the rack?

Triaging to ServiceOps Radar (Pending) in case this is something where we have actual actions to take, if that's not forecast to be the case please move to Radar (Awareness)

In T421711#12132686, @ops-monitoring-bot wrote:

Cookbook cookbooks.sre.k8s.renumber-node started by cgoubert@cumin2003 Renumbering for host wikikube-worker1339.eqiad.wmnet completed:

• wikikube-worker1339.eqiad.wmnet (FAIL)
  • Successfully cordoned node wikikube-worker1339.eqiad.wmnet
  • Successfully reimaged node wikikube-worker1339.eqiad.wmnet
  • Failed to run puppet agent on deployment servers
  • Successfully ran puppet agent on deployment servers
  • Successfully ran puppet agent on registry servers
  • Pooled and uncordoned node wikikube-worker1339.eqiad.wmnet

• wikikube-worker1339.eqiad.wmnet (PASS)
  • Host wikikube-worker1339.eqiad.wmnet depooled from wikikube-eqiad
  • Host wikikube-worker1339.eqiad.wmnet pooled in wikikube-eqiad

• wikikube-worker1339 (PASS)
  • Downtimed on Icinga/Alertmanager
  • Disabled Puppet
  • Host successfully migrated to the new VLAN
  • Removed from Puppet and PuppetDB if present and deleted any certificates
  • Removed from Debmonitor if present
  • Forced UEFI HTTP Boot for next reboot
  • Host rebooted via Redfish
  • Host up (Debian installer)
  • Host up (new fresh trixie OS)
  • Generated Puppet certificate
  • Signed new Puppet certificate
  • Run Puppet in NOOP mode to populate exported resources in PuppetDB
  • Found Nagios_host resource for this host in PuppetDB
  • Downtimed the new host on Icinga/Alertmanager
  • Removed previous downtime on Alertmanager (old OS)
  • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202607171433_cgoubert_2308257_wikikube-worker1339.out
  • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
  • Rebooted
  • Automatic Puppet run was successful
  • Forced a re-check of all Icinga services for the host
  • Icinga status is optimal
  • Icinga downtime removed
  • Updated Netbox data from PuppetDB

In T432108#12132725, @Dzahn wrote:

Looks like this caused T432442 on gerrit and releases hosts which was kind of my concern with that patch. Have not looked yet if it was only temporary though.

I fixed it as soon as it popped up, it should be good now

That's me, https://gerrit.wikimedia.org/r/c/operations/puppet/+/1310589 required a manual kill of stunnel4 and a puppet rerun, that's now done and fixed.

Onboarding WDQSv2 to rate limiting/REST Gateway is in scope for PP3.5 Preferred API Pathways, specifically PP3.5.1 WDQS Cost-Based Rate Limiting

In T432110#12119181, @Dreamy_Jazz wrote:

It would also be nice if the alert for these failing jobs is fired once per job. For example, a failure on Sunday caused 4 alerts to be sent to our Slack channel until I was on my PC for Monday to check the failing job

For the record, systemctl restart doesn't work either, so I don't think it's a puppet problem.

That seems to keep happening, killing and restarting properly for now, but we should investigate further.

Yeah that's on me, don't worry about it.

You should now be able to reach linked-artifacts through the service mesh, using http://localhost:6074 as a base URL, so curl https://localhost:6074/revisions/v1/article_topics/enwiki/534366/1358198637

Just ran

cgoubert@deploy1003:~$ mwscript-k8s --comment="T430531" --file=apiportalwiki_recalculatecount.txt --follow -- extensions/CentralAuth/maintenance/RecalculateGlobalEditCountForUsers.php --wiki=testwiki apiportalwiki_recalculatecount.txt

Survey is now generated, and we have a Grafana dashboard

We had a few Gerrit crashes yesterday that probably explain the 503s y'all got cc @hashar @ABran-WMF

In T431108#12094536, @Dbrant wrote:

In T431108#12085309, @Jgiannelos wrote:

@Dbrant the service we want is liftwing, right? Can you provide some example request?

I believe so; More details about the service url are here:
https://phabricator.wikimedia.org/T392833#11987986

And here is an example request:
curl https://linked-artifacts.discovery.wmnet:30443/revisions/v1/article_topics/enwiki/534366/1358198637

...where the parameters in the path are .../[wiki]/[pageid]/[revisionid]

Copying my comment from gerrit here for ease of retrieval:

Given PDFs are arbitrary binary input, we have application and infrastructure level security and reliability concerns that require consideration.
Application-side, we need a security review of the pdf-parse and backing pdfjs-dist libraries. This would also help determine the level of sandboxing necessary to consider a production deployment.
Infrastructure-side, we would want to isolate the PDF handling as much as possible. Ideally this would happen in a separate microservice that only does PDF handling, or at least in a separate citoid deployment, to mitigate potential vulnerabilities and resource exhaustion. Both solutions would also require a stronger sandbox than a standard container.

There should be some of us around during tomorrow's yes, but I'll let @RLazarus decide whether it's worth doing at a different time considering there's a little more than just puppet merges.

I've taken the liberty to add some necessary patches removing LVS service definitions, DNS entries, and mw-on-k8s mesh config. Order of operations should look more or less like:

I'm closing this task presuming the file upload worked on a retry. Feel free to reopen if this error presents itself again.

In T430629#12070503, @Mvolz wrote:

Could I get access to the restricted task?

This should now be fixed for this use case, general discussion continues in parent. Feel free to reopen if issues persist.

For now, I've removed any X-W-D handling on api.wikimedia.org. Other endpoints status:

• /w/rest.php: There is an argument to remove it from this endpoint because the REST Gateway rewrites the paths. This means hitting mw-debug directly will probably break.
• /api/rest_v1: This endpoint does not have X-W-D handling configured.
• /w/api.php (Action API): This endpoint has X-W-D handling configured, but the REST Gateway does not meaningfully rewrite the paths. Hitting mw-debug directly works identically to going through the REST Gateway, but bypasses rate limiting.

In T428909#12069858, @hnowlan wrote:

I think given the API portal is deprecated and the /core/ APIs are in maintenance mode I would be of the opinion that we can just drop api-gateway-specific support for XWD entirely. It was mostly used for test deployments of the portal (T262396) - if the linkrecommendation folks are using it then that's another matter, but if the bug in T430600 was an accidental discovery I would vote for dropping the support entirely.

In T430531#12067418, @matmarex wrote:

Is there a list of affected users (those who existed or had edits on apiportalwiki) still available somewhere, or is that data lost for good and we need to recalculate all users?

I think the simplest way to go about this is to remove X-Wikimedia-Debug from the plugin chain for api.wikimedia.org/{core,service,feed} in the ATS configuration.
It is not currently needed as even the /core API endpoints can't currently be tested on these paths using X-W-D.

The reason that happens is the X-Wikimedia-Debug header (or the extension which just sets that header) bypasses ATS at the edge where the routing to the rest gateway for api.wikimedia.org API endpoints is done. It forces you to the mw-debug MediaWiki-On-Kubernetes deployment, which redirects everything under api.wikimedia.org to the API Doc page.

Created T430531: Recalculate edit count following apiportalwiki deletion to try to sum up and track that work, because it seems off topic for this task.
Resolving here because the original issue is now fixed.

Eventstreams does benefit from the global override, and the others should be migrated. The override is happening on the deployment servers only, so moving eventgate and system charts to use docker.registry wouldn't change the ability of local devs to set the registry themselves, and we don't expect to change the charts defaulting to the external URL either.