Thanks for the heads up @VRiley-WMF.

Not strictly, but it would be nice to have for peace of mind. This may be a task that @Blake can work on in the coming quarter as prep for the next switchover.

Tagging in @KCVelaga_WMF as we discussed this briefly in Lisbon.

In T412585#11458421, @matmarex wrote:

I haven't been able to find this in the related tasks: what does the error response you'll get look like? and is there some way to "force" an error response to test client libraries?

Yes but it calls restGatewayGet which routes through the rest-gateway instead of directly to AWS

Thanks @taavi for pointing out this is client-side JS and not internal.

I've linked the tasks I'd created as children of T410198: Determine the source of internal requests going through the API gateway., feel free to dedupe as wanted.

In T410764#11432344, @Tgr wrote:

In T410764#11432069, @Clement_Goubert wrote:

If you rename them, the next time the alert fires, it will create a new task instead of appending to the existing open one.

IMO that's not ideal; these tasks will clutter up search results and will be hard to navigate. Couldn't it use a custom Maniphest field rather than the title for identifying which task is related to a script?

As far as I can tell from logstash, calls identified as internal (that get no ratelimit_key) are definitely from Wikifeeds and MediaWiki itself, and all to either page-analytics or device-analytics. device-analytics calls seem to also originate from PageViewInfo

In T410764#11429889, @Tgr wrote:

In general I think it would be nice to rename these auto-generated tasks and put something about the specific failure reason in their title. They are going to be very unhelpful when searching for related issues in the future.

cc @Kappakayala for SRE Manager approval

Thanks for all the support @brouberol <3

Probably a good task to pair on with @Blake or @jasmine_ on tracking down what's used where in puppet.

Hmm I'd like to be able to actually see a failed run, so I'll change the job definition to keep runs for longer so I can inspect the actual kubernetes objects next time it fails, CR uploaded.

In T341553#11423856, @Dreamy_Jazz wrote:

In T341553#11423316, @Tgr wrote:

In T341553#11422966, @Dreamy_Jazz wrote:

Not sure if it has already been mentioned, but mwscript-k8s doesn't seem to support running maintenance scripts that are not defined in public code.

Last time I tried, you could do it with --file.

--file has the following documentation which to me implied I could not use it to define the script but only input files:

--file FILE           Copy a text file into the MediaWiki container (in the script's working directory) to be used as script input. Format:
                       path/to/local-file.txt[:remote-file.txt] -- omit colon section to use the same filename (with any leading path stripped).
                       Pass --file again to copy multiple files.

I also read https://wikitech.wikimedia.org/wiki/Maintenance_scripts#Not_yet_supported as saying that non-text files (i.e. non .txt files) are not supported as input

Perhaps the documentation needs updating to indicate this functionality or at least say that the argument also supports more than just "text file[s] .. to be used as script input"?

In T327663#11416513, @Raine wrote:

Is anything still needed beyond the functionality in sudo cookbook -d sre.discovery.datacenter status all? That provides the following table:

Service                       Type           eqiad     codfw
=================================================================
apertium                      Active/Active  pooled    pooled    
api-gateway                   Active/Active  pooled    pooled    
apt                           Active/Passive pooled              
apus                          Active/Active  pooled    pooled    
citoid                        Active/Active  pooled    pooled    
config-master                 Active/Active  pooled    pooled    
cxserver                      Active/Active  pooled    pooled    
device-analytics              Active/Active  pooled    pooled    
docker-registry               Active/Passive           pooled    
echostore                     Active/Active  pooled    pooled    
eventgate-analytics           Active/Active  pooled    pooled    
[...]

Rebalance done on kafka-main-eqiad

In T411104#11409999, @Urbanecm_WMF wrote:

In T411104#11409929, @Clement_Goubert wrote:

Added documentation of FOREACHWIKI_IGNORE_ERRORS https://wikitech.wikimedia.org/w/index.php?title=Maintenance_scripts&oldid=2365511

Thank you!

The issue with adding some type of log like you're mentioning is that set +e will exit mwscript-mwcron immediately without giving the opportunity to log what wiki was being used, and the mwscript-k8s wrapper is not aware of what's happening within the loop, neither does the kubernetes Pod the script is running inside of.

Do we have to use set +e though? Perhaps we can check $? manually if it is non-zero, which would give us the ability to perform any additional logging we want. Or maybe there is a reason why checking $? is not advisable?

Added documentation of FOREACHWIKI_IGNORE_ERRORS https://wikitech.wikimedia.org/w/index.php?title=Maintenance_scripts&oldid=2365511

Deployed and tested quickly, looks like it's fixed for me, resolving.
Feel free to reopen if there are still issues.

In T411066#11408456, @aaron wrote:

Maybe related to https://gerrit.wikimedia.org/r/c/operations/puppet/+/1198941

In trafficserver/gateway-check.lua.conf, "wikifunctions.org" is not in the "group0"/"group1"/"group2" config (which is right), so the rest gateway was not used for wikifunctions.org/w/api.php until that patch.

Incidentally, in profile/trafficserver/backend.yaml I see an early "target: http://www.wikifunctions.org/w/api.php" rule before the generic "target: 'http://(.*)/w/api.php'" rule. The former does not have the gateway-check.lua plugin and is meant to cause the use of a special mw-on-k8 cluster. There is no specific rule for wikifunctions.org, so it falls into the generic rule. I'm not sure why it 404s, though it might be related to rest-gateway/values.yaml not having domains/FQDNs for wikifunctions.org / www.wikifunctions.org.

In T410748#11406974, @Daimona wrote:

In T410748#11405819, @Clement_Goubert wrote:

The commands should be run on deployment.eqiad.wmnet, these are in the task just for ease of copy/pasting, a more complete troubleshooting documentation is on Wikitech

Thank you, I didn't realize there were more instructions there.

I had tried the command there but I don't have permissions to run them, that's why I wondered if I was doing something wrong. (I am in restricted but not deployment). In fact, the only deployer on the team is currently OOO (@cmelo). Is there anything else I could try? I still think it was probably just a fluke, but double-checking wouldn't hurt.

All ServiceOps hosts have been migrated to the new switch.

Waiting until T405950: eqiad row C/D Service Ops host migrations is done with moving the kafka-main nodes so we don't run into a network blip if the rebalance takes a while

The commands should be run on deployment.eqiad.wmnet, these are in the task just for ease of copy/pasting, a more complete troubleshooting documentation is on Wikitech

In T410612#11400490, @MoritzMuehlenhoff wrote:

This broke Puppet runs on the puppetservers:

Error: Could not retrieve catalog from remote server: Error 500 on SERVER: Server Error: Evaluation Error: Error while evaluating a Function Call, User 'blake' lacks an api token for HP: please add it in profile::conftool::hiddenparma::api_tokens (file: /srv/puppet_code/environments/production/modules/profile/manifests/conftool/requestctl_client.pp, line: 34, column: 17) on node puppetserver2001.codfw.wmnet

See the inline comment in data.yaml: https://github.com/wikimedia/operations-puppet/blob/production/modules/admin/data/data.yaml#L68

I'd like to do the actual implementation of this with @Blake in the upcoming months.

@KOfori Could you approve this ?

@Kappakayala and @hnowlan being OOO, @mark could I get approval for this please?

Tagging:

• Content-Transform-Team for Wikifeeds
• Growth-Team for GrowthExperiments
• PageViewInfo for PageViewInfo

According to the rest-gateway logs, both MediaWiki itself and Wikifeeds are making direct calls to rest-gateway for the page-analytics_pageviews route.

In T405950#11379425, @RobH wrote:

@Clement_Goubert,

Is it possible that I could send the commands for this or do we need someone in your team? If we need someone in your team, could we schedule an hour or so for this tomorrow (Tuesday, Nov 18thy) at 17:00GMT start time? (So 9:00 AM Pacific?)

Apart from the drain (which I've not done and you've detailed someone in your team should do) the other steps to move a host is about 5 minutes total per host, with network connectivity loss of approximately 1 minute or less. (We ping the host during the move and it misses less than 12 ping sequence numbers.)

Hmm, we should probably also figure out a way to route these to mw-api-int instead of mw-api-ext somehow. I have to think about this.

The kafka-main rebalance question is now pretty critical to figure out. One of the broker's certificates is expiring in 2 days, so the brokers need a roll-restart to pick up the new one. As far as I can tell, this operation also requires topics to be balanced, which is not the case at the moment.

Silencing for 3 months.

Reopening for followup discussion of https://gerrit.wikimedia.org/r/c/operations/deployment-charts/+/1204865

In T408223#11369270, @Scott_French wrote:

I was checking some graphs this evening while finalizing the plan for winding down PHP_ENGINE routing tomorrow, and I ran into something odd. Compare mw-api-ext traffic served by its next release over the last 3 days in eqiad vs. codfw (note: you can see the same effect on the set of releases that share an endpoints object with main but it's much noisier).

Specifically, what shifted a bunch of mw-api-ext traffic from codfw to eqiad around 15:00 UTC on the 11th and 9:00 UTC on the 12th?

I then realized this has to be when https://gerrit.wikimedia.org/r/1198936 and https://gerrit.wikimedia.org/r/1198937 were applied across the CDN fleet.

Although we created rest-gateway-ro.discovery.wmnet and updated multi-dc.lua to ensure A/A routing to rest-gateway works, rest-gateway itself does not implement A/A routing. Stated differently, any mw-api-ext-bound traffic routed via rest-gateway will always hit the primary DC on the upstream side.

@Blake will handle this one

Puppet updated, but we've got some work to do so the hosts can be racked in E/F (see https://phabricator.wikimedia.org/T405285#11350683 )
We'll get back to you on this asap.

Puppet updated

Puppet updated

Puppet updated

After thinking about it a bit, I tend to agree with @pmiazga
Adding Traffic for their opinion

With a little bit of tweaking to the regex for api-gateway we now have correctly labeled metrics, and a (somewhat) useful rate limit graph

I merged the mapping for the rest-gateway and exported metrics look pretty good:

cgoubert@deploy2002:/srv/deployment-charts/helmfile.d/services/rest-gateway$ curl http://localhost:9090/metrics | grep service_ | grep -v '#'
[...]
ratelimit_service_rest_gateway_near_limit{policy="experiment-2025-shadow",unit="HOUR",user_class="anon"} 1
ratelimit_service_rest_gateway_over_limit{policy="experiment-2025-shadow",unit="HOUR",user_class="anon"} 10
ratelimit_service_rest_gateway_shadow_mode{policy="experiment-2025-shadow",unit="HOUR",user_class="anon"} 10
ratelimit_service_rest_gateway_total_hits{policy="experiment-2025-shadow",unit="HOUR",user_class="anon"} 13
ratelimit_service_rest_gateway_within_limit{policy="experiment-2025-shadow",unit="HOUR",user_class="anon"} 3