Change #1052791 had a related patch set uploaded (by Ottomata; author: Ottomata):

[operations/puppet@production] mediawiki.org - Apache rewrite /beacon/event -> /w/beacon/event.php

https://gerrit.wikimedia.org/r/1052791

In T353817#9961942, @Ottomata wrote:

@Joe, to not squat w/beacon.php, in case there are other usages of /beacon that we'd want to route, would this perhaps be better at w/beacon/event.php ?

/beacon/statsv powers statsv.

Indeed! I should have said:

"in case there are other usages of /beacon that we'd want to route to a PHP endpoint"

:)

In T353817#9959609, @Joe wrote:

Personal opinion, we should never ever turn off DirectorySlash. It's a perfect landmine waiting to be stepped onto, apart from the other risks.

Fair enough. I don't think we need to turn off DirectorySlash. I was pointing out the cause as-is.

Can we use a rewrite rule for /beacon/event?foo to /beacon/event/index.php? I would assume rewrites run well before Apache tries to serve a directory index, as it would replace the default destination.

My personal take on the matter is that the new endpoint should've been created under /w, we should've made it restricted to mediawiki.org there, and we should've just had a rewrite rule to direct traffic to it.

We have no precedence (with the exception of stuff we long-deprecated and removed for good reasons) for extensions, let alone individual wikis, creating additional top-level entry points. This would thus add non-trivial complexity that require more people in more places long-term to be aware of and see this. I think that's quite expensive and worth avoiding. Even if we do that, it would still be very far from a "normal" entrypoint without a lot more development time spent.

I think based on what is needed here and how we do things normally in MediaWiki, this would be unexpected as an entrypoint. Perhaps with the same information you'd agree, or maybe we have different intuition/preferences when it comes to MediaWiki.

Are there specific costs or downsides with a standalone entrypoint under /beacon/ as currently developed?

In T353817#9961942, @Ottomata wrote:

Move docroot/mediawiki.org/beacon/event/index.php to w/beacon.php

@Joe, to not squat w/beacon.php, in case there are other usages of /beacon that we'd want to route, would this perhaps be better at w/beacon/event.php

I would advise against creating directory structures under w/ as this adds significant long-term complexity and uncertainty to where we allow PHP execution, what directories MW core supports, and what directory we can assume as the base path in multiverison and MediaWiki core Setup.php.

If there are other beacons within MediaWiki PHP in the future, it stands to reason they can share at least a route destination. A condtional statement is not expensive. Creating new entrypoints and requiring further awareness at every level (including monitoring, profiling, debugging, etc) outweighs that easily.

If SRE don't want this as standalone entrypoint in the document root, I would recommend making it a rest.php entrypoint instead, with the Apache rewrite rewriting it to rest.php/eventlogging/v0/beacon/?querystring or some such. I don't see it working out well long-term to have a hybrid where it's kinda-MediaWiki and kinda-not. This would require a new iteration of code rewriting and testing in the EventLogging extension.

If SRE don't want this as standalone entrypoint in the document root, I would recommend making it a rest.php entrypoint instead, with the Apache rewrite rewriting it to rest.php/eventlogging/v0/beacon/?querystring

I think this is a better solution anyway. The main reason for doing this in mediawiki-config docroot was to avoid having to maintain a special rewrite rule. If we have to maintain one anyway, we might as well put the endpoint in EventLogging and rewrite there as you suggest.

I think that will satisfy both you and @Joe, yes?

Change #1052782 abandoned by Ottomata:

[operations/mediawiki-config@master] EventLoggingLegacyProxy - move endpoint to w/beacon/event.php

Reason:

https://phabricator.wikimedia.org/T353817#9962760

https://gerrit.wikimedia.org/r/1052782

Change #1052816 had a related patch set uploaded (by Ottomata; author: Ottomata):

[mediawiki/extensions/EventLogging@master] WIP - Add RestApiLegacyEventBeacon endpoint

https://gerrit.wikimedia.org/r/1052816

I think this is a waste of time and resources to make such an endpoint part of the rest api, including the fact it's going to get published and other clients might start using it. But more importantly, it's yet another specialized endpoint that I can't just see clearly in the same place where others are. I've learned the hard way when moving to k8s that php files under docroots often are sour surprises. Even as part of the rest api, this will at a minimum need to be restricted to just mediawiki.org (how?) and monitored somehow.
It also introduces yet another set of rewrite rules for stuff under rest.php, which we managed to avoid up to now.

I won't start a war of opinions though, I think this is not the best solution but @Ottomata has been already the victim of our bikeshedding here and I will break the circle. I'm ok with any solution at this point :)

get published and other clients might start using it.

I will mark it deprecated outright. It also already is restricted to MediaWikiPingback events only.

this will at a minimum need to be restricted to just mediawiki.org (how?)

I guess just raising exception if being called anywhere but mediawiki.org? Also only having rewrite rule for mediawiki.org

Change #1055443 had a related patch set uploaded (by Ottomata; author: Ottomata):

[operations/mediawiki-config@master] Remove docroot/mediawiki.org/beacon/event/index.php

https://gerrit.wikimedia.org/r/1055443

Change #1052816 merged by jenkins-bot:

[mediawiki/extensions/EventLogging@master] Add Rest\Handler\LegacyBeaconHandler endpoint

https://gerrit.wikimedia.org/r/1052816

@CCicalese_WMF pinged me today on Slack. Apparently MediaWikiPingback events have not been received since July 3rd!

https://grafana-rw.wikimedia.org/d/000000018/eventlogging-schema?orgId=1&var-schema=MediaWikiPingback&from=1719061110385&to=1721653110385

I'm not sure what might be causing this; we haven't done any deployments related to this since June 26th. Looking into it...

Mentioned in SAL (#wikimedia-analytics) [2024-07-22T13:27:14Z] <ottomata> restarting eventlogging-processor on eventlog1003 - something is wrong with the consumer...https://phabricator.wikimedia.org/T353817#10002205

• The eventlogging-client-side topic had plenty of raw events for MediaWikiPingback. So events were being forwarded by varnishkafka just fine.
• The eventlogging_MediaWikiPingback topic had only a handful of events in the last 7 days.
• eventlogging-client-side looks like it has no 'bytes out' since July 3. This means that the kafka consumer is not actually consuming.
• I checked journalctl logs for eventlogging-processer@client-side-* on eventlog1003, nothing relevant.
• I tcpdump port 9092 on eventlog1003, no traffic. The Kafka consumer just looked stuck?
• I restarted eventlogging-processer@client-side-*, and now events look like they are flowing in!

I don't know why this happened. I guess this is just yet another reason to decom the legacy eventlogging backend!

+1 to decom

Update: Add Rest\Handler\LegacyBeaconHandler endpoint (1052816) has been merged and is riding the MW train this week.

Next week we can have another attempt at the decom.

LegacyBeaconHandler is deployed.

Verified that

curl -k -v 'https://www.mediawiki.org/w/rest.php/eventlogging/v0/beacon?%7B%22schema%22%3A%22Test%22%2C%22revision%22%3A15047841%2C%22wiki%22%3A%22472d1e0653ccb71526c091ecd2ea4bbe%22%2C%22event%22%3A%7B%22OtherMessage%22%3A%22OttoTest%5Cu0020Message%22%7D%7D;';

Results an event in the eventlogging_Test kafka topic.

Now we need a rewrite rule... :)

Change #1052791 merged by Ottomata:

[operations/puppet@production] mediawiki.org - Rewrite /beacon/event -> EventLogging rest handler

https://gerrit.wikimedia.org/r/1052791

Mentioned in SAL (#wikimedia-operations) [2024-07-30T17:13:25Z] <otto@deploy1003> Started scap sync-world: mediawiki.org - Apache Rewrite /beacon/event -> EventLogging rest handler - T353817

Mentioned in SAL (#wikimedia-operations) [2024-07-30T17:18:56Z] <otto@deploy1003> Finished scap: mediawiki.org - Apache Rewrite /beacon/event -> EventLogging rest handler - T353817 (duration: 05m 56s)

Deployed the rewrite rule today:

[@deploy1003:/home/otto]↥ $ curl -v -H 'Host: www.mediawiki.org' 'https://mw-api-int-ro.discovery.wmnet:4446/beacon/event?%7B%22schema%22%3A%22Test%22%2C%22revision%22%3A15047841%2C%22wiki%22%3A%22472d1e0653ccb71526c091ecd2ea4bbe%22%2C%22event%22%3A%7B%22OtherMessage%22%3A%22OttoTest%5Cu0020Message%22%7D%7D;';

The requested path (/beacon/event) was not inside the REST API base path (/w/rest.php)

Either the rule needs to actually rewrite the path that MW sees...or something else...

MW REST API explicitly fails of the URI path does not start with what it expects.

I think I need to use the Proxy flag. I need MW to see the request as the target of the rewrite.

I've been trying to test this on mwdebug1002 following https://wikitech.wikimedia.org/wiki/Application_servers/Runbook#Testing_config, but something isn't quite right with the Apache mod_proxy settings I think:

I've tried each of the following:

ProxyPass "/beacon/event" "https://www.mediawiki.org/w/rest.php/eventlogging/v0/beacon"

RewriteRule ^/beacon/event$ https://www.mediawiki.org/w/rest.php/eventlogging/v0/beacon [P]

<Location "/beacon/event">
    ProxyPass "https://www.mediawiki.org/w/rest.php/eventlogging/v0/beacon"
</Location>

All give the same result:

curl -i --connect-to ::$HOSTNAME 'https://www.mediawiki.org/beacon/event?%7B%22schema%22%3A%22Test%22%2C%22revision%22%3A15047841%2C%22wiki%22%3A%22472d1e0653ccb71526c091ecd2ea4bbe%22%2C%22event%22%3A%7B%22OtherMessage%22%3A%22OttoTest%5Cu0020Message%22%7D%7D;';

...
HTTP/1.1 500 Internal Server Error

In /var/log/apache2.log, I see:

Jul 30 18:46:16 mwdebug1002 apache2[17666]: [proxy:warn] [pid 17666:tid 140703483356928] [client 2620:0:861:101:10:64:0:46:41014] AH01144: No protocol handler was valid for the URL /beacon/event (scheme 'https'). If you are using a DSO version of mod_proxy, make sure the proxy submodules are included in the configuration using LoadModule.

It looks like mod_proxy needs some other module enabled to handle proxying to https?

I believe we'd need modules:

• proxy_http
• ssl
• SSLProxyEngine on

I'm at an impasse again. @Joe would appreciate advice.

Should I abort the REST API solution after all?

@Joe is going to have a go at getting a working rewrite rule with the MW REST API. If not, I'll move this back to a php file in mediawiki-config.

I can confirm locally that

RewriteRule ^/beacon/event$ /w/rest.php/eventlogging/v0/beacon [P]

Works with proxy_http module enabled. In prod, I think we need either:

• we need mod ssl and SSLProxyEngine on to allow the request to be proxied to https://www.mediawiki.org/w/rest.php/eventlogging/v0/beacon
• to explicitly proxy the request to the local non https apache port? I'm not sure how to do that.

@Joe got back to me in IRC. After trying a few things, he suggests going back to the standalone docroot PHP script approach.

Will do...

Actually, idea:

• we already have special VCL based handling of /beacon/event.
• Would it be worse to do custom routing to mediawiki.org/w/rest.php/EventLogging/v0/beacon in frontends (varnish or ATS?) than in Apache? If so, perhaps a proxy routing there would be easier to do now than in our MW Apache setup?

I'll ask the Traffic team. cc @ssingh ?

Discussed in #wikimedia-traffic IRC. Traffic prefers routing to be in Apache if possible. So I'll proceed with the php docroot file + mod_rewrite solution.

Change #1063219 had a related patch set uploaded (by Ottomata; author: Ottomata):

[mediawiki/extensions/EventLogging@master] Revert Rest LegacyBeaconHandler back to EventLoggingLegacyConverter

https://gerrit.wikimedia.org/r/1063219

Change #1063219 abandoned by Ottomata:

[mediawiki/extensions/EventLogging@master] Revert Rest LegacyBeaconHandler back to EventLoggingLegacyConverter

https://gerrit.wikimedia.org/r/1063219

Change #1063211 had a related patch set uploaded (by Ottomata; author: Ottomata):

[mediawiki/extensions/EventLogging@master] Revert Rest LegacyBeaconHandler back to EventLoggingLegacyConverter

https://gerrit.wikimedia.org/r/1063211

Change #1063222 had a related patch set uploaded (by Ottomata; author: Ottomata):

[operations/mediawiki-config@master] mediawiki.org/beacon/event/index.php - use EventLoggingLegacyConverter::submitEvent

https://gerrit.wikimedia.org/r/1063222

Change #1063224 had a related patch set uploaded (by Ottomata; author: Ottomata):

[operations/puppet@production] Rewrite mediawiki.org/beacon/event to /beacon/event/index.php

https://gerrit.wikimedia.org/r/1063224

Change #1055443 abandoned by Ottomata:

[operations/mediawiki-config@master] Remove docroot/mediawiki.org/beacon/event/index.php

Reason:

Ie194d0142b97c3288bc6c15584f7a5e161fcbdc9 instead

https://gerrit.wikimedia.org/r/1055443

Alright. I've prepared the following patches:

• EventLogging - Revert Rest LegacyBeaconHandler back to EventLoggingLegacyConverter (1063211)
• mediawiki-config - mediawiki.org/beacon/event/index.php - use EventLoggingLegacyConverter::submitEvent (1063222)
• puppet - Rewrite mediawiki.org/beacon/event to /beacon/event/index.php (1063224)

I've left the mediawiki-config PHP file in docroot/mediawiki.org/beacon/event/index.php, and made the Apache rule rewrite to this as suggested by @Krinkle in T353817#9962622. This is counter to @Joe's recommendation in T353817#9959609 to put the PHP file in /w/beacon.php. We need a RewriteRule wherever this endpoint lives, so ultimately it doesn't really matter. I think mediawiki.org/beacon/event/index.php is advantageous because /w/beacon.php would be available on all wikis, and we only want this endpoint on mediawiki.org.

@Joe if you have strong objections to this, lets discuss.

Change #1063211 merged by jenkins-bot:

[mediawiki/extensions/EventLogging@master] Revert Rest LegacyBeaconHandler back to EventLoggingLegacyConverter

https://gerrit.wikimedia.org/r/1063211

I just manually copied the docroot/mediawiki.org/beacon/event/index.php file to mwdebug1002 and tried to test. I got a surprising error.

EventLoggingLegacyConverter: Failed proxying legacy EventLogging event query string to WMF Event Platform JSON: Error: Class 'MediaWiki\Extension\EventLogging\EventLogging' not found</code></p></div>

My logstash fu is poor, so I wasn't able to find the stack trace, but I assume it is via the call to EventLoggingLegacyConverter::submitEvent on line 35, which calls EventLogging::submit.

Do I have to do something other than

require MWMultiVersion::getMediaWiki( 'extensions/EventLogging/includes/Libs/Legacy/EventLoggingLegacyConverter.php' );

use MediaWiki\Extension\EventLogging\Libs\Legacy\EventLoggingLegacyConverter;

to load classes from the EventLogging extension?

Note: to reproduce

curl -H 'X-Wikimedia-Debug: backend=mwdebug1002.eqiad.wmnet' -v 'https://www.mediawiki.org/test-beacon/event/?%7B%22schema%22%3A%22Test%22%2C%22revision%22%3A15047841%2C%22wiki%22%3A%22472d1e0653ccb71526c091ecd2ea4bbe%22%2C%22event%22%3A%7B%22OtherMessage%22%3A%22OttoTest%5Cu0020Message%22%7D%7D;';

As of now though 2024-12-05, the index.php file is an old one and will cause a different error. https://gerrit.wikimedia.org/r/c/operations/mediawiki-config/+/1063222/3/docroot/mediawiki.org/beacon/event/index.php#35 will need manually copied to mwdebug1002 to fully reproduce.

Ah ha! I am bootstrapping MW in the wrong way. Reedy helped me here.

I need to load includes/WebStart.php.

Change #1063222 merged by jenkins-bot:

[operations/mediawiki-config@master] mediawiki.org/beacon/event/index.php - use EventBus->send

https://gerrit.wikimedia.org/r/1063222

Mentioned in SAL (#wikimedia-operations) [2024-12-11T16:10:41Z] <otto@deploy2002> Started scap sync-world: Backport for [[gerrit:1063222|mediawiki.org/beacon/event/index.php - use EventBus->send (T353817)]]

Mentioned in SAL (#wikimedia-operations) [2024-12-11T16:16:21Z] <otto@deploy2002> otto: Backport for [[gerrit:1063222|mediawiki.org/beacon/event/index.php - use EventBus->send (T353817)]] synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug)

Mentioned in SAL (#wikimedia-operations) [2024-12-11T16:22:17Z] <otto@deploy2002> Finished scap sync-world: Backport for [[gerrit:1063222|mediawiki.org/beacon/event/index.php - use EventBus->send (T353817)]] (duration: 11m 36s)

mediawiki-config - mediawiki.org/beacon/event/index.php (1063222)

^ is merged and deployed.

puppet - Rewrite mediawiki.org/beacon/event to /beacon/event/index.php (1063224)

This is next.

Instructions for deploying apache rewrite rule from @Scott_French (thank you!)

1. Find a window where there is no risk anyone else will deploy (coordinate that in -operations as needed)
2. Merge your change and puppet-merge
3. [optional, but IMO a good idea] Test the change by running puppet on one of your preferred mwdebug hosts and testing against that host
4. Run puppet on the deployment server if that has not already happened
5. scap sync-world --pause-after-testserver-sync and during the pause, test against mwdebug.discovery.wmnet (continuing if that does what you want)

Change #1103366 had a related patch set uploaded (by Ottomata; author: Ottomata):

[operations/puppet@production] httpbb - add mediawiki.org/beacon/event test for legacy EventLogging beacon

https://gerrit.wikimedia.org/r/1103366

In slack, @Joe advised:

if you want to make sure the behaviour of your patch is correct and that it will stay correct in the future, you should add one HTTPbb test that a) you can run yourself when testing things after syncing to the test servers b) will be tested by scap during every release and c) will be checked against our backends regularly by our monitoring.

https://wikitech.wikimedia.org/wiki/Httpbb

Done here:
httpbb - add mediawiki.org/beacon/event test for legacy EventLogging beacon (1103366)

This test will fail until the apache rewrite rule is deployed, so let's wait until we do that to merge it.

Change #1063224 merged by Ottomata:

[operations/puppet@production] Rewrite mediawiki.org/beacon/event to /beacon/event/index.php

https://gerrit.wikimedia.org/r/1063224

Mentioned in SAL (#wikimedia-operations) [2024-12-16T18:31:08Z] <otto@deploy2002> Started scap sync-world: T353817 - Apache rewrite mediawiki.org/beacon/event to /beacon/event/index.php

Mentioned in SAL (#wikimedia-operations) [2024-12-16T18:35:10Z] <otto@deploy2002> otto: T353817 - Apache rewrite mediawiki.org/beacon/event to /beacon/event/index.php synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug)

Mentioned in SAL (#wikimedia-operations) [2024-12-16T19:03:00Z] <swfrench@deploy2002> Started scap sync-world: T353817 - Additional deployment to clear remaining diffs

Mentioned in SAL (#wikimedia-operations) [2024-12-16T19:05:51Z] <swfrench@deploy2002> Finished scap sync-world: T353817 - Additional deployment to clear remaining diffs (duration: 02m 51s)

Change #1103366 merged by Scott French:

[operations/puppet@production] httpbb - add mediawiki.org/beacon/event test for legacy EventLogging beacon

https://gerrit.wikimedia.org/r/1103366

Just verified via (httpbb on deploy2002) that I can produce MediaWikiPingback events to https://www.mediawiki.org/beacon/event:

[@deploy2002:/home/otto] $ cat httpbb_mediawikipingback.yaml
https://www.mediawiki.org:
- path: '/beacon/event?%7B%22schema%22%3A%22MediaWikiPingback%22%2C%22revision%22%3A15781718%2C%22wiki%22%3A%22538005c667d2152d695df1be4a1c0eb4%22%2C%22event%22%3A%7B%22database%22%3A%22mysql%22%2C%22MediaWiki%22%3A%221.31.1%22%2C%22PHP%22%3A%227.4.33%22%2C%22OS%22%3A%22Linux%5Cu00204.4.400-icpu-101%22%2C%22arch%22%3A64%2C%22machine%22%3A%22x86_64%22%2C%22serverSoftware%22%3A%22otto%22%7D%7D;'
  assert_status: 204

[@deploy2002:/home/otto] $ httpbb ./httpbb_mediawikipingback.yaml --hosts=mwdebug.discovery.wmnet --https_port=4444

 [@stat1010:/home/otto] $ kafkacat -u -q -C -b kafka-jumbo1007.eqiad.wmnet:9092 -t eventlogging_MediaWikiPingback -o end | grep -v canary | grep otto

{"schema":"MediaWikiPingback","revision":15781718, [...], "serverSoftware":"otto" [...] }

Change #1105076 had a related patch set uploaded (by Ottomata; author: Ottomata):

[operations/puppet@production] Disable varnish handling of /beacon/event on cp1100

https://gerrit.wikimedia.org/r/1105076

Change #1105078 had a related patch set uploaded (by Ottomata; author: Ottomata):

[operations/puppet@production] Disable varnish handling of /beacon/event to decommission eventlogging backend

https://gerrit.wikimedia.org/r/1105078

Submitted the next 2 patches:

• Disable varnish handling of /beacon/event on cp1100 (1105076)
• Disable varnish handling of /beacon/event to decommission eventlogging backend (1105078)

The cp1100 will allow us to test routing from just one frontend. If that works, we can move on to do it all!

Change #1105076 merged by Ottomata:

[operations/puppet@production] Disable varnish handling of /beacon/event on cp1100

https://gerrit.wikimedia.org/r/1105076

• Merged Disable varnish handling of /beacon/event on cp1100 (1105076)

Tested by sshing to cp1100, then:

$ curl -k -H 'Host: www.mediawiki.org'  -v 'https://127.0.0.1/beacon/event?%7B%22schema%22%3A%22Test%22%2C%22revision%22%3A15047841%2C%22wiki%22%3A%22472d1e0653ccb71526c091ecd2ea4bbe%22%2C%22event%22%3A%7B%22OtherMessage%22%3A%22OttoTest%5Cu0020Message%22%7D%7D;';

< HTTP/2 204
# Verified Test event makes it to Kafka!

$ curl -k -H 'Host: www.mediawiki.org'  -v 'https://127.0.0.1/beacon/event?%7B%22schema%22%3A%22MediaWikiPingback%22%2C%22revision%22%3A15781718%2C%22wiki%22%3A%22538005c667d2152d695df1be4a1c0eb4%22%2C%22event%22%3A%7B%22database%22%3A%22mysql%22%2C%22MediaWiki%22%3A%221.31.1%22%2C%22PHP%22%3A%227.4.33%22%2C%22OS%22%3A%22Linux%5Cu00204.4.400-icpu-101%22%2C%22arch%22%3A64%2C%22machine%22%3A%22x86_64%22%2C%22serverSoftware%22%3A%22otto%22%7D%7D;';

< HTTP/2 204
# Verified MediaWikiPingback event makes it to Kafka!

Mentioned in SAL (#wikimedia-operations) [2025-01-07T17:50:39Z] <ottomata> Disable varnish handling of /beacon/event to decommission eventlogging backend [puppet] - T238230 T353817

Change #1105078 merged by Ottomata:

[operations/puppet@production] Disable varnish handling of /beacon/event to decommission eventlogging backend

https://gerrit.wikimedia.org/r/1105078

Disable varnish handling of /beacon/event to decommission eventlogging backend (1105078) has been deployed.

Tested locally:

curl -k  -v 'https://www.mediawiki.org/beacon/event?%7B%22schema%22%3A%22MediaWikiPingback%22%2C%22revision%22%3A15781718%2C%22wiki%22%3A%22538005c667d2152d695df1be4a1c0eb4%22%2C%22event%22%3A%7B%22database%22%3A%22mysql%22%2C%22MediaWiki%22%3A%221.31.1%22%2C%22PHP%22%3A%227.4.33%22%2C%22OS%22%3A%22Linux%5Cu00204.4.400-icpu-101%22%2C%22arch%22%3A64%2C%22machine%22%3A%22x86_64%22%2C%22serverSoftware%22%3A%22otto%22%7D%7D;';

< HTTP/2 204
# Verified MediaWikiPingback event makes it to Kafka!

MediaWikiPingback events are flowing through EventGate! https://grafana.wikimedia.org/goto/qKlDsWDHg?orgId=1
No change in event volume in kafka topic: https://grafana.wikimedia.org/goto/jfR5sWDNg?orgId=1

Looking good!

No raw eventlogging events coming in!

https://grafana.wikimedia.org/goto/ntrotVvNg?orgId=1

We can decom!

Change #1115111 had a related patch set uploaded (by Krinkle; author: Ottomata):

[operations/mediawiki-config@master] mediawiki.org/beacon/event - don't raise error on failure

https://gerrit.wikimedia.org/r/1115111

Change #1115111 merged by jenkins-bot:

[operations/mediawiki-config@master] mediawiki.org/beacon/event - don't raise error on failure

https://gerrit.wikimedia.org/r/1115111

Mentioned in SAL (#wikimedia-operations) [2025-02-18T15:15:26Z] <otto@deploy2002> Started scap sync-world: Backport for [[gerrit:1115111|mediawiki.org/beacon/event - don't raise error on failure (T383939 T353817)]]

Mentioned in SAL (#wikimedia-operations) [2025-02-18T15:20:06Z] <otto@deploy2002> otto: Backport for [[gerrit:1115111|mediawiki.org/beacon/event - don't raise error on failure (T383939 T353817)]] synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug)

Mentioned in SAL (#wikimedia-operations) [2025-02-18T15:27:38Z] <otto@deploy2002> Finished scap sync-world: Backport for [[gerrit:1115111|mediawiki.org/beacon/event - don't raise error on failure (T383939 T353817)]] (duration: 12m 12s)