tchin opened https://gitlab.wikimedia.org/repos/data-engineering/eventgate-wikimedia/-/merge_requests/24

Migrate to service-utils (eventgate > 1.9.0)

tchin merged https://gitlab.wikimedia.org/repos/data-engineering/eventgate-wikimedia/-/merge_requests/24

Migrate to service-utils (eventgate > 1.9.0)

Change #1189260 had a related patch set uploaded (by TChin; author: TChin):

[operations/deployment-charts@master] [eventgate-*] Bump to v1.22.0 (service-utils)

https://gerrit.wikimedia.org/r/1189260

Change #1189260 merged by jenkins-bot:

[operations/deployment-charts@master] [eventgate-*] Bump to v1.22.0 (service-utils)

https://gerrit.wikimedia.org/r/1189260

Change #1189876 had a related patch set uploaded (by TChin; author: TChin):

[mediawiki/extensions/EventLogging@master] Use Node 20

https://gerrit.wikimedia.org/r/1189876

Change #1189876 merged by jenkins-bot:

[mediawiki/extensions/EventLogging@master] Use Node 20

https://gerrit.wikimedia.org/r/1189876

tchin opened https://gitlab.wikimedia.org/repos/data-engineering/service-utils/-/merge_requests/16

Add service name as a static label for metrics

tchin opened https://gitlab.wikimedia.org/repos/data-engineering/eventgate/-/merge_requests/8

Fix winston default logger

tchin opened https://gitlab.wikimedia.org/repos/data-engineering/eventgate-wikimedia/-/merge_requests/25

Remove service-runner abstraction over the metrics interface

tchin merged https://gitlab.wikimedia.org/repos/data-engineering/service-utils/-/merge_requests/16

Add service name as a static label for metrics

tchin merged https://gitlab.wikimedia.org/repos/data-engineering/eventgate/-/merge_requests/8

Fix winston default logger

tchin merged https://gitlab.wikimedia.org/repos/data-engineering/eventgate-wikimedia/-/merge_requests/25

Remove service-runner abstraction over the metrics interface

Change #1190681 had a related patch set uploaded (by TChin; author: TChin):

[operations/deployment-charts@master] [eventgate-*] Bump to v1.23.0 (service-utils)

https://gerrit.wikimedia.org/r/1190681

Change #1190681 merged by jenkins-bot:

[operations/deployment-charts@master] [eventgate-*] Bump to v1.23.0 (service-utils)

https://gerrit.wikimedia.org/r/1190681

Deployed to eventgate-logging-external for codfw, it works in the sense that it didn't blow up, but will have to fix some stuff before I deploy the rest of it. Logs export fine in ECS, but for some reason a lot of fields are being dropped. Metrics show up in the dashboards but some need renaming, and I also forgot to add metrics for the express routes.

Also these three metrics stopped being reported, which I don't really know why since from what I can tell it's a Kubernetes metric
nodejs_process_heap_used_bytes
nodejs_process_heap_total_bytes
nodejs_process_heap_rss_bytes

tchin opened https://gitlab.wikimedia.org/repos/data-engineering/service-utils/-/merge_requests/17

Refactor wikimedia ECS middleware

tchin merged https://gitlab.wikimedia.org/repos/data-engineering/service-utils/-/merge_requests/17

Refactor wikimedia ECS middleware

tchin opened https://gitlab.wikimedia.org/repos/data-engineering/eventgate/-/merge_requests/9

Add express metrics and bump service utils

tchin merged https://gitlab.wikimedia.org/repos/data-engineering/eventgate/-/merge_requests/9

Add express metrics and bump service utils

tchin opened https://gitlab.wikimedia.org/repos/data-engineering/eventgate-wikimedia/-/merge_requests/26

Bump eventgate to 1.9.2

tchin merged https://gitlab.wikimedia.org/repos/data-engineering/eventgate-wikimedia/-/merge_requests/26

Bump eventgate to 1.9.2

Change #1191234 had a related patch set uploaded (by TChin; author: TChin):

[operations/deployment-charts@master] [eventgate-*] Bump to v1.24.0

https://gerrit.wikimedia.org/r/1191234

Change #1191234 merged by jenkins-bot:

[operations/deployment-charts@master] [eventgate-*] Bump to v1.24.0

https://gerrit.wikimedia.org/r/1191234

tchin opened https://gitlab.wikimedia.org/repos/data-engineering/eventgate/-/merge_requests/10

Fix nested http request logs

tchin merged https://gitlab.wikimedia.org/repos/data-engineering/eventgate/-/merge_requests/10

Fix nested http request logs

tchin opened https://gitlab.wikimedia.org/repos/data-engineering/eventgate-wikimedia/-/merge_requests/27

Bump eventgate to 1.9.3

tchin merged https://gitlab.wikimedia.org/repos/data-engineering/eventgate-wikimedia/-/merge_requests/27

Bump eventgate to 1.9.3

Change #1191506 had a related patch set uploaded (by TChin; author: TChin):

[operations/deployment-charts@master] [eventgate_*] Bump eventgate to v1.25.0

https://gerrit.wikimedia.org/r/1191506

Change #1191506 merged by jenkins-bot:

[operations/deployment-charts@master] [eventgate_*] Bump eventgate to v1.25.0

https://gerrit.wikimedia.org/r/1191506

Fully deployed to eventgate-logging-external. Logs seem fine. No log spam, the duplicate dropped fields are fixed, and it's fully ingested into logstash in ECS format now. Metrics also look good. It's a bit fuzzy because this happened during the DC switchover, but looking at the dashboard it seems like all metrics still match except the only one lost is the one I stated before that's under Memory usage (sum over all pods). That doesn't concern me that much since the service is also now being picked up by the new service-utils metrics dashboard so we still have memory reporting.

Deployed to eventgate-analytics-external and it looks stable. Proceeding to deploy to the remaining instances.

Hmmm ok everything is deployed now and it works fine, but I can't tell if the p99 performance got worse, or the express metrics are broken somehow (either broken beforehand and now fixed or vice-versa). What makes me suspicious is that when you look at the latency quantiles by HTTP method from before the deployment, every deployment and every instance had a GET and POST p99 of almost exactly 9.90ms. After the deployment, it's actually correlated with the amount of events it's received. I'm assuming this means that something was actually fixed somewhere, but because of this, alerts are being fired on the passive DC because of the bursty nature of events there and the latency increase that's correlated with it.

Hm, okay! Indeed something does look fishy about the metrics prior to this change.

jforrester opened https://gitlab.wikimedia.org/repos/abstract-wiki/wikifunctions/function-evaluator/-/merge_requests/455

Upgrade @wikimedia/service-utils from v1.4.0 to v1.4.2

jforrester opened https://gitlab.wikimedia.org/repos/abstract-wiki/wikifunctions/function-orchestrator/-/merge_requests/499

Upgrade @wikimedia/service-utils from v1.4.0 to v1.4.2

dmartin merged https://gitlab.wikimedia.org/repos/abstract-wiki/wikifunctions/function-evaluator/-/merge_requests/455

Upgrade @wikimedia/service-utils from v1.4.0 to v1.4.2

dmartin merged https://gitlab.wikimedia.org/repos/abstract-wiki/wikifunctions/function-orchestrator/-/merge_requests/499

Upgrade @wikimedia/service-utils from v1.4.0 to v1.4.2

Change #1224078 had a related patch set uploaded (by Jforrester; author: Jforrester):

[operations/deployment-charts@master] wikifunctions: Upgrade evaluators from 2025-12-10-150641 to 2026-01-06-172129

https://gerrit.wikimedia.org/r/1224078

Change #1224079 had a related patch set uploaded (by Jforrester; author: Jforrester):

[operations/deployment-charts@master] wikifunctions: Upgrade orchestrator from 2025-12-10-133418 to 2026-01-07-132737

https://gerrit.wikimedia.org/r/1224079

Change #1224078 merged by jenkins-bot:

[operations/deployment-charts@master] wikifunctions: Upgrade evaluators from 2025-12-10-150641 to 2026-01-07-132938

https://gerrit.wikimedia.org/r/1224078

Change #1224079 merged by jenkins-bot:

[operations/deployment-charts@master] wikifunctions: Upgrade orchestrator from 2025-12-10-133418 to 2026-01-07-132737

https://gerrit.wikimedia.org/r/1224079