Page MenuHomePhabricator
Create Task
Maniphest T347916

Investigate sharp increase in lost Arc Lamp samples (arclamp_client_error.exception)
Closed, ResolvedPublic
Actions

Description

https://grafana.wikimedia.org/d/yVf-D1RWk/arc-lamp?orgId=1

Screenshot 2023-10-02 at 16.22.55.png (1×1 px, 228 KB)

@fgiunchedi mentioned this ramp up likely correlates with MW-on-k8s rollout.

Details

SubjectRepoBranchLines +/-
Profiler: Enable logging of caught Redis exceptions to Logstashoperations/mediawiki-configmaster+1 -0
Customize query in gerrit

Related Objects

Event Timeline

Krinkle created this task.Oct 2 2023, 11:24 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptOct 2 2023, 11:24 PM
gerritbot added a comment.Oct 2 2023, 11:24 PM

Change 962725 had a related patch set uploaded (by Krinkle; author: Krinkle):

[operations/mediawiki-config@master] Profiler: Enable logging of caught Redis exceptions to Logstash

https://gerrit.wikimedia.org/r/962725

gerritbot added a project: Patch-For-Review.Oct 2 2023, 11:24 PM
gerritbot added a comment.Oct 3 2023, 2:31 AM

Change 962725 merged by jenkins-bot:

[operations/mediawiki-config@master] Profiler: Enable logging of caught Redis exceptions to Logstash

https://gerrit.wikimedia.org/r/962725

Krinkle mentioned this in T347926: Excimer UI profile lost when requested from mw-on-k8s.Oct 3 2023, 2:46 AM
Maintenance_bot removed a project: Patch-For-Review.Oct 3 2023, 3:10 AM
fgiunchedi mentioned this in T347987: PHP Warning: RedisException: Connection timed out.Oct 3 2023, 1:11 PM
fgiunchedi added a comment.Oct 3 2023, 2:44 PM

Allowing egress redis to arclamp hosts did the trick, no exceptions anymore from arclamp

fgiunchedi closed this task as Resolved.Oct 3 2023, 2:59 PM
fgiunchedi claimed this task.

Optimistically resolving

Krinkle added a comment.Oct 3 2023, 5:22 PM

Quoting here for reference, as it tagged a different webperf issue by accident:

In T347926#9220705, @gerritbot wrote:

Change 963024 merged by Filippo Giunchedi:

[operations/deployment-charts@master] services: fix xenon/arclamp redis egress rules

https://gerrit.wikimedia.org/r/963024

lmata added a project: SRE Observability (FY2023/2024-Q2).Oct 9 2023, 2:10 PM
lmata moved this task from Inbox to Done on the SRE Observability (FY2023/2024-Q2) board.Jan 26 2024, 1:08 AM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits