Page MenuHomePhabricator
Create Task
Maniphest T414085

Proof of Concept: SquareOne OnCall Dashboard
Open, In Progress, MediumPublic
Actions

Description

Create incident-focused SquareOne dashboards (could be more than one but no more than 3) that serve as the primary entry points for on-call responders during incidents.

Design

  • Infobox with useful oncall information
  • Traffic status (text+upload)
    • Precooked Turnilo URLs
  • Edge Caches status
  • Mediawiki status (user facing, ie mw-web, mw-ext-int)
  • Kubernetes status
  • Database status
Motivation

Recently there was an alert for

FIRING: ATSBackendErrorsHigh: ATS: elevated 5xx errors from swift.discovery.wmnet in eqiad #page - https://wikitech.wikimedia.org/wiki/Apache_Traffic_Server#Debugging - https://grafana.wikimedia.org/d/1T_4O08Wk/ats-backends-origin-servers-overview?orgId=1&viewPanel=12&var-site=eqiad&var-cluster=upload&var-origin=swift.discovery.wmnet - https://alerts.wikimedia.org/?q=alertname%3DATSBackendErrorsHigh

While there were indeed errors on eqiad, the actual problem was excess traffic on eqsin and magru. Given that we have seen other similar alerts
which had the same root cause (https://logstash.wikimedia.org/goto/6f87337a5da6ea02c30a2499f7711a14). We should consider attacghing a more diagnostic dashboard that helps quickly identify the underlying traffic issues rather than just the symptom of elevated errors.

Drafts

Related Objects
Search...

Event Timeline

jijiki created this task.Jan 8 2026, 1:00 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptJan 8 2026, 1:00 PM
jijiki added projects: serviceops-deprecated, SRE Observability.Jan 8 2026, 1:00 PM
• Marostegui subscribed.Jan 9 2026, 9:03 AM
ABran-WMF subscribed.Jan 9 2026, 9:03 AM
jijiki changed the task status from Open to In Progress.Jan 9 2026, 12:53 PM
jijiki triaged this task as Medium priority.
jijiki edited projects, added ServiceOps new; removed serviceops-deprecated.
jijiki moved this task from Inbox to In Progress on the ServiceOps new board.Jan 9 2026, 12:59 PM
jijiki renamed this task from Create a dashboard to easily visualise upload/media issues to Create a dashboard to easily visualise capacity issues for OnCallers.Jan 12 2026, 5:15 PM
jijiki updated the task description. (Show Details)
jijiki renamed this task from Create a dashboard to easily visualise capacity issues for OnCallers to Proof of Concept: SquareOne OnCall Dashboard.Jan 12 2026, 5:44 PM
jijiki added a parent task: T411202: Proof of Concept: MediaWiki SquareOne Dashboards.
hnowlan moved this task from Inbox to Radar on the SRE Observability board.Jan 14 2026, 3:15 PM
jijiki mentioned this in T414663: SquareOne Dashboards: Guided Incident Response.Jan 15 2026, 10:46 AM
jijiki updated the task description. (Show Details)Jan 15 2026, 10:54 AM
jijiki claimed this task.Jan 15 2026, 10:59 AM
jijiki added a parent task: T414663: SquareOne Dashboards: Guided Incident Response.
jijiki created subtask T414665: Proof of Concept: SquareOne CDN Dashboards.Jan 15 2026, 11:07 AM
jijiki removed a parent task: T414663: SquareOne Dashboards: Guided Incident Response.
jijiki edited parent tasks, added: T414663: SquareOne Dashboards: Guided Incident Response; removed: T411202: Proof of Concept: MediaWiki SquareOne Dashboards.Jan 15 2026, 11:10 AM
jijiki changed the status of subtask T414665: Proof of Concept: SquareOne CDN Dashboards from Open to In Progress.
Joe subscribed.Jan 15 2026, 12:17 PM

I like the idea of having a full-stack view of traffic problems, especially on the media-serving side of things, which I'd focus on more at first. I would actually create two separate dashboards for text and media-serving.

I like a lot less the "square one" name, but I won't shave that yak :)

jijiki added a comment.Jan 15 2026, 12:28 PM
In T414085#11524639, @Joe wrote:

I like the idea of having a full-stack view of traffic problems, especially on the media-serving side of things, which I'd focus on more at first. I would actually create two separate dashboards for text and media-serving.

I agree, it became quite evident as I was working on the OnCall one. I created some drafts last week, and opened T414665. I will add this to the description too for clarity. Thanks!

I like a lot less the "square one" name, but I won't shave that yak :)

Naming things is hard. I was wondering too what better naming scheme would work for us, but decided to revisit it later.

jijiki updated the task description. (Show Details)Jan 15 2026, 12:39 PM
jijiki edited projects, added Incident Tooling; removed SRE Observability.Jan 22 2026, 12:46 PM
MLechvien-WMF moved this task from In Progress to Backlog on the ServiceOps new board.Tue, Feb 10, 11:54 AM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits