Page MenuHomePhabricator
Feed Advanced Search

Apr 4 2024

lmata moved T343028: Audit & convert stats for mediawiki core from Inbox to Done on the SRE Observability (FY2023/2024-Q3) board.
Apr 4 2024, 6:11 PM · SRE Observability (FY2023/2024-Q3), Observability-Metrics
lmata moved T317887: Upgrade to Grafana 9 from Inbox to Done on the SRE Observability (FY2023/2024-Q3) board.
Apr 4 2024, 6:11 PM · SRE Observability (FY2023/2024-Q3), Observability-Metrics
lmata moved T354905: migrate MediaWiki.timing.editResponseTime to statslib from Inbox to Done on the SRE Observability (FY2023/2024-Q3) board.
Apr 4 2024, 6:11 PM · MW-1.42-notes (1.42.0-wmf.18; 2024-02-13), MediaWiki-Platform-Team, Patch-For-Review, SRE Observability (FY2023/2024-Q3), Observability-Metrics
lmata moved T352665: Upgrade Grafana hosts to Bookworm from Up next to Done on the SRE Observability (FY2023/2024-Q3) board.
Apr 4 2024, 6:11 PM · Patch-For-Review, SRE Observability (FY2023/2024-Q3)
lmata moved T354217: Make arc-lamp aware of the new names of request life cycle methods. from Up next to Done on the SRE Observability (FY2023/2024-Q3) board.
Apr 4 2024, 6:11 PM · SRE Observability (FY2023/2024-Q3), Patch-For-Review, Arc-Lamp
lmata moved T357547: ☂️ Northward Datacentre Switchover (March 2024) from Up next to Done on the SRE Observability (FY2023/2024-Q3) board.
Apr 4 2024, 6:11 PM · Patch-For-Review, Datacenter-Switchover, Data-Persistence, SRE Observability (FY2023/2024-Q3), collaboration-services, observability, serviceops, DC-Ops, Traffic
lmata edited projects for T350192: On-call batphone escalation configuration holidays FY2023-24, added: SRE Observability (FY2023/2024-Q4); removed SRE Observability (FY2023/2024-Q3).
Apr 4 2024, 6:09 PM · SRE Observability (FY2023/2024-Q4)
lmata closed T354570: Create alert Review for FY2023/2024-Q3 as Resolved.
Apr 4 2024, 2:31 PM · SRE Observability (FY2023/2024-Q3)
lmata moved T354570: Create alert Review for FY2023/2024-Q3 from Up next to Done on the SRE Observability (FY2023/2024-Q3) board.
Apr 4 2024, 2:30 PM · SRE Observability (FY2023/2024-Q3)

Apr 3 2024

lmata added a comment to T361412: Write PHPUnit tests for excimer-ui-client (WikimediaDebug).

@Krinkle I'll ask someone to take a look then, thanks!

Apr 3 2024, 6:56 PM · SRE Observability (FY2023/2024-Q4), MediaWiki-Platform-Team (Radar), WikimediaDebug
lmata edited projects for T361257: Document Icinga Migration Strategy and Communicate to SRE Teams, added: SRE Observability (FY2023/2024-Q4); removed SRE Observability.
Apr 3 2024, 2:12 PM · SRE Observability (FY2023/2024-Q4)
lmata moved T361412: Write PHPUnit tests for excimer-ui-client (WikimediaDebug) from Inbox to FY2023/2024-Q4 on the SRE Observability board.
Apr 3 2024, 2:11 PM · SRE Observability (FY2023/2024-Q4), MediaWiki-Platform-Team (Radar), WikimediaDebug
lmata added a comment to T361412: Write PHPUnit tests for excimer-ui-client (WikimediaDebug).

Hola @Krinkle, is any assistance needed from our end?

Apr 3 2024, 2:11 PM · SRE Observability (FY2023/2024-Q4), MediaWiki-Platform-Team (Radar), WikimediaDebug
lmata moved T360595: beta-scap-sync-world fails: logstash_checker.py: KeyError: 'aggregations' from Inbox to Radar on the observability board.
Apr 3 2024, 2:07 PM · Infrastructure-Foundations, SRE Observability, observability, Beta-Cluster-Infrastructure, Release-Engineering-Team
lmata moved T360595: beta-scap-sync-world fails: logstash_checker.py: KeyError: 'aggregations' from Inbox to Radar on the SRE Observability board.
Apr 3 2024, 2:06 PM · Infrastructure-Foundations, SRE Observability, observability, Beta-Cluster-Infrastructure, Release-Engineering-Team
lmata added a project to T361229: titan200[12] RAM/SSD upgrade coordination: SRE Observability (FY2023/2024-Q4).
Apr 3 2024, 2:06 PM · SRE Observability (FY2023/2024-Q4), SRE, observability, ops-codfw
lmata moved T361229: titan200[12] RAM/SSD upgrade coordination from Inbox to In progress on the SRE Observability (FY2023/2024-Q4) board.
Apr 3 2024, 2:06 PM · SRE Observability (FY2023/2024-Q4), SRE, observability, ops-codfw
lmata added a project to T361251: titan100[12] ram/ssd upgrade coordination: SRE Observability (FY2023/2024-Q4).
Apr 3 2024, 2:05 PM · SRE Observability (FY2023/2024-Q4), SRE, observability, ops-eqiad
lmata moved T361251: titan100[12] ram/ssd upgrade coordination from Inbox to In progress on the SRE Observability (FY2023/2024-Q4) board.
Apr 3 2024, 2:05 PM · SRE Observability (FY2023/2024-Q4), SRE, observability, ops-eqiad

Apr 2 2024

lmata moved T357614: Alert in need of triage: Number of requests triggering circuit breakers due to excessive memory usage (instance graphite1005) from Inbox to Up next on the SRE Observability (FY2023/2024-Q4) board.
Apr 2 2024, 6:59 PM · SRE Observability (FY2023/2024-Q4), sre-alert-triage
lmata edited projects for T357614: Alert in need of triage: Number of requests triggering circuit breakers due to excessive memory usage (instance graphite1005), added: SRE Observability (FY2023/2024-Q4); removed SRE Observability.

@andrea.denisse could you disable this alert please?

Apr 2 2024, 6:58 PM · SRE Observability (FY2023/2024-Q4), sre-alert-triage
lmata added a comment to T357614: Alert in need of triage: Number of requests triggering circuit breakers due to excessive memory usage (instance graphite1005).

@bking ack! thank you for the confirmation!

Apr 2 2024, 6:57 PM · SRE Observability (FY2023/2024-Q4), sre-alert-triage

Mar 27 2024

lmata edited projects for T354908: evaluate and migrate in-use parsoid metrics to statslib, added: SRE Observability (FY2023/2024-Q4); removed SRE Observability (FY2023/2024-Q3).
Mar 27 2024, 6:03 PM · Content-Transform-Team, OKR-Work, Content-Transform-Team-WIP, SRE Observability (FY2023/2024-Q4), MediaWiki-Platform-Team (Radar), Observability-Metrics
lmata edited projects for T350592: EPIC: migrate in use metrics and dashboards to statslib, added: SRE Observability (FY2023/2024-Q4); removed SRE Observability (FY2023/2024-Q3).
Mar 27 2024, 6:02 PM · Epic, SRE Observability (FY2023/2024-Q4), MW-1.42-notes (1.42.0-wmf.15; 2024-01-23), MediaWiki-Platform-Team (Radar), Observability-Metrics
lmata moved T361110: New VictorOps user request from Inbox to Done on the SRE Observability (FY2023/2024-Q3) board.
Mar 27 2024, 5:51 PM · SRE Observability (FY2023/2024-Q3)
lmata closed T361110: New VictorOps user request as Resolved.
Mar 27 2024, 5:51 PM · SRE Observability (FY2023/2024-Q3)
lmata added a comment to T361110: New VictorOps user request.

I sent an invite and invited you to the SRE team, I also set up a basic Batphone schedule for you doing 8 am-8 pm, but please let me know how you need this tweaked to whatever period you want to be notified in. Thanks!

Mar 27 2024, 5:51 PM · SRE Observability (FY2023/2024-Q3)
lmata added a comment to T358322: Query regarding AQS 2.0 SLOs.

Saw this patch: https://gerrit.wikimedia.org/r/c/operations/grafana-grizzly/+/1009472
is there any we can do to assist?

Mar 27 2024, 3:50 PM · observability
lmata moved T359293: Alert in need of triage: ProbeDown (instance centrallog1002:6514) from Inbox to Backlog on the SRE Observability board.
Mar 27 2024, 3:14 PM · SRE Observability, sre-alert-triage
lmata removed a project from T361110: New VictorOps user request: observability.
Mar 27 2024, 3:06 PM · SRE Observability (FY2023/2024-Q3)
lmata claimed T361110: New VictorOps user request.

Will send an invite shortly.

Mar 27 2024, 3:06 PM · SRE Observability (FY2023/2024-Q3)

Mar 26 2024

lmata moved T326419: Expand kafka-logging using hosts kafka-logging[12]00[45] from In progress to Done on the SRE Observability (FY2023/2024-Q3) board.
Mar 26 2024, 4:02 PM · SRE Observability (FY2023/2024-Q3), User-herron, Observability-Logging

Mar 25 2024

lmata updated the task description for T350192: On-call batphone escalation configuration holidays FY2023-24.
Mar 25 2024, 2:14 PM · SRE Observability (FY2023/2024-Q4)

Mar 22 2024

lmata updated the task description for T350192: On-call batphone escalation configuration holidays FY2023-24.
Mar 22 2024, 3:55 PM · SRE Observability (FY2023/2024-Q4)
lmata updated the task description for T350192: On-call batphone escalation configuration holidays FY2023-24.
Mar 22 2024, 3:55 PM · SRE Observability (FY2023/2024-Q4)

Mar 19 2024

lmata updated the task description for T350592: EPIC: migrate in use metrics and dashboards to statslib.
Mar 19 2024, 11:13 PM · Epic, SRE Observability (FY2023/2024-Q4), MW-1.42-notes (1.42.0-wmf.15; 2024-01-23), MediaWiki-Platform-Team (Radar), Observability-Metrics
lmata moved T337677: Thumbor logging improvements from Inbox to Watching on the Observability-Logging board.
Mar 19 2024, 3:31 PM · Observability-Logging, Thumbor
lmata moved T325143: Logstash dashboard for thumbor from Inbox to Watching on the Observability-Logging board.
Mar 19 2024, 3:31 PM · Observability-Logging, Thumbor, observability, Platform Team Workboards (Platform Engineering Reliability), Wikimedia-Logstash, Thumbor Migration
lmata changed the status of T326419: Expand kafka-logging using hosts kafka-logging[12]00[45] from Open to In Progress.
Mar 19 2024, 3:29 PM · SRE Observability (FY2023/2024-Q3), User-herron, Observability-Logging
lmata added a project to T360414: Phase out cergen for Observability services: SRE Observability (FY2023/2024-Q4).

This all ties into the greater scheme of Puppet 5 and Buster deprecations, so next quarter ideally.

Mar 19 2024, 3:26 PM · Patch-For-Review, SRE Observability (FY2023/2024-Q4), observability, SRE
lmata added a comment to T360414: Phase out cergen for Observability services.

hi @MoritzMuehlenhoff, what timeline do you need us to meet for this change?

Mar 19 2024, 3:11 PM · Patch-For-Review, SRE Observability (FY2023/2024-Q4), observability, SRE

Mar 16 2024

lmata awarded T355960: Migrate MediaWiki.resourceloader* metrics to statslib a Love token.
Mar 16 2024, 12:02 AM · MW-1.42-notes (1.42.0-wmf.22; 2024-03-12), MediaWiki-Platform-Team, SRE Observability (FY2023/2024-Q3), Observability-Metrics

Mar 12 2024

lmata added a comment to T355960: Migrate MediaWiki.resourceloader* metrics to statslib.

It was interesting to learn :) about the weirdness of gauges (especially in the scenario of no data) and how smart Graphite would fill up the gaps if we wanted to.

This patch was merged yesterday meaning it will ride this week's train, I'll leave it open until Friday before resolved.

Mar 12 2024, 10:13 PM · MW-1.42-notes (1.42.0-wmf.22; 2024-03-12), MediaWiki-Platform-Team, SRE Observability (FY2023/2024-Q3), Observability-Metrics
lmata removed a project from T348796: MediaWiki: Define new metric type - Histogram: SRE Observability (FY2023/2024-Q3).
Mar 12 2024, 3:03 PM · Observability-Metrics, MediaWiki-libs-Stats

Mar 11 2024

lmata added a project to T359612: logstash1011 is down: SRE Observability (FY2023/2024-Q3).
Mar 11 2024, 5:57 PM · SRE Observability (FY2023/2024-Q3), Observability-Logging
lmata added a comment to T359632: install (2) 1.92TB SSDs from decom into prometheus100[56].

thank you for all the help and care @Jclark-ctr and @RobH

Mar 11 2024, 5:29 PM · ops-eqiad, SRE, procurement
lmata added a project to T359168: Knowledge sharing MWP - SRE O11Y mw.track(): SRE Observability (FY2023/2024-Q3).
Mar 11 2024, 4:59 PM · SRE Observability (FY2023/2024-Q4), MediaWiki-Platform-Team, Observability-Metrics
lmata added a project to T356815: Migrate MediaWiki.stable_pcache to statslib: SRE Observability (FY2023/2024-Q3).
Mar 11 2024, 4:57 PM · MW-1.42-notes (1.42.0-wmf.25; 2024-04-02), SRE Observability (FY2023/2024-Q3), Observability-Metrics

Mar 6 2024

lmata updated the task description for T350592: EPIC: migrate in use metrics and dashboards to statslib.
Mar 6 2024, 9:08 PM · Epic, SRE Observability (FY2023/2024-Q4), MW-1.42-notes (1.42.0-wmf.15; 2024-01-23), MediaWiki-Platform-Team (Radar), Observability-Metrics
lmata created T359485: Migrate MediaWiki.extension.Phonos to statslib.
Mar 6 2024, 9:07 PM · Observability-Metrics
lmata created T359484: Migrate MediaWiki.ExternalGuidance to statslib.
Mar 6 2024, 9:07 PM · Observability-Metrics
lmata created T359482: Migrate MediaWiki.firefox.100.browsertime.statistics to statslib.
Mar 6 2024, 9:06 PM · Observability-Metrics
lmata updated the task description for T359481: Migrate MediaWiki.FileImporter to statslib.
Mar 6 2024, 9:05 PM · Observability-Metrics
lmata created T359481: Migrate MediaWiki.FileImporter to statslib.
Mar 6 2024, 9:05 PM · Observability-Metrics
lmata created T359480: Migrate MediaWiki.global_blocking to statslib.
Mar 6 2024, 9:03 PM · GlobalBlocking, Trust and Safety Product Team, Observability-Metrics
lmata created T359479: Migrate MediaWiki.globalwatchlist to statslib.
Mar 6 2024, 9:02 PM · Observability-Metrics
lmata created T359478: Migrate MediaWiki.hooks.*.count to statslib.
Mar 6 2024, 9:01 PM · Observability-Metrics
lmata created T359477: Migrate MediaWiki.html2wt to statslib.
Mar 6 2024, 9:00 PM · Observability-Metrics
lmata created T359476: Migrate MediaWiki.htmloutputrendererhelper to statslib.
Mar 6 2024, 8:59 PM · Observability-Metrics
lmata created T359475: Migrate MediaWiki.html_input_transform.original_html to statslib.
Mar 6 2024, 8:59 PM · Observability-Metrics
lmata created T359474: Migrate MediaWiki.ipinfo_address_copy to statslib.
Mar 6 2024, 8:58 PM · Trust and Safety Product Team, Observability-Metrics
lmata created T359473: Migrate MediaWiki.jobexecutor.PublishStashedFile.exec.mean to statslib.
Mar 6 2024, 8:58 PM · Observability-Metrics
lmata created T359472: Migrate MediaWiki.jobqueue to statslib.
Mar 6 2024, 8:57 PM · Observability-Metrics
lmata created T359471: Migrate MediaWiki.extension.PageTriage to statslib.
Mar 6 2024, 8:56 PM · Moderator-Tools-Team, PageTriage, Observability-Metrics
lmata created T359470: Migrate MediaWiki.extdist to statslib.
Mar 6 2024, 8:55 PM · Observability-Metrics
lmata created T359468: Migrate MediaWiki.errors.fatal.rate to statslib.
Mar 6 2024, 8:54 PM · Observability-Metrics
lmata renamed T359467: (mw.track) Migrate MediaWiki.EntitySchema to statslib from Migrate <metric(s)> to statslib to Migrate MediaWiki.EntitySchema to statslib.
Mar 6 2024, 8:54 PM · Observability-Metrics
lmata created T359467: (mw.track) Migrate MediaWiki.EntitySchema to statslib.
Mar 6 2024, 8:53 PM · Observability-Metrics
lmata created T359466: Migrate MediaWiki.electronpdf to statslib.
Mar 6 2024, 8:53 PM · Observability-Metrics
lmata created T359465: Migrate MediaWiki.editstash to statslib.
Mar 6 2024, 8:52 PM · MW-1.43-notes (1.43.0-wmf.10; 2024-06-18), MW-Interfaces-Team, Observability-Metrics
lmata created T359464: Migrate MediaWiki.editor.milestones.timing.1.median to statslib.
Mar 6 2024, 8:52 PM · Observability-Metrics
lmata created T359463: Migrate MediaWiki.discussiontools.addReplyLinks to statslib.
Mar 6 2024, 8:51 PM · MW-1.43-notes (1.43.0-wmf.4; 2024-05-07), Observability-Metrics
lmata created T359462: Migrate MediaWiki.Cognate.Repo to statslib.
Mar 6 2024, 8:51 PM · Observability-Metrics
lmata created T359461: (Analytics?) Migrate MediaWiki.CodeMirror to statslib.
Mar 6 2024, 8:50 PM · Observability-Metrics
lmata created T359460: Migrate MediaWiki.chrome.100.browsertime.statistics to statslib.
Mar 6 2024, 8:49 PM · Observability-Metrics
lmata created T359459: Migrate MediaWiki.bouncehandler to statslib.
Mar 6 2024, 8:48 PM · Patch-For-Review, Observability-Metrics
lmata updated the task description for T350592: EPIC: migrate in use metrics and dashboards to statslib.
Mar 6 2024, 8:46 PM · Epic, SRE Observability (FY2023/2024-Q4), MW-1.42-notes (1.42.0-wmf.15; 2024-01-23), MediaWiki-Platform-Team (Radar), Observability-Metrics
lmata created T359458: (mw.track) Migrate MediaWiki.BlockNotices.* to statslib.
Mar 6 2024, 8:46 PM · Trust and Safety Product Team, Observability-Metrics
lmata created T359457: Migrate MediaWiki.Parsoid.extension.* to statslib.
Mar 6 2024, 8:45 PM · Observability-Metrics
lmata created T359455: Migrate MediaWiki.Parsoid.entry.wt2html to statslib.
Mar 6 2024, 8:43 PM · Observability-Metrics
lmata created T359454: Migrate MediaWiki.Parsoid.entry.html2wt to statslib.
Mar 6 2024, 8:42 PM · Observability-Metrics
lmata created T359453: Migrate MediaWiki.Parsoid.html2wt to statslib.
Mar 6 2024, 8:40 PM · Observability-Metrics
lmata updated the task description for T350592: EPIC: migrate in use metrics and dashboards to statslib.
Mar 6 2024, 2:55 PM · Epic, SRE Observability (FY2023/2024-Q4), MW-1.42-notes (1.42.0-wmf.15; 2024-01-23), MediaWiki-Platform-Team (Radar), Observability-Metrics
lmata created T359402: Migrate MediaWiki.parsoidhelper.stashing to statslib.
Mar 6 2024, 2:55 PM · Observability-Metrics
lmata updated the task description for T350592: EPIC: migrate in use metrics and dashboards to statslib.
Mar 6 2024, 2:43 PM · Epic, SRE Observability (FY2023/2024-Q4), MW-1.42-notes (1.42.0-wmf.15; 2024-01-23), MediaWiki-Platform-Team (Radar), Observability-Metrics
lmata created T359398: Migrate MediaWiki.resourceloader_startup_bytes to statslib.
Mar 6 2024, 2:42 PM · Observability-Metrics
lmata created T359397: Migrate MediaWiki.resourceloader_module_decodedsize_bytes to statslib.
Mar 6 2024, 2:42 PM · MediaWiki-Platform-Team (Radar), Observability-Metrics
lmata created T359396: Migrate MediaWiki.resourceloader_build to statslib.
Mar 6 2024, 2:41 PM · Observability-Metrics
lmata created T359395: Migrate MediaWiki.Parsoid.html2wt to statslib.
Mar 6 2024, 2:39 PM · Observability-Metrics
lmata created T359394: Migrate MediaWiki.parsoidhtmlhelper. to statslib.
Mar 6 2024, 2:37 PM · Observability-Metrics
lmata created T359393: Migrate MediaWiki.Parsoid.langconv to statslib.
Mar 6 2024, 2:36 PM · Observability-Metrics
lmata created T359392: Migrate MediaWiki.Parsoid.linting to statslib.
Mar 6 2024, 2:35 PM · Observability-Metrics
lmata created T359391: Migrate MediaWiki.ParsoidOutputAccess.Cache to statslib.
Mar 6 2024, 2:35 PM · Observability-Metrics
lmata created T359390: Migrate MediaWiki.parsoid_pcache.*.reason.*.rate to statslib.
Mar 6 2024, 2:34 PM · Observability-Metrics
lmata created T359389: Migrate MediaWiki.parsoid.*.reason.*.rate to statslib.
Mar 6 2024, 2:34 PM · Observability-Metrics
lmata created T359388: Migrate MediaWiki.Parsoid.wt2html.* to statslib.
Mar 6 2024, 2:33 PM · Observability-Metrics
lmata updated the task description for T350592: EPIC: migrate in use metrics and dashboards to statslib.
Mar 6 2024, 2:30 PM · Epic, SRE Observability (FY2023/2024-Q4), MW-1.42-notes (1.42.0-wmf.15; 2024-01-23), MediaWiki-Platform-Team (Radar), Observability-Metrics
lmata created T359386: Migrate MediaWiki.articleplaceholder to statslib.
Mar 6 2024, 2:29 PM · Observability-Metrics
lmata created T359385: Migrate MediaWiki.arclamp to statslib.
Mar 6 2024, 2:29 PM · Observability-Metrics
lmata created T359383: Migrate MediaWiki.api to statslib.
Mar 6 2024, 2:27 PM · MW-Interfaces-Team, Observability-Metrics