Based on my understanding of the task, it appears that the only viable way to test it is by implementing it in a live environment. While this may seem spooky, I don't think we have another environment to simulate this effectively to test the sampling.
- Queries
- All Stories
- Search
- Advanced Search
- Transactions
- Transaction Logs
Feed Advanced Search
Advanced Search
Advanced Search
Jan 26 2024
Jan 26 2024
lmata edited projects for T353691: Reload thanos-rule on new pyrra rules deployed, added: SRE Observability (FY2023/2024-Q3); removed SRE Observability (FY2023/2024-Q2).
lmata removed a project from T353716: Pyrra: cleanup output-rules when config is removed: SRE Observability (FY2023/2024-Q2).
lmata edited projects for T288622: All Prometheus based alerts move from Icinga to alert manager exclusively, added: SRE Observability (FY2023/2024-Q3); removed SRE Observability (FY2023/2024-Q2).
lmata edited projects for T321808: Port most/all Icinga checks to Prometheus/Alertmanager, added: SRE Observability (FY2023/2024-Q3); removed SRE Observability (FY2023/2024-Q2).
lmata removed a project from T274372: Improve Automation for Alert Reviews: SRE Observability (FY2023/2024-Q2).
Jan 19 2024
Jan 19 2024
lmata awarded T350591: Audit legacy mediawiki stats used in production dashboards a Party Time token.
Jan 17 2024
Jan 17 2024
Jan 16 2024
Jan 16 2024
We are back to regular on-call
Jan 15 2024
Jan 15 2024
batphone enabled for MLK
lmata updated the task description for T350192: On-call batphone escalation configuration holidays FY2023-24.
Jan 12 2024
Jan 12 2024
lmata moved T326419: Expand kafka-logging using hosts kafka-logging[12]00[45] from Inbox to Up next on the SRE Observability (FY2023/2024-Q3) board.
lmata moved T317887: Upgrade to Grafana 9 from Inbox to Up next on the SRE Observability (FY2023/2024-Q2) board.
hi @colewhite, a friendly reminder that the silence is expiring on 2024-02-01.
lmata edited projects for T350591: Audit legacy mediawiki stats used in production dashboards, added: SRE Observability (FY2023/2024-Q3); removed SRE Observability (FY2023/2024-Q2).
lmata edited projects for T349626: Migrate SRE repositories to GitLab - operations/alerts, added: SRE Observability (FY2023/2024-Q3); removed SRE Observability (FY2023/2024-Q2).
lmata removed a project from T325775: improve email template for alertmanager notifications: SRE Observability (FY2023/2024-Q2).
lmata lowered the priority of T317240: Improve AlertManager alert titles as sent to VictorOps from High to Medium.
Lowering priority due to lack of activity, we can revisit this if it continues to be a pressing matter.
Based on the lack of recent feedback indicating that this issue persists, we have decided to downgrade its severity. By doing so, we can focus our resources on more pressing concerns that require immediate attention.
lmata removed a project from T317240: Improve AlertManager alert titles as sent to VictorOps: SRE Observability (FY2023/2024-Q2).
lmata edited projects for T333615: Upgrade alert* hosts to Bookworm, added: SRE Observability (FY2023/2024-Q3); removed SRE Observability (FY2023/2024-Q2).
lmata edited projects for T302373: Upgrade prometheus-statsd-exporter, added: SRE Observability (FY2023/2024-Q3); removed SRE Observability (FY2023/2024-Q2).
lmata removed a project from T351935: Audit Prometheus metrics size/label values: SRE Observability (FY2023/2024-Q2).
lmata removed a project from T343029: Audit & convert stats for mediawiki extensions : SRE Observability (FY2023/2024-Q2).
lmata removed a project from T343045: Audit & convert stats for mediawiki modules: SRE Observability (FY2023/2024-Q2).
lmata moved T343020: Converting MediaWiki Metrics to StatsLib from Up next to Epics In Progress on the SRE Observability (FY2023/2024-Q2) board.
lmata edited projects for T350192: On-call batphone escalation configuration holidays FY2023-24, added: SRE Observability (FY2023/2024-Q3); removed SRE Observability (FY2023/2024-Q2).
lmata edited projects for T343028: Audit & convert stats for mediawiki core, added: SRE Observability (FY2023/2024-Q3); removed SRE Observability (FY2023/2024-Q2).
I'm resolving this one on my understanding that this has already been deployed. Please reopen if that's not the case, and there's work pending. Thanks!
lmata closed T343024: Configure MediaWiki to use new StatsLib in production, a subtask of T343020: Converting MediaWiki Metrics to StatsLib, as Resolved.
lmata moved T240685: MediaWiki Prometheus support from Inbox to Epics In Progress on the SRE Observability (FY2023/2024-Q3) board.
Jan 12 2024, 5:39 PM · SRE Observability (FY2023/2024-Q4), MW-1.41-notes (1.41.0-wmf.28; 2023-09-26), MW-1.40-notes (1.40.0-wmf.27; 2023-03-13), MW-1.38-notes (1.38.0-wmf.19; 2022-01-24), MediaWiki-libs-Stats, Platform Team Workboards (External Code Reviews), serviceops, SRE, MediaWiki-General, observability
lmata edited projects for T240685: MediaWiki Prometheus support, added: SRE Observability (FY2023/2024-Q3); removed SRE Observability (FY2023/2024-Q2).
Jan 12 2024, 5:39 PM · SRE Observability (FY2023/2024-Q4), MW-1.41-notes (1.41.0-wmf.28; 2023-09-26), MW-1.40-notes (1.40.0-wmf.27; 2023-03-13), MW-1.38-notes (1.38.0-wmf.19; 2022-01-24), MediaWiki-libs-Stats, Platform Team Workboards (External Code Reviews), serviceops, SRE, MediaWiki-General, observability
lmata edited projects for T354905: migrate MediaWiki.timing.editResponseTime to statslib, added: SRE Observability (FY2023/2024-Q3); removed SRE Observability (FY2023/2024-Q2).
lmata edited projects for T354906: migrate mw.performance.save to statslib, added: SRE Observability (FY2023/2024-Q3); removed SRE Observability (FY2023/2024-Q2).
lmata edited projects for T354907: (mw.track) migrate MediaWiki.RevisionSlider.timing.init to statslib, added: SRE Observability (FY2023/2024-Q3); removed SRE Observability (FY2023/2024-Q2).
lmata edited projects for T354908: evaluate and migrate in-use parsoid metrics to statslib, added: SRE Observability (FY2023/2024-Q3); removed SRE Observability (FY2023/2024-Q2).
Jan 12 2024, 5:13 PM · SRE Observability (FY2023/2024-Q4), MW-1.41-notes (1.41.0-wmf.28; 2023-09-26), MW-1.40-notes (1.40.0-wmf.27; 2023-03-13), MW-1.38-notes (1.38.0-wmf.19; 2022-01-24), MediaWiki-libs-Stats, Platform Team Workboards (External Code Reviews), serviceops, SRE, MediaWiki-General, observability
Jan 11 2024
Jan 11 2024
lmata raised the priority of T350592: EPIC: migrate in use metrics and dashboards to statslib from Medium to High.
lmata edited projects for T350592: EPIC: migrate in use metrics and dashboards to statslib, added: SRE Observability (FY2023/2024-Q3); removed SRE Observability (FY2023/2024-Q2).
lmata updated the task description for T350592: EPIC: migrate in use metrics and dashboards to statslib.
lmata updated the task description for T354908: evaluate and migrate in-use parsoid metrics to statslib.
lmata renamed T354908: evaluate and migrate in-use parsoid metrics to statslib from migrate MediaWiki.Parsoid.html2wt.setup to statslib to migrate top used parsoid to statslib.
lmata updated the task description for T354908: evaluate and migrate in-use parsoid metrics to statslib.
lmata updated the task description for T350592: EPIC: migrate in use metrics and dashboards to statslib.
Jan 10 2024
Jan 10 2024
lmata moved T354217: Make arc-lamp aware of the new names of request life cycle methods. from Inbox to Up next on the SRE Observability (FY2023/2024-Q3) board.
lmata removed a project from T278309: Move librenms deployment to Debian package: SRE Observability (FY2023/2024-Q4).
lmata moved T277816: Improve Logstash's rate-limiting capabilities from Inbox to Up next on the SRE Observability (FY2023/2024-Q3) board.
lmata edited projects for T277816: Improve Logstash's rate-limiting capabilities, added: SRE Observability (FY2023/2024-Q3); removed SRE Observability (FY2023/2024-Q2), observability.
lmata moved T353912: Observability Bookworm upgrades from Inbox to Up next on the SRE Observability (FY2023/2024-Q3) board.
lmata moved T354570: Create alert Review for FY2023/2024-Q3 from Inbox to Up next on the SRE Observability (FY2023/2024-Q3) board.
lmata edited projects for T354255: Alert in need of triage: AlertLintProblem (instance localhost:9123), added: SRE Observability (FY2023/2024-Q3); removed SRE Observability.
lmata edited projects for T354217: Make arc-lamp aware of the new names of request life cycle methods., added: SRE Observability (FY2023/2024-Q3); removed observability.
lmata edited projects for T354762: [pint,karma] Find a way to forward AlertLintProblem to the right team (ex. using the team=wmcs label), added: Observability-Alerting; removed observability.
Jan 9 2024
Jan 9 2024
lmata updated the task description for T350592: EPIC: migrate in use metrics and dashboards to statslib.
lmata renamed T350592: EPIC: migrate in use metrics and dashboards to statslib from Audit and prioritize metrics for conversion to statslib that rely on grafana alerting to migrate in use metrics and dashboards to statslib.
Keeping this task open for further snapshots as the project evolves.
lmata raised the priority of T350592: EPIC: migrate in use metrics and dashboards to statslib from Low to Medium.
Jan 8 2024
Jan 8 2024
In T307958#9390852, @fgiunchedi wrote:This is essentially what https://alerts.wikimedia.org/triage/ displays now, for hide_alerts_older_than: '1200h' alerts. The app also offers the user a button to open a task
lmata edited projects for T353912: Observability Bookworm upgrades, added: SRE Observability (FY2023/2024-Q3); removed SRE Observability.
lmata moved T352665: Upgrade Grafana hosts to Bookworm from Inbox to Up next on the SRE Observability (FY2023/2024-Q3) board.
lmata edited projects for T352665: Upgrade Grafana hosts to Bookworm, added: SRE Observability (FY2023/2024-Q3); removed SRE Observability (FY2023/2024-Q2).
Batphone has been removed, and the business-hours on-call rota is enabled again in Splunk on-call.
Dec 23 2023
Dec 23 2023
@MatthewVernon, IT does look weird. I think it's just the UI; when I added Batphone to the escalation path instead of the EMEA/Americas rotation, it seems to have expanded the Batphone list as the escalation and the on/off-calls folks according to their individual schedule set within the Batphone rotation.
lmata updated the task description for T350192: On-call batphone escalation configuration holidays FY2023-24.
lmata updated the task description for T350192: On-call batphone escalation configuration holidays FY2023-24.
Dec 20 2023
Dec 20 2023
fgiunchedi awarded T343024: Configure MediaWiki to use new StatsLib in production a Party Time token.
Dec 14 2023
Dec 14 2023
Dec 6 2023
Dec 6 2023
lmata removed a project from T328707: Update arclamp to active/active architecture: SRE Observability (FY2023/2024-Q3).
It seems like this is not a priority, so we'll postpone it for now. We can revisit it when the time comes.
lmata edited projects for T343529: Prometheus doesn't reload or alert on expired client certificates, added: SRE Observability (FY2023/2024-Q3); removed SRE Observability (FY2023/2024-Q2).
lmata edited projects for T326419: Expand kafka-logging using hosts kafka-logging[12]00[45], added: SRE Observability (FY2023/2024-Q3); removed SRE Observability (FY2023/2024-Q2).
lmata edited projects for T352517: Put logging-hd[12]00[1-3] in service, added: SRE Observability (FY2023/2024-Q3); removed observability.
lmata added a project to T352756: Gap in metrics rendered from Thanos Rules: SRE Observability (FY2023/2024-Q2).
lmata moved T352783: Change data platform-related IRC channels to improve communication from Inbox to Radar on the observability board.
Dec 5 2023
Dec 5 2023
lmata added a project to T351935: Audit Prometheus metrics size/label values: SRE Observability (FY2023/2024-Q2).
lmata added a project to T351936: Stop exporting unit state metrics for timers: SRE Observability (FY2023/2024-Q2).
Raising priority based on recent conversations with the team and the intent to address this in the near future as part of risk mitigations to the logging pipeline.
lmata moved T277816: Improve Logstash's rate-limiting capabilities from Backlog to Prioritized on the Observability-Logging board.
lmata closed T352128: No on-call page notification when shift override was set on November 27 as Resolved.
It seems like the issue is resolved.
Nov 29 2023
Nov 29 2023
lmata awarded T351934: Label value spam in ncredir_requests_total metric a Like token.
lmata moved T352128: No on-call page notification when shift override was set on November 27 from Inbox to Prioritized on the Incident Tooling board.
lmata moved T322636: Provide mechanism to join/leave oncall from Inbox to Backlog on the Incident Tooling board.
Nov 28 2023
Nov 28 2023
lmata added a comment to T349159: Transfer Arc Lamp alerts (Grafana and cron errors_mailto) to SRE O11y.
In T349159#9361905, @Krinkle wrote:Arc Lamp metrics are visualised in Grafana but the alerts are defined in AlertManager, not in Grafana.
Nov 27 2023
Nov 27 2023
lmata added a project to T351927: Decide and tweak Thanos retention: SRE Observability (FY2023/2024-Q2).
Adding to ongoing quarter for visibility.
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits