Page MenuHomePhabricator
Create Task
Maniphest T429088

staging.webrequest.page_view.dev0 taking up most space on kafka-jumbo
Closed, ResolvedPublic
Actions

Description

2026-06-14-084703_2526x1009_scrot.png (2,526×1,009 px, 116 KB)

  • Rootcause: A staging topic on kafka-jumbo was created with an insufficient/default number of partitions for its size, leading to three brokers (the replication factor) filling up their filesystem
  • Mitigation: me and @ABran-WMF have mitigate the issue by extending the vg, though a proper fix is of course needed

on kafka-jumbo1012 / kafka-jumbo1015 / kafka-jumbo1018

sudo lvextend -r -L +3T /dev/vg1/srv

incident doc: https://docs.google.com/document/d/1PrMCQ4W1Li1VQ3SaGIokp1PQ3gPvy3n7UAhgHJol0IA/edit?tab=t.0

Related Objects
Search...

Event Timeline

fgiunchedi created this task.Jun 14 2026, 6:49 AM
Restricted Application added a subscriber: • Aklapper. · View Herald TranscriptJun 14 2026, 6:49 AM
fgiunchedi triaged this task as Unbreak Now! priority.Jun 14 2026, 6:49 AM
fgiunchedi updated the task description. (Show Details)Jun 14 2026, 6:51 AM
JMonton-WMF added a comment.Jun 14 2026, 7:25 PM

Sorry about that. This is a new application we are working on, I was testing it with real traffic and left it running. It produces around 8% of the webrequest_frontend_text traffic, so I didn't expect it to grow a lot inside the Kafka cluster, but the topic was automatically created with only 1 partition (replica 3), making it grow on only 3 brokers.

It's ok to clean or remove the topic, or stop the application, it's a staging release under development.

We'll talk on Monday about ways of avoiding this situation.

elukey subscribed.Jun 15 2026, 7:14 AM

@JMonton-WMF Hi! Could you please stop the application? I am not sure which one it is :)

After that we'll clear the topic to regain space.

MLechvien-WMF lowered the priority of this task from Unbreak Now! to Needs Triage.Jun 15 2026, 8:15 AM
MLechvien-WMF edited projects, added: Wikimedia-Incident, Kafka-Infrastructure, Data-Platform-SRE; removed: SRE.
MLechvien-WMF triaged this task as Unbreak Now! priority.EditedJun 15 2026, 8:17 AM
MLechvien-WMF added a project: Data-Engineering.
MLechvien-WMF subscribed.

Incident was only mitigated by expanding filesystem but full rootcausing and resolution is needed, so keeping UBN priority and triaging to relevant teams

fgiunchedi updated the task description. (Show Details)Jun 15 2026, 8:21 AM
brouberol added a subtask: T429127: [kafka-jumbo] Ensure partitions are within acceptable size.Jun 15 2026, 8:32 AM
MLechvien-WMF added a project: Incident Severity 3.Jun 15 2026, 8:39 AM
fgiunchedi lowered the priority of this task from Unbreak Now! to Medium.Jun 15 2026, 9:19 AM

Issue has been fixed, though I'm not sure how / by whom

2026-06-15-111849_3780x1773_scrot.png (3,780×1,773 px, 115 KB)

JMonton-WMF added a subscriber: brouberol.Jun 15 2026, 9:37 AM

I believe @brouberol reduced the retention of the topic yesterday afternoon, which reduced the storage.
I stopped the application a few hours ago, and I believe @brouberol has removed the topic.

fgiunchedi closed this task as Resolved.Jun 15 2026, 9:55 AM
fgiunchedi claimed this task.
In T429088#12017792, @JMonton-WMF wrote:

I believe @brouberol reduced the retention of the topic yesterday afternoon, which reduced the storage.
I stopped the application a few hours ago, and I believe @brouberol has removed the topic.

Thank you ! I'm boldly resolving the task since we have subtasks / action items, feel free to reopen as needed

Gehel edited projects, added: Data-Platform-SRE (2026-06-05 - 2026-06-26); removed: Data-Platform-SRE.Jun 16 2026, 1:25 PM
Gehel moved this task from Backlog - project to Done on the Data-Platform-SRE (2026-06-05 - 2026-06-26) board.
Gehel moved this task from Done to Reported on the Data-Platform-SRE (2026-06-05 - 2026-06-26) board.Fri, Jun 19, 7:28 AM
brouberol closed subtask T429127: [kafka-jumbo] Ensure partitions are within acceptable size as Resolved.Tue, Jun 23, 9:20 AM
MLechvien-WMF moved this task from Active investigation to Complete on the Wikimedia-Incident board.Tue, Jun 23, 1:39 PM
MLechvien-WMF updated the task description. (Show Details)Tue, Jun 23, 1:42 PM
BTullis mentioned this in T431112: The webrequest_sampled_live data pipeline and its query tools have become mission-critical and require re-engineering for resilience.Fri, Jul 3, 12:11 PM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits