Clement_Goubert (claime)
Senior SRE

Projects

MW-on-K8s
Component
Security
Tag
serviceops
Group
serviceops-radar
Group

Calendar

User Details

User Since: Jul 26 2022, 2:11 PM (129 w, 6 d)
Availability: Available
IRC Nick: claime
LDAP User: Clément Goubert
MediaWiki User: CGoubert-WMF [ Global Accounts ]

Recent Activity
View All

Today

Clement_Goubert added a comment to T384233: Unexpected utilization increase in udp_localhost-info kafka-logging topic.

rsyslog container was added to mercurius on the 7th https://gerrit.wikimedia.org/r/c/operations/deployment-charts/+/1105800 but it again doesn't line up with the beginning of the slope

Mon, Jan 20, 4:30 PM · Observability-Logging, MW-on-K8s, serviceops

Clement_Goubert added a comment to T384233: Unexpected utilization increase in udp_localhost-info kafka-logging topic.

I enabled logging for mw-jobrunner through rsyslog on the 13th https://gerrit.wikimedia.org/r/c/operations/deployment-charts/+/1110786 but it looks like the increase of udp_localhost-info preceeds that a bit?

Mon, Jan 20, 4:27 PM · Observability-Logging, MW-on-K8s, serviceops

Fri, Jan 17

Clement_Goubert created T384002: Error creating a new ipblock in existing scope.

Fri, Jan 17, 1:06 PM · Hiddenparma

Mon, Jan 13

Clement_Goubert closed T293943: Enable mediawiki appserver metrics for jobrunner hosts as Resolved.

Logs are now appearing in logstash, as well as benthos metrics in the Application Servers RED - k8s dashboard. Other metrics can be found in the mw-jobrunner service dashboard

Mon, Jan 13, 3:23 PM · Wikimedia-Performance-recommendation, Observability-Metrics, serviceops

Clement_Goubert added a comment to T293943: Enable mediawiki appserver metrics for jobrunner hosts.

While investigating if this task could be closed, I realized we were not logging the same way in mw-jobrunner than the rest of mw-on-k8s, meaning we didn't get benthos metrics for this deployment. The linked patch should fix that.

Mon, Jan 13, 3:01 PM · Wikimedia-Performance-recommendation, Observability-Metrics, serviceops

Clement_Goubert added a comment to T187078: Re-consider ` >/dev/null 2>&1` as output of many cron'd MW maintenance scripts.

In T187078#10446147, @andrea.denisse wrote:

I think that having a list of the MW maintenance scripts that have this behavior would make it easier to work on them.

Mon, Jan 13, 11:00 AM · serviceops, WMF-General-or-Unknown, SRE

Wed, Jan 8

Clement_Goubert added a comment to T383032: wikifunction httpbb tests fail because of title case issue.

Re-prioritizing this to High, as the alert is actually critical, and would mask other wikifunctions httpbb alerts if they were to happen.

Wed, Jan 8, 9:52 AM · Abstract Wikipedia team (25Q3 (Jan–Mar)), serviceops

Clement_Goubert triaged T383032: wikifunction httpbb tests fail because of title case issue as High priority.

Wed, Jan 8, 9:51 AM · Abstract Wikipedia team (25Q3 (Jan–Mar)), serviceops

Clement_Goubert added a comment to T376519: Steady-state sizing of mw-web and mw-api-ext.

SGTM, thanks for doing the maths

Wed, Jan 8, 9:39 AM · Datacenter-Switchover, serviceops

Mon, Jan 6

Clement_Goubert added a comment to T355292: Port videoscaling to kubernetes.

@tstarling @TheDJ This has been flagged in T382517: PHP Warning seen by logspam-watch but not by mediawiki-errors logstash page and is due to the mw-videoscaler deployment missing the rsyslog sidecar. A patch is currently in review and will be deployed soon.

Mon, Jan 6, 9:43 AM · Patch-For-Review, Video, TimedMediaHandler, MW-on-K8s, serviceops

Clement_Goubert created T383032: wikifunction httpbb tests fail because of title case issue.

Mon, Jan 6, 8:56 AM · Abstract Wikipedia team (25Q3 (Jan–Mar)), serviceops

Dec 19 2024

Clement_Goubert closed T365265: Create a per-release deployment of statsd-exporter for mw-on-k8s as Resolved.

There will be a deployement for the future mw-cron and maybe for mw-videoscaler (@hnowlan will be able to weigh in on this after the holidays), but we can handle that in separates issues, the main deployments of mediawiki are indeed covered. Resolving.

Dec 19 2024, 3:00 PM · SRE Observability (FY2024/2025-Q2), Patch-For-Review, MW-on-K8s, serviceops, Observability-Metrics

Clement_Goubert closed T365265: Create a per-release deployment of statsd-exporter for mw-on-k8s, a subtask of T359640: mediawiki_resourceloader_build_seconds_bucket big metric on Prometheus ops, as Resolved.

Dec 19 2024, 3:00 PM · SRE Observability (FY2024/2025-Q1), Patch-For-Review, MediaWiki-Platform-Team (Radar), Observability-Metrics

Dec 17 2024

Clement_Goubert added a comment to T382334: mediawiki_job_translationnotifications fails on for mediawikiwiki and metawiki.

FTR, direct link to that patch which is part of the work on T378458: Modernize code for the Translation notifications extension

Dec 17 2024, 4:26 PM · Unplanned-Sprint-Work, LPL Essential (LPL Essential 2024 Nov-Dec), serviceops-radar, TranslationNotifications, Wikimedia-production-error

Clement_Goubert created T382334: mediawiki_job_translationnotifications fails on for mediawikiwiki and metawiki.

Dec 17 2024, 12:49 PM · Unplanned-Sprint-Work, LPL Essential (LPL Essential 2024 Nov-Dec), serviceops-radar, TranslationNotifications, Wikimedia-production-error

Dec 16 2024

Clement_Goubert added a comment to T352650: WE 5.4 KR - Hypothesis 5.4.4 - Q3 FY24/25 - Migrate current-generation dumps to run on kubernetes.

You probably also want to if-guard the service definition in templates/service.yaml.tpl

Dec 16 2024, 12:49 PM · Data-Engineering, Patch-For-Review, Data-Platform-SRE, Epic, MW-on-K8s, Dumps-Generation, Release-Engineering-Team, serviceops

Clement_Goubert added a comment to T352650: WE 5.4 KR - Hypothesis 5.4.4 - Q3 FY24/25 - Migrate current-generation dumps to run on kubernetes.

In T352650#10405691, @BTullis wrote:

[...]

I see various references to mwscript and mercurius here, so if you (or someone) could let me know the current direction of travel regarding these two things, I'd be grateful.

Dec 16 2024, 12:04 PM · Data-Engineering, Patch-For-Review, Data-Platform-SRE, Epic, MW-on-K8s, Dumps-Generation, Release-Engineering-Team, serviceops

Dec 13 2024

Sdrqaz awarded T359901: HTTP 429 error on private wikis trying to create account via Special:CreateAccount a Like token.

Dec 13 2024, 1:21 PM · serviceops, SRE

Dec 12 2024

alaa awarded T359901: HTTP 429 error on private wikis trying to create account via Special:CreateAccount a Like token.

Dec 12 2024, 7:33 PM · serviceops, SRE

Clement_Goubert closed T359901: HTTP 429 error on private wikis trying to create account via Special:CreateAccount as Resolved.

The problem should be resolved for all private wikis now.

Dec 12 2024, 5:13 PM · serviceops, SRE

Clement_Goubert changed the status of T359901: HTTP 429 error on private wikis trying to create account via Special:CreateAccount from Open to In Progress.

Dec 12 2024, 4:57 PM · serviceops, SRE

Clement_Goubert added a comment to T359901: HTTP 429 error on private wikis trying to create account via Special:CreateAccount.

In T359901#10400510, @Krd wrote:

The account was created in the meantime. I suggest I test this at the next opportunity and report here. Thank you!

Dec 12 2024, 4:57 PM · serviceops, SRE

Clement_Goubert merged task T382048: blank 429 error attempting to create accounts on checkuserwiki into T359901: HTTP 429 error on private wikis trying to create account via Special:CreateAccount.

Dec 12 2024, 4:49 PM · serviceops, SRE

Clement_Goubert merged T382048: blank 429 error attempting to create accounts on checkuserwiki into T359901: HTTP 429 error on private wikis trying to create account via Special:CreateAccount.

Dec 12 2024, 4:49 PM · serviceops, SRE

Clement_Goubert renamed T359901: HTTP 429 error on private wikis trying to create account via Special:CreateAccount from HTTP 429 error on VRT wiki trying to create account via Special:CreateAccount to HTTP 429 error on private wikis trying to create account via Special:CreateAccount.

Dec 12 2024, 4:49 PM · serviceops, SRE

Clement_Goubert added a comment to T359901: HTTP 429 error on private wikis trying to create account via Special:CreateAccount.

I've tweaked a rate limiting rule, could you please try again?

Dec 12 2024, 1:20 PM · serviceops, SRE

Clement_Goubert added a comment to T359901: HTTP 429 error on private wikis trying to create account via Special:CreateAccount.

Sorry for the delay in responding.

Dec 12 2024, 12:18 PM · serviceops, SRE

Clement_Goubert added a project to T379788: Decommission kubernetes20[07-14].codfw.wmnet: decommission-hardware.

Dec 12 2024, 11:30 AM · decommission-hardware, SRE, DC-Ops, ops-codfw, serviceops

Dec 11 2024

Clement_Goubert updated the task description for T379788: Decommission kubernetes20[07-14].codfw.wmnet.

Dec 11 2024, 4:55 PM · decommission-hardware, SRE, DC-Ops, ops-codfw, serviceops

Clement_Goubert updated the task description for T379788: Decommission kubernetes20[07-14].codfw.wmnet.

Dec 11 2024, 4:51 PM · decommission-hardware, SRE, DC-Ops, ops-codfw, serviceops

Dec 5 2024

Clement_Goubert assigned T379788: Decommission kubernetes20[07-14].codfw.wmnet to jasmine_.

Dec 5 2024, 2:34 PM · decommission-hardware, SRE, DC-Ops, ops-codfw, serviceops

Dec 2 2024

Clement_Goubert updated subscribers of T380958: httpb fails upon deployment of 1.44.0-wmf.5.

In T381250#10370809, @Ladsgroup wrote:

Now erroring with:

11:13:16 Check 'check_testservers_baremetal-1_of_1' failed: Sending to 4 hosts...
https://boardgovcom.wikimedia.org/wiki/Main_Page (/srv/deployment/httpbb-tests/appserver/test_remnant.yaml:43)
  mwdebug2002.codfw.wmnet
    Status code: expected 200, got 503.
    Body: expected to contain 'Board Governance Committee', got '<!DOCTYPE html>\n<html lang="en">\n<meta charset="ut'... (1953 characters total).
===
FAIL: 131 requests sent to each of 4 hosts. 1 request with failed assertions.

Dec 2 2024, 12:31 PM · Deployments, serviceops, Wikimedia-production-error, Release-Engineering-Team

Clement_Goubert added a comment to T381250: Uncaught MediaWiki\Config\ConfigException: Translate: Message group subscriptions (TranslateEnableMessageGroupSubscription) are enabled but Echo extension is not installed.

From T381252: legalteam wiki reliably returns 500s

Currently, any request to https://legalteam.wikimedia.org is returning 500 with this:
Uncaught MediaWiki\Config\ConfigException: Translate: Message group subscriptions (TranslateEnableMessageGroupSubscription) are enabled but Echo extension is not installed in /srv/mediawiki/php-1.44.0-wmf.5/extensions/Translate/src/HookHandler.php:438
This wiki is used in health checks, so needs to be fixed ASAP.

Dec 2 2024, 10:46 AM · Unplanned-Sprint-Work, Regression, LPL Essential (LPL Essential 2024 Nov-Dec), Wikimedia-production-error

Clement_Goubert triaged T381250: Uncaught MediaWiki\Config\ConfigException: Translate: Message group subscriptions (TranslateEnableMessageGroupSubscription) are enabled but Echo extension is not installed as Unbreak Now! priority.

Dec 2 2024, 10:43 AM · Unplanned-Sprint-Work, Regression, LPL Essential (LPL Essential 2024 Nov-Dec), Wikimedia-production-error

Clement_Goubert merged task T381252: legalteam wiki reliably returns 500s into T381250: Uncaught MediaWiki\Config\ConfigException: Translate: Message group subscriptions (TranslateEnableMessageGroupSubscription) are enabled but Echo extension is not installed.

Dec 2 2024, 10:42 AM · serviceops, Release-Engineering-Team

Clement_Goubert merged T381252: legalteam wiki reliably returns 500s into T381250: Uncaught MediaWiki\Config\ConfigException: Translate: Message group subscriptions (TranslateEnableMessageGroupSubscription) are enabled but Echo extension is not installed.

Dec 2 2024, 10:42 AM · Unplanned-Sprint-Work, Regression, LPL Essential (LPL Essential 2024 Nov-Dec), Wikimedia-production-error

Nov 27 2024

Clement_Goubert added a comment to T341553: Allow running one-off scripts manually.

In T341553#10362280, @Urbanecm_WMF wrote:

Usually, I find the output of kubectl get job unhelpful. For example, for T379146, I just executed userOptions.php with various arguments. When I request the list of jobs I executed, I get this:
...
@RLazarus Do you think it would be possible to add more details to the list of jobs?

Nov 27 2024, 5:14 PM · MW-on-K8s, serviceops

Nov 26 2024

Clement_Goubert closed T380350: wikikube-worker13[13-27] implementation tracking as Resolved.

Nov 26 2024, 5:48 PM · serviceops

Clement_Goubert updated the task description for T380350: wikikube-worker13[13-27] implementation tracking.

Nov 26 2024, 5:48 PM · serviceops

Clement_Goubert closed T380350: wikikube-worker13[13-27] implementation tracking, a subtask of T378185: Q2:rack/setup/install wikikube-worker13[13-28], as Resolved.

Nov 26 2024, 5:48 PM · SRE, ops-eqiad, DC-Ops

Clement_Goubert updated the task description for T380350: wikikube-worker13[13-27] implementation tracking.

Nov 26 2024, 3:45 PM · serviceops

Clement_Goubert updated the task description for T380350: wikikube-worker13[13-27] implementation tracking.

Nov 26 2024, 3:44 PM · serviceops

Clement_Goubert moved T375842: decommission mw[1349-1413] from Incoming 🐫 to 🛠 Upgrades and Hardware on the serviceops board.

Nov 26 2024, 1:36 PM · SRE, DC-Ops, ops-eqiad, serviceops, decommission-hardware

Clement_Goubert added a project to T375842: decommission mw[1349-1413]: serviceops.

Nov 26 2024, 1:36 PM · SRE, DC-Ops, ops-eqiad, serviceops, decommission-hardware

Nov 25 2024

Clement_Goubert awarded T377011: kubestage200[3-4] implementation tracking a Like token.

Nov 25 2024, 4:48 PM · serviceops

Clement_Goubert changed the status of T380350: wikikube-worker13[13-27] implementation tracking, a subtask of T378185: Q2:rack/setup/install wikikube-worker13[13-28], from Open to Stalled.

Nov 25 2024, 4:36 PM · SRE, ops-eqiad, DC-Ops

Clement_Goubert changed the status of T380350: wikikube-worker13[13-27] implementation tracking from Open to Stalled.

Because of T375845: WikiKube clusters close to exhausting Calico IPPool allocations, putting these nodes in production needs to wait for T379599: Reevaluate the requirement for dedicated sessionstore/kask nodes in wikikube clusters to be completed to have enough ip blocks to proceed
.

Nov 25 2024, 4:35 PM · serviceops

Clement_Goubert edited projects for T380027: Decommission kubernetes10[09-14], added: decommission-hardware, ops-eqiad; removed Patch-For-Review.

Nov 25 2024, 3:54 PM · SRE, ops-eqiad, decommission-hardware, DC-Ops, serviceops

Clement_Goubert updated the task description for T380027: Decommission kubernetes10[09-14].

Nov 25 2024, 3:12 PM · SRE, ops-eqiad, decommission-hardware, DC-Ops, serviceops

Clement_Goubert renamed T380350: wikikube-worker13[13-27] implementation tracking from wikikube-worker13[13-28] implementation tracking to wikikube-worker13[13-27] implementation tracking.

Nov 25 2024, 12:07 PM · serviceops

Clement_Goubert closed T379454: Degraded RAID on wikikube-worker1256 as Resolved.

Host reimaged, RAID ok, repooled

Nov 25 2024, 11:56 AM · serviceops, DC-Ops, ops-eqiad

Nov 22 2024

Clement_Goubert edited projects for T380473: Decommission parse20[01-20], added: decommission-hardware, ops-codfw; removed Patch-For-Review.

Nov 22 2024, 4:23 PM · SRE, DC-Ops, ops-codfw, decommission-hardware, serviceops

Clement_Goubert updated the task description for T380473: Decommission parse20[01-20].

Nov 22 2024, 2:59 PM · SRE, DC-Ops, ops-codfw, decommission-hardware, serviceops

Clement_Goubert closed T376966: wikikube-worker21[56-70] implementation tracking, a subtask of T376965: Q2:rack/setup/install wikikube-worker21[56-70], as Resolved.

Nov 22 2024, 11:26 AM · SRE, serviceops, ops-codfw, DC-Ops

Clement_Goubert closed T376966: wikikube-worker21[56-70] implementation tracking as Resolved.

Nov 22 2024, 11:26 AM · serviceops

Clement_Goubert updated the task description for T376966: wikikube-worker21[56-70] implementation tracking.

Nov 22 2024, 11:16 AM · serviceops

Clement_Goubert closed T377028: wikikube-worker21[36-55] implementation tracking as Resolved.

Nov 22 2024, 11:10 AM · serviceops

Clement_Goubert closed T377028: wikikube-worker21[36-55] implementation tracking, a subtask of T377027: Q2:rack/setup/install wikikube-worker21[36-55], as Resolved.

Nov 22 2024, 11:10 AM · Patch-For-Review, SRE, serviceops, ops-codfw, DC-Ops

Clement_Goubert updated the task description for T377028: wikikube-worker21[36-55] implementation tracking.

Nov 22 2024, 11:09 AM · serviceops

Clement_Goubert added a comment to T379599: Reevaluate the requirement for dedicated sessionstore/kask nodes in wikikube clusters.

I'm good with removing them as well.

Nov 22 2024, 11:06 AM · Data-Persistence, serviceops, Prod-Kubernetes

Clement_Goubert updated the task description for T376966: wikikube-worker21[56-70] implementation tracking.

Nov 22 2024, 11:03 AM · serviceops

Clement_Goubert added a comment to T379454: Degraded RAID on wikikube-worker1256.

Re-imaging because I accidentaly overwrote the partition table on the good disk with the partition table on the new disk...

Nov 22 2024, 10:34 AM · serviceops, DC-Ops, ops-eqiad

Clement_Goubert updated the task description for T377028: wikikube-worker21[36-55] implementation tracking.

Nov 22 2024, 10:19 AM · serviceops

Clement_Goubert updated the task description for T376966: wikikube-worker21[56-70] implementation tracking.

Nov 22 2024, 10:19 AM · serviceops

Nov 21 2024

Clement_Goubert updated the task description for T380350: wikikube-worker13[13-27] implementation tracking.

Nov 21 2024, 2:23 PM · serviceops

Clement_Goubert renamed T380027: Decommission kubernetes10[09-14] from Decommission kubernetes10[07-14] to Decommission kubernetes10[09-14].

Nov 21 2024, 2:21 PM · SRE, ops-eqiad, decommission-hardware, DC-Ops, serviceops

Clement_Goubert triaged T380473: Decommission parse20[01-20] as Low priority.

Nov 21 2024, 2:15 PM · SRE, DC-Ops, ops-codfw, decommission-hardware, serviceops

Clement_Goubert created T380473: Decommission parse20[01-20].

Nov 21 2024, 2:13 PM · SRE, DC-Ops, ops-codfw, decommission-hardware, serviceops

Clement_Goubert updated the task description for T376966: wikikube-worker21[56-70] implementation tracking.

Nov 21 2024, 2:12 PM · serviceops

Clement_Goubert updated the task description for T376966: wikikube-worker21[56-70] implementation tracking.

Nov 21 2024, 12:21 PM · serviceops

Clement_Goubert added a comment to T376966: wikikube-worker21[56-70] implementation tracking.

wikikube-worker2159.codfw.wmnet is in C4 and blocked by management switch being down

Nov 21 2024, 12:16 PM · serviceops

Clement_Goubert added a comment to T376966: wikikube-worker21[56-70] implementation tracking.

wikikube-worker2157.codfw.wmnet has the same issue as T380265: hw troubleshooting: Link down for wikikube-worker2140.codfw.wmnet

Nov 21 2024, 12:14 PM · serviceops

Clement_Goubert added a comment to T380265: hw troubleshooting: Link down for wikikube-worker2140.codfw.wmnet.

I have the same issue on wikikube-worker2157.codfw.wmnet, the interface in netbox is eno12409np1 but it has no link, whereas eno12399np0 does.

Nov 21 2024, 11:39 AM · SRE, serviceops, ops-codfw, DC-Ops

Clement_Goubert changed the status of T376966: wikikube-worker21[56-70] implementation tracking, a subtask of T376965: Q2:rack/setup/install wikikube-worker21[56-70], from Open to In Progress.

Nov 21 2024, 11:31 AM · SRE, serviceops, ops-codfw, DC-Ops

Clement_Goubert changed the status of T376966: wikikube-worker21[56-70] implementation tracking from Open to In Progress.

Nov 21 2024, 11:31 AM · serviceops

Clement_Goubert updated the task description for T376966: wikikube-worker21[56-70] implementation tracking.

Nov 21 2024, 11:31 AM · serviceops

Nov 20 2024

Clement_Goubert reopened T380265: hw troubleshooting: Link down for wikikube-worker2140.codfw.wmnet as "Open".

@Papaul sorry for the misunderstanding, but it's not resolved. The interface that is supposed to have the link according to Netbox doesn't. I don't know if the best course of action is to change the connection in Netbox to be to eno12399np0 and reprovision the server?

Nov 20 2024, 3:55 PM · SRE, serviceops, ops-codfw, DC-Ops

Clement_Goubert reopened T380265: hw troubleshooting: Link down for wikikube-worker2140.codfw.wmnet, a subtask of T377028: wikikube-worker21[36-55] implementation tracking, as Open.

Nov 20 2024, 3:55 PM · serviceops

Clement_Goubert added a comment to T380265: hw troubleshooting: Link down for wikikube-worker2140.codfw.wmnet.

Yes, eno12409np1 was the one where the IPs were originally mounted when I encountered the issue. In order to troubleshoot, I changed the config in /etc/network/interfaces to mount the IPs on eno12399np0, and that interface has the link up.

Nov 20 2024, 3:49 PM · SRE, serviceops, ops-codfw, DC-Ops

Clement_Goubert added a comment to T380265: hw troubleshooting: Link down for wikikube-worker2140.codfw.wmnet.

i just managed to mount the ip adresses on the other interface eno12399np0 and the link is up. Looks like the wrong one got provisioned?

Nov 20 2024, 3:39 PM · SRE, serviceops, ops-codfw, DC-Ops

Clement_Goubert changed the status of T377028: wikikube-worker21[36-55] implementation tracking from In Progress to Stalled.

All done and pooled except 2140 waiting on T380265: hw troubleshooting: Link down for wikikube-worker2140.codfw.wmnet

Nov 20 2024, 1:57 PM · serviceops

Clement_Goubert changed the status of T377028: wikikube-worker21[36-55] implementation tracking, a subtask of T377027: Q2:rack/setup/install wikikube-worker21[36-55], from In Progress to Stalled.

Nov 20 2024, 1:57 PM · Patch-For-Review, SRE, serviceops, ops-codfw, DC-Ops

Clement_Goubert updated the task description for T377028: wikikube-worker21[36-55] implementation tracking.

Nov 20 2024, 1:41 PM · serviceops

Clement_Goubert changed the status of T377028: wikikube-worker21[36-55] implementation tracking, a subtask of T377027: Q2:rack/setup/install wikikube-worker21[36-55], from Open to In Progress.