Page MenuHomePhabricator
Create Task
Maniphest T371881

low rate of mw-memcached errors
Closed, ResolvedPublic
Actions

Assigned To
jijiki
Authored By
jijiki
Aug 6 2024, 10:16 AM
Tags
Referenced Files
F59360187: image.png
May 16 2025, 5:04 PM
F59360185: image.png
May 16 2025, 5:04 PM
F57113304: image.png
Aug 6 2024, 10:16 AM
F57113246: image.png
Aug 6 2024, 10:16 AM
Subscribers
Aklapper
cmooney
jijiki
JMeybohm
• mszabo
Muehlenhoff

Description

We have observed a low rate of memcached errors in production, eg 61/3hrs, which may or may not be worth investigating. While the rate is *very* low, we could dig a little bit further just in case there is an underlying problem or mcrouter behaviour we should be aware of or mitigate.

image.png (1×2 px, 212 KB)

zooming in a bit, we get:

image.png (778×2 px, 115 KB)

6 Aug 2024 - 08:33-08:48
Rack: B5
Host: parse2006
Pod: mcrouter-main-lcqjh
Container: mcrouter-main
Notes: Resources wise the pod looks alright, the host itself is experiencing TCP retransmits. https://grafana.wikimedia.org/goto/iDpzTPrIg?orgId=1

I will keep adding some data, and check if there is some sort of a pattern here

Details

Related Changes in Gerrit:
SubjectRepoBranchLines +/-
k8s::mediawiki_runner: allow outgoing connections to memcachedoperations/puppetproduction+11 -0
k8s::mediawiki_runner: allow outgoing connections to memcachedoperations/puppetproduction+14 -0
memcached: Switch to profile::memcached::firewall_src_setsoperations/puppetproduction+2 -1
mediawiki::memcached enable performance cpu governoroperations/puppetproduction+2 -2
memcached: add option to switch to the performance cpu governoroperations/puppetproduction+27 -18
Customize query in gerrit

Related Objects
Search...

StatusSubtypeAssignedTask
 ResolvedjijikiT371881 low rate of mw-memcached errors
OpenNoneT374366 Race condition in iptables rules during puppet runs on k8s nodes
 ResolvedakosiarisT372878 Re-IP wikikube servers in codfw row A/B moving to per-rack subnets
 ResolvedJhancock.wmT372916 Relabel codfw kubernetes nodes
 ResolvedNoneT373401 Relabel codfw kubernetes nodes
 ResolvedJhancock.wmT373457 Relabel codfw kubernetes nodes
 DuplicateNoneT373491 Relabel codfw kubernetes nodes
 DuplicateNoneT373505 Relabel codfw kubernetes nodes
 ResolvedJhancock.wmT373591 Relabel codfw kubernetes nodes
 DuplicateNoneT373669 Relabel codfw kubernetes nodes mw2295,mw2296,mw2297
 DuplicateNoneT373699 Relabel codfw kubernetes nodes mw237[789]
 ResolvedMoritzMuehlenhoffT373819 Issues reimaging kubernetes workers due to user conflicts in systemd-timesyncd
ResolvedJhancock.wmT373916 Relabel codfw kubernetes nodes
 InvalidNoneT373934 Update iDRAC on mw2260.codfw.wmnet
 ResolvedPRODUCTION ERRORClement_GoubertT373982 wikikube-worker2080.codfw.wmnet can't auth to registry
 ResolvedJMeybohmT374019 kubernetes2035 (renamed to wikikube-worker2087) reporting "Comm Error: Backplane 0"
 ResolvedJhancock.wmT374249 Relabel codfw kubernetes nodes
 ResolvedJhancock.wmT374258 Comm Error: backplane 0 when reimaging wikikube-worker2095
 ResolvedJhancock.wmT374380 Relabel codfw kubernetes nodes
 ResolvedJhancock.wmT374622 Relabel codfw kubernetes nodes mw2390 and mw2394-mw2399
 ResolvedJhancock.wmT374733 Relabel codfw kubernetes nodes
 ResolvedJhancock.wmT375398 Relabel codfw kubernetes nodes mw2424 and mw2425
 ResolvedNoneT381244 Relabel codfw kubernetes nodes
 OpenNoneT365687 Improve calico-typha firewall rules

Event Timeline

jijiki created this task.Aug 6 2024, 10:16 AM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptAug 6 2024, 10:16 AM
jijiki claimed this task.Aug 6 2024, 10:18 AM
jijiki triaged this task as Low priority.
jijiki updated the task description. (Show Details)
jijiki updated the task description. (Show Details)Aug 6 2024, 11:23 AM
• mszabo subscribed.Aug 21 2024, 3:32 PM
jijiki added a project: SRE.Oct 10 2024, 1:34 PM
jijiki removed a project: SRE.Oct 10 2024, 2:27 PM
jijiki moved this task from Incoming 🐫 to Stalled 🐌 on the serviceops-deprecated board.Nov 7 2024, 11:30 AM
jijiki added a subscriber: cmooney.May 16 2025, 5:04 PM

Adding thos graphs as notes, though I do not think they are the cause. Chatted with @cmooney as well if network issues could be the culprit, though we deduced it may not be it.

image.png (1×2 px, 436 KB)

image.png (369×936 px, 41 KB)

jijiki raised the priority of this task from Low to Medium.May 16 2025, 5:17 PM

I have observed that the rate those errors surface has increased, I will keep working on it and update.

gerritbot added a comment.May 21 2025, 3:40 PM

Change #1148900 had a related patch set uploaded (by Effie Mouzeli; author: Effie Mouzeli):

[operations/puppet@production] memcached: add option to switch to the performance cpu governor

https://gerrit.wikimedia.org/r/1148900

gerritbot added a project: Patch-For-Review.May 21 2025, 3:40 PM
gerritbot added a comment.May 21 2025, 3:49 PM

Change #1148901 had a related patch set uploaded (by Effie Mouzeli; author: Effie Mouzeli):

[operations/puppet@production] mediawiki::memcached enable performance cpu governor

https://gerrit.wikimedia.org/r/1148901

gerritbot added a comment.May 22 2025, 8:34 AM

Change #1148900 merged by Effie Mouzeli:

[operations/puppet@production] memcached: add option to switch to the performance cpu governor

https://gerrit.wikimedia.org/r/1148900

gerritbot added a comment.May 22 2025, 8:35 AM

Change #1148901 merged by Effie Mouzeli:

[operations/puppet@production] mediawiki::memcached enable performance cpu governor

https://gerrit.wikimedia.org/r/1148901

Maintenance_bot removed a project: Patch-For-Review.May 22 2025, 9:31 AM
jijiki added a subtask: T374366: Race condition in iptables rules during puppet runs on k8s nodes.May 28 2025, 12:51 PM
jijiki reopened subtask T374366: Race condition in iptables rules during puppet runs on k8s nodes as Open.
jijiki mentioned this in T374366: Race condition in iptables rules during puppet runs on k8s nodes.
jijiki added subscribers: JMeybohm, Muehlenhoff.May 29 2025, 1:10 PM

A chunk of the memcached errors we are observing are due to T374366, which we have known for a long time and attempted to mitigate (to no avail yet). After chatting with @Muehlenhoff and @JMeybohm, we will attempt to firstly add rules to allow access from k8s nodes to memcached, since all nodes are running the mw-mcrouter ds already. As the migration to nftables progresses, we will revisit.

gerritbot added a comment.Jun 5 2025, 10:25 AM

Change #1153981 had a related patch set uploaded (by Muehlenhoff; author: Muehlenhoff):

[operations/puppet@production] memcached: Switch to profile::memcached::firewall_src_sets

https://gerrit.wikimedia.org/r/1153981

gerritbot added a project: Patch-For-Review.Jun 5 2025, 10:25 AM
gerritbot added a comment.Jun 12 2025, 8:50 AM

Change #1153981 abandoned by Muehlenhoff:

[operations/puppet@production] memcached: Switch to profile::memcached::firewall_src_sets

Reason:

replaced by 1156269

https://gerrit.wikimedia.org/r/1153981

Maintenance_bot removed a project: Patch-For-Review.Jun 12 2025, 9:31 AM
gerritbot added a comment.Jul 14 2025, 12:28 PM

Change #1169104 had a related patch set uploaded (by Effie Mouzeli; author: Effie Mouzeli):

[operations/puppet@production] k8s::mediawiki_runner: allow outgoing connections to memcached

https://gerrit.wikimedia.org/r/1169104

gerritbot added a project: Patch-For-Review.Jul 14 2025, 12:28 PM
gerritbot added a comment.Jul 14 2025, 1:38 PM

Change #1169118 had a related patch set uploaded (by Effie Mouzeli; author: Effie Mouzeli):

[operations/puppet@production] k8s::mediawiki_runner: allow outgoing connections to memcached

https://gerrit.wikimedia.org/r/1169118

gerritbot added a comment.Jul 14 2025, 1:42 PM

Change #1169104 abandoned by Effie Mouzeli:

[operations/puppet@production] k8s::mediawiki_runner: allow outgoing connections to memcached

Reason:

in favour of Icc0f0cfb8832809df6bb64baa015e1831b371718

https://gerrit.wikimedia.org/r/1169104

gerritbot added a comment.Jul 15 2025, 11:11 AM

Change #1169118 merged by Effie Mouzeli:

[operations/puppet@production] k8s::mediawiki_runner: allow outgoing connections to memcached

https://gerrit.wikimedia.org/r/1169118

Maintenance_bot removed a project: Patch-For-Review.Jul 15 2025, 11:31 AM
jijiki closed this task as Resolved.Jan 22 2026, 11:53 AM

I am closing this as it has not manifested itself for a long time

Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits