Page MenuHomePhabricator
Create Task
Maniphest T246470

Measure segfaults in mediawiki and parsoid servers
Open, MediumPublic
Actions

Description

We are aware that daily some mediawiki servers segfault, for example

php-fpm7.2[18796]: segfault at 5ffddb105c ip 00005575f7234ae3 sp 00007ffd32aad530 error 4 in php-fpm7.2[5575f6f92000+410000]

php-fpm7.2[18795]: segfault at 215 ip 00007f8b49b4d3a9 sp 00007ffd32aad490 error 4 in libmemcached.so.11.0.0[7f8b49b3c000+30000]

It might make sense to use mtail and measure those segfaults, in the future we could potentially alert on this and avoid running into issues when we deploy php extentions and other related updates

Details

SubjectRepoBranchLines +/-
mtail: add counter for kernel trapsoperations/puppetproduction+6 -0
Customize query in gerrit

Related Objects
Search...

Event Timeline

jijiki created this task.Feb 28 2020, 6:13 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptFeb 28 2020, 6:13 PM
jijiki triaged this task as Medium priority.Feb 28 2020, 6:13 PM
CDanis subscribed.Jun 12 2020, 5:41 PM

We do have this data in mtail already but it isn't the easiest to work with: https://w.wiki/Ttx

There's also something odd happening there where the rate of segfaults seems to decline after 16:00 UTC and reset at 00:00 UTC. I don't think that's accurate in reality:

P11485 Masterwork From Distant Lands
1✔️ root@centrallog2001.codfw.wmnet /srv/syslog 🕐⁉️ ls mw*/syslog.log-20200612 | xargs -n10 -P3 grep --line-buffered -hP 'mw\d{4} kernel: \[[0-9.]+\] .*: segfault at' | awk '$7 ~ /php-fpm7\.2/ {print $3}' | cut -d: -f1 | sort | uniq -c
2 12 00
3 18 01
4 9 02
5 16 03
6 14 04
7 12 05
8 3 06
9 3 07
10 14 08
11 15 09
12 13 10
13 25 11
14 22 12
15 4 13
16 3 14
17 5 15
18 7 16
19 8 17
20 18 18
21 21 19
22 4 20
23 30 21
24 16 22
25 3 23

CDanis added a subscriber: elukey.Jun 12 2020, 5:42 PM
jijiki added a subscriber: User-jijiki.Aug 13 2020, 7:18 PM
jijiki moved this task from Incoming 🐫 to 🔦Unused2 on the serviceops board.Aug 17 2020, 11:45 PM
Aklapper added a project: User-jijiki.Sep 18 2020, 7:56 AM
Aklapper removed a subscriber: User-jijiki.
elukey added a subtask: T236253: systemd-coredump can make a system unresponsive.Sep 17 2021, 8:52 AM
elukey added a project: observability.

Getting back to this, added the Observability tag to get some feedback from the team as well.

gerritbot added a comment.Sep 17 2021, 9:45 AM

Change 721773 had a related patch set uploaded (by Elukey; author: Elukey):

[operations/puppet@production] mtail: add counter for kernel traps

https://gerrit.wikimedia.org/r/721773

gerritbot added a project: Patch-For-Review.Sep 17 2021, 9:45 AM
elukey added a comment.Sep 17 2021, 9:53 AM

It seems that we do have an mtail metric for segfaults, but when checking Thanos I see a never-increasing counter, even if I can find segfaults during the recent days (maybe I am missing something trivial). When we clear out this issue, and possibly update mtail for traps as well, we could add those to dashboards.

lmata edited projects, added Observability-Logging; removed observability.Sep 21 2021, 3:16 PM
lmata edited projects, added Observability-Metrics; removed Observability-Logging.
gerritbot added a comment.Sep 29 2021, 9:41 AM

Change 721773 merged by Elukey:

[operations/puppet@production] mtail: add counter for kernel traps

https://gerrit.wikimedia.org/r/721773

Stashbot added a comment.Sep 29 2021, 9:54 AM

Mentioned in SAL (#wikimedia-operations) [2021-09-29T09:54:21Z] <godog> bounce mtail on centrallog* - T246470

fgiunchedi subscribed.Sep 29 2021, 9:55 AM

While investigating this with @elukey I noticed mtail_lines_total has stopped increasing for centrallog in march (!), a separate but related problem for sure (to be tackled in another task)

fgiunchedi mentioned this in T292051: mtail was not working on centrallog, need to improve monitoring.Sep 29 2021, 10:08 AM
Maintenance_bot removed a project: Patch-For-Review.Sep 29 2021, 10:10 AM
herron subscribed.Sep 29 2021, 8:20 PM
In T246470#7387278, @fgiunchedi wrote:

While investigating this with @elukey I noticed mtail_lines_total has stopped increasing for centrallog

This has been fixed, and I can see trap metrics working now:

traps{binary="php-fpm7.2", cluster="syslog", hostname="mw1367", instance="centrallog2001:3903", job="mtail", prog="kernel.mtail", prometheus="ops", site="codfw"} 1
traps{binary="php-fpm7.2", cluster="syslog", hostname="mw1442", instance="centrallog1001:3903", job="mtail", prog="kernel.mtail", prometheus="ops", site="eqiad"} 1
traps{binary="php-fpm7.2", cluster="syslog", hostname="mw1442", instance="centrallog2001:3903", job="mtail", prog="kernel.mtail", prometheus="ops", site="codfw"} 1
traps{binary="php-fpm7.2", cluster="syslog", hostname="wtp1038", instance="centrallog2001:3903", job="mtail", prog="kernel.mtail", prometheus="ops", site="codfw"} 1
elukey added a comment.Oct 1 2021, 9:36 AM

I can confirm, thanks a lot for the quick fix!

jijiki moved this task from 🔦Unused2 to 🥋Good First Task on the serviceops board.Oct 17 2022, 4:22 PM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL