Ores mwparserfromhell causes celery segfaults
Closed, ResolvedPublic
Actions

Assigned To

Authored By

	elukey
	Nov 27 2021, 11:22 AM

Description

This is similar to T222866, celery on some ores nodes is segfaulting for the same text:

$1 = {ob_base = {ob_refcnt = 2, ob_type = 0x7f523de284e0 <TokenizerType>}, text = {
    object = '<noinclude>{{위키백과:질문방/보존2|년=2021|월=11|주=47}}</noinclude>\n\n== 파일 삭제 요청에 대한 질문 ==\n\n 파일 삭제 요청은 어디서 하나요? --\'\'\'[[사:밀크맛 우유|<span style="text-shadow: 0px 0px 5px #FF83AB; color: snow;">MILK</span>]] [[사토:밀크맛 우유|<span style="text-shadow: 0px 0px 5px #FF83AB; color: skyblue;"> ⍤⃝♬</span>]]\'\'\' 2021년 11월 22일 (월) 15:49 (KST)\n\n:{{핑|밀크맛 우유}}[[백:문서 관리 요청]]에서 하시면 됩니다.ㅡ [[사:Cream Latte|<span style="background: linear-gradient(135deg, slateblue, hotpink); -webkit-background-clip: text; -webkit-text-fill-color: transparent; white-space:nowrap; -webkit-text-stroke:0px white;">\'\'\'˖ ִֶָᶤ C.l ִֶָꨄˊ˗\'\'\'</span>]] 2021년 11월 22일 (월) 17:50 (KST)\n:{{ㄷ|밀크맛 우유}}일문 삭제하듯이 {{틀|ㅅ}} 쓰시면 됩니다. --[[사용자:KeySpace|<span style="color:Blue;font-family:Courier new, serif;">Key</span><span style="color:brown;font-family:Courier new, serif;">.S</span>]]<span style="font-size:85%">([[사토: KeySpace|토론]]/[[특: 기여/KeySpace|기여]]/[[특: 이메일보내기/KeySpace|이메일]])</span> 2021년 11월 22일 (월) 21:07 (KST)\n\n== 최초 ==\n\n위키백과 최초로 차단된 사람이 누구인가요?\n2021년 11월 2...(truncated), length = 25854, kind = 4, data = 0x560e6b9d9ad8}, topstack = 0x7f5226df9990, head = 3085, global = 1,
  depth = 2, route_state = 0, route_context = 524288, bad_routes = 0x7f52256ec428, skip_style_tags = 0}

Details

Subject	Repo	Branch	Lines +/-
Bump mwparserfromhell dependency to 0.6.3	mediawiki/services/ores/deploy	master	+2 -2
Update mwparserfromhell to 0.6.3	research/ores/wheels	master	+3 -3
ores::worker: disable coredumps	operations/puppet	production	+0 -1

Customize query in gerrit

Related Objects

Mentioned In: rORESWHEELSdafbea43fbd0: Update mwparserfromhell to 0.6.3
Mentioned Here: T222866: Ores hosts: mwparserfromhell tokenizer random segfault

Event Timeline

elukey created this task.Nov 27 2021, 11:22 AM

elukey created this object with visibility "WMF-NDA (Project)".

Restricted Application added a subscriber: Aklapper. · View Herald TranscriptNov 27 2021, 11:22 AM

Trying to follow up with the same procedure done in T222866#5179563. In this case, the parameters are: length = 25854, kind = 4, data = 0x560e6b9d9ad8

Starting address: 0x560e6b9d9ad8
End address: 0x560e6b9d9ad8 + (4 * 25854) = 0x560E6B9F2ED0

dump memory esbugmem 0x560e6b9d9ad8 0x560E6B9F2ED0

Following pep-0393, kind = 4 should be UCS-4 so 4 bytes for each character.

Indeed with the following script I was able to cause a segfault:

import mwparserfromhell
import time
import os
import glob

# The file is binary memory dump, so read it as binary and then decode it as utf16 which for the purposes of this reproduction is close enough to UCS2.
with open('esbugmem', 'rb') as f:
    data = f.read()
    text = data.decode('utf32')
    print(mwparserfromhell.parse(text))

The good thing is that I cannot reproduce the same problem with the last version of mwparserfromhell, so we should upgrade asap in my opinion. More precisely, we run version 0.5.4 and 0.6+ seems to be unaffected.

elukey added subscribers: • ACraze, kevinbazira, klausman, calbon.Nov 27 2021, 11:44 AM

To limit the damage, I have disabled coredumps in ORES celery's systemd unit since it was causing too many 10G files to be created on the root partition of the worker nodes, ending up in alarms.

I have created https://gerrit.wikimedia.org/r/c/research/ores/wheels/+/742195/ as follow up, to update mwparserfromhell, following https://wikitech.wikimedia.org/wiki/ORES/Deployment#Update_wheels

elukey changed the visibility from "WMF-NDA (Project)" to "Custom Policy".Nov 27 2021, 12:19 PM

elukey added a subscriber: Halfak.Nov 27 2021, 12:27 PM

Created https://gerrit.wikimedia.org/r/c/mediawiki/services/ores/deploy/+/742215

Deployed to Beta and checked some scores on https://ores-beta.wmflabs.org, nothing horrible registered in the celery logs on deployment-ores01.deployment-prep.eqiad1.wikimedia.cloud.

The number of segfaults is not a lot, but I'd proceed with the deployment of one prod node anyway (given how much damage a wrong score can cause): thanos link

elukey mentioned this in rORESWHEELSdafbea43fbd0: Update mwparserfromhell to 0.6.3.Nov 27 2021, 5:45 PM

deployed to ores1001, will proceed with the rest tomorrow if nothing comes up.

Change deployed fleetwide, will keep monitoring metrics but so far everything green.

elukey changed the visibility from "Custom Policy" to "Public (No Login Required)".Nov 29 2021, 8:14 AM

akosiaris awarded a token.Nov 29 2021, 4:54 PM

isarantopoulos moved this task from Unsorted to 2023-2024 Q3 Done on the Machine-Learning-Team board.Nov 20 2023, 11:39 AM

Ores mwparserfromhell causes celery segfaultsClosed, ResolvedPublicActions

Description

Details

Related Objects

Event Timeline

Ores mwparserfromhell causes celery segfaults
Closed, ResolvedPublic
Actions