Page MenuHomePhabricator

The Tokenizer strips backslashes from \x
Closed, ResolvedPublic


See code:

case 'x':
	$chr = substr( $code, $offset + 2, 2 );

	if ( preg_match( '/^[0-9A-Fa-f]{2}$/', $chr ) ) {
		$token .= chr( hexdec( $chr ) );
		// \xXX -- 2 done later
		$offset += 2;
	} else {
		$token .= 'x';

It tries to parse hex codepoints in literals, and that's OK. However, when it cannot parse a hex, it will add 'x' to the string, and not '\x' as it found. This is usually equivalent, but sometimes it's not. For instance, inside regexps: see


Related Gerrit Patches:
mediawiki/extensions/AbuseFilter : masterTokenizer: don't strip backslashes from \x

Event Timeline

Daimona created this task.Nov 16 2019, 3:11 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptNov 16 2019, 3:11 PM

Change 551312 had a related patch set uploaded (by Daimona Eaytoy; owner: Daimona Eaytoy):
[mediawiki/extensions/AbuseFilter@master] Tokenizer: don't strip backslashes from \x

Change 551312 merged by jenkins-bot:
[mediawiki/extensions/AbuseFilter@master] Tokenizer: don't strip backslashes from \x

Daimona closed this task as Resolved.Nov 22 2019, 1:48 PM
Daimona removed a project: Patch-For-Review.