Page MenuHomePhabricator
Create Task
Maniphest T324170

php8.2 deprecate use of 'HTML-ENTITIES' on mb_convert_encoding
Closed, ResolvedPublic
Actions

Description

- Mbstring:
  . Use of QPrint, Base64, Uuencode, and HTML-ENTITIES 'text encodings' is
    deprecated for all Mbstring functions. Unlike all the other text
    encodings supported by Mbstring, these do not encode a sequence of
    Unicode codepoints, but rather a sequence of raw bytes. It is not
    clear what the correct return values for most Mbstring functions should
    be when one of these non-encodings is specified. Further, PHP has
    separate, built-in implementations of all of them; for example, UUencoded
    data can be handled using convert_uuencode/convert_uudecode.

From https://github.com/php/php-src/commit/9308974f8cc6c1046f228be5320fe067913ba987

Codesearch (best-effort): https://codesearch.wmcloud.org/search/?q=mb_%5Cw%2B%5Cs*%5C(%5B%5E%22%27%5D*%5B%22%27%5D(BASE64%7CHTML-ENTITIES%7CHTML%7CQUOTED-PRINTABLE%7CQPRINT%7CUUENCODE)%5B%22%27%5D&i=fosho&files=%5C.php&excludeFiles=&repos=

Details

Show related patches Customize query in gerrit

Related Objects

Event Timeline

Umherirrender created this task.Nov 30 2022, 11:22 PM
cscott subscribed.Dec 1 2022, 3:08 PM

For the Parsoid project it looks like it's the Zest library which uses HTML-ENTITIES. I can take a look and release a PHP 8.2-compatible version.

Jgiannelos moved this task from Needs Triage to Bugs & Crashers on the Parsoid board.Dec 1 2022, 3:09 PM
ssastry triaged this task as Medium priority.Dec 1 2022, 3:13 PM
TheDJ subscribed.EditedDec 21 2022, 7:39 PM

This is a common workaround for the fact that the default for a DOM document is Latin-1. We have HTML that is utf-8 encoded, then escape the utf-8 to entities, so that it becomes valid latin-1 and then feed the DOMDocument.

This is required in these cases, likely because we are not feeding our DOMDocument proper HTML5 files, and possibly in older libxml versions, the loadHTML/libxml was unable to read the meta declaration from the html document ?

The proper way to fix this issue is

  • feeding it an ACTUAL utf-8 HTML document (instead of html fragments)
  • prepending the '<?xml encoding="UTF-8"> to force the parser to utf-8
  • prepending <meta charset="utf-8"/> or <meta http-equiv="content-type" content="text/html; charset=utf-8"> to the html (these last things might be what didn't work in the past in libxml ???)
Reedy moved this task from Backlog to Other on the PHP 8.2 support board.Jan 8 2023, 11:42 PM
• 1313 removed projects: PHP 8.2 support, Parsoid, Refreshed, HtmlFormatter, MediaWiki-extensions-RandomImage, CommonsMetadata.Jan 12 2023, 4:28 AM
Izno added projects: PHP 8.2 support, Parsoid, Refreshed, HtmlFormatter, MediaWiki-extensions-RandomImage, CommonsMetadata.Jan 12 2023, 6:02 AM
gerritbot added a comment.Jan 12 2023, 7:38 PM

Change 879625 had a related patch set uploaded (by Umherirrender; author: Umherirrender):

[HtmlFormatter@master] Parse html as whole document to avoid encoding issues

https://gerrit.wikimedia.org/r/879625

gerritbot added a project: Patch-For-Review.Jan 12 2023, 7:38 PM
gerritbot added a comment.Jan 12 2023, 8:06 PM

Change 879632 had a related patch set uploaded (by Umherirrender; author: Umherirrender):

[mediawiki/libs/Zest@master] tests: Use utf-8 html document for testing

https://gerrit.wikimedia.org/r/879632

gerritbot added a comment.Jan 12 2023, 8:41 PM

Change 879639 had a related patch set uploaded (by Umherirrender; author: Umherirrender):

[mediawiki/extensions/CommonsMetadata@master] Parse html as whole document to avoid encoding issues

https://gerrit.wikimedia.org/r/879639

gerritbot added a comment.Jan 12 2023, 8:49 PM

Change 879641 had a related patch set uploaded (by Umherirrender; author: Umherirrender):

[mediawiki/extensions/RandomImage@master] Parse html as whole document to avoid encoding issues

https://gerrit.wikimedia.org/r/879641

gerritbot added a comment.Jan 12 2023, 8:52 PM

Change 879642 had a related patch set uploaded (by Umherirrender; author: Umherirrender):

[mediawiki/skins/Refreshed@master] Parse html as whole document to avoid encoding issues

https://gerrit.wikimedia.org/r/879642

gerritbot added a comment.Jan 19 2023, 9:03 PM

Change 879642 merged by jenkins-bot:

[mediawiki/skins/Refreshed@master] Parse html as whole document to avoid encoding issues

https://gerrit.wikimedia.org/r/879642

Umherirrender mentioned this in R1893:4b095fef87fb: Parse html as whole document to avoid encoding issues.Jan 19 2023, 9:30 PM
gerritbot added a comment.Jan 20 2023, 8:21 PM

Change 879625 merged by jenkins-bot:

[HtmlFormatter@master] Parse html as whole document to avoid encoding issues

https://gerrit.wikimedia.org/r/879625

gerritbot added a comment.Jan 21 2023, 10:12 PM

Change 879639 merged by jenkins-bot:

[mediawiki/extensions/CommonsMetadata@master] Parse html as whole document to avoid encoding issues

https://gerrit.wikimedia.org/r/879639

ReleaseTaggerBot added a project: MW-1.40-notes (1.40.0-wmf.20; 2023-01-23).Jan 21 2023, 11:00 PM
gerritbot added a comment.Feb 5 2023, 7:09 PM

Change 879641 merged by jenkins-bot:

[mediawiki/extensions/RandomImage@master] Parse html as whole document to avoid encoding issues

https://gerrit.wikimedia.org/r/879641

kostajh mentioned this in T330501: Deprecated: mb_convert_encoding(): Handling HTML entities via mbstring is deprecated; use htmlspecialchars, htmlentities, or mb_encode_numericentity/mb_decode_numericentity instead in vendor/wikimedia/html-formatter/src/HtmlFormatter.php.Feb 24 2023, 1:36 PM
kostajh merged a task: T330501: Deprecated: mb_convert_encoding(): Handling HTML entities via mbstring is deprecated; use htmlspecialchars, htmlentities, or mb_encode_numericentity/mb_decode_numericentity instead in vendor/wikimedia/html-formatter/src/HtmlFormatter.php.
kostajh added subscribers: kostajh, Aklapper.
kostajh added a comment.Feb 24 2023, 1:39 PM

We need a new release of wikimedia/html-formatter on Packagist so that we can update MediaWiki core's composer.json from 3.0.1 to whatever the new version will be. Looking at the README and the mediawiki.org docs (https://www.mediawiki.org/wiki/HtmlFormatter), I don't see notes about how to do this or who usually does it. Does anyone know?

Umherirrender mentioned this in T330528: Release new version of HTMLFormatter.Feb 24 2023, 8:24 PM
In T324170#8644166, @kostajh wrote:

We need a new release of wikimedia/html-formatter on Packagist so that we can update MediaWiki core's composer.json from 3.0.1 to whatever the new version will be. Looking at the README and the mediawiki.org docs (https://www.mediawiki.org/wiki/HtmlFormatter), I don't see notes about how to do this or who usually does it. Does anyone know?

Have created T330528 as request for a new release

gerritbot added a comment.Feb 27 2023, 4:45 PM

Change 879632 merged by jenkins-bot:

[mediawiki/libs/Zest@master] tests: Use utf-8 html document for testing

https://gerrit.wikimedia.org/r/879632

Maintenance_bot removed a project: Patch-For-Review.Feb 27 2023, 5:10 PM
gerritbot added a comment.Feb 27 2023, 11:08 PM

Change 892414 had a related patch set uploaded (by Jack Phoenix; author: Umherirrender):

[mediawiki/skins/Refreshed@REL1_35] Parse html as whole document to avoid encoding issues

https://gerrit.wikimedia.org/r/892414

gerritbot added a project: Patch-For-Review.Feb 27 2023, 11:08 PM
gerritbot added a comment.Feb 27 2023, 11:12 PM

Change 892414 merged by jenkins-bot:

[mediawiki/skins/Refreshed@REL1_35] Parse html as whole document to avoid encoding issues

https://gerrit.wikimedia.org/r/892414

ashley mentioned this in R1893:27ad71b0dc6c: Parse html as whole document to avoid encoding issues.Feb 27 2023, 11:29 PM
Maintenance_bot removed a project: Patch-For-Review.Feb 27 2023, 11:30 PM
alistair3149 subscribed.Feb 28 2023, 7:47 PM
Samwilson mentioned this in T332112: Deprecated: mb_convert_encoding().Mar 15 2023, 8:05 AM
Umherirrender unsubscribed.Jul 12 2023, 7:57 PM
DonPaolo subscribed.Aug 22 2023, 7:33 PM

The bug is still present in 1.39.4, it appears only on non-big cell phones, I see it with Pixel 5 simulator in chrome developer tools.

Reedy added projects: MW-1.39-release, MW-1.40-release.Aug 22 2023, 7:36 PM
gerritbot added a comment.Aug 22 2023, 8:00 PM

Change 951493 had a related patch set uploaded (by Reedy; author: Reedy):

[HtmlFormatter@master] HtmlFormatter: Add #[\ReturnTypeWillChange] to ease migration

https://gerrit.wikimedia.org/r/951493

gerritbot added a project: Patch-For-Review.Aug 22 2023, 8:00 PM
Reedy subscribed.Aug 22 2023, 8:01 PM
In T324170#9111431, @DonPaolo wrote:

The bug is still present in 1.39.4, it appears only on non-big cell phones, I see it with Pixel 5 simulator in chrome developer tools.

FWIW, no one ever stated it was fixed ;)

gerritbot added a comment.Aug 22 2023, 8:20 PM

Change 951493 merged by jenkins-bot:

[HtmlFormatter@master] HtmlFormatter: Add #[\ReturnTypeWillChange] to ease migration

https://gerrit.wikimedia.org/r/951493

Maintenance_bot removed a project: Patch-For-Review.Aug 22 2023, 8:31 PM
gerritbot added a comment.Aug 22 2023, 8:35 PM

Change 951494 had a related patch set uploaded (by Reedy; author: Reedy):

[HtmlFormatter@master] HtmlFormatter: Add #[\ReturnTypeWillChange] to correct function to ease migration

https://gerrit.wikimedia.org/r/951494

gerritbot added a project: Patch-For-Review.Aug 22 2023, 8:35 PM
gerritbot added a comment.Aug 23 2023, 3:37 AM

Change 951494 merged by jenkins-bot:

[HtmlFormatter@master] HtmlFormatter: Add #[\ReturnTypeWillChange] to more functions to ease migration

https://gerrit.wikimedia.org/r/951494

Maintenance_bot removed a project: Patch-For-Review.Aug 23 2023, 4:10 AM
Reedy mentioned this in T344811: Test failure in MW Core with HTMLFormatter updates.Aug 23 2023, 1:49 PM
Reedy added a comment.Sep 19 2023, 12:31 PM

Guess I need to still do some backports... yay

gerritbot added a comment.Sep 19 2023, 12:41 PM

Change 958422 had a related patch set uploaded (by Reedy; author: Umherirrender):

[mediawiki/extensions/RandomImage@REL1_39] Parse html as whole document to avoid encoding issues

https://gerrit.wikimedia.org/r/958422

gerritbot added a project: Patch-For-Review.Sep 19 2023, 12:41 PM

Change 958423 had a related patch set uploaded (by Reedy; author: Umherirrender):

[mediawiki/extensions/CommonsMetadata@REL1_39] Parse html as whole document to avoid encoding issues

https://gerrit.wikimedia.org/r/958423

gerritbot added a comment.Sep 19 2023, 2:18 PM

Change 958423 merged by jenkins-bot:

[mediawiki/extensions/CommonsMetadata@REL1_39] Parse html as whole document to avoid encoding issues

https://gerrit.wikimedia.org/r/958423

gerritbot added a comment.Sep 19 2023, 2:18 PM

Change 958422 merged by jenkins-bot:

[mediawiki/extensions/RandomImage@REL1_39] Parse html as whole document to avoid encoding issues

https://gerrit.wikimedia.org/r/958422

Reedy added a comment.Sep 19 2023, 2:21 PM
In T324170#9178170, @Reedy wrote:

Guess I need to still do some backports... yay

Or not. I've done HtmlFormatter into REL1_39/REL1_40... I think the two above are about all that was missing...

Reedy added projects: ContentTranslation, EImage.Sep 19 2023, 2:22 PM

ContentTranslation and EImage are the only two left per https://codesearch.wmcloud.org/search/?q=mb_%5Cw%2B%5Cs*%5C(%5B%5E%22%27%5D*%5B%22%27%5D(BASE64%7CHTML-ENTITIES%7CHTML%7CQUOTED-PRINTABLE%7CQPRINT%7CUUENCODE)%5B%22%27%5D&i=fosho&files=%5C.php&excludeFiles=&repos= from the description...

gerritbot added a comment.Sep 19 2023, 2:25 PM

Change 958944 had a related patch set uploaded (by Reedy; author: Reedy):

[mediawiki/extensions/ContentTranslation@master] Parse html as whole document to avoid encoding issues

https://gerrit.wikimedia.org/r/958944

gerritbot added a comment.Sep 19 2023, 2:33 PM

Change 958949 had a related patch set uploaded (by Reedy; author: Reedy):

[mediawiki/extensions/EImage@master] Parse html as whole document to avoid encoding issues

https://gerrit.wikimedia.org/r/958949

gerritbot added a comment.Sep 19 2023, 9:57 PM

Change 958949 merged by jenkins-bot:

[mediawiki/extensions/EImage@master] Parse html as whole document to avoid encoding issues

https://gerrit.wikimedia.org/r/958949

Reedy mentioned this in rEEIM2d6d6e2a39ae: Parse html as whole document to avoid encoding issues.Sep 19 2023, 9:58 PM
gerritbot added a comment.Oct 12 2023, 12:06 PM

Change 958944 merged by jenkins-bot:

[mediawiki/extensions/ContentTranslation@master] Parse html as whole document to avoid encoding issues

https://gerrit.wikimedia.org/r/958944

Maintenance_bot removed a project: Patch-For-Review.Oct 12 2023, 12:10 PM
ReleaseTaggerBot added a project: MW-1.42-notes (1.42.0-wmf.1; 2023-10-17).Oct 12 2023, 1:00 PM
Pginer-WMF closed this task as Resolved.Nov 20 2023, 11:33 AM
Pginer-WMF claimed this task.
Pginer-WMF moved this task from Needs Triage to Upstream/Other teams on the ContentTranslation board.
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL