We need to ensure that UTF-8 input is:
In some cases we may need to normalize on output as well, due to old
data being corrupt. Or, we can do a one-time pass on the database
to clean it up.
Version: 1.4.x
Severity: normal
| Status | Subtype | Assigned | Task | ||
|---|---|---|---|---|---|
| Invalid | None | T5985 character conversion (tracking) | |||
| Declined | None | T5969 Unicode (UTF-8, utf8) compatibility (tracking) | |||
| Resolved | • brooke | T3225 False unicode-encoding in Database creates un-reachable image-description-page | |||
| Resolved | None | T2202 Milestone: MediaWiki 1.4 (tracking) | |||
| Invalid | Aklapper | T3503 Images with broken unicode filenames | |||
| Resolved | None | T2215 File-Uploads from MacOS X has Problems with UTF-8 | |||
| Resolved | None | T2209 [DO NOT USE] HTML validity (tracking) | |||
| Resolved | • brooke | T2240 Need to perform Unicode normalization |
I'm working on a 'pure PHP' normalizer, though it's likely to be relatively slow. If too bad we may want a DSO extension for high-performance
sites (see comments in bug 215 on some external resources).
I've checked in some more or less functional normalization routines, in includes/normal.
Probably will want to have WebRequest call UtfNormal::toNFC() on input, or at least some input.
And/or put it in title/username normalization.
Additionally we'll want to check for broken UTF-8; these routines can probably be extended to do that too.
Now checking all(?) input for broken UTF-8 and normalizing to form C on input. Could use optimization but that's a
separate issue. :)
Checked into CVS for 1.4.