HomePhabricator

Validate and normalize file contents in FFS

Description

Validate and normalize file contents in FFS

  1. Only accept valid UTF-8. Further work could be done to convert

things on the fly, but it is unclear whether that extra complexity
is needed right now and worth the effort. Further work is probably
needed to use better exceptions (MWException is being deprecated)
and handle those exceptions appropriately.

  1. Normalize the input to the standard MediaWiki Unicode normalization

which is NFC. There is probably a small (unmeasured) performance penalty
here, but that should be negligible because:

  • parsing should only happen when updating group definitions (known issues exist)
  • we are normalizing the whole file before parsing it, not individual messages

This should prevent any kind of unexpected issues with search,
translation memory, insertables, no-change diffs and many other things.

Bug: T87503
Change-Id: Ib8e0348dd562c7b82b07705fc07d87476f49f961

Details

Provenance
NikerabbitAuthored on
Amire80Committed on Feb 18 2015, 10:10 AM
Parents
rETRA59d53e05c655: Split language details to subpage on Special:SupportedLanguages
Branches
Unknown
Tags
Unknown
ChangeId
Ib8e0348dd562c7b82b07705fc07d87476f49f961