Change Details

```name=Error message Invariant failed: Bad UTF-8 (full string verification) ``` ```lines=12,name=Stack Trace ``` ##### Notes The bad UTF-8 can also be seen directly in the source for: https://ja.wikipedia.org/wiki/Wikipedia:%E5%89%8A%E9%99%A4%E8%A8%98%E9%8C%B2/%E9%81%8E%E5%8E%BB%E3%83%AD%E3%82%B0_2004%E5%B9%B411%E6%9C%88 This task is forked from T237467, which ended up being an issue with `Language::commafy` generating bad UTF-8. In contrast, in this task the bad UTF-8 is coming directly from the DB. As described in T237467#6566785, we need the following mitigations: 1. Bad UTF-8 is not supposed to make it past PST to get stored in the DB in the first place. So we need to track down how it got in there and clean it up; also perhaps cleaning up other articles that managed to get saved with bad UTF-8. 2. Fix core to plug this hole so that bad UTF-8 is not stored in the DB. 3. Validate wikitext source we get from the DB and fix up bad UTF-8 we get, downgrading this from a crasher to a warning. (The assertion is still appropriate if we encounter bad UTF-8 later, since that would be generated by Parsoid from valid inputs; but Parsoid operates under the assumption that all of its inputs are valid.)

```name=Error message Invariant failed: Bad UTF-8 (full string verification) ``` ##### Notes The bad UTF-8 can also be seen directly in the source for: https://ja.wikipedia.org/wiki/Wikipedia:%E5%89%8A%E9%99%A4%E8%A8%98%E9%8C%B2/%E9%81%8E%E5%8E%BB%E3%83%AD%E3%82%B0_2004%E5%B9%B411%E6%9C%88 This task is forked from T237467, which ended up being an issue with `Language::commafy` generating bad UTF-8. In contrast, in this task the bad UTF-8 is coming directly from the DB. As described in T237467#6566785, we need the following mitigations: 1. Bad UTF-8 is not supposed to make it past PST to get stored in the DB in the first place. So we need to track down how it got in there and clean it up; also perhaps cleaning up other articles that managed to get saved with bad UTF-8. 2. Fix core to plug this hole so that bad UTF-8 is not stored in the DB. 3. Validate wikitext source we get from the DB and fix up bad UTF-8 we get, downgrading this from a crasher to a warning. (The assertion is still appropriate if we encounter bad UTF-8 later, since that would be generated by Parsoid from valid inputs; but Parsoid operates under the assumption that all of its inputs are valid.)

```name=Error message Invariant failed: Bad UTF-8 (full string verification) ``` ```lines=12,name=Stack Trace ``` ##### Notes The bad UTF-8 can also be seen directly in the source for: https://ja.wikipedia.org/wiki/Wikipedia:%E5%89%8A%E9%99%A4%E8%A8%98%E9%8C%B2/%E9%81%8E%E5%8E%BB%E3%83%AD%E3%82%B0_2004%E5%B9%B411%E6%9C%88 This task is forked from T237467, which ended up being an issue with `Language::commafy` generating bad UTF-8. In contrast, in this task the bad UTF-8 is coming directly from the DB. As described in T237467#6566785, we need the following mitigations: 1. Bad UTF-8 is not supposed to make it past PST to get stored in the DB in the first place. So we need to track down how it got in there and clean it up; also perhaps cleaning up other articles that managed to get saved with bad UTF-8. 2. Fix core to plug this hole so that bad UTF-8 is not stored in the DB. 3. Validate wikitext source we get from the DB and fix up bad UTF-8 we get, downgrading this from a crasher to a warning. (The assertion is still appropriate if we encounter bad UTF-8 later, since that would be generated by Parsoid from valid inputs; but Parsoid operates under the assumption that all of its inputs are valid.)