About ~150 tests fails in a test run (noticed this over the last month in rt-testing where only 99.99% of tests succeed whereas it used to be 100% earlier). This has probably been a problem from the beginning of testreduce and something we haven't noticed. After about 5 failing test runs, these test titles are ignore and successful test result % goes back to 100% again (and we forget about the failed titles).
Here is an example of a failing insert.
INSERT INTO results ( page_id, commit_hash, result ) VALUES ( 27797, '509155d5', '<testsuites>\n<testsuite name=\"Roundtrip article enwiki:Talk:List of awards and nominations received by Blackpink\">\n<testcase name=\"enwiki:Talk:List of awards and nominations received by Blackpink character 817\">\n<skipped type=\"insignificantWikitextDiff\">\n------\n== 2 new Apan awards ==\n\n++++++\n==2 new Apan awards==\n\n</skipped>\n</testcase>\n<testcase name=\"enwiki:Talk:List of awards and nominations received by Blackpink character 2440\">\n<skipped type=\"insignificantWikitextDiff\">\n------\n:::{{done}} '''<span style="color:#f535aa">—</span> [[User:Paper9oll|<span style="background:#f535aa; color:#fff; padding: 2px">Paper9oll</span>]] <span style="color: #f535aa">([[User_talk:Paper9oll|📣]] • [[Special:Contributions/Paper9oll|📝]])</span>''' 15:29, 24 January 2021 (UTC)\n++++++\n:::{{done}} '''<span style="color:#f535aa">—</span> [[User:Paper9oll|<span style="background:#f535aa; color:#fff; padding: 2px">Paper9oll</span>]] <span style="color: #f535aa">([[User_talk:Paper9oll|📣]] • [[Special:Contributions/Paper9oll|📝]])</span>''' 15:29, 24 January 2021 (UTC)\n\n</skipped>\n</testcase>\n</testsuite>\n<perfstats>\n<perfstat type=\"time:total\">1388.9099119901657</perfstat>\n<perfstat type=\"size:wt:raw\">43990</perfstat>\n<perfstat type=\"size:wt:gzip\">7607</perfstat>\n<perfstat type=\"size:html:raw\">2723</perfstat>\n<perfstat type=\"size:html:gzip\">1098</perfstat>\n</perfstats>\n</testsuites>' ) ON DUPLICATE KEY UPDATE id = LAST_INSERT_ID( id ), result = VALUES( result )