With the work that the #test_platform do in T420590 we are decreasing the feedback loop time for developers. The changes we do will also affect the gate pipeline (the tests that happens before the actual merge).
Here are some jobs before our changes (some jobs are cached, that why some metrics are really fast):
This [[ https://gerrit.wikimedia.org/r/c/mediawiki/core/+/1188790 | patch ]] 2th of Feb:
{F74800811 width=800}
This [[ https://gerrit.wikimedia.org/r/c/mediawiki/core/+/1224892 | patch ]] 9th of January:
{F74800888 width=800}
Here are the statistics for the slowest jobs in January:
|job |Count| min |max |average |stddev| median |p75| p90| p95 |p99
|[[ https://releng-data.wmcloud.org/jobs?sql=SELECT%0D%0A++%22%3Ca+href%3D%2F-%2Fdashboards%2Fjenkins%3Frepo_name%3D%22+%7C%7C+repo+%7C%7C+%22%26date_start%3D%22+%7C%7C+%3Adate_start+%7C%7C+%22%3E%22+%7C%7C+repo+%7C%7C+%22%3C%2Fa%3E%22+as+%27Repo+name%27%2C%0D%0A++count%28*%29+as+count%2C%0D%0A++printf%28%0D%0A++++%27%2502d%3A%2502d%27%2C%0D%0A++++cast%28min%28time+%2F+1000%29+%2F+60+as+integer%29%2C%0D%0A++++cast%28min%28time+%2F+1000%29+%25+60+as+integer%29%0D%0A++%29+as+min%2C%0D%0A++printf%28%0D%0A++++%27%2502d%3A%2502d%27%2C%0D%0A++++cast%28max%28time+%2F+1000%29+%2F+60+as+integer%29%2C%0D%0A++++cast%28max%28time+%2F+1000%29+%25+60+as+integer%29%0D%0A++%29+as+max%2C%0D%0A++printf%28%0D%0A++++%27%2502d%3A%2502d%27%2C%0D%0A++++cast%28avg%28time+%2F+1000%29+%2F+60+as+integer%29%2C%0D%0A++++cast%28avg%28time+%2F+1000%29+%25+60+as+integer%29%0D%0A++%29+as+average%2C%0D%0A++printf%28%0D%0A++++%27%2502d%3A%2502d%27%2C%0D%0A++++cast%28%0D%0A++++++%28%0D%0A++++++++SQRT%28%0D%0A++++++++++AVG%28time+%2F+1000+*+time+%2F+1000%29+-+AVG%28time+%2F+1000%29+*+AVG%28time+%2F+1000%29%0D%0A++++++++%29%0D%0A++++++%29+%2F+60+as+integer%0D%0A++++%29%2C%0D%0A++++cast%28%0D%0A++++++%28%0D%0A++++++++SQRT%28%0D%0A++++++++++AVG%28time+%2F+1000+*+time%29+-+AVG%28time+%2F+1000%29+*+AVG%28time+%2F+1000%29%0D%0A++++++++%29%0D%0A++++++%29+%25+60+as+integer%0D%0A++++%29%0D%0A++%29+as+stddev%2C%0D%0A++printf%28%0D%0A++++%27%2502d%3A%2502d%27%2C%0D%0A++++cast%28median%28time+%2F+1000%29+%2F+60+as+integer%29%2C%0D%0A++++cast%28median%28time+%2F+1000%29+%25+60+as+integer%29%0D%0A++%29+as+median%2C%0D%0A++printf%28%0D%0A++++%27%2502d%3A%2502d%27%2C%0D%0A++++cast%28percentile%28time+%2F+1000%2C+75%29+%2F+60+as+integer%29%2C%0D%0A++++cast%28percentile%28time+%2F+1000%2C+75%29+%25+60+as+integer%29%0D%0A++%29+as+p75%2C%0D%0A++printf%28%0D%0A++++%27%2502d%3A%2502d%27%2C%0D%0A++++cast%28percentile%28time+%2F+1000%2C+90%29+%2F+60+as+integer%29%2C%0D%0A++++cast%28percentile%28time+%2F+1000%2C+90%29+%25+60+as+integer%29%0D%0A++%29+as+p90%2C%0D%0A++printf%28%0D%0A++++%27%2502d%3A%2502d%27%2C%0D%0A++++cast%28percentile%28time+%2F+1000%2C+95%29+%2F+60+as+integer%29%2C%0D%0A++++cast%28percentile%28time+%2F+1000%2C+95%29+%25+60+as+integer%29%0D%0A++%29+as+p95%2C%0D%0A++printf%28%0D%0A++++%27%2502d%3A%2502d%27%2C%0D%0A++++cast%28percentile%28time+%2F+1000%2C+99%29+%2F+60+as+integer%29%2C%0D%0A++++cast%28percentile%28time+%2F+1000%2C+99%29+%25+60+as+integer%29%0D%0A++%29+as+p99%0D%0AFROM%0D%0A++builds%0D%0A++JOIN+jobs+on+jobs.id+%3D+builds.job_id%0D%0AWHERE%0D%0A++time+IS+NOT+NULL%0D%0A++and+time+%3E+0.0%0D%0A++AND+datetime%28timestamp+%2F+1000%2C+%27unixepoch%27%29+%3E%3D+date%28%3Adate_start%29%0D%0A++AND+datetime%28timestamp+%2F+1000%2C+%27unixepoch%27%29+%3C%3D+date%28%3Adate_end%29%0D%0A++AND+result+%3D+%27SUCCESS%27%0D%0A++AND+success_cache_hit+%3D+false%0D%0A++AND+name+%3D+%3Ajob_name%0D%0A++AND+repo+%3D+%3Arepo_name%0D%0Agroup+by%0D%0A++repo%0D%0Ahaving%0D%0A++count%28*%29+%3E+10%0D%0Aorder+by%0D%0A++avg%28time%29+desc%2C%0D%0A++count&date_start=2026-01-01&date_end=2026-02-01&job_name=quibble-for-mediawiki-core-vendor-postgres-php83&repo_name=mediawiki%2Fcore | quibble-for-mediawiki-core-vendor-postgres-php83]] |139 |09:42| 19:38| 16:29| 01:30| 16:35| 17:26| 18:29| 18:48| 19:35
|[[ https://releng-data.wmcloud.org/jobs?sql=SELECT%0D%0A++%22%3Ca+href%3D%2F-%2Fdashboards%2Fjenkins%3Frepo_name%3D%22+%7C%7C+repo+%7C%7C+%22%26date_start%3D%22+%7C%7C+%3Adate_start+%7C%7C+%22%3E%22+%7C%7C+repo+%7C%7C+%22%3C%2Fa%3E%22+as+%27Repo+name%27%2C%0D%0A++count%28*%29+as+count%2C%0D%0A++printf%28%0D%0A++++%27%2502d%3A%2502d%27%2C%0D%0A++++cast%28min%28time+%2F+1000%29+%2F+60+as+integer%29%2C%0D%0A++++cast%28min%28time+%2F+1000%29+%25+60+as+integer%29%0D%0A++%29+as+min%2C%0D%0A++printf%28%0D%0A++++%27%2502d%3A%2502d%27%2C%0D%0A++++cast%28max%28time+%2F+1000%29+%2F+60+as+integer%29%2C%0D%0A++++cast%28max%28time+%2F+1000%29+%25+60+as+integer%29%0D%0A++%29+as+max%2C%0D%0A++printf%28%0D%0A++++%27%2502d%3A%2502d%27%2C%0D%0A++++cast%28avg%28time+%2F+1000%29+%2F+60+as+integer%29%2C%0D%0A++++cast%28avg%28time+%2F+1000%29+%25+60+as+integer%29%0D%0A++%29+as+average%2C%0D%0A++printf%28%0D%0A++++%27%2502d%3A%2502d%27%2C%0D%0A++++cast%28%0D%0A++++++%28%0D%0A++++++++SQRT%28%0D%0A++++++++++AVG%28time+%2F+1000+*+time+%2F+1000%29+-+AVG%28time+%2F+1000%29+*+AVG%28time+%2F+1000%29%0D%0A++++++++%29%0D%0A++++++%29+%2F+60+as+integer%0D%0A++++%29%2C%0D%0A++++cast%28%0D%0A++++++%28%0D%0A++++++++SQRT%28%0D%0A++++++++++AVG%28time+%2F+1000+*+time%29+-+AVG%28time+%2F+1000%29+*+AVG%28time+%2F+1000%29%0D%0A++++++++%29%0D%0A++++++%29+%25+60+as+integer%0D%0A++++%29%0D%0A++%29+as+stddev%2C%0D%0A++printf%28%0D%0A++++%27%2502d%3A%2502d%27%2C%0D%0A++++cast%28median%28time+%2F+1000%29+%2F+60+as+integer%29%2C%0D%0A++++cast%28median%28time+%2F+1000%29+%25+60+as+integer%29%0D%0A++%29+as+median%2C%0D%0A++printf%28%0D%0A++++%27%2502d%3A%2502d%27%2C%0D%0A++++cast%28percentile%28time+%2F+1000%2C+75%29+%2F+60+as+integer%29%2C%0D%0A++++cast%28percentile%28time+%2F+1000%2C+75%29+%25+60+as+integer%29%0D%0A++%29+as+p75%2C%0D%0A++printf%28%0D%0A++++%27%2502d%3A%2502d%27%2C%0D%0A++++cast%28percentile%28time+%2F+1000%2C+90%29+%2F+60+as+integer%29%2C%0D%0A++++cast%28percentile%28time+%2F+1000%2C+90%29+%25+60+as+integer%29%0D%0A++%29+as+p90%2C%0D%0A++printf%28%0D%0A++++%27%2502d%3A%2502d%27%2C%0D%0A++++cast%28percentile%28time+%2F+1000%2C+95%29+%2F+60+as+integer%29%2C%0D%0A++++cast%28percentile%28time+%2F+1000%2C+95%29+%25+60+as+integer%29%0D%0A++%29+as+p95%2C%0D%0A++printf%28%0D%0A++++%27%2502d%3A%2502d%27%2C%0D%0A++++cast%28percentile%28time+%2F+1000%2C+99%29+%2F+60+as+integer%29%2C%0D%0A++++cast%28percentile%28time+%2F+1000%2C+99%29+%25+60+as+integer%29%0D%0A++%29+as+p99%0D%0AFROM%0D%0A++builds%0D%0A++JOIN+jobs+on+jobs.id+%3D+builds.job_id%0D%0AWHERE%0D%0A++time+IS+NOT+NULL%0D%0A++and+time+%3E+0.0%0D%0A++AND+datetime%28timestamp+%2F+1000%2C+%27unixepoch%27%29+%3E%3D+date%28%3Adate_start%29%0D%0A++AND+datetime%28timestamp+%2F+1000%2C+%27unixepoch%27%29+%3C%3D+date%28%3Adate_end%29%0D%0A++AND+result+%3D+%27SUCCESS%27%0D%0A++AND+success_cache_hit+%3D+false%0D%0A++AND+name+%3D+%3Ajob_name%0D%0A++AND+repo+%3D+%3Arepo_name%0D%0Agroup+by%0D%0A++repo%0D%0Ahaving%0D%0A++count%28*%29+%3E+10%0D%0Aorder+by%0D%0A++avg%28time%29+desc%2C%0D%0A++count&date_start=2026-01-01&date_end=2026-02-01&job_name=quibble-with-gated-extensions-selenium-php83&repo_name=mediawiki%2Fcore | quibble-with-gated-extensions-selenium-php83 ]] |205 |17:58| 29:06 |22:55 |01:21| 22:58 |23:58| 25:25 |26:03 |27:40
|[[ https://releng-data.wmcloud.org/jobs?sql=SELECT%0D%0A++%22%3Ca+href%3D%2F-%2Fdashboards%2Fjenkins%3Frepo_name%3D%22+%7C%7C+repo+%7C%7C+%22%26date_start%3D%22+%7C%7C+%3Adate_start+%7C%7C+%22%3E%22+%7C%7C+repo+%7C%7C+%22%3C%2Fa%3E%22+as+%27Repo+name%27%2C%0D%0A++count%28*%29+as+count%2C%0D%0A++printf%28%0D%0A++++%27%2502d%3A%2502d%27%2C%0D%0A++++cast%28min%28time+%2F+1000%29+%2F+60+as+integer%29%2C%0D%0A++++cast%28min%28time+%2F+1000%29+%25+60+as+integer%29%0D%0A++%29+as+min%2C%0D%0A++printf%28%0D%0A++++%27%2502d%3A%2502d%27%2C%0D%0A++++cast%28max%28time+%2F+1000%29+%2F+60+as+integer%29%2C%0D%0A++++cast%28max%28time+%2F+1000%29+%25+60+as+integer%29%0D%0A++%29+as+max%2C%0D%0A++printf%28%0D%0A++++%27%2502d%3A%2502d%27%2C%0D%0A++++cast%28avg%28time+%2F+1000%29+%2F+60+as+integer%29%2C%0D%0A++++cast%28avg%28time+%2F+1000%29+%25+60+as+integer%29%0D%0A++%29+as+average%2C%0D%0A++printf%28%0D%0A++++%27%2502d%3A%2502d%27%2C%0D%0A++++cast%28%0D%0A++++++%28%0D%0A++++++++SQRT%28%0D%0A++++++++++AVG%28time+%2F+1000+*+time+%2F+1000%29+-+AVG%28time+%2F+1000%29+*+AVG%28time+%2F+1000%29%0D%0A++++++++%29%0D%0A++++++%29+%2F+60+as+integer%0D%0A++++%29%2C%0D%0A++++cast%28%0D%0A++++++%28%0D%0A++++++++SQRT%28%0D%0A++++++++++AVG%28time+%2F+1000+*+time%29+-+AVG%28time+%2F+1000%29+*+AVG%28time+%2F+1000%29%0D%0A++++++++%29%0D%0A++++++%29+%25+60+as+integer%0D%0A++++%29%0D%0A++%29+as+stddev%2C%0D%0A++printf%28%0D%0A++++%27%2502d%3A%2502d%27%2C%0D%0A++++cast%28median%28time+%2F+1000%29+%2F+60+as+integer%29%2C%0D%0A++++cast%28median%28time+%2F+1000%29+%25+60+as+integer%29%0D%0A++%29+as+median%2C%0D%0A++printf%28%0D%0A++++%27%2502d%3A%2502d%27%2C%0D%0A++++cast%28percentile%28time+%2F+1000%2C+75%29+%2F+60+as+integer%29%2C%0D%0A++++cast%28percentile%28time+%2F+1000%2C+75%29+%25+60+as+integer%29%0D%0A++%29+as+p75%2C%0D%0A++printf%28%0D%0A++++%27%2502d%3A%2502d%27%2C%0D%0A++++cast%28percentile%28time+%2F+1000%2C+90%29+%2F+60+as+integer%29%2C%0D%0A++++cast%28percentile%28time+%2F+1000%2C+90%29+%25+60+as+integer%29%0D%0A++%29+as+p90%2C%0D%0A++printf%28%0D%0A++++%27%2502d%3A%2502d%27%2C%0D%0A++++cast%28percentile%28time+%2F+1000%2C+95%29+%2F+60+as+integer%29%2C%0D%0A++++cast%28percentile%28time+%2F+1000%2C+95%29+%25+60+as+integer%29%0D%0A++%29+as+p95%2C%0D%0A++printf%28%0D%0A++++%27%2502d%3A%2502d%27%2C%0D%0A++++cast%28percentile%28time+%2F+1000%2C+99%29+%2F+60+as+integer%29%2C%0D%0A++++cast%28percentile%28time+%2F+1000%2C+99%29+%25+60+as+integer%29%0D%0A++%29+as+p99%0D%0AFROM%0D%0A++builds%0D%0A++JOIN+jobs+on+jobs.id+%3D+builds.job_id%0D%0AWHERE%0D%0A++time+IS+NOT+NULL%0D%0A++and+time+%3E+0.0%0D%0A++AND+datetime%28timestamp+%2F+1000%2C+%27unixepoch%27%29+%3E%3D+date%28%3Adate_start%29%0D%0A++AND+datetime%28timestamp+%2F+1000%2C+%27unixepoch%27%29+%3C%3D+date%28%3Adate_end%29%0D%0A++AND+result+%3D+%27SUCCESS%27%0D%0A++AND+success_cache_hit+%3D+false%0D%0A++AND+name+%3D+%3Ajob_name%0D%0A++AND+repo+%3D+%3Arepo_name%0D%0Agroup+by%0D%0A++repo%0D%0Ahaving%0D%0A++count%28*%29+%3E+10%0D%0Aorder+by%0D%0A++avg%28time%29+desc%2C%0D%0A++count&date_start=2026-01-01&date_end=2026-02-01&job_name=quibble-for-mediawiki-core-vendor-mysql-php82&repo_name=mediawiki%2Fcore | quibble-for-mediawiki-core-vendor-mysql-php82 ]]| 145 |09:10| 17:42| 13:42| 01:49 |14:02| 15:00 |15:38 |15:54 |16:37
|quibble-for-mediawiki-core-vendor-mysql-php83| 270| 08:00| 14:50| 11:27| 01:44 | 11:36 | 12:29 | 13:14 | 13:34| 14:19
|quibble-for-mediawiki-core-vendor-mysql-php84|94 | 08:31 | 15:47| 12:03| 01:31| 12:07| 13:01 | 13:35 | 13:49 | 14:22
|quibble-for-mediawiki-core-vendor-mysql-php85|13| 07:27| 12:33| 10:01 | 01:17 | 10:01| 10:53| 11:09| 11:44 | 12:24
Developer gate pipeline feedback time (using [[ https://releng-data.wmcloud.org/jobs?sql=SELECT+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++%0D%0A++++month%2C+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++%0D%0A++++printf%28++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++%0D%0A++++++%27%2502d%3A%2502d%27%2C++++++++++++++++++++++++++++++++++++++++++%0D%0A++++++cast%28max%28job_median%29+%2F+60+as+integer%29%2C%0D%0A++++++cast%28max%28job_median%29+%25+60+as+integer%29++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++%0D%0A++++%29+as+%22Gate+pipeline+median+feedback+time%22%2C%0D%0A++++name+as+%22Slowest+job%22++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++%0D%0A++FROM+%28++++++++++++++++++++++++++++++++++++++++++++++++++++%0D%0A++++SELECT%0D%0A++++++strftime%28%27%25Y-%25m%27%2C+datetime%28timestamp+%2F+1000%2C+%27unixepoch%27%29%29+as+month%2C+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++%0D%0A++++++name%2C%0D%0A++++++median%28time+%2F+1000%29+as+job_median++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++%0D%0A++++FROM++++++++++++++++++++++++++++++++++++++++++++++++++++%0D%0A++++++builds%0D%0A++++++JOIN+jobs+ON+jobs.id+%3D+builds.job_id%0D%0A++++WHERE++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++%0D%0A++++++time+IS+NOT+NULL%0D%0A++++++AND+time+%3E+0.0+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++%0D%0A++++++AND+result+%3D+%27SUCCESS%27++++++++++++++++++++++++++++++++%0D%0A++++++AND+success_cache_hit+%3D+false%0D%0A++++++AND+timestamp+%3E%3D+1767225600000%0D%0A++++++AND+timestamp+%3C+1798761600000%0D%0A++++++AND+name+IN+%28%0D%0A++++++++%27quibble-with-gated-extensions-vendor-mysql-php83%27%2C++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++%0D%0A++++++++%27quibble-with-gated-extensions-selenium-php83%27%2C%0D%0A++++++++%27quibble-with-Wikibase-extensions-browser-tests-only-vendor-php83%27%2C++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++%0D%0A++++++++%27quibble-with-GrowthExperiments-extensions-browser-tests-only-vendor-php83%27%2C+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++%0D%0A++++++++%27quibble-for-mediawiki-core-vendor-sqlite-php83%27%2C%0D%0A++++++++%27quibble-for-mediawiki-core-vendor-postgres-php83%27%2C++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++%0D%0A++++++++%27quibble-for-mediawiki-core-vendor-mysql-php82%27%2C+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++%0D%0A++++++++%27quibble-for-mediawiki-core-vendor-mysql-php83%27%2C%0D%0A++++++++%27quibble-for-mediawiki-core-vendor-mysql-php84%27++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++%0D%0A++++++%29++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++%0D%0A++++GROUP+BY%0D%0A++++++month%2C+name++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++%0D%0A++%29+++++++++++++++++++++++++++++++++++++++++++++++++++++++++%0D%0A++GROUP+BY%0D%0A++++month%0D%0A++ORDER+BY%0D%0A++++month | this query ]]):
|month |Gate pipeline median feedback time| Slowest job
|2026-01 |22:58 |quibble-with-gated-extensions-selenium-php83
|2026-02 |24:11 |quibble-with-gated-extensions-selenium-php83
|2026-03 |17:22 |quibble-for-mediawiki-core-vendor-postgres-php83
|2026-04 |15:51 |quibble-for-mediawiki-core-vendor-postgres-php83
Since we are doing a lot of work to make sure test jobs are fast, we should also keep an eye on the gates pre merge job. Let's see if there's any easy wins we can implement/fix.
When we end the work for speeding up the jobs we should also report our wins on the gated jobs.