CI performance issues
Closed, DuplicatePublic
Actions

Description

https://integration.wikimedia.org/zuul/ is currently showing puppet CRs queued up to 18 minutes on the test-prio queue.
This performance issue is blocking the normal workflow of the SRE team.

Details

Related Gerrit Patches:

mediawiki/core : master	Make LocalisationCache a service
mediawiki/core : REL1_34	Make LocalisationCache a service
integration/config : master	Move puppet jobs to dedicated small node
mediawiki/core : master	Revert "Make LocalisationCache a service"

Related Objects

Mentioned In: T225730: Reduce runtime of MW shared gate Jenkins jobs to 5 min
T231183: L10n cache is completely broken
T231220: Figure out how to make LocalisationCache a service without hurting performance
Mentioned Here: T231198: Wikibase and WikibaseLexeme tests now take more than 1 hour

Event Timeline

Vgutierrez created this task.Aug 26 2019, 1:44 PM

Restricted Application added a subscriber: Aklapper. · View Herald TranscriptAug 26 2019, 1:44 PM

Paladox added a subscriber: Paladox.Aug 26 2019, 1:44 PM

For context, the actual time to run the tests for operations/puppet is under one minute for most patches.

Either Zuul or jenkins are broken, and this has been a constant pain in the last few weeks for everyone involved.

Triaging to UBN! because this is effectively an outage of the service.

Restricted Application added a subscriber: Liuxinyu970226. · View Herald TranscriptAug 26 2019, 1:46 PM

Daimona added a subscriber: Daimona.Aug 26 2019, 3:25 PM

Change 532399 had a related patch set uploaded (by Ladsgroup; owner: Ladsgroup):
[mediawiki/core@master] Revert "Make LocalisationCache a service"

https://gerrit.wikimedia.org/r/532399

gerritbot added a project: Patch-For-Review.Aug 26 2019, 4:30 PM

Krinkle added a project: Continuous-Integration-Infrastructure.Aug 26 2019, 5:31 PM

Simetrical mentioned this in T231220: Figure out how to make LocalisationCache a service without hurting performance.Aug 26 2019, 5:48 PM

Simetrical mentioned this in T231183: L10n cache is completely broken.Aug 26 2019, 5:51 PM

Change 532437 had a related patch set uploaded (by Thcipriani; owner: Thcipriani):
[integration/config@master] Move puppet jobs to dedicated small node

https://gerrit.wikimedia.org/r/532437

Change 532399 merged by jenkins-bot:
[mediawiki/core@master] Revert "Make LocalisationCache a service"

https://gerrit.wikimedia.org/r/532399

hashar closed this task as a duplicate of T231198: Wikibase and WikibaseLexeme tests now take more than 1 hour.Aug 27 2019, 7:22 AM

Restricted Application removed a subscriber: Liuxinyu970226. · View Herald TranscriptAug 27 2019, 7:22 AM

The root cause was a faulty patch merged in mediawiki/core on Friday. It roughly doubled the time it takes to run the tests so that eg Wikibase changes occupied execution slot for up to an hour. In turn that starved the very thin pool of executors we currently have which thus delayed execution of jobs for non MediaWiki repo.

Anyway, that has been fixed by reverting the faulty code.

Change 532679 had a related patch set uploaded (by simetrical; owner: simetrical):
[mediawiki/core@master] Make LocalisationCache a service

https://gerrit.wikimedia.org/r/532679

gerritbot added a project: Patch-For-Review.Aug 27 2019, 10:10 AM

Change 532437 merged by jenkins-bot:
[integration/config@master] Move puppet jobs to dedicated small node

https://gerrit.wikimedia.org/r/532437

Change 532679 merged by jenkins-bot:
[mediawiki/core@master] Make LocalisationCache a service

https://gerrit.wikimedia.org/r/532679

Change 541624 had a related patch set uploaded (by Jforrester; owner: simetrical):
[mediawiki/core@REL1_34] Make LocalisationCache a service

https://gerrit.wikimedia.org/r/541624

Change 541624 merged by jenkins-bot:
[mediawiki/core@REL1_34] Make LocalisationCache a service