My suggestion: Disable this job for a week. Compare the metrics between the two weeks to have better understanding of how much this job makes all other jobs slower. Like it is now, we cannot have a job that makes all other jobs slower.

Lets push this to later.

I think there are other tasks that is about how we can make Castor faster. With the graphs https://releng-data.wmcloud.org/-/dashboards/ci-by-repo-and-job we can see that the queue has increasing over time (month by month).

I think this wasn't the plan. Lets close it for now and come up with a new idea.

https://releng-data.wmcloud.org/-/dashboards/ci-by-repo-and-job

I checked what triggers the mwext-codehealth-master-non-voting:

I got some another example but this time it's not a single job. I'm looking at the wait time for https://integration.wikimedia.org/ci/job/quibble-with-gated-extensions-selenium-php83/36785/console

Hi @Jdforrester-WMF do you remember the reason we set the cache for npm differently in quibble and the node24-test job? Not sure if its a bug or a feature.

Let me know @Mhurd if this could be something for you!

I don't there's any bug on our side. The reason its 0 is that p75 yet isn't impacted.

So I checked against the crux API and the p75 is 0. But we see an increase in URLs with bad CLS:

Yes but I could only see the increase in time for that job, so I think something happened there.

Lets park this and merge if we feel its needed.

Let's skip this for now since its fast enough.

Closing this as resolved since we have one month of data where we are under 10 minutes.

I read this like jobs that have a median time of 0,1-0,2 is ok. Job that has an higher median and high total time is the ones we should look at.

And looking at top 10 for 28 of May:

Here's a list from the last days calculating which jobs spends most time in Castor saving data. To the left is the total time spent for that job.

Also this job is interesting:

Did a summary:

So the toplist of jobs running longer than 1 minute (number of times and job):

51 /srv/jenkins/workspace/mwext-codehealth-master-non-voting
 3 /srv/jenkins/workspace/quibble-with-Wikibase-extensions-browser-tests-only-vendor-php83
 1 /srv/jenkins/workspace/quibble-apitests-only-vendor-php83

Here are jobs taking longer than 1 minute:

Hi! I think this is breaking Quibble when I try to run the tests in CI for the job integration-quibble-fullrun-extensions-phpunit-php83

For me there's two things:

• I think it's a code smell that we have tests in Cite that do not run in CI when do changes for Cite. Instead the tests runs when you do a change in Citoid. That's why I think the Cite tests that needs Citoid should live in Citoid instead. It would make sense in the way that it runs when you push your change.
• There's the issue where we insert content three times and they collide (that you identified). I can make patch for that where we insert the content before the test starts. If that works, we can do a follow up and enable parallel tests again (I will make the CI run X amount of times to verify it fixes the flakiness). In the long run I think we should fix so Cypress also can use the replacement for MWBot in that use the Mediawiki API. That would make it easier to setup test data the way we want, instead of doing it in Cypress.

@pwangai I think this is ready to be closed!

Would it make sense to move the tests that needs Citoid to the Citoid extension instead of Cite?

Cite added the parallel extension and it made those tests faster. If you run Cypress, check that plugin.

We did two things:

• We moved out GrowthExperiments and Wikibase to two separate jobs
• We made wdio tests run as fast as possible (running in parallel, no video by default, making tests smarter)

This was done in another way: we moved out GrowthExperinent and Wikibase to it separate jobs.

Closing let me know if you have any questions about the work we did!

I merged the parallel setup, that is enough for now.