Page MenuHomePhabricator

Random GitLab CI failures with ContainersNotReady
Open, Needs TriagePublic

Description

There are currently some random pipelines failing in the mwbot-rs repository.

Note that tests ending with MW API HTTP 412 errors (Error: Http(reqwest::Error { kind: Status(412), url: "xxxx" })) are known mwbot-rs issues and not related to GitLab.

A few runs listed in the above link are GitLab CI errors:

  1. https://gitlab.wikimedia.org/repos/mwbot-rs/mwbot/-/jobs/402560
  2. https://gitlab.wikimedia.org/repos/mwbot-rs/mwbot/-/jobs/402565
  3. https://gitlab.wikimedia.org/repos/mwbot-rs/mwbot/-/jobs/402566
  4. https://gitlab.wikimedia.org/repos/mwbot-rs/mwbot/-/jobs/402603
  5. https://gitlab.wikimedia.org/repos/mwbot-rs/mwbot/-/jobs/402635
Waiting for pod gitlab-runner/runner-9e2abdumz-project-860-concurrent-0-ha8p8w3p to be running, status is Pending
Waiting for pod gitlab-runner/runner-9e2abdumz-project-860-concurrent-0-ha8p8w3p to be running, status is Pending
	ContainersNotReady: "containers with unready status: [build helper istio-proxy]"
	ContainersNotReady: "containers with unready status: [build helper istio-proxy]"

I have no idea how to trigger these errors and they seemed to be just random.

Probable upstream bug: https://gitlab.com/gitlab-org/gitlab/-/issues/399823

Details

Request URL
https://gitlab.wikimedia.org/repos/mwbot-rs/mwbot/-/jobs/402603

Event Timeline

taavi changed the subtype of this task from "Production Error" to "Task".Nov 23 2024, 3:20 PM
taavi removed a project: Wikimedia-production-error.
Jelto subscribed.

All jobs listed above have been executed on the Digital Ocean Cloud Runners. We had similar issues in the past when some components in the Kubernetes cluster were not ready.

I'll add Release-Engineering-Team tag because they are maintaining the Cloud Runner infrastructure.

@XtexChooser you could try running your jobs on the WMCS non-Kubernetes runners, that should work if you don't use any special features from the cloud runners. Just use tags: [ wmcs ] in you job definition:

rust-coverage:
  tags:
    - wmcs