components-api failing to connect to internal api
Closed, ResolvedPublicBUG REPORT
Actions

Assigned To

None

Authored By

	DamianZaremba
	Jun 4 2026, 10:36 PM

Project Tags

Referenced Files

None

Subscribers

Description

Deployments:

20260604-221112-gh1x39kyle
20260604-221508-uz9kuzoba5
20260604-221902-c3j2g2emcg
20260604-222120-85mxn8qwb5
20260604-222612-b8x95li2oj

All failed due to an internal api issue, example

tools.cluebotng@tools-bastion-15:~$ toolforge components deployment show 20260604-222612-b8x95li2oj
Deployment ID: 20260604-222612-b8x95li2oj
Created: 20260604-222612
Status: failed
Long status: 
  Got exception: Some builds failed to start: backup-database(error:HTTPSConnectionPool(host='api.svc.tools.eqiad1.wikimedia.cloud', port=30003): Read timed out. (read timeout=20)) core(error:HTTPSConnectionPool(host='api.svc.tools.eqiad1.wikimedia.cloud', port=30003): Read timed out. (read timeout=20)) grafana-alloy(error:HTTPSConnectionPool(host='api.svc.tools.eqiad1.wikimedia.cloud', port=30003): Read timed out. (read timeout=20)) pushgateway(error:HTTPSConnectionPool(host='api.svc.tools.eqiad1.wikimedia.cloud', port=30003): Read timed out. (read timeout=20)) redis(error:HTTPSConnectionPool(host='api.svc.tools.eqiad1.wikimedia.cloud', port=30003): Read timed out. (read timeout=20)) report-interface(error:HTTPSConnectionPool(host='api.svc.tools.eqiad1.wikimedia.cloud', port=30003): Read timed out. (read timeout=20))

Builds:
  backup-database(failed): id:no-id-yet unexpected HTTPSConnectionPool(host='api.svc.tools.eqiad1.wikimedia.cloud', port=30003): Read timed out. (read timeout=20)
  bot(skipped): id:cluebotng-buildpacks-pipelinerun-xcspw Reusing existing build
  core(failed): id:no-id-yet unexpected HTTPSConnectionPool(host='api.svc.tools.eqiad1.wikimedia.cloud', port=30003): Read timed out. (read timeout=20)
  grafana-alloy(failed): id:no-id-yet unexpected HTTPSConnectionPool(host='api.svc.tools.eqiad1.wikimedia.cloud', port=30003): Read timed out. (read timeout=20)
  irc-relay(skipped): id:cluebotng-buildpacks-pipelinerun-q9tsb Reusing existing build
  prune-backups(skipped): id:no-build-needed Component re-uses build from backup-database
  pushgateway(failed): id:no-id-yet unexpected HTTPSConnectionPool(host='api.svc.tools.eqiad1.wikimedia.cloud', port=30003): Read timed out. (read timeout=20)
  redis(failed): id:no-id-yet unexpected HTTPSConnectionPool(host='api.svc.tools.eqiad1.wikimedia.cloud', port=30003): Read timed out. (read timeout=20)
  report-interface(failed): id:no-id-yet unexpected HTTPSConnectionPool(host='api.svc.tools.eqiad1.wikimedia.cloud', port=30003): Read timed out. (read timeout=20)
  report-review-import(skipped): id:no-build-needed Component re-uses build from report-interface
  report-send-whitelisted-to-review(skipped): id:no-build-needed Component re-uses build from report-interface

Related Objects

Mentioned In: T428449: [jobs-api] is not listing all container images for a tool
T428427: cluebotng-review tool stores 180G of backups

Event Timeline

DamianZaremba created this task.Jun 4 2026, 10:36 PM

Restricted Application added a project: cloud-services-team. · View Herald TranscriptJun 4 2026, 10:36 PM

Restricted Application added a subscriber: Aklapper. · View Herald Transcript

BLiviero-WMF added a project: tools-platform-team.Jun 4 2026, 11:02 PM

This is still happening, so doesn't appear to be a transient issue.

Unfortunately it makes deploying new release quite painful as the process needs to be done step-by-step-by-hand to work around this.

Also getting failures when interacting with the build api directly;

tools.cluebotng@tools-bastion-15:~$ toolforge build start --ref main -i core https://github.com/cluebotng/external-core.git
ReadTimeout: HTTPSConnectionPool(host='api.svc.tools.eqiad1.wikimedia.cloud', port=30003): Read timed out. (read timeout=20)
Please report this issue to the Toolforge admins if it persists: https://w.wiki/6Zuu

DamianZaremba mentioned this in T428427: cluebotng-review tool stores 180G of backups.Jun 8 2026, 10:51 AM

DamianZaremba mentioned this in T428449: [jobs-api] is not listing all container images for a tool.Jun 8 2026, 1:10 PM

I think this has been resolved @DamianZaremba . can you check on your end?

I can't replicate this at the moment on any tool, so it appears to have resolved it's self.

It would be good to understand why it happened, especially given the direct API usage was also affected.

Resolving for now. If it happens again we can dig deeper, as we would also have more data available from logs.

dcaro moved this task from Backlog to Done on the tools-platform-team board.Jul 8 2026, 12:32 PM

components-api failing to connect to internal apiClosed, ResolvedPublicBUG REPORTActions

Description

Related Objects

Event Timeline

components-api failing to connect to internal api
Closed, ResolvedPublicBUG REPORT
Actions