Page MenuHomePhabricator

integration-agent-docker-1035 free disk space flapping, causing Gerrit patches to not merge
Closed, DuplicatePublic

Description

Been flapping for a while, -releng:

19:10:50 <wmf-insecte> maintenance-disconnect-full-disks build 503787 integration-agent-docker-1035 (/: 31%, /srv: 12%, /var/lib/docker: 100%): RECOVERY disk space OK
19:15:47 <wmf-insecte> maintenance-disconnect-full-disks build 503788 integration-agent-docker-1033 (/: 30%, /srv: 59%, /var/lib/docker: 99%): OFFLINE due to disk space
19:18:33 <jinxer-wm> (Queue (Jenkins jobs + Zuul functions) alert) resolved: <no value>   - https://alerts.wikimedia.org/?q=alertname%3DQueue+%28Jenkins+jobs+%2B+Zuul+functions%29+alert
19:18:48 <jinxer-wm> (Queue (Jenkins jobs + Zuul functions) alert) firing: <no value>   - https://alerts.wikimedia.org/?q=alertname%3DQueue+%28Jenkins+jobs+%2B+Zuul+functions%29+alert
19:20:55 <wmf-insecte> maintenance-disconnect-full-disks build 503789 integration-agent-docker-1033 (/: 30%, /srv: 8%, /var/lib/docker: 94%): RECOVERY disk space OK
19:23:00 ⇐ ollie_wmde quit (~ollie@cpc141210-aztw34-2-0-cust203.18-1.cable.virginm.net) Ping timeout: 240 seconds
19:23:48 <jinxer-wm> (Queue (Jenkins jobs + Zuul functions) alert) resolved: <no value>   - https://alerts.wikimedia.org/?q=alertname%3DQueue+%28Jenkins+jobs+%2B+Zuul+functions%29+alert
19:25:40 <wmf-insecte> maintenance-disconnect-full-disks build 503790 integration-agent-docker-1035 (/: 31%, /srv: 17%, /var/lib/docker: 100%): OFFLINE due to disk space
19:26:23 ⇐ Daimona quit (~Daimona@wikipedia/Daimona-Eaytoy) Ping timeout: 264 seconds
19:28:48 <jinxer-wm> (Queue (Jenkins jobs + Zuul functions) alert) firing: <no value>   - https://alerts.wikimedia.org/?q=alertname%3DQueue+%28Jenkins+jobs+%2B+Zuul+functions%29+alert
19:30:37 <wmf-insecte> maintenance-disconnect-full-disks build 503791 integration-agent-docker-1035 (/: 31%, /srv: 12%, /var/lib/docker: 99%): RECOVERY disk space OK
19:33:48 <jinxer-wm> (Queue (Jenkins jobs + Zuul functions) alert) resolved: <no value>   - https://alerts.wikimedia.org/?q=alertname%3DQueue+%28Jenkins+jobs+%2B+Zuul+functions%29+alert
19:40:37 <wmf-insecte> maintenance-disconnect-full-disks build 503793 integration-agent-docker-1035 (/: 31%, /srv: 19%, /var/lib/docker: 100%): OFFLINE due to disk space
19:44:21 <Jdlrobson> Hey there, several Gerrit patches are not merging due to a "No space left on device" CI issue. I assume this relates to the above? Is this being looked in to?
19:45:34 <wmf-insecte> maintenance-disconnect-full-disks build 503794 integration-agent-docker-1035 (/: 31%, /srv: 12%, /var/lib/docker: 99%): RECOVERY disk space OK

Event Timeline

Jdlrobson triaged this task as Unbreak Now! priority.Jun 27 2023, 8:58 PM
Jdlrobson subscribed.

This is blocking several patches and thus productivity in the web team today. Hope you don't mind me marking as UBN.

Jdlrobson lowered the priority of this task from Unbreak Now! to Needs Triage.Jun 27 2023, 10:01 PM
Jdlrobson added a subscriber: hashar.

The above error seems to be fixed now (possibly by T340092 ? ) There was an "Installation failed, reverting ./composer.json and ./composer.lock to their original content." Maybe this has been resolved? I saw some activity from @hashar in the releng IRC channel.