Page MenuHomePhabricator
Create Task
Maniphest T438232

Request to increase the size of /srv on production deployment servers
Closed, ResolvedPublic
Actions

Description

Human deployers and automated processes have hit the problem described in T438148 several times over the last couple of weeks. We've reached the point where /srv just needs to be bigger on deploy1003 and friends. It's currently 277G. I suggest increasing by >= 30GB.

Related Objects
Search...

Event Timeline

dancy created this task.Wed, Sep 16, 6:58 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptWed, Sep 16, 6:58 PM
thcipriani edited projects, added: Release-Engineering-Team (Radar); removed: Release-Engineering-Team.Wed, Sep 16, 8:27 PM
thcipriani added a parent task: T438148: Image building is running out of space in `/srv`.
Jdforrester-WMF subscribed.Wed, Sep 16, 8:28 PM
MLechvien-WMF moved this task from Inbox to Needs Info / Blocked on the ServiceOps board.Thu, Sep 17, 11:03 AM
thcipriani mentioned this in T438148: Image building is running out of space in `/srv`.Thu, Sep 17, 4:24 PM
MLechvien-WMF subscribed.Mon, Sep 21, 6:13 PM

@dancy Does this need Serviceops support, or do you have the means to control this?

dancy added a comment.Mon, Sep 21, 6:16 PM
In T438232#12347005, @MLechvien-WMF wrote:

@dancy Does this need Serviceops support, or do you have the means to control this?

I will need support. I can't do it myself.

MLechvien-WMF triaged this task as Medium priority.Mon, Sep 21, 6:21 PM
MLechvien-WMF edited projects, added: Serviceops-easywins, ServiceOps (Next quarter); removed: ServiceOps.

Ack. Setting to Medium as I understand there's a workaround mentioned in the other task

Clement_Goubert subscribed.Tue, Sep 29, 10:14 AM

We can allocate about 80GB more, but that's it, the vg would be full afterwards. I will allocate 50GB as a first step (so we keep some margin on the vg) and we'll go from there.

Stashbot added a comment.Tue, Sep 29, 10:17 AM

Mentioned in SAL (#wikimedia-operations) [2026-09-29T10:17:28Z] <claime> deploy2003:~$ sudo lvextend -L +50GB --resizefs /dev/vg0/srv # T438232

Stashbot added a comment.Tue, Sep 29, 10:17 AM

Mentioned in SAL (#wikimedia-operations) [2026-09-29T10:17:53Z] <claime> deploy1003:~$ sudo lvextend -L +50GB --resizefs /dev/vg0/srv # T438232

Stashbot added a comment.Tue, Sep 29, 10:19 AM

Mentioned in SAL (#wikimedia-operations) [2026-09-29T10:19:19Z] <claime> deploy2002:~$ sudo lvextend -L +50GB --resizefs /dev/vg0/srv # T438232

Clement_Goubert closed this task as Resolved.Tue, Sep 29, 10:19 AM
Clement_Goubert claimed this task.
cgoubert@deploy1003:~$ sudo lvextend -L +50GB --resizefs /dev/vg0/srv
  Size of logical volume vg0/srv changed from <281.92 GiB (72171 extents) to <331.92 GiB (84971 extents).
  Logical volume vg0/srv successfully resized.
resize2fs 1.47.0 (5-Feb-2023)
Filesystem at /dev/mapper/vg0-srv is mounted on /srv; on-line resizing required
old_desc_blocks = 36, new_desc_blocks = 42
The filesystem on /dev/mapper/vg0-srv is now 87010304 (4k) blocks long.
cgoubert@deploy1003:~$ df -h /srv
Filesystem           Size  Used Avail Use% Mounted on
/dev/mapper/vg0-srv  326G  182G  128G  59% /srv

Done on all deploy servers.

dancy added a comment.Tue, Sep 29, 2:59 PM

Thanks @Clement_Goubert! Combined with the daily scap clean-images, I think 50GB will be plenty.

Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits