Page MenuHomePhabricator
Create Task
Maniphest T397017

Turn down mwmaint production servers
Closed, ResolvedPublic
Actions

Description

  • Copy homedirs to /srv/homedirs/mwmaint{1002,2002} on deploy1003
  • Communicate upcoming turn down to wikitech-l
  • Restrict access to ops group members (soft turndown)
  • Decommission mwmaint servers

Timeline TBD

Details

Related Changes in Gerrit:
Show related patches Customize query in gerrit

Related Objects
Search...

Event Timeline

Clement_Goubert created this task.Jun 16 2025, 10:57 AM
Clement_Goubert triaged this task as Medium priority.
gerritbot added a comment.Jun 25 2025, 10:27 AM

Change #1163731 had a related patch set uploaded (by Clément Goubert; author: Clément Goubert):

[operations/puppet@production] P::mediawiki::maintenance: rsync to deployment server

https://gerrit.wikimedia.org/r/1163731

gerritbot added a project: Patch-For-Review.Jun 25 2025, 10:27 AM
Clement_Goubert claimed this task.Jun 25 2025, 10:27 AM
Clement_Goubert updated the task description. (Show Details)
gerritbot added a comment.Jun 25 2025, 10:45 AM

Change #1163731 merged by Clément Goubert:

[operations/puppet@production] P::mediawiki::maintenance: rsync to deployment server

https://gerrit.wikimedia.org/r/1163731

Maintenance_bot removed a project: Patch-For-Review.Jun 25 2025, 11:31 AM
Clement_Goubert updated the task description. (Show Details)Jun 25 2025, 2:10 PM
Clement_Goubert updated the task description. (Show Details)Jun 25 2025, 2:16 PM
Urbanecm_WMF subscribed.Jun 25 2025, 2:53 PM

hey, before we plug the switch, would it be possible to adjust the default mysql prompt on the deployment machine? I like mwmaint tells me where am I connected, while deploy* doesn't do that.

gerritbot added a comment.Jun 26 2025, 9:49 AM

Change #1164139 had a related patch set uploaded (by Clément Goubert; author: Clément Goubert):

[operations/puppet@production] O:kubernetes::deployment_server: mariadb client

https://gerrit.wikimedia.org/r/1164139

gerritbot added a project: Patch-For-Review.Jun 26 2025, 9:49 AM
Clement_Goubert changed the status of subtask T397847: Should deployment servers include mariadb::maintenance profile from Open to In Progress.Jun 26 2025, 9:52 AM
gerritbot added a comment.Jun 26 2025, 9:58 AM

Change #1164139 merged by Clément Goubert:

[operations/puppet@production] O:kubernetes::deployment_server: mariadb client

https://gerrit.wikimedia.org/r/1164139

Maintenance_bot removed a project: Patch-For-Review.Jun 26 2025, 10:31 AM
Clement_Goubert closed subtask T397847: Should deployment servers include mariadb::maintenance profile as Resolved.Jun 26 2025, 10:50 AM
In T397017#10947620, @Urbanecm_WMF wrote:

hey, before we plug the switch, would it be possible to adjust the default mysql prompt on the deployment machine? I like mwmaint tells me where am I connected, while deploy* doesn't do that.

Did adding profile::mariadb::client to deployment servers fix your prompt issue?

Urbanecm_WMF added a comment.Jun 26 2025, 10:52 AM
In T397017#10949871, @Clement_Goubert wrote:
In T397017#10947620, @Urbanecm_WMF wrote:

hey, before we plug the switch, would it be possible to adjust the default mysql prompt on the deployment machine? I like mwmaint tells me where am I connected, while deploy* doesn't do that.

Did adding profile::mariadb::client to deployment servers fix your prompt issue?

Yes, thank you!

gerritbot added a comment.Jul 8 2025, 11:27 AM

Change #1167195 had a related patch set uploaded (by Clément Goubert; author: Clément Goubert):

[operations/puppet@production] check_user: Use deploy instead of mwmaint

https://gerrit.wikimedia.org/r/1167195

gerritbot added a project: Patch-For-Review.Jul 8 2025, 11:27 AM

Change #1167196 had a related patch set uploaded (by Clément Goubert; author: Clément Goubert):

[operations/puppet@production] mwaint: Remove from scap

https://gerrit.wikimedia.org/r/1167196

gerritbot added a comment.Jul 8 2025, 11:27 AM

Change #1167197 had a related patch set uploaded (by Clément Goubert; author: Clément Goubert):

[operations/puppet@production] mwmaint: deprecate mwmaint servers

https://gerrit.wikimedia.org/r/1167197

gerritbot added a comment.Jul 8 2025, 11:35 AM

Change #1167195 abandoned by Clément Goubert:

[operations/puppet@production] check_user: Use deploy instead of mwmaint

Reason:

Script is broken and will likely be removed

https://gerrit.wikimedia.org/r/1167195

gerritbot added a comment.Jul 9 2025, 10:05 AM

Change #1167196 merged by Clément Goubert:

[operations/puppet@production] mwaint: Remove from scap

https://gerrit.wikimedia.org/r/1167196

Stashbot added a comment.Jul 9 2025, 10:14 AM

Mentioned in SAL (#wikimedia-operations) [2025-07-09T10:14:25Z] <claime> Cutting off access to mwmaint servers - T397017

gerritbot added a comment.Jul 9 2025, 10:16 AM

Change #1167197 merged by Clément Goubert:

[operations/puppet@production] mwmaint: deprecate mwmaint servers

https://gerrit.wikimedia.org/r/1167197

Maintenance_bot removed a project: Patch-For-Review.Jul 9 2025, 10:32 AM
Clement_Goubert updated the task description. (Show Details)Jul 9 2025, 3:11 PM
gerritbot added a comment.Jul 23 2025, 8:20 AM

Change #1171983 had a related patch set uploaded (by Clément Goubert; author: Clément Goubert):

[operations/dns@master] wmnet: Remove maintenance.eqiad.wmnet record

https://gerrit.wikimedia.org/r/1171983

gerritbot added a project: Patch-For-Review.Jul 23 2025, 8:20 AM
gerritbot added a comment.Jul 24 2025, 9:56 AM

Change #1171983 merged by Clément Goubert:

[operations/dns@master] wmnet: Remove maintenance.eqiad.wmnet record

https://gerrit.wikimedia.org/r/1171983

Maintenance_bot removed a project: Patch-For-Review.Jul 24 2025, 10:32 AM
Clement_Goubert added a subtask: T400442: decommission mwmaint1002.eqiad.wmnet.Jul 25 2025, 8:07 AM
hashar subscribed.Aug 7 2025, 8:40 AM

As part of deploying the train, deploy1003 had a disk space alarm for /srv:

CRITICAL: DISK CRITICAL - free space: /srv 8607 MB (3% inode=69%)
df -h /srv
Filesystem           Size  Used Avail Use% Mounted on
/dev/mapper/vg0-srv  277G  255G  8.3G  97% /srv
du -d2 -m -x /srv 2>/dev/null |sort -rn|head -n10
101093	/srv
37541	/srv/mediawiki-staging
36861	/srv/deployment
26615	/srv/homedirs
13592	/srv/homedirs/mwmaint1002
13024	/srv/homedirs/mwmaint2002
10120	/srv/mediawiki-staging/php-1.45.0-wmf.12
10087	/srv/mediawiki-staging/php-1.45.0-wmf.11
10021	/srv/mediawiki-staging/php-1.45.0-wmf.13
7917	/srv/deployment/analytics

The homedirs consume 26G or 10% of the available disk space. That is not that much in the grand scheme of things :)

Clement_Goubert added a comment.Aug 7 2025, 11:02 AM

I plan on removing the homedirs soon-ish, but I will add 30GB to the vg to negate the issue in the meantime.

Clement_Goubert closed this task as Resolved.Aug 7 2025, 11:03 AM
Clement_Goubert updated the task description. (Show Details)
Clement_Goubert added a comment.Aug 7 2025, 11:20 AM
root@deploy1003:~# lvextend -L+30G /dev/vg0/srv
  Size of logical volume vg0/srv changed from <281.92 GiB (72171 extents) to <311.92 GiB (79851 extents).
  Logical volume vg0/srv successfully resized.
root@deploy1003:~# resize2fs /dev/vg0/srv
resize2fs 1.46.2 (28-Feb-2021)
Filesystem at /dev/vg0/srv is mounted on /srv; on-line resizing required
old_desc_blocks = 36, new_desc_blocks = 39
The filesystem on /dev/vg0/srv is now 81767424 (4k) blocks long.

root@deploy1003:~# df -h /srv
Filesystem           Size  Used Avail Use% Mounted on
/dev/mapper/vg0-srv  306G  255G   37G  88% /srv
hashar added a comment.Aug 7 2025, 2:21 PM

I am happy to see there was plenty of disk space on the volume group \o/ Thank you!

hashar mentioned this in T401647: deploy1003 running out of disk space.Aug 19 2025, 8:03 AM
VRiley-WMF closed subtask T400442: decommission mwmaint1002.eqiad.wmnet as Resolved.Sep 9 2025, 10:56 AM
gerritbot added a comment.Sep 22 2025, 4:53 PM

Change #1190309 had a related patch set uploaded (by Jasmine; author: Jasmine):

[operations/dns@master] wmnet: remove mwmaint discovery aliases since turning down production servers [0]

https://gerrit.wikimedia.org/r/1190309

gerritbot added a project: Patch-For-Review.Sep 22 2025, 4:53 PM
gerritbot added a comment.Oct 3 2025, 5:00 PM

Change #1190309 merged by Jasmine:

[operations/dns@master] wmnet: remove mwmaint discovery aliases since turning down production servers [0]

https://gerrit.wikimedia.org/r/1190309

Maintenance_bot removed a project: Patch-For-Review.Oct 3 2025, 5:31 PM
hashar unsubscribed.Oct 3 2025, 7:43 PM
gerritbot added a comment.Dec 17 2025, 5:32 PM

Change #1219189 had a related patch set uploaded (by Krinkle; author: Krinkle):

[operations/puppet@production] scap: Remove unused mwmaint config, obsolete wikitech/php7 comments

https://gerrit.wikimedia.org/r/1219189

gerritbot added a project: Patch-For-Review.Dec 17 2025, 5:32 PM
gerritbot added a comment.Dec 19 2025, 1:27 AM

Change #1219189 merged by RLazarus:

[operations/puppet@production] scap: Remove unused mwmaint config, obsolete wikitech/php7 comments

https://gerrit.wikimedia.org/r/1219189

Maintenance_bot removed a project: Patch-For-Review.Dec 19 2025, 1:30 AM
gerritbot added a comment.Jan 19 2026, 11:15 AM

Change #1228454 had a related patch set uploaded (by Muehlenhoff; author: Muehlenhoff):

[operations/puppet@production] Remove ferm/rsync/tcpircbot settings

https://gerrit.wikimedia.org/r/1228454

gerritbot added a project: Patch-For-Review.Jan 19 2026, 11:15 AM
gerritbot added a comment.Jan 19 2026, 11:37 AM

Change #1228457 had a related patch set uploaded (by Muehlenhoff; author: Muehlenhoff):

[operations/puppet@production] Fix the description of what restricted does after mwmaint* decom

https://gerrit.wikimedia.org/r/1228457

gerritbot added a comment.Jan 19 2026, 12:08 PM

Change #1228457 merged by Muehlenhoff:

[operations/puppet@production] Fix the description of what restricted does after mwmaint* decom

https://gerrit.wikimedia.org/r/1228457

gerritbot added a comment.Jan 19 2026, 1:18 PM

Change #1228454 merged by Muehlenhoff:

[operations/puppet@production] Remove ferm/rsync/tcpircbot settings

https://gerrit.wikimedia.org/r/1228454

Maintenance_bot removed a project: Patch-For-Review.Jan 19 2026, 1:33 PM
gerritbot added a comment.Jan 19 2026, 2:16 PM

Change #1228507 had a related patch set uploaded (by Marostegui; author: Marostegui):

[operations/puppet@production] production-m5.sql.erb: Remove old mwmaint IP

https://gerrit.wikimedia.org/r/1228507

gerritbot added a project: Patch-For-Review.Jan 19 2026, 2:16 PM
gerritbot added a comment.Jan 20 2026, 6:32 AM

Change #1228507 merged by Marostegui:

[operations/puppet@production] production-m5.sql.erb: Remove old mwmaint IP

https://gerrit.wikimedia.org/r/1228507

Marostegui created subtask T415008: Remove old mwmaint grants from production.Jan 20 2026, 6:35 AM
Marostegui closed subtask T415008: Remove old mwmaint grants from production as Resolved.
Maintenance_bot removed a project: Patch-For-Review.Jan 20 2026, 7:31 AM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits