Draining ganeti1029.eqiad.wmnet of running VMs

Draining ganeti1029.eqiad.wmnet of running VMs

Draining ganeti1029.eqiad.wmnet of running VMs

Draining ganeti1030.eqiad.wmnet of running VMs

I would like to check in on this and see if there has been any further warnings?

@fnegri @VRiley-WMF did this need to be reopened.

idrac shows The system inlet temperature is greater than the upper warning threshold. Sun 02 Mar 2025 05:30:16

The temperature remains very close to the threshold, and the alert has been firing intermittently since my previous comment.

Understood, I'm currently investigating this

Is there a timeframe for us to take this server down?

I think this one is tricky to depool, there are some notes at https://wikitech.wikimedia.org/wiki/Dumps/Dump_servers#Maintenance/downtime

It's probably a good exercise to try and route all traffic to clouddumps1002 because I think no one has done it recently :)

Routing the traffic to the other host would also clarify if the high temperature is somehow related to the user load on this server.

In T383723#10637026, @fnegri wrote:

Finally, I don't have a clear picture of which clients are connecting to clouddumps hosts. Is it only Cloud VPS VMs using profile::wmcs::nfsclient, or there's more?

The stat* physical hosts also mount the NFS volumes:

$ ssh stat1009.eqiad.wmnet
$ mount | grep nfs
clouddumps1002.wikimedia.org:/ on /mnt/nfs/dumps-clouddumps1002.wikimedia.org type nfs4 (ro,relatime,vers=4.2,rsize=8192,wsize=8192,namlen=255,hard,proto=tcp,timeo=14,retrans=2,sec=sys,clientaddr=10.64.21.17,local_lock=none,addr=208.80.154.71)
clouddumps1001.wikimedia.org:/ on /mnt/nfs/dumps-clouddumps1001.wikimedia.org type nfs4 (ro,relatime,vers=4.2,rsize=8192,wsize=8192,namlen=255,hard,proto=tcp,timeo=14,retrans=2,sec=sys,clientaddr=10.64.21.17,local_lock=none,addr=208.80.154.142)

These are the "Production NFS" clients from https://wikitech.wikimedia.org/wiki/Portal:Data_Services/Admin/Dumps#Interfaces

I see there's an Nginx server listening as well.

In normal operation one of the 2 hosts serves NFS and the other serves https://dumps.wikimedia.org and the rsync mirrors. I think the nginx process always runs on both hosts and the dumps.wikimedia.org CNAME DNS record is used to control which is the active host at any given time.

clouddumps1001 is also mounted from an-launcher1002, and this path is used for running a few DAGs.

I also found this reference in PAWS: https://gitlab.wikimedia.org/repos/cloud/paws/-/blob/main/paws/production.yaml?ref_type=heads#L18-22

Without this, dumps becomes inaccessible and can hang the host

I checked that this is still in use this by logging into https://hub-paws.wmcloud.org

I did a quick check of which hostnames have this share mounted.

btullis@clouddumps1001:~$ for a in $(ss -tano state established sport nfs|awk '{print $4}'|cut -f1 -d":"|grep -v Address|sort|uniq); do host $a | grep -v alias | awk '{print $5}'; done
wdqs2009.codfw.wmnet.
wcqs2001.codfw.wmnet.
wdqs2010.codfw.wmnet.
stat1010.eqiad.wmnet.
wdqs1024.eqiad.wmnet.
an-launcher1002.eqiad.wmnet.
stat1009.eqiad.wmnet.
stat1011.eqiad.wmnet.
stat1008.eqiad.wmnet.
nat.cloudgw.eqiad1.wikimediacloud.org.
mail.toolsbeta.wmcloud.org.
instance-toolsbeta-mail-2.toolsbeta.wmcloud.org.
beta.math.wmflabs.org.
instance-math24.math.wmcloud.org.
mardi.math.wmflabs.org.
login.tools.wmflabs.org.
dev.toolforge.org.
instance-tools-bastion-12.tools.wmcloud.org.
instance-tools-checker-5.tools.wmcloud.org.
checker.tools.wmflabs.org.
instance-tools-bastion-13.tools.wmcloud.org.
bastion.toolforge.org.
instance-tools-mail-4.tools.wmcloud.org.
mail.tools.wmcloud.org.
instance-tools-sgebastion-10.tools.wmcloud.org.
login-buster.toolforge.org.
nat.cloudgw.codfw1dev.wikimediacloud.org.

PAWS doesn't show up on that list, so I presume that it's NAT'd by nat.cloudgw.eqiad1.wikimediacloud.org.

So from the Data-Platform-SRE side, I think that we would be OK with downtime. Processes trying to access this path will just hang, waiting for it to return.

But I think that the NFS clients in WMCS might have more of an issue, as @fnegri indicated.

Change #1131051 had a related patch set uploaded (by FNegri; author: FNegri):

[operations/puppet@production] Failover all dumps traffic to clouddumps1002

https://gerrit.wikimedia.org/r/1131051

Change #1131051 merged by FNegri:

[operations/puppet@production] Failover all dumps traffic to clouddumps1002

https://gerrit.wikimedia.org/r/1131051

@dcaro is concerned about active NFS mounts from pods, those might require a restart of the NFS server (unless Puppet is doing it automatically).

This is what Puppet did on NFS workers, the symlinks were updated but the service was not restarted:

2025-03-26T13:19:32.034169+00:00 tools-k8s-worker-nfs-43 puppet-agent[355401]: Applying configuration version '(673839ca92) jelto - aptrepo: upgrade gitlab-ce and
gitlab-runner to 17.8'
2025-03-26T13:19:38.036936+00:00 tools-k8s-worker-nfs-43 puppet-agent[355401]: (/Stage[main]/Profile::Wmcs::Nfsclient/File[/public/dumps/public]/target) target cha
nged '/mnt/nfs/dumps-clouddumps1001.wikimedia.org/' to '/mnt/nfs/dumps-clouddumps1002.wikimedia.org/'
2025-03-26T13:19:38.041939+00:00 tools-k8s-worker-nfs-43 puppet-agent[355401]: (/Stage[main]/Profile::Wmcs::Nfsclient/File[/public/dumps/incr]/target) target changed '/mnt/nfs/dumps-clouddumps1001.wikimedia.org/other/incr' to '/mnt/nfs/dumps-clouddumps1002.wikimedia.org/other/incr'
2025-03-26T13:19:38.046222+00:00 tools-k8s-worker-nfs-43 puppet-agent[355401]: (/Stage[main]/Profile::Wmcs::Nfsclient/File[/public/dumps/pagecounts-all-sites]/targ
et) target changed '/mnt/nfs/dumps-clouddumps1001.wikimedia.org/other/pagecounts-all-sites' to '/mnt/nfs/dumps-clouddumps1002.wikimedia.org/other/pagecounts-all-sites'
2025-03-26T13:19:38.049072+00:00 tools-k8s-worker-nfs-43 puppet-agent[355401]: (/Stage[main]/Profile::Wmcs::Nfsclient/File[/public/dumps/pagecounts-raw]/target) ta
rget changed '/mnt/nfs/dumps-clouddumps1001.wikimedia.org/other/pagecounts-raw' to '/mnt/nfs/dumps-clouddumps1002.wikimedia.org/other/pagecounts-raw'
2025-03-26T13:19:38.052357+00:00 tools-k8s-worker-nfs-43 puppet-agent[355401]: (/Stage[main]/Profile::Wmcs::Nfsclient/File[/public/dumps/pageviews]/target) target
changed '/mnt/nfs/dumps-clouddumps1001.wikimedia.org/other/pageviews' to '/mnt/nfs/dumps-clouddumps1002.wikimedia.org/other/pageviews'
2025-03-26T13:19:39.359024+00:00 tools-k8s-worker-nfs-43 puppet-agent[355401]: Applied catalog in 7.44 seconds

In T383723#10678670, @fnegri wrote:

@dcaro is concerned about active NFS mounts from pods, those might require a restart of the NFS server (unless Puppet is doing it automatically).

This is what Puppet did on NFS workers, the symlinks were updated but the service was not restarted:

2025-03-26T13:19:32.034169+00:00 tools-k8s-worker-nfs-43 puppet-agent[355401]: Applying configuration version '(673839ca92) jelto - aptrepo: upgrade gitlab-ce and
gitlab-runner to 17.8'
2025-03-26T13:19:38.036936+00:00 tools-k8s-worker-nfs-43 puppet-agent[355401]: (/Stage[main]/Profile::Wmcs::Nfsclient/File[/public/dumps/public]/target) target cha
nged '/mnt/nfs/dumps-clouddumps1001.wikimedia.org/' to '/mnt/nfs/dumps-clouddumps1002.wikimedia.org/'
2025-03-26T13:19:38.041939+00:00 tools-k8s-worker-nfs-43 puppet-agent[355401]: (/Stage[main]/Profile::Wmcs::Nfsclient/File[/public/dumps/incr]/target) target changed '/mnt/nfs/dumps-clouddumps1001.wikimedia.org/other/incr' to '/mnt/nfs/dumps-clouddumps1002.wikimedia.org/other/incr'
2025-03-26T13:19:38.046222+00:00 tools-k8s-worker-nfs-43 puppet-agent[355401]: (/Stage[main]/Profile::Wmcs::Nfsclient/File[/public/dumps/pagecounts-all-sites]/targ
et) target changed '/mnt/nfs/dumps-clouddumps1001.wikimedia.org/other/pagecounts-all-sites' to '/mnt/nfs/dumps-clouddumps1002.wikimedia.org/other/pagecounts-all-sites'
2025-03-26T13:19:38.049072+00:00 tools-k8s-worker-nfs-43 puppet-agent[355401]: (/Stage[main]/Profile::Wmcs::Nfsclient/File[/public/dumps/pagecounts-raw]/target) ta
rget changed '/mnt/nfs/dumps-clouddumps1001.wikimedia.org/other/pagecounts-raw' to '/mnt/nfs/dumps-clouddumps1002.wikimedia.org/other/pagecounts-raw'
2025-03-26T13:19:38.052357+00:00 tools-k8s-worker-nfs-43 puppet-agent[355401]: (/Stage[main]/Profile::Wmcs::Nfsclient/File[/public/dumps/pageviews]/target) target
changed '/mnt/nfs/dumps-clouddumps1001.wikimedia.org/other/pageviews' to '/mnt/nfs/dumps-clouddumps1002.wikimedia.org/other/pageviews'
2025-03-26T13:19:39.359024+00:00 tools-k8s-worker-nfs-43 puppet-agent[355401]: Applied catalog in 7.44 seconds

It's not that the service needs need restarting (that would not help), but if there's any processes (ex. pods) that have a file open from before the link changed that handle will still point to the old server, so when the old nfs server goes away, that file handle will get stuck, and might require the pods from restarting (they might just fail after 5 min or so by themselves).

It's not that the service not need restarting, but if there's any processes (ex. pods) that have a file open from before the link changed that handle will still point to the old server

I see, is there any way of finding out how many processes have a handle pointing to the old server? I guess we can just wait a few hours and hope that most of those will be closed by then.

I'm also a bit concerned by the fact there are still many open connections to the "depooled" server and I have no idea how many of those are "open but idle" and how many are in active use:

fnegri@clouddumps1001:~$ ss -tano state established sport nfs |wc -l
122

fnegri@clouddumps1001:~$ for a in $(ss -tano state established sport nfs|awk '{print $4}'|cut -f1 -d":"|grep -v Address|sort|uniq); do host $a | grep -v alias | awk '{print $5}'; done
wdqs2009.codfw.wmnet.
wcqs2001.codfw.wmnet.
wdqs2010.codfw.wmnet.
stat1010.eqiad.wmnet.
wdqs1024.eqiad.wmnet.
an-launcher1002.eqiad.wmnet.
stat1009.eqiad.wmnet.
stat1011.eqiad.wmnet.
stat1008.eqiad.wmnet.
nat.cloudgw.eqiad1.wikimediacloud.org.
instance-toolsbeta-mail-2.toolsbeta.wmcloud.org.
mail.toolsbeta.wmcloud.org.
beta.math.wmflabs.org.
instance-math24.math.wmcloud.org.
mardi.math.wmflabs.org.
login.tools.wmflabs.org.
dev.toolforge.org.
instance-tools-bastion-12.tools.wmcloud.org.
instance-tools-checker-5.tools.wmcloud.org.
checker.tools.wmflabs.org.
bastion.toolforge.org.
instance-tools-bastion-13.tools.wmcloud.org.
mail.tools.wmcloud.org.
instance-tools-mail-4.tools.wmcloud.org.
instance-tools-sgebastion-10.tools.wmcloud.org.
login-buster.toolforge.org.
nat.cloudgw.codfw1dev.wikimediacloud.org.

There are still some spikes of network activity on clouddumps1001, I will wait until tomorrow to see if that continues overnight or not:

Network activity did not decrease much after I merged my patch:

NFS connections did not decrease either:

fnegri@clouddumps1001:~$ ss -tano state established sport nfs |wc -l
122

@VRiley-WMF sorry for the lack of updates, I had to prioritize other things. This server is still alerting, and I didn't manage to depool it completely, there are still some connections going to it.

After discussing with @Andrew we think we're ready to shut it down anyway so that you can to try to swap the fans, but we would like to coordinate with you so that the host does not remain offline for more than a few hours.

Would you be available this week or the next, at a time between 15:00 UTC and 17:00 UTC? If that does not work for you we can find other slots!

I'm available for this activity at anytime. 15:00 UTC - 17:00 UTC works for me.

Great! I talked with @VRiley-WMF and the plan is:

1. I'm gonna shut down the server tomorrow for about 1 hour, to check if there's any unexpected impact, then take it back online.
2. If the 1-hour downtime does not cause immediate trouble, we'll schedule a longer 6-hour downtime for @VRiley-WMF to do more troubleshooting (possibly later tomorrow or later this week).

Icinga downtime and Alertmanager silence (ID=591bcb32-8025-4bce-af2c-49d023d1b4ca) set by fnegri@cumin1002 for 1 day, 0:00:00 on 1 host(s) and their services with reason: down for maintenance

clouddumps1001.wikimedia.org

I'm gonna shut down the server tomorrow for about 1 hour, to check if there's any unexpected impact, then take it back online.

I did this and it caused some issues on both Cloud VPS and Toolforge, I created T391369: If the inactive clouddumps host goes down, it causes a ripple effect on Cloud VPS and Toolforge.

We need to fix that before we can shut down clouddumps1001 for longer.

@fnegri I had looked at this briefly kinda looks like might be a bad intake sensor and might not be over heating comparing this with clouddumps1002 all other sensor readings look identical except inlet air We should comp air inlet air temps from any neighboring device before assuming that is bad though. If we are able to relocate the server we should move lower in rack.

Thanks @Jclark-ctr, do you think there is a way to disable the sensor so that it will not trigger the alert? We could also silence that alert indefinitely, only for this specific host, but it's a bit of hack.

I still want to fix T391369: If the inactive clouddumps host goes down, it causes a ripple effect on Cloud VPS and Toolforge because we might have to take this server offline for other reasons in the future.

We should comp air inlet air temps from any neighboring device before assuming that is bad though.

The next two servers in the rack (an-worker1150 and clouddb1016) also have a similar Inlet Temp, so maybe it's not a bad sensor.

https://grafana.wikimedia.org/goto/C3i1lGJHR?orgId=1

@fnegri. I installed several blanking panels on the hot aisle to help prevent hot air bleeding into cold aisle between Row B and Row C. After stepping back to assess the broader airflow pattern, I noticed that the overhead air vent near it the fin may also be contributing to the issue. the Far left one is pointed in the wrong direction and is the one that would be pointed directly at that rack. Lets see if this makes a difference

Also updated idrac firmware from 5.00.20.00 to 7.00.00.181

No alerts for 4 days and temps and fan speeds have dropped closing this ticket for Temp
The system inlet temperature is greater than the upper warning threshold. Thu Apr 17 2025 20:45:16

Made some more adjustments will see how this effects cooling

Mentioned in SAL (#wikimedia-cloud) [2025-05-23T09:00:01Z] <dhinus> failover dumps_dist_active_vps to clouddumps1001 (T383723)

I reverted my change from last month and moved back NFS traffic to clouddumps1001: https://gerrit.wikimedia.org/r/c/operations/puppet/+/1149610