Cluster in warning also blocks ceph reboots for {T431659}, not sure if we want to --force though ?

Good point @dcaro, ok thank you! So as far as Toolforge is concerned we can move to bind-mounting /mnt/nfs with mount propagation, I'll send out patches

In T430933#12132469, @Andrew wrote:

I think I'm behind on something. When you say...

should not be hardcoded in puppet anymore

Are you talking about the profile::openstack::base::nova::compute::id values in hiera, or a different nova ID somewhere else in puppet?

Thank you for the feedback @hnowlan @ssingh ! re: exclusion list I'm also expecting that moving to ratio-based paging will carry more signal (i.e. we can shrink the exclusion list)

Thank you @Odinaldo for reaching out and the useful information. I don't think 60GB is problematic, I'm looking forward to the size estimation for the full database. Also in terms of sizing I'd like to know if you have a sense of system load in terms of requests per second coming from users? thank you

A side effect of the toolsbeta deployment from yesterday was T432115: [jobs-emailer, toolsbeta] No emails sent in the last hour i.e. pods getting stuck while starting. toolforge volume-admission expects dumps mountpoint directories to be present, therefore we'll have to make sure to not remove them via puppet.

In T431682#12119607, @VRiley-WMF wrote:

Thanks @fgiunchedi! I will be running through this today. I just got back from vacation and catching up on a few different things here. I will let you know when this first one is done.

I'm going through https://wikitech.wikimedia.org/wiki/Server_Lifecycle#Move_existing_server_between_rows/racks,_changing_IPs and figuring out ownership for the various steps, cc @cmooney @ayounsi

@VRiley-WMF I thought we could start with one host, cloudvirt1048 and test-run the whole procedure there. Once we have nailed the process then we can do the rest in batch, what do you think?

As of now dumps_use_nfs_lb: true is deployed to toolsbeta, unfortunately for k8s workers that means a roll-reboot because containers keep a reference to /mnt/nfs mountpoints. That in turn means that pre-reboot the mount attempts will keep contacting the old clouddumps nfs server address. A reboot is the easiest way to ensure new mounts hit dumps-nfs lb address.

I forgot to attach a task number, however:

During the last reboots of clouddumps hosts we have not observed toolforge/cloudvps unavailability. At any rate, the service will be improving when T411248: Plan to make clouddumps more resilient and easier to operate is complete

This is still an issue: SRE got paged for 10 5xx req/s on swift which was doing 2k req/s at the time (esams). I very much doubt it is worth paging engineers on absolute number of errors, irrespective of service traffic

In T429013#12109258, @cmooney wrote:

Thanks @fgiunchedi.

This isn't so urgent we want to cause stress for you guys. So 2 or 3 is also fine if you want to move some of those hosts first. But 1 obviously works for us too :)

Agreed re: 'sshd is quite reliable enough'.

ok two paths as I see it:

This is done in the sense that maintenance aggregate is no longer a thing. I have updated https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Maintenance#cloudvirtXXXX with commands to list maint status

In T424658#12101239, @VRiley-WMF wrote:

@fgiunchedi I was curious to know if there is an estimated date for these moves? Will we be doing it one server at a time.

Sweet, thank you !

And another one for your eyes @Andrew https://gitlab.wikimedia.org/repos/sre/wikitech-static-docker/-/merge_requests/6

+1 LGTM

Tentatively resolving, feel free to reopen if sth is amiss

I did some more testing of https://gerrit.wikimedia.org/r/c/operations/puppet/+/1308126 (general scaffolding to support dumps-nfs.w.o) and https://gerrit.wikimedia.org/r/c/operations/puppet/+/1308128 (cloud vps support).

In T411248#12084087, @fgiunchedi wrote:

In T411248#12080727, @ayounsi wrote:

In T411248#12076020, @fgiunchedi wrote:

In T411248#12074721, @ayounsi wrote:

@fgiunchedi maybe silly question but to make sure we're preserving public IPs, why can't we use the same IPs for dumps and nfs dumps? Like we do for other services listening on different ports (eg. gerrit-http/gerrit-ssh)?

Not a silly question at all -- please double check my reasoning, service is not in production yet (more on that later) and there's still time to change with basically-zero impact.

• The service is fundamentally different than http/rsync in my mind, it is public only because cloud/prod both access it. Unlike http/rsync which are public by design
• Easier to match/isolate the load balanced nfs traffic from http/rsync

Having said that, I don't feel very strongly nor have stronger motivations, maybe some other folks do

Thanks, then let's be frugal about IPs and use the same one for both protocols if that's ok with you.

Using ports + IPs seems fine enough to match/isolate traffic (ACLs, analytics).

SGTM, I'll be moving the service to the same IP as http/rsync

In T431303#12093395, @MatthewVernon wrote:

I've just deleted and re-created my OoO starting 21 July. Thanks for looking into this :)

Ok I spent some time debugging the script (https://script.google.com/home/projects/1k0oMpC0CtdiwLN9zCbvFM_g2z5LtdtMDfV4QMJWLgMi7gljqfJAJb0iI) and it is definitely doing something, even recently:

I'm not sure there's anything immediate to do here as trying again worked. I suspect the proper and more time-intensive fix is to move away from wmcs-drain-hypervisor and onto cookbooks, either using cli or openstacksdk clients.

In T429738#12085068, @taavi wrote:

Moving the alerting to be based on Istio-level metrics means we lose visibility into any potential connectivity issues between HAProxy and Istio, right?

+1 to just get rid of renaming/broken functionality

Current situation wrt aggregates ceph, network-ovs and maintenance

In T424802#12058115, @fgiunchedi wrote:

In T424802#12054459, @Andrew wrote:

I'm not sure why this is happening. Most likely it's due to a different cookbook (maybe wmcs.openstack.cloudvirt.safe_reboot) failing and leaving things in an inconsistent state.

As far as I know the 'maintenance' aggregate doesn't do anything specific, it was just created as in indicator to human eyeballs that a host is getting worked on. The real action is in the 'ceph' aggregate which allows new VMs to be scheduled. So we could potentially do away with the maintenance aggregate entirely.

Ok now I'm even more confused, the set_maintenance and unset_maintenance cookbooks operate on maintenance aggregate to supposedly stop/start VMs being scheduled on the hypervisor.

The tool availability metric will need to be a recording rule because istio_requests_total is huge. And the same per-tool availability metric can also feed the KR metrics for webservice unavailability we were talking about yesterday (cc @CCiufo-WMF @aputhin)

Indeed now with the work by @taavi on T392356: Replace ingress-nginx before upstream EOL date we do have per-tool status codes as seen by istio, i.e. the metric used in https://grafana.wmcloud.org/d/fnhp8st/tool-error-rates based on istio_requests_total

Could be yeah, I don't know if puppet agent is actually supposed to be doing its own crl management/refresh.

On balance I like option 2, which ensures truly end-to-end monitoring of container build + deploy. In other words the wikitech_static_update_timestamp_seconds metric going stale (time() - wikitech_static_update_timestamp_seconds) means something went wrong, either the container failed to build or failed to deploy. Either way we can take a look when/if that happens, @Andrew I sent the MR your way, please let me know what you think

On the specifics on why exactly this happens, speaking as a former o11y member, I'm not going to look deeper into icinga-issued pages because IMHO we shouldn't be doing that anymore in the first place.

To provide some context/history: the "re-page on acked but not resolved incidents" is a VO setting which we set to 24h and can be disabled (cfr T259465: VictorOps behavior on long-ack'd incidents). I am +1 on changing the behavior to not re-page

In T411248#12080727, @ayounsi wrote:

In T411248#12076020, @fgiunchedi wrote:

In T411248#12074721, @ayounsi wrote:

@fgiunchedi maybe silly question but to make sure we're preserving public IPs, why can't we use the same IPs for dumps and nfs dumps? Like we do for other services listening on different ports (eg. gerrit-http/gerrit-ssh)?

Not a silly question at all -- please double check my reasoning, service is not in production yet (more on that later) and there's still time to change with basically-zero impact.

• The service is fundamentally different than http/rsync in my mind, it is public only because cloud/prod both access it. Unlike http/rsync which are public by design
• Easier to match/isolate the load balanced nfs traffic from http/rsync

Having said that, I don't feel very strongly nor have stronger motivations, maybe some other folks do

Thanks, then let's be frugal about IPs and use the same one for both protocols if that's ok with you.

Using ports + IPs seems fine enough to match/isolate traffic (ACLs, analytics).

I looked at, and reported, logs as part of clinic duty though did not touch the db FWIW

Ok so I spent some time getting a better understanding of NFS and how Linux clients reacts to a server failover. Below my findings:

All steps done, I'm tentatively resolving though @Rscout please reach out and reopen if something is amiss

@Rsilvola FYI this is pending your approval

In T411248#12074721, @ayounsi wrote:

@fgiunchedi maybe silly question but to make sure we're preserving public IPs, why can't we use the same IPs for dumps and nfs dumps? Like we do for other services listening on different ports (eg. gerrit-http/gerrit-ssh)?

In T430304#12072406, @KFrancis wrote:

Hi all, the NDA is complete. Thanks!

In T430304#12070658, @Monrac5 wrote:

In T430304#12068953, @fgiunchedi wrote:

Thank you all!

@Monrac5 we'd need to verify your ssh public key out of band. please let me know when it would be a good time for doing so synchronously (e.g. google meet)

@fgiunchedi would tomorrow at 11:30 am CEST work for you? i could also meet anytime before that or after 1 pm CEST

I'll be starting the trials on toolsbeta and specifically toolsbeta-test-k8s-worker-nfs-8.toolsbeta by overriding its hiera values with the following:

hosts are in service. I have updated https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Maintenance#new_cloudvirt_install with the instructions, manual for now

In T430304#12070030, @Milimetric wrote:

Approved (NOTE: I initially thought there was no NDA on file, but see from the comments it's now available, so you can update that in the task description)

https://gerrit.wikimedia.org/r/c/operations/puppet/+/1306672 is the bandaid I propose for now to get unblocked

@Rscout we need to verify your ssh key out of band, please let me know when it would be a good time for a quick google meet. feel also free to send a meeting invite my way: fgiunchedi@wikimedia.org

@Milimetric @Ahoelzl @Ottomata I'm seeking analytics-privatedata-users approval for Mona, an former WMDE intern. thank you !

Thank you all!

Would we have capacity (power, space) to move two hosts to their final allocation in E4/F4 ?

@KFrancis I could not find an NDA on file for Mona Thierse, would you mind arranging one? thank you so much!

Hello @Monrac5, thank you for reaching out -- just to confirm: you are not part of WMDE staff, correct ?