In T408892#11426444, @Papaul wrote:

@ssingh yes we have to depool the site, yes 10 AM CT

In T408892#11424869, @Papaul wrote:

@ssingh We are planning on doing the first phase(loopback IP change on core routers and management router) of the ULSFO refresh next week Dec 09th at 10:00am. Please let me know if this work for you an your team.

Thanks .

@Dzahn has freed up some inodes. We were not out of disk space, we were out of inodes. We are trying to free up some more but for now, we should be running again. As @AntiCompositeNumber mentioned, there has been a steady rise for a while now so we should look into that.

And while there is that fallback mechanism to the old system, this is something to keep in mind.

Yeah I think that makes sense if we want to exert control over upstream issues and how it reflects to the proxy itself. We can approach this in two ways:

Yeah, there are certainly other files. I think you can remove your comment that has the SVG file contents otherwise it makes it difficult to read the task.

Once https://gerrit.wikimedia.org/r/c/operations/software/homer/deploy/+/1207915 is merged tomorrow or Monday, we will enable BGP, test the anycast address and then switch the backend to use that. But since the VMs are up and running with the desired role, marking this as resolved.

Hi @RoyZuo: we have tried to debug this on the CDN side and can't seem to find anything there that can point us to the problem. Can you upload any file at all, or is it simply this file, which given an SVG and 36.7Mb can be leading to some issues on the app layer.

In T409860#11388223, @MoritzMuehlenhoff wrote:

In T409860#11388216, @ssingh wrote:

Oh wow, thanks @MoritzMuehlenhoff! But what was the issue for my understanding?

Basically just magru being affected by https://phabricator.wikimedia.org/T396864. I ran makevm and once it had created the VM (but not yet initiated the reimage), I moved the install3004 VM to a different node than the node where the new hcaptchy-proxy700X was added. After that the reimage proceeds as usual. We'll have the underlying issue fixed once the new dnsmasq release is out.

Oh wow, thanks @MoritzMuehlenhoff! But what was the issue for my understanding?

In T367732#11382492, @ayounsi wrote:

@ssingh started working on this with https://gerrit.wikimedia.org/r/1206424 in T410047: No free IPs on public1-ulsfo vlan (Nov 2025) boldly assigning the task to him :)

Related: T410411.

Tagging Traffic for this is perfectly fine, thanks @A_smart_kitten.

In T409860#11379750, @MoritzMuehlenhoff wrote:

In T409860#11372373, @ssingh wrote:

hcaptcha-proxy3001 worked just fine but hcaptcha-proxy3002 does not come up after reboot (tried twice). A manual start also did not work and sudo gnt-instance info hcaptcha-proxy3002.wikimedia.org on ganeti3005.esams.wmnet also wasn't really helpful.

This was caused by https://phabricator.wikimedia.org/T396864. I've shuffled the VM to a different node. Simply running the reimage cookbook on the node should fix it.

Once dnsmasq 2.92 is out, this will no longer affected routed Ganeti: https://phabricator.wikimedia.org/T396864#11113708

In T410047#11379165, @cmooney wrote:

In T410047#11379157, @ssingh wrote:

Yeah, good point about the LVS IPs since we no longer need them given Liberica. I will be checking that with Valentin today.

It's more than they aren't "needed", they aren't being used. They are just incorrectly marked as in use in Netbox and need to be tidied up (also in puppet where they are referenced).

In T410047#11379141, @cmooney wrote:

In T410047#11379108, @ssingh wrote:

My plan for now to unblock the hCaptcha work was to decommission one of the Wikidough hosts in ulsfo -- which should be fine since they both average ~14 rps between them -- and then use that IP to create the hCaptcha VM. The reasoning for doing so is that hCaptcha is more critical service than Wikidough right now and I don't want it to be running on a single VM in ulsfo. But let me know what you think about this, in general, and if I should not go down this path!

Why not just re-use the LVS IPs? I think it's better to keep Wikidough with two hosts in case one fails?

In terms of expanding the vlan I think we should do it anyway, but I agree it should not hold up hCaptcha.

In T410047#11379134, @ayounsi wrote:

You can use 198.35.26.5/28. It's marked as reserved for infra, but we don't need it (and we will even less need it after the network upgrade).

In T410047#11374122, @cmooney wrote:

@ssingh I made a patch and can kick off the changes in Netbox and on the routers next week for this.

However I wonder what your thoughts are, how many more public IPs do you need in the short term? Reason I ask is this vlan will be doubled (quadrupled actually as we will add another public vlan for the second rack) during the T408510: ULSFO: switch refresh work which is coming up in the next few months. That work will require most hosts to be reimaged while we change the network setup to a L3 POP.

So an option, if removing unused IPs like the LVS from the vlan now gives enough for the proxy VMs, is to decline this task and increase the subnet size as planned during the larger job?

Actually I discussed with @Papaul in relation to our plans for ulsfo, and we both agree that work would be a lot simpler if we make this change now. We can discuss the way forward next week.

hcaptcha-proxy3001 worked just fine but hcaptcha-proxy3002 does not come up after reboot (tried twice). A manual start also did not work and sudo gnt-instance info hcaptcha-proxy3002.wikimedia.org on ganeti3005.esams.wmnet also wasn't really helpful.

In T405623#11368198, @RobH wrote:

All cp hosts in rows C/D have been migrated as of today (last ones done) and all that is left in Traffic realm for migration is dns1006 and lvs1020 via T405602.

We moved the existing cp hosts in C/D either last Thursday, Friday, or today. For each one I put in maint mode via icinga cookbook then logged into the cp host directly and depooled it, had the cable migrated, and then when ping resumed repooled the host. Each host was depooled for less than 5 minutes. I did not depool more than 1 host at a time from either text or upload.

In T405623#11298598, @RobH wrote:

Please note this migration has shifted from Oct 15th start date to Nov 1 start date.

sudo cookbook sre.ganeti.makekevm --vcpus 2 --memory 2 --disk 20 --network public --os trixie -t T409860 --cluster <site> --group <group> <hostname>

In T409860#11367296, @MoritzMuehlenhoff wrote:

One thing to consider, when we actually apply the role on DCs enabled with routed ganeti (esams and magru) we need to set profile::bird::routed_ganeti_apt: true to enable the apt component with the Bird package that has been enabled for routed ganeti (via hieradata/role/magru|esams/foo.yaml

Hi @Monneyboi: Thanks for reporting! Setting a user-agent will fix the error as your own example above shows (thanks for trying that!) And additionally, like you mentioned, the reason this policy is now being enforced, is to ensure fair-use of infrastructure; as such, user-agents are one of the ways of identifying or classifying the request, and thus we are requiring them to be set.

Once the VMs are up, we will need to enable BGP for all of them in Netbox and then run homer.

Initial role can be insetup::traffic_nftables. We will reimage to hcaptcha::proxy role later, with Debian bookworm as routed Ganeti setups (magru/esams) do not have the patched bird packaged for trixie yet and we don't want to wait. (We can reimage to trixie later.)

Hi @Jdrewniak: Daniel has already commented on the questions from Traffic's end (and as it related to the CDN and DNS) and what he has mentioned is correct per our understanding as well. We also set up some redirects under T408168 so we can work with those domains if required (wikipedia25.org).

In T409735#11362769, @cmooney wrote:

Thanks @ssingh. I'm just reading about this RFC for the first time, I wonder longer term might it be a goal to automate the ingestion of data from such feeds to update our maps automatically? Obviously not as part of this work, just an idle thought.

In T352245#11360536, @Scott_French wrote:

In T117618#11359861, @sbassett wrote:

In T117618#11224896, @ssingh wrote:

Commenting from Traffic's side: this is in some ways, a trivial patch for us because we are simply setting an additional header. The challenge here, though, is understanding the header itself and the associated ramifications of setting it and also keeping it updated. For that, the Security should be/needs to be consulted, so this patch currently blocks on that happening.

Would you like to have a chat about this? I think we'd be fine with @Bawolff's suggested CSP in T117618#11072208, if we're fine breaking favicons. I'm not sure of what the best way to test this would be.

Thanks for filing this task @cmooney! The geofeed link above is very helpful. So it seems from the above (57.141.8.0/24, 57.141.8.0/24), we are missing the entries in the geo-maps file so they default to codfw. (We have 57.141.4.0/24 and 57.141.5.0/24 in the geo-maps).

Ah interesting, that explains why we couldn't see the verification option on the Search Console. So just to confirm, you are set for both?

Thanks @akosiaris, that sounds good. We would like to get this done in Q3 to resolve this blocker and to deploy Liberica everywhere, so please do factor that in for your planning. Thank you!

@JKelsoteel-WMF: Can you please try to log in to wikibooks.org as well so we can see the text of the DNS record that needs to be verified?

Thanks @JKelsoteel-WMF, we will be picking this up today.

In T408168#11342081, @SCampos-WMF wrote:

Hi @BCornwall thank you for this. I have one last update request, apologies for the back and forth. I just heard from Fundraising that we had to adjust one of the UTM parameters since they weren’t able to track it correctly. Could you please update the destination URL one last time to this one: https://donate.wikimedia.org/?appeal=WP25&pym_appeal=WP25&wmf_campaign=vvid&wmf_medium=vvid&wmf_source=vvid

Thank you again for your patience and help!

In T409330#11346176, @cmooney wrote:

Thanks for the task @ssingh !

I agree this is definitely a major gap. In terms of the alertmanager rule you list it does make sense we should have another one (or expand it) to also cover transport / private WAN circuit. So we can absolutely do that.

Historically these alerts were triggered for us by LibreNMS. Checking there it's pretty obvious why those are no longer firing - they are turned off!

It's not at all clear to me when or why those were disabled. But in any event I have re-enabled them now which I think should make alerting work again. This setup may also explain why my attempts to get LibreNMS to alert at lower-than-line-rate for eqsin didn't work - the damn things were disabled completely!

Ultimately we can move these to alertmanager, we can work on what those alerts look like. And potentially move to basing them on dropped outbound packets taking QoS priority into account (T384052).

In T390813#11345599, @Papaul wrote:

@ssingh @Vgutierrez planning on doing this on Nov 19th @10:am CT. Thank you

In T408689#11323157, @SLyngshede-WMF wrote:

@ssingh - for manager sign off

@Ottomata - Group access approval

Thanks for the help @Jhancock.wm. Marking this as resolved for now.

Brett pointed out that the regression was introduced in https://gerrit.wikimedia.org/r/q/I9fab3e43a39456432eb148df91faffba54b1926e.

Sorry this took a while but this should now be resolved. Thanks to Giuseppe for taking care of it in https://gerrit.wikimedia.org/r/c/operations/puppet/+/1198424, which we will build on further as required, with Traffic responsible for ensuring parity.

Yeah I missed this in the previous fix. I am going to take this tomorrow since now essentially we have to guard the includes as well.

In T404826#11303839, @Krinkle wrote:

In T407966#11299911, @ssingh merged:

[operations/puppet@production] varnish: add conditional to varnish::common::vcl for beta

https://gerrit.wikimedia.org/r/1198132

Note that while this silenced the alert for Puppet, it did not resolve the Varnish compilation failure. Beta is kept online only by Varnish's stale memory of the last working config, because reloads fail to apply new changes:

In T390813#11302841, @Papaul wrote:

@ssingh thanks for the update. I am planning on doing it before Thanksgiving any day during the week of November 17th works for me. Let me know if that woks for you and I can get back with you on the exact day and time.

Sorry about this, this should now be fixed. And glad to see that Traffic was added automatically, thanks to @bd808 and @Ladsgroup for their work on this!

In T404826#11298704, @dancy wrote:

https://gerrit.wikimedia.org/r/c/operations/puppet/+/1197986 has caused puppet to break on deployment-cache-upload08.deployment-prep. Please help!

This is because of:

In T390813#11296819, @Papaul wrote:

@ssingh @Vgutierrez hello just checking in to see if you have a day and time for this for drmrs.
Thanks

Hi @CRoslof: This is another ticket that we would like to take up and will need your help with so that we can reflect it in downstream services as well. Let me know if I should create a Zendesk thread for tracking by other Legal members? Thanks a lot of for bearing with us and helping us clean the ownership.

It looks like spicerack should check that alerts for the downtimed host have been resolved (not in firing state) before deleting the silence/downtime with ALERTS{alertstate="firing", instance=~"cp5018:.*"}

Thanks for filing this task! I think this is a good idea to reduce the manual updates to this list, and something we have failed to keep updated. We will triage this after discussion in Traffic.

Nice job indeed in pursuing this over the years, Brett!

[Adding Raine @kamila as well.]

Thanks for filing the task, @JVanderhoop-WMF. As per the discussion on Slack, the above sounds good.

Thanks to @Jhancock.wm for the help with this!

FWIW doing one or two hosts is more than enough. We will reimage them again anyway so it doesn't make sense IMO for you both to spend time upgrading all of them to trixie. If one or two reimage fine, please leave the rest to us.

In T407320#11276943, @colewhite wrote:

Do you happen to have a trixie host available that we can try the existing package on?

I was also looped into a new request today. As part of the birthday initiative, the Fundraising team is developing a customized donation portal under the donate.wiki domain. Would it be possible to set up a redirect for this new portal as well? I don’t have the final destination URL yet, but we’d like to create the domain donate.wikipedia25.org to redirect to the donation portal once it’s ready. Is this something you could help with too?

In T405499#11275363, @cmooney wrote:

In T405499#11273763, @ssingh wrote:

FWIW we have typically reimaged for this in the past. I am not suggesting, just sharing! And given that this is lvs1020, that might be OK? (Leaving to you both for the final decision.)

This is lvs1018. I'm comfortable enough either way, can I leave the decision with traffic?

I was thinking of trying to schedule this for tomorrow, Thurs Oct 16th if that worked for you guys?

Thanks for working on this! We will try our best to follow up on our end in making sure that Puppet is not broken on the cache hosts in Beta.

FWIW we have typically reimaged for this in the past. I am not suggesting, just sharing! And given that this is lvs1020, that might be OK? (Leaving to you both for the final decision.)