Ok, thanks for dealing with this.

I remember trying with either --sudo-project-id <project> and --os-project-id <project> in the cmdline with no luck.

The record for wmcloud.org was created:

BTW: the pull request to update this upstream is: https://github.com/publicsuffix/list/pull/970

I couldn't create the TXT record int he wmcloud.org zone, either using horizon or the cmdline:

Ok, I think we can safely assume we are talking only about openstack databases: designate, glance, keystone, neutron, nova*.

In T218570#5888668, @Marostegui wrote:

Will m5 content be eventually migrated to those new hosts?

I created a wikitech page:

General hard drive failure doesn't sound good. Please @Jclark-ctr @Cmjohnson advice how to proceed.

The neutron router implements the address scope mechanism by looking at the input/output interface of packets. Since the networks we are interested in are external (truly physically external) to neutron, all packets circulate using the same output interface and thus the get applied the general NAT.
This would help us if neutron were implementing the address scope mechanism by evaluating source/destination address of packets, which is not the case.

This was accepted by the WMCS team.

Mention: T244986: cloudvirt1009: Device not healthy -SMART-

Apparently the issue resolved itself:

This task might be duplicated: T244222: CloudVPS: hiera refactor

This ticket refers to the legacy kubernetes cluster, which is in the process of being deprecated.

Bumping this. I'm not sure if we are still interested in this.

some related openstack docs: https://docs.openstack.org/liberty/networking-guide/scenario-classic-ovs.html

Work here is done. Please reopen if required.

This is done. Please reopen if required.

For the record, yes, we are aware of the change in the document root for webservices changing the URL scheme. This is a change that tool developers should change in their apps.

TODO: after last changes, requests to https://tools.wmflabs.org/ no longer redirect to the admin tool. The end in fourohfour domains for it to handle. We should consider adding some special handing for this case in fourohfour. CC @bd808

In T244222#5851742, @aborrero wrote:

Proposal:

• drop project/host hieradata from operations/puppet.git. Declare we have horizon for that. I doubt it is properly working anyway (for the same problem we can't use $::wmcs_deployment)
• introduce per-deployment hierakeys in operations/puppet.git. But have the search hierarchy non-dependant on hiera lookups, but have it harcoded (we have different files for different deployments anyway)

This way we:

• eliminate the potentially dangerous facts gathering
• eliminate a duplicity in how hiera is stablished (horizon vs ops/puppet.git)
• effectively support per-deployment hiera data, which is the ultimate goal we are after.

Will try to create a patch for this.

• drop project/host hieradata from operations/puppet.git. Declare we have horizon for that. I doubt it is properly working anyway (for the same problem we can't use $::wmcs_deployment)
• introduce per-deployment hierakeys in operations/puppet.git. But have the search hierarchy non-dependant on hiera lookups, but have it harcoded (we have different files for different deployments anyway)

Trying to merge https://gerrit.wikimedia.org/r/c/operations/puppet/+/569230 I discovered the following issue:

I tried installing in Buster the package from Experimental as @Epantaleo suggested. APT was unable to install the package due to missing dependencies, but I checked and this is the kind of problem that can be solved by building the package directly in the target release (backporting). This is what I did, for future reference.

In T238096#5840766, @Chicocvenancio wrote:

Does this include updates to the toolforge prometheus instance? I wanted to funnel PAWS metrics there but it was such an old version when I tried it I gave up on getting it to work.

The server is still online and serving requests as usual.
We have a project to introduce a replacement in this task: T242607: Create in-cloud puppetmaster for codfw1dev

Closing task now, feel free to reopen if required.

Things are far better now:

Ok, I got to this point:

I discovered there are many tools with same name (jarbot-ii, jarbot-iii) doing apparently the same thing. No orphan procs, but actual tools!

I used this to try detecting the procs:

I just discovered that codfw1dev-ns0.wikimedia.org exists indeed.

+1 for cloudinfra.

@russblau could you please describe what behavior would you like to see / you we expecting?

Both error page handling (T103662) and this debug setup is under review. This is affected by our current effort to introduce the new kubernetes cluster into Toolforge, which has a new ingress mechanism and new error page handling mechanism.
You can read more here: https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Networking_and_ingress
Moreover, when we finally introduce the new domain toolforge.org (something we plan to do this quarter), error handling and debugging options may need to be reworked completely on our side.

I had a look a this today.

BTW this server has active workloads (pooled) at the moment. Please @Jclark-ctr coordinate with WMCS before shutting server down.

@Jclark-ctr I believe this server may need the BBU checked/replaced, but I may be wrong.