The root of this seems to be that keystone has stopped logging with the name 'keystone,' instead using the name '<frozen importlib._bootstrap>'

We can likely have cloudinit add a public cumin key to all hosts on creation, but I have a few thoughts:

Upstream has acknowledged this and think it's fixed.

We are attempting to only get the puppet package from the wikimedia repo (this is set by cloud-init at creation time)

The base image is based on a trixie VM with our puppet classes already applied (that happens at build time). So shouldn't /that/ have already downgraded puppet in the base image?

For starters we should probably look for places that take a --project arg and convert them to either --project-id or --project-name depending on what the code does.

@Nokib_Sarkar have you seen this happen on multiple occasions, or just several times on the 7th specifically? (I want to make sure it's not a side-effect of maintenance activity.)

Do you have any theory (you being @elukey and @fgiunchedi) about why that happened on this exact instance? I just checked and we have around 100 running Trixie VMs so presumably cloud-init works properly most of the time.

This is done, and your creds should be in your envvars as TOOL_ELASTICSEARCH_USER and TOOL_ELASTICSEARCH_PASSWORD

How much disk is there to play with?

~400T of unreplicated space, split in 3 nodes

The only thing left to do here (that I know if) is relative links being messed up in the initial wikitech-static landing page. Search works, and once you navigate to a valid page the links work.

This is fine but I have to figure out how to do it! Docs seem to be at https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin#Granting_a_tool_write_access_to_Elasticsearch

cloudcontrol nodes not in C8 (i.e. 1006/1007) though didn't seem to give up trying to connect to rabbitmq01.eqiad1.wikimediacloud.org:5671 whereas cloudcontrol1011 stopped trying to talk to rabbitmq01 as expected.

decom script is failing:

note to self: the old flavor (g4.cores8.ram24.disk20.ephemeral90.4xiops) is available in other projects (according to tofu-infra: ["integration", "search", "gitlab-runners", "wikiapiary", "zuul3"]) so I'm leaving it as is.

I think I've fixed both (!) things that were wiping out your ptr record. Please re-open if it vanishes again!

I'm pretty sure the issue was that tofu was removing the by-hand record and then the ip-updater adding the instance- record. I've added this to tofu, let's see if it persists now.

I have not used it before, but 'designate-manage service clean' seems to be the tool needed. Now I see:

In T379550#11298228, @Andrew wrote:

I just encountered a variation on this: new user dpogorzelski was in the bastion project with the proper role, but 'project-bastion' didn't appear in 'groups dpogorzelski'. It's not unusual for that to take a few minutes but this time it had been hours.

In T421739#11767301, @Andrew wrote:

Item #2 is already handled by the code. I don't know how/why my last attempt was clobbered; trying again.

Item #2 is already handled by the code. I don't know how/why my last attempt was clobbered; trying again.

For #2, taavi points out that there is a boilerplate description for auto-created records

Hm, we have a bot that maintains those instance- addresses, it must've clobbered the one I made by hand. I will need to think about this a bit.

Current status:

Thank you @Don-vip! For better or worse there turns out to be no automated upgrade path; users will basically have to do a dump and import into a new engine. That said, if you want to try that (and, better yet, document it) I will ping you when the new engine is available.

Hi folks! Any idea when this is likely to happen? I will need to coordinate for openstack nodes which use a bespoke openstack repo hosted on the mirror.

+1 this seems fine; would you like us to also remove the older 8-core flavor or do you expect to use both in the future?

+1 approved

I've deleted all the things I can find in our rackspace account -- it should be empty or effectively empty. Over to you, Rob!

because they are declared classic and not quorum by default.

users had not actually allocated a floating IP, but I have now done so. It is: 185.15.56.85

eqiad folks: these hosts are untested hardware with a novel drive configuration. I do not expect partman to work on the first go!

eqiad folks: these hosts are untested hardware with a novel drive configuration. I do not expect partman to work on the first go!

eqiad folks: these hosts are untested hardware with a novel drive configuration. I do not expect partman to work on the first go!

In T420532#11730157, @bd808 wrote:

I will say that putting hyphens in the project name makes my brain hurt, but mostly because I still think of the project name as the primary key and I know that the S3 gateway we use does not allow hyphens in names.

+1, seems good

In T420213#11719139, @taavi wrote:

FWIW I see some value in having the cumin authorized_keys entries have an IP restriction on a host not accessible to most people, but otherwise agree with the proposal.

This is all done except for the cloudvirtlocal hosts.

This VM is now shut off. They will still bill us for the space, but let's give this a week before we delete things and close out the account.

Just as soon as I can get pwstore working I will shut down this host and wait for screams.

This was discussed and approved during today's weekly meeting.

It's been a long time since we discussed this and no one is working on it. The privacy implications are a bit messy so I'm just closing.

In T417393#11701437, @fgiunchedi wrote:

Thank you, I looked at cloudvirt.drain though I couldn't find an option specifically to make sure the destination host is not in the rack we are draining. Maybe not a huge issue though? The scenario I'm thinking about is we're draining a cloudvirt and all/most VMs migrate to another cloudvirt in the same rack, of course things would converge eventually at the risk of moving VMs a bunch of times.

decom script says:

2004-dev is up and working now, thanks to @taavi and a reboot.

Oh, to check the maintenance state of a host you want to look at the host aggregates. Docs for that here: https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Host_aggregates

In T417393#11696203, @fgiunchedi wrote:

Plan is to grab another announced maint window on Tues March 17th to resume the testing.

I have also opened subtasks for the remaining racks, one notable difference is that those do contain cloudvirt hosts. @Andrew what's the recommended procedure to temporarily drain a rack of VMs and then put them back? So far I found wmcs.openstack.cloudvirt.drain cookbook mentioned on wikitech

It seems that the network services on 1006 were manually (or via cookbook) set to down. That would certainly explain the failover.

In T418813#11689829, @dduvall wrote:

Thanks, @Andrew

In T418813#11675877, @Andrew wrote:

Hey folks, sorry about the not-very-coherent response on this. The bottom line is that compute+storage resources are not an issue, we can definitely provide what you need.

The thing that is in flux our commitment to magnum:

Since the resources will be needed regardless of using Magnum (the volume requirements MAY be different without Magnum, not sure) can we go ahead with the quota increase and have the Magnum discussion in a different venue?

+1

This is resolved in codfw1dev. Task is still open because I'm trying to track (and fix?) the issue upstream.

So -- I reiterate that I don't think a cloud-vps project is the right way to handle things like this. If I were solving the issues you're solving, I would definitely look for a hosted service rather than building my own infra from scratch, and I would start /that/ process by talking to other affiliates and asking how they are solving the problem.