https://openstack-browser.toolforge.org/project/5238620953c040e7a9effbe47d4e0932 is the UUID named page for the "wikiqlever" project. Stashbot thinks that @Physikerwelt should have used !log 5238620953c040e7a9effbe47d4e0932 start setting up puppet master following https://wikitech.wikimedia.org/wiki/Help:Project_puppetserver based on the LDAP lookup data. This is really poor user experience (same for the UUID in the openstack-browser URL honestly) that we should figure out how to fix.

In T379030#10291065, @bd808 wrote:

https://openstack-browser.toolforge.org/project/5238620953c040e7a9effbe47d4e0932 is the UUID named page for the "wikiqlever" project. Stashbot thinks that @Physikerwelt should have used !log 5238620953c040e7a9effbe47d4e0932 start setting up puppet master following https://wikitech.wikimedia.org/wiki/Help:Project_puppetserver based on the LDAP lookup data. This is really poor user experience (same for the UUID in the openstack-browser URL honestly) that we should figure out how to fix.

I tried that with the next log entry, which went https://sal.toolforge.org/__all__ but not at https://wikitech.wikimedia.org/wiki/Nova_Resource:Wikiqlever

It works for elder projects such as math

https://wikitech.wikimedia.org/wiki/Nova_Resource:Math/SAL

In T379030#10307609, @Physikerwelt wrote:

I tried that with the next log entry, which went https://sal.toolforge.org/__all__

https://sal.toolforge.org/5238620953c040e7a9effbe47d4e0932 is the project specific log collection in the Toolforge backend.

but not at https://wikitech.wikimedia.org/wiki/Nova_Resource:Wikiqlever

Stashbot current does not have any way to figure out that the OpenStack project id 5238620953c040e7a9effbe47d4e0932 is connected with the OpenStack project name Wikiqlever. This is basically the bug.

It works for elder projects such as math

https://wikitech.wikimedia.org/wiki/Nova_Resource:Math/SAL

Correct. Both the OpenStack project id and name are math for that project as it was created prior to the change from T274268: Wind down use of project ID and project name equivalency in OpenStack. The hooks that populate the LDAP directory with information about projects created the https://ldap.toolforge.org/group/project-math entry. For new projects we are now creating LDAP records like https://ldap.toolforge.org/group/project-5238620953c040e7a9effbe47d4e0932 which have no connection at all to the human readable project name in their data.

The stupidest thing I can think of is moving https://wikitech.wikimedia.org/wiki/Nova_Resource:Wikiqlever to https://wikitech.wikimedia.org/wiki/Nova_Resource:5238620953c040e7a9effbe47d4e0932. One can then use displaytitle, etc to make it look nicer. I guess there was a reason for changing from semantic identifiers to identifiers with different desired properties. If we don't enforce uniqueness in project names anymore, it would be hard for the bot the guess the right project.

Ah the SAL subpage page is already at the place where I would have expected it to be https://wikitech.wikimedia.org/wiki/Nova_Resource:5238620953c040e7a9effbe47d4e0932/SAL

I just changed https://wikitech.wikimedia.org/wiki/Nova_Resource:Wikiqlever to include that other page. Maybe we can document that, and is that good enough?

Also in emails, e.g., regarding failed puppet runs

Puppet is having issues on the "puppetmaster.5238620953c040e7a9effbe47d4e0932.eqiad1.wikimedia.cloud (172.16.4.20)" instance in project

5238620953c040e7a9effbe47d4e0932 in Wikimedia Cloud VPS.

the semantic name is not shown, which will most likely cause people who are involved in more than 1 project to ignore it. However, the project name seems still to be unique as it can be used for ssh

Hostname puppetmaster.wikiqlever.eqiad1.wikimedia.cloud

Change #1090854 had a related patch set uploaded (by Arturo Borrero Gonzalez; author: Arturo Borrero Gonzalez):

[operations/puppet@production] openstack: wmfkeystonehooks: create LDAP groups with project name

https://gerrit.wikimedia.org/r/1090854

First, some context: There isn't actually a change in ldap from one format to another; in all cases ldap is using the canonical ID of the project. It's just that IDs and names have recently diverged.

I agree that the new IDs are essentially human unreadable and shouldn't appear in most user interactions, but I think we should nevertheless continue to use canonical openstack IDs for all backend references to projects -- we've already tripped over this in one big way and I don't like the idea of obfuscating the ID/name distinction further in code. Ideally I'd expect things to use IDs on the backend/internally and then do a last-minute lookup when displaying to the user. I'm pretty sure that ldap representation counts as 'internal' rather than 'UI' even though wmcs staff is used to reading ldap with our bare eyes.

The proposed patch seems hazardous because if we start using names rather than IDs in ldap, that is actually a big change -- even though it feels like it's putting things back the way it was, it's actually changing ldap to store a new kind of thing that nothing that consumes ldap will be expecting.

So, I think the correct way to address this is to move forward towards making (the rest of) our user tools understand about what an ID is vs a name. I'm also not sure that things like stashbot should be looking at ldap at all when they can talk directly to openstack which is the actual source of truth.

In T379030#10318678, @Andrew wrote:

The proposed patch seems hazardous because if we start using names rather than IDs in ldap, that is actually a big change -- even though it feels like it's putting things back the way it was, it's actually changing ldap to store a new kind of thing that nothing that consumes ldap will be expecting.

This fear of using Project Name rather than Project ID feels like a direct contradiction of the IRC conversation I documented in https://phabricator.wikimedia.org/T366679#9993083. Names are just as unique as UUIDs at least for the purpose of interrogating the active projects.

At least two of the things that consume LDAP for OpenStack project data, my Stashbot tool and my brain, are both broken by the current UUID state. I am not coming up with reasons to keep mirroring the Keystone project membership data into LDAP at all if it is going to be obfuscated. I think humans have thought at least a bit about this before because someone decided that we should have DNS records using the Project Name in addition to records using the Project ID. A dual recording in LDAP as well would at least restore the ability for Stashbot and my eyes to read the data there if dropping the UUID group naming is undesirable.

From an non-wmf perspective, the most important aspect is documentation. It only takes 5 seconds to C&P the project id, but it might take several hours to find out that the configuration was changed.

I think my opinion is that I'm happy if openstack internally has uuids for projects ids, which is what upstream does.
Then, everything outside openstack, including our tooling, we can keep using project names. LDAP, stashbot, SAL, DNS, etc they all qualify in this category of being "our tooling".

With this, we may get the best of both words: we retain the native upstream openstack behavior of using uuids, but we leverage the fact that project names are unique, to simplify and make 'developer-friendly' some of our tooling.

I would be happy if the only place where we can see project uuids is deep inside the openstack database.

It's definitely the case that we enforce unique project names in eqiad1. So the only situation I can think of where names would be ambiguous (and IDs unambiguous) is if we ever build our long-awaited second region.

The only other (mild) concern I have is with format of project names, but that's probably imaginary... I imagine that as long as we constrain project name to valid dns domain names that will also restrict it to valid ldap keys? If so, think we can go forward with Arturo's plan although it still strikes me as moderately incorrect :p

In T379030#10322740, @Andrew wrote:

It's definitely the case that we enforce unique project names in eqiad1. So the only situation I can think of where names would be ambiguous (and IDs unambiguous) is if we ever build our long-awaited second region.

There are a number of implementation details for that, like who will be enforcing uniqueness of uuids across regions, or how the LDAP backend would be used for different regions.

The simpler examples I can think of are:

• different regions have different accounts (and LDAP backend). This is the case with codfw1dev. We basically don't care here if the names/uuids are duplicated.
• different regions share the same keystone (and LDAP backend). I guess the same keystone implies same openstack DB, therefore organic uniqueness for names/uuids.

Do you have something else in mind?

The only other (mild) concern I have is with format of project names, but that's probably imaginary... I imagine that as long as we constrain project name to valid dns domain names that will also restrict it to valid ldap keys? If so, think we can go forward with Arturo's plan although it still strikes me as moderately incorrect :p

You definitely have a number of considerations in your radar that I don't. I would feel more confident if you take over the patch https://gerrit.wikimedia.org/r/1090854 and related changes that may be needed.

I am not coming up with reasons to keep mirroring the Keystone project membership data into LDAP at all if it is going to be obfuscated.

Up until this week, I would have said that keystone projects are in ldap for pam/sssd and sudo rule management. @bd808 can you tell me more about what kinds of things you use ldap for directly? I think I understand the stashbot case (which can potentially be altered to talk to keystone directly).

Change #1093997 had a related patch set uploaded (by Andrew Bogott; author: Andrew Bogott):

[labs/tools/stashbot@master] Get openstack project list from keystone

https://gerrit.wikimedia.org/r/1093997

In T379030#10344421, @Andrew wrote:

I am not coming up with reasons to keep mirroring the Keystone project membership data into LDAP at all if it is going to be obfuscated.

Up until this week, I would have said that keystone projects are in ldap for pam/sssd and sudo rule management. @bd808 can you tell me more about what kinds of things you use ldap for directly? I think I understand the stashbot case (which can potentially be altered to talk to keystone directly).

https://ldap.toolforge.org/user/bd808 is another consumer.

I personally quite often query the directory when I am looking for information about a particular Developer account:

$ ldap cn=BryanDavis + | grep memberOf:
memberOf: cn=tools.my-first-django-tool,ou=servicegroups,dc=wikimedia,dc=org
memberOf: cn=tools.my-first-flask-tool,ou=servicegroups,dc=wikimedia,dc=org
memberOf: cn=tools.my-first-flask-oauth-tool,ou=servicegroups,dc=wikimedia,dc=org
memberOf: cn=tools.esbackup,ou=servicegroups,dc=wikimedia,dc=org
memberOf: cn=tools.keystone-browser,ou=servicegroups,dc=wikimedia,dc=org
memberOf: cn=tools.bd808-pywikibot,ou=servicegroups,dc=wikimedia,dc=org
memberOf: cn=tools.bash,ou=servicegroups,dc=wikimedia,dc=org
memberOf: cn=tools.mysql-php-session-test,ou=servicegroups,dc=wikimedia,dc=org
memberOf: cn=tools.toolforge,ou=servicegroups,dc=wikimedia,dc=org
memberOf: cn=tools.versions,ou=servicegroups,dc=wikimedia,dc=org
memberOf: cn=tools.tool-db-usage,ou=servicegroups,dc=wikimedia,dc=org
memberOf: cn=tools.static,ou=servicegroups,dc=wikimedia,dc=org
memberOf: cn=tools.phabulous,ou=servicegroups,dc=wikimedia,dc=org
memberOf: cn=tools.csp-report,ou=servicegroups,dc=wikimedia,dc=org
memberOf: cn=tools.phab-ban,ou=servicegroups,dc=wikimedia,dc=org
memberOf: cn=tools.hatjitsu,ou=servicegroups,dc=wikimedia,dc=org
memberOf: cn=tools.docker-registry,ou=servicegroups,dc=wikimedia,dc=org
memberOf: cn=tools.cloudvps,ou=servicegroups,dc=wikimedia,dc=org
memberOf: cn=tools.www,ou=servicegroups,dc=wikimedia,dc=org
memberOf: cn=tools.iw,ou=servicegroups,dc=wikimedia,dc=org
memberOf: cn=tools.bikeshed,ou=servicegroups,dc=wikimedia,dc=org
memberOf: cn=tools.paws-public,ou=servicegroups,dc=wikimedia,dc=org
memberOf: cn=tools.help,ou=servicegroups,dc=wikimedia,dc=org
memberOf: cn=toolsbeta.test4,ou=servicegroups,dc=wikimedia,dc=org
memberOf: cn=tools.wikistream,ou=servicegroups,dc=wikimedia,dc=org
memberOf: cn=tools.xn--dk8hv9g,ou=servicegroups,dc=wikimedia,dc=org
memberOf: cn=tools.toolhub,ou=servicegroups,dc=wikimedia,dc=org
memberOf: cn=tools.doodle,ou=servicegroups,dc=wikimedia,dc=org
memberOf: cn=tools.unicornify,ou=servicegroups,dc=wikimedia,dc=org
memberOf: cn=tools.example,ou=servicegroups,dc=wikimedia,dc=org
memberOf: cn=tools.pygments,ou=servicegroups,dc=wikimedia,dc=org
memberOf: cn=project-download,ou=groups,dc=wikimedia,dc=org
memberOf: cn=project-striker,ou=groups,dc=wikimedia,dc=org
memberOf: cn=tools.jouncebot,ou=servicegroups,dc=wikimedia,dc=org
memberOf: cn=tools.replag,ou=servicegroups,dc=wikimedia,dc=org
memberOf: cn=tools.vpsalertmanager,ou=servicegroups,dc=wikimedia,dc=org
memberOf: cn=tools.toolhub-gadget-demo,ou=servicegroups,dc=wikimedia,dc=org
memberOf: cn=tools.os-deprecation,ou=servicegroups,dc=wikimedia,dc=org
memberOf: cn=tools.toolinfo-scraper,ou=servicegroups,dc=wikimedia,dc=org
memberOf: cn=project-toolhub,ou=groups,dc=wikimedia,dc=org
memberOf: cn=project-devportal,ou=groups,dc=wikimedia,dc=org
memberOf: cn=tools.cdnjs-beta,ou=servicegroups,dc=wikimedia,dc=org
memberOf: cn=tools.officewikibot,ou=servicegroups,dc=wikimedia,dc=org
memberOf: cn=tools.convert,ou=servicegroups,dc=wikimedia,dc=org
memberOf: cn=tools.bridgebot,ou=servicegroups,dc=wikimedia,dc=org
memberOf: cn=tools.bd808-ruby,ou=servicegroups,dc=wikimedia,dc=org
memberOf: cn=tools.toolhub-extension-demo,ou=servicegroups,dc=wikimedia,dc=org
memberOf: cn=tools.admin-beta,ou=servicegroups,dc=wikimedia,dc=org
memberOf: cn=tools.stashbot,ou=servicegroups,dc=wikimedia,dc=org
memberOf: cn=tools.sal,ou=servicegroups,dc=wikimedia,dc=org
memberOf: cn=ciadmin,ou=groups,dc=wikimedia,dc=org
memberOf: cn=tools.sal-test,ou=servicegroups,dc=wikimedia,dc=org
memberOf: cn=tools.docker-ratelimit,ou=servicegroups,dc=wikimedia,dc=org
memberOf: cn=tools.mwdemo,ou=servicegroups,dc=wikimedia,dc=org
memberOf: cn=tools.wp-trending,ou=servicegroups,dc=wikimedia,dc=org
memberOf: cn=tools.python-toolforge,ou=servicegroups,dc=wikimedia,dc=org
memberOf: cn=tools.bd808-buildpack,ou=servicegroups,dc=wikimedia,dc=org
memberOf: cn=tools.k8s-status,ou=servicegroups,dc=wikimedia,dc=org
memberOf: cn=tools.ifttt-dev,ou=servicegroups,dc=wikimedia,dc=org
memberOf: cn=tools.ifttt,ou=servicegroups,dc=wikimedia,dc=org
memberOf: cn=tools.bd808-test2,ou=servicegroups,dc=wikimedia,dc=org
memberOf: cn=tools.extloc,ou=servicegroups,dc=wikimedia,dc=org
memberOf: cn=tools.toolviews,ou=servicegroups,dc=wikimedia,dc=org
memberOf: cn=project-admin,ou=groups,dc=wikimedia,dc=org
memberOf: cn=project-admin-monitoring,ou=groups,dc=wikimedia,dc=org
memberOf: cn=project-clouddb-services,ou=groups,dc=wikimedia,dc=org
memberOf: cn=project-cloudinfra,ou=groups,dc=wikimedia,dc=org
memberOf: cn=project-metricsinfra,ou=groups,dc=wikimedia,dc=org
memberOf: cn=project-paws,ou=groups,dc=wikimedia,dc=org
memberOf: cn=project-project-proxy,ou=groups,dc=wikimedia,dc=org
memberOf: cn=project-quarry,ou=groups,dc=wikimedia,dc=org
memberOf: cn=project-redirects,ou=groups,dc=wikimedia,dc=org
memberOf: cn=project-testlabs,ou=groups,dc=wikimedia,dc=org
memberOf: cn=project-toolsbeta,ou=groups,dc=wikimedia,dc=org
memberOf: cn=project-wmflabsdotorg,ou=groups,dc=wikimedia,dc=org
memberOf: cn=tools.automated-toolforge-tests,ou=servicegroups,dc=wikimedia,dc=org
memberOf: cn=tools.fourohfour,ou=servicegroups,dc=wikimedia,dc=org
memberOf: cn=tools.jobs,ou=servicegroups,dc=wikimedia,dc=org
memberOf: cn=tools.openstack-browser,ou=servicegroups,dc=wikimedia,dc=org
memberOf: cn=tools.openstack-browser-dev,ou=servicegroups,dc=wikimedia,dc=org
memberOf: cn=tools.quarry,ou=servicegroups,dc=wikimedia,dc=org
memberOf: cn=tools.security,ou=servicegroups,dc=wikimedia,dc=org
memberOf: cn=tools.toolschecker,ou=servicegroups,dc=wikimedia,dc=org
memberOf: cn=toolsbeta.fourohfour,ou=servicegroups,dc=wikimedia,dc=org
memberOf: cn=toolsbeta.test3,ou=servicegroups,dc=wikimedia,dc=org
memberOf: cn=tools.iw-beta,ou=servicegroups,dc=wikimedia,dc=org
memberOf: cn=tools.admin,ou=servicegroups,dc=wikimedia,dc=org
memberOf: cn=project-bastion,ou=groups,dc=wikimedia,dc=org
memberOf: cn=wmf,ou=groups,dc=wikimedia,dc=org
memberOf: cn=project-tools,ou=groups,dc=wikimedia,dc=org
memberOf: cn=tools.xn--9s9h,ou=servicegroups,dc=wikimedia,dc=org
memberOf: cn=tools.gitlab-account-approval,ou=servicegroups,dc=wikimedia,dc=org
memberOf: cn=toolsbeta.admin,ou=servicegroups,dc=wikimedia,dc=org
memberOf: cn=tools.wikibugs,ou=servicegroups,dc=wikimedia,dc=org
memberOf: cn=tools.wikibugs-testing,ou=servicegroups,dc=wikimedia,dc=org
memberOf: cn=tools.github-ratelimit,ou=servicegroups,dc=wikimedia,dc=org
memberOf: cn=tools.gitlab-webhooks,ou=servicegroups,dc=wikimedia,dc=org
memberOf: cn=toolsbeta.gitlab-webhooks-beta,ou=servicegroups,dc=wikimedia,dc=org
memberOf: cn=tools.schedule-deployment,ou=servicegroups,dc=wikimedia,dc=org
memberOf: cn=tools.containers,ou=servicegroups,dc=wikimedia,dc=org
memberOf: cn=tools.disabled-tools,ou=servicegroups,dc=wikimedia,dc=org
memberOf: cn=tools.ircservserv,ou=servicegroups,dc=wikimedia,dc=org
memberOf: cn=project-deployment-prep,ou=groups,dc=wikimedia,dc=org
memberOf: cn=project-675d02a3344846919fd7fdee700b53a2,ou=groups,dc=wikimedia,dc=org
memberOf: cn=tools.jenkins-build-stats,ou=servicegroups,dc=wikimedia,dc=org
memberOf: cn=tools.toolforge-standards-committee,ou=servicegroups,dc=wikimedia,dc=org

In theory things that are using the LDAP directory for authentication either directly or indirectly via idp.wikimedia.org can also use the cn=project-*,ou=groups,dc=wikimedia,dc=org groups for authorization. In practice I am not sure that this actually happens, but I also don't really know how to check all IDP consumers.

git grep 'cn=project-' in operations/puppet.git hits on a number of python scripts:

• modules/openldap/files/offboard-user.py
• modules/profile/files/toolforge/checker/toolschecker.py
• modules/profile/files/wmcs/services/maintain_dbusers/maintain_dbusers.py

profile::wmcs::metricsinfra::alertmanager::karma shows that they are also used in /etc/prometheus-configurator/config.d/karma_acls_config.yaml.

I still haven't decided what to do about this.

https://gerrit.wikimedia.org/r/c/labs/tools/stashbot/+/1093997 will resolve the specific stashbot issue but doesn't affect what's stored in ldap.

Right now a VM's fqdn is made out of the project ID at build time, then once the VMs are running that fqdn is broken down to produce the project ID for use on VMs for puppet and also to key into ldap. So if we want human-readable ldap we need to change at least the following things to use project name rather than project id:

• ldap groups
• fqdns
• ldap/sudo integration
• everything project-specific that happens in puppet (e.g. hiera lookups)
• all the horizon UIs that make things that are used by any of the above (again, hiera, enc, sudo rules)

and we'd also need to retcon or rebuild the VMs and projects that are currently using IDs. That's a total of 13 VMs at the moment.

What I expected to do, before this task existed, was leave IDs as the internal reference and convert IDs to names for user-facing interactions. That's (for example) what the above gerrit patch does for stashbot.

The idea of considering the raw ldap data itself as user-facing is throwing me for a loop. Could we address that issue partially by amending the ldap schema to include the human-readable project name in the cn=project- entries?

In T274268: Wind down use of project ID and project name equivalency in OpenStack, @Bstorm wrote:

This should include trying to make it so that we can still use project names and not have to manually look up ids, hopefully.

I think this the core disagreement/source of friction. The change to use OpenStack native UUIDs for project ids is fine, but not at the same time changing user facing information, including both FQDNs and LDAP groups, to use project names instead of the formerly equivalent project id is causing friction and toil for humans.

In T379030#10553421, @Andrew wrote:

The idea of considering the raw ldap data itself as user-facing is throwing me for a loop. Could we address that issue partially by amending the ldap schema to include the human-readable project name in the cn=project- entries?

I would ask for that inclusion to be making the group names cn=project-{name} rather than cn=project-{id}-{name} or cn=project-{name}-{id} because there is no currently described benefit of showing the id to anyone outside of OpenStack database internals and comprehensive reporting.

The URL structure for https://openstack-browser.toolforge.org/project/675d02a3344846919fd7fdee700b53a2 is a similar mistake that we should correct.

In T379030#10322740, @Andrew wrote:

The only other (mild) concern I have is with format of project names, but that's probably imaginary... I imagine that as long as we constrain project name to valid dns domain names that will also restrict it to valid ldap keys? If so, think we can go forward with Arturo's plan although it still strikes me as moderately incorrect :p

A cn is per schema a DirectoryString. A DirectoryString attribute is "a string of one or more arbitrary characters from the Universal Character Set (UCS)." This functionally means that a cn is a non-empty UTF-8 string. It will hold whatever weirdness you want. And as you imply we currently want these strings to also be valid DNS labels which means they will be 1-63 ASCII characters taken from the set [a-zA-Z0-9-].

Server fqdn is a bit of a puzzle. Right now the fqdn is generated by cloud-init, based on openstack metadata. The openstack metadata provides projct ID but not project name.

So, options are...

1. write a custom dynamic metadata service to provide the project name. (I have written a dynamic metadata service once before but it was impossible to keep stable so I decomissioned it. Could try again.)
2. provide openstack credentials to VMs on startup (via static metadata) so that the instance can query openstack and look up its own project name. That should be possible, and the 'observer' creds are already public. Unfortunately the lookup can only happen after cloud-init completes; I'm not if that will break other existing cloud-init setups.
3. go back to hacking keystone so that project_id==project_name
4. leave things as they are

#2 is probably the best option here, but it'll be ugly :(

In T379030#10560684, @Andrew wrote:

1. write a custom dynamic metadata service to provide the project name. (I have written a dynamic metadata service once before but it was impossible to keep stable so I decomissioned it. Could try again.)

https://docs.openstack.org/nova/2024.2/admin/vendordata.html#dynamicjson makes it sound relatively boring to add a microservice that responds with custom metadata, but I have never messed with things at this level of OpenStack internals. Was that "respond to a POST with a JSON blob" what you found practical challenges with before, or is this maybe a different way to inject data into the cloud-init system?

In T379030#10560835, @bd808 wrote:

https://docs.openstack.org/nova/2024.2/admin/vendordata.html#dynamicjson makes it sound relatively boring to add a microservice that responds with custom metadata, but I have never messed with things at this level of OpenStack internals. Was that "respond to a POST with a JSON blob" what you found practical challenges with before, or is this maybe a different way to inject data into the cloud-init system?

Yes, the microservice approach was what I found unreliable. I don't remember the specifics as it was years ago, only that it never worked right and felt extremely jerry-rigged so I dropped it.

There is more background than you might like about extra vendor data vs. cloud-init here https://bugs.launchpad.net/cloud-init/+bug/1841104 and here https://github.com/canonical/cloud-init/issues/5221 -- the summary is that it seems to mostly still not work very well, if at all.

Change #1120683 had a related patch set uploaded (by Andrew Bogott; author: Andrew Bogott):

[operations/puppet@production] vendordata.txt: include rudimentary clouds.yaml in initial VM

https://gerrit.wikimedia.org/r/1120683

Change #1120684 had a related patch set uploaded (by Andrew Bogott; author: Andrew Bogott):

[operations/puppet@production] nova vendordata: set fqdn from project_name rather than project_id

https://gerrit.wikimedia.org/r/1120684

The above two patches implement option #2. They leave /etc/hosts subsequently unmanaged which is a bit fragile but probably OK.

In T379030#10560867, @Andrew wrote:

Yes, the microservice approach was what I found unreliable.

Ack. It does look like there has been some amount of confusion historically between the OpenStack folks and the cloud-init folks on the spec for the data. Thank you for giving me that extra context.

Change #1121344 had a related patch set uploaded (by Andrew Bogott; author: Andrew Bogott):

[operations/puppet@production] validatecloudvpsfqdn.py: Support projects with project_name in fqdn

https://gerrit.wikimedia.org/r/1121344

Change #1121345 had a related patch set uploaded (by Andrew Bogott; author: Andrew Bogott):

[operations/puppet@production] wmfkeystonehooks: use project name instead of project id for ldap key

https://gerrit.wikimedia.org/r/1121345

Change #1121346 had a related patch set uploaded (by Andrew Bogott; author: Andrew Bogott):

[operations/puppet@production] Add wmcs_project_id custom fact and handling in realm

https://gerrit.wikimedia.org/r/1121346

aborrero opened https://gitlab.wikimedia.org/repos/cloud/toolforge/volume-admission/-/merge_requests/24

volumes: mount /etc/openstack/clouds.yaml

aborrero opened https://gitlab.wikimedia.org/repos/cloud/toolforge/lima-kilo/-/merge_requests/228

toolforge: introduce /etc/openstack/clouds.yaml

Change #1121369 had a related patch set uploaded (by Andrew Bogott; author: Andrew Bogott):

[operations/puppet@production] cloud-vps instance: populate /etc/openstack/project_id

https://gerrit.wikimedia.org/r/1121369

Here is the list of VMs currently running with project IDs in the fqdn/puppet cert:

664e04b3-34b7-4b87-9296-bfb1cbb5f872 beta
b2c96f28-03bb-4ff4-ab83-f79cc943d31d qlever1
57d26b02-dcb3-447e-b2eb-ac667852932c puppetmaster
fa6f811e-e18e-4f6c-bbc2-5cdf464bad1d ci-components
5e5c3bad-f1c7-49e5-b846-edaf111af83c ci2
60ffad65-edf9-4edd-b531-cfc030dde6ab k3s-node00
6799679b-ac3f-4e07-8bd4-8c23640a5886 k3s-node01
241f3730-4e8d-48c9-90f7-8c67e271d273 crux-metrics
eb6f591f-69b2-4f16-a7eb-3740b5fd074f db
b54d0173-24e7-426a-97db-7d9ca9489009 queue
c2f528c1-6668-4619-91f9-b6ef53d3d8a3 wikipeoplestats-prod01
6356dbde-37f8-40fa-ae3e-26c5913702be primary-app
01ffdbd4-eb45-45d2-8dc2-6d667d4b456d defectdojo

Change #1121369 merged by Andrew Bogott:

[operations/puppet@production] cloud-vps instance: populate /etc/openstack/project_id

https://gerrit.wikimedia.org/r/1121369

Change #1121423 had a related patch set uploaded (by Andrew Bogott; author: Andrew Bogott):

[operations/puppet@production] validatecloudvpsfqdn.py: Only support projects with project_name in fqdn

https://gerrit.wikimedia.org/r/1121423

Change #1120683 merged by Andrew Bogott:

[operations/puppet@production] vendordata.txt: include rudimentary clouds.yaml in initial VM

https://gerrit.wikimedia.org/r/1120683

Change #1121346 merged by Andrew Bogott:

[operations/puppet@production] Add wmcs_project_id custom fact and handling in realm

https://gerrit.wikimedia.org/r/1121346

Change #1121344 merged by Andrew Bogott:

[operations/puppet@production] validatecloudvpsfqdn.py: Support projects with project_name in fqdn

https://gerrit.wikimedia.org/r/1121344

Change #1121691 had a related patch set uploaded (by Andrew Bogott; author: Andrew Bogott):

[operations/puppet@production] openstack_project_id.rb: discard trailing newline

https://gerrit.wikimedia.org/r/1121691

Change #1121691 merged by Andrew Bogott:

[operations/puppet@production] openstack_project_id.rb: discard trailing newline

https://gerrit.wikimedia.org/r/1121691

Change #1121765 had a related patch set uploaded (by Andrew Bogott; author: Andrew Bogott):

[openstack/horizon/wmf-puppet-dashboard@main] Use project name (not tenant_id) for labels and VM fqdn.

https://gerrit.wikimedia.org/r/1121765

Change #1121765 merged by Andrew Bogott:

[openstack/horizon/wmf-puppet-dashboard@main] Use project name (not tenant_id) for labels and VM fqdn.

https://gerrit.wikimedia.org/r/1121765

Change #1120684 merged by Andrew Bogott:

[operations/puppet@production] nova vendordata: set fqdn from project_name rather than project_id

https://gerrit.wikimedia.org/r/1120684

Change #1121776 had a related patch set uploaded (by Andrew Bogott; author: Andrew Bogott):

[operations/puppet@production] Upgrade eqiad1 horizon release

https://gerrit.wikimedia.org/r/1121776

Change #1121776 merged by Andrew Bogott:

[operations/puppet@production] Upgrade eqiad1 horizon release

https://gerrit.wikimedia.org/r/1121776

Change #1121345 merged by Andrew Bogott:

[operations/puppet@production] wmfkeystonehooks: use project name instead of project id for ldap key

https://gerrit.wikimedia.org/r/1121345

Change #1121777 had a related patch set uploaded (by Andrew Bogott; author: Andrew Bogott):

[operations/puppet@production] puppet_enc: lookup roles using project_id in the url

https://gerrit.wikimedia.org/r/1121777

Change #1121777 merged by Andrew Bogott:

[operations/puppet@production] puppet_enc: lookup roles using project_id in the url

https://gerrit.wikimedia.org/r/1121777

Change #1121778 had a related patch set uploaded (by Andrew Bogott; author: Andrew Bogott):

[openstack/horizon/wmf-sudo-dashboard@main] Try to clarify the distinction between project name and id

https://gerrit.wikimedia.org/r/1121778

Change #1121778 merged by Andrew Bogott:

[openstack/horizon/wmf-sudo-dashboard@main] Try to clarify the distinction between project name and id

https://gerrit.wikimedia.org/r/1121778

Change #1121785 had a related patch set uploaded (by Andrew Bogott; author: Andrew Bogott):

[operations/puppet@production] Update horizon version on eqiad1

https://gerrit.wikimedia.org/r/1121785

I believe this to be resolved now. I adjusted the fqdn of the VMs listed above, and hand-edited ldap to use project names instead of ids anywhere where it was previously using IDs. I need to do a couple of tests for new projects, but here are the notes from what I've done so far:

now:

- VM fqdns are hostname.<project id>.domain

- $wmcs_project is project id, determined from puppet certname

- project name unknown to puppet

- ldap keys are project-<project id>

- puppet enc lookups are based on project id ($wmcs_project) and fqdm


after:

- VM fqdns are hostname.<project name>.domain

https://gerrit.wikimedia.org/r/c/operations/puppet/+/1120684

- $wmcs_project is project name, determined from puppet certname

- ldap keys are project-<project name>

https://gerrit.wikimedia.org/r/c/operations/puppet/+/1121345/3

TODO: hand edit existing ldap entries that are uuid-based

- project_id available (but untrusted) via /etc/openstack/project_id and $wmcs_project_id

https://gerrit.wikimedia.org/r/c/operations/puppet/+/1121369/3

https://gerrit.wikimedia.org/r/c/operations/puppet/+/1121346

- puppet enc lookups are based on project id ($wmcs_project_id) and fqdn

https://gerrit.wikimedia.org/r/c/operations/puppet/+/1121347/3

- update all fqdns in the enc database to replace project IDs with names



TODO: what about existing VMs with uuid ID in fqdn?

- update /var/lib/cloud/instances/873c0e0d-945b-4f68-946d-9102276aaac0/obj.pkl to have manage_etc_hosts false

- sed -i "s/<projectid>/<projectname>/g" /etc/hosts

Change #1121785 merged by Andrew Bogott:

[operations/puppet@production] Update horizon version on eqiad1

https://gerrit.wikimedia.org/r/1121785

Change #1121423 merged by Andrew Bogott:

[operations/puppet@production] validatecloudvpsfqdn.py: Only support projects with project_name in fqdn

https://gerrit.wikimedia.org/r/1121423

fnegri closed https://gitlab.wikimedia.org/repos/cloud/toolforge/lima-kilo/-/merge_requests/228

toolforge: introduce /etc/openstack/clouds.yaml

dcaro closed https://gitlab.wikimedia.org/repos/cloud/toolforge/volume-admission/-/merge_requests/24

volumes: mount /etc/openstack/clouds.yaml

Change #1090854 abandoned by Andrew Bogott:

[operations/puppet@production] openstack: wmfkeystonehooks: create LDAP groups with project name

Reason:

addressed elsewhere

https://gerrit.wikimedia.org/r/1090854