In T358496#9799629, @dcaro wrote:

In T358496#9796170, @Andrew wrote:

In T358496#9790877, @dcaro wrote:

This all seems correct, although I reiterate that the interesting part is the scope creation or management. We don't currently have a good way to create databases that are explicitly tied to a particular tool -- even with the bespoke/by hand approach we take now there's nothing to keep the tool and trove ACLS in sync.

Can you elaborate on this?

The current database-for-a-tool solution is that we've created trove-only openstack projects to manage databases used by toolforge tools. Those projects may or may not have the same members as the tool that project is supporting; it's entirely ad-hoc. If we have a model where a tool corresponds directly to an openstack tenant then we can put the trove DBs in there and have consistent access and membership between tools and trove.

Interesting, I thought that the openstack projects created for trove were already mapped to a tool (and the auth was through ldap matching the user to that tool that then matches the openstack tenant).
Just to verify, as of today, the trove database for a tool, is in an arbitrary openstack project that is managed by whomever requested to create the database right? (so completely independent of the LDAP tool group)

That's correct. Of course the owner of the db project can manage project access, so ideally they keep things in sync manually.

In T358496#9793278, @Slst2020 wrote:

Here, we are only talking about allowing tools to set up and use s3-style buckets, correct?

In T358496#9790877, @dcaro wrote:

This all seems correct, although I reiterate that the interesting part is the scope creation or management. We don't currently have a good way to create databases that are explicitly tied to a particular tool -- even with the bespoke/by hand approach we take now there's nothing to keep the tool and trove ACLS in sync.

Can you elaborate on this?

In T358496#9790100, @dcaro wrote:

<snip>

I'm still thinking on use the cases

This task is specifically tackling this one:

• As a tool, I want to be able to access the s3 buckets I created (from horizon) from within toolforge

I'm now learned that new prod puppservers also use the 'gitpuppet' user. So eliminating that user will increase the diff with prod rather than shrink it. So that's not the right path forward. Probably I should just figure out a fix for case 1.

Rivers change course, civilizations rise and fall, and I have finally done some work on this task.

Reimaging cloudcontrol2006-dev works now, thanks!

I'm closing this as invalid since those hosts have come and gone :)

In T358496#9782046, @fnegri wrote:

The per-tool service user would not have a password, only app credentials.

If I understand it correctly, this might be entirely impossible in the current Keystone model (it could be a bug or intended behaviour). Is it a problem though to have a password associated with the service tool user, if that password is stored securely and only available to the agent that uses it to create the associated app credentials?

I'm hitting a roadblock with the service user plan -- because of keystone's belt-and-suspenders approach to security, I can override the policy to allow an admin user to create app creds for another user (e.g. novaadmin creating creds for tool.mytool) but there's an explicit check in the code comparing context ID to cred ID and erroring out. IMO this is a keystone bug (https://launchpad.net/bugs/2065212) but it's unlikely to be changed upstream anytime soon.

So many questions!

My favorite option is 'Automatic creation of per-tool keystone project'. Since that's a simple extension of 'On-demand creation of per-tool keystone project' I'm going to start with that (with a cli tool rather than an API endpoint for now).

I'm striking out the 'keystone projects in ldap' option because keystone doesn't really support that one.

I think this is now cleaned up and resolved for now. In the future, I suspect that deleting canary VMs before deleting hypervisors will prevent them from showing up here, but openstack resource provider delete might be needed.

Ok, I think I found them! These deleted hosts can be cleaned up with

Removing hardware records from the DB seems a little bit dangerous as that could leave dangling references elsewhere (for instance in the action log which keeps track of any previous actions a VM took, including a reference to where the VM was at the time.)

Inasmuch as Trove works for this, the integration is also working.

I (Andrew) am accepting this task to investigate deprecation warnings in these services and (probably) take over maintenance of python-flask-keystone.

From a meeting about these services today:

The toolforge exim server is using an experimental feature to support forwarding to gmail. That build is here: https://gitlab.wikimedia.org/repos/sre/exim4-arc -- it will likely become part of the main exim build soon.

I've built a new puppetserver in this project, wdqspuppetserver-1. Nothing was using the old one so probably this effort was in vain.

It is safe to reimage cloudbackup1003 on April 30.

Everything seems happy now. Thanks!

This is taavi messing with ovs

In T361804#9734320, @dcaro wrote:

In T361804#9727897, @Andrew wrote:

I'm late to this, but I also agree with B5. Do we need another rule regarding what to do with existing code when applying new checks or standards and would result in a reformat?

I would say no, specially as this decision does not force anyone to change anything.

A rule of the thumb though would be to separate those changes in it's own MR (if the changes are big), or in a first commit inside the MR (if they are small).

In T363125#9734178, @taavi wrote:

B: Fishbowl wiki hosted on wikikube, accounts in ldap. This option could be a final state OR a temporary state on the way to the SUL option.
Con (long-term): Allowing r/w ldap access from wikitech/wikikube may continue to introduce surprising edge cases for product maintenance.

I don't think Wikitech will require r/w access after T359544: Disable SSH key management on Wikitech is done.

...I just checked and Bobcat is still using greenlet 2.0.2 so this is likely not fixed in bobcat :(

I think this is the same issue (but different log message) as T352635

I've been seeing this crash periodically since we upgraded to A -- if this is the same failure then believe this is a bug in the python threading library that we're using and the full queue is a symptom of a stuck listener.

I'm late to this, but I also agree with B5. Do we need another rule regarding what to do with existing code when applying new checks or standards and would result in a reformat?

codfw1dev is now running bobcat. The only (minor) issue I'm aware of so far is T350807

Coincidentally, I just did a dist-upgrade that pulled in this new package. The 0.14 package installs its binary here:

These are now in service and working fine.

puppetserver is upgraded but everything in this project is Buster so puppet 7 will be unhappy until that's fixed.

This project was managed by jbond -- for now I will do this upgrade.

10:28 AM taavi, jhathaway, moritzm, is the puppet-dev project effectively defunct now that jbond has departed? It's unmarked on the purge page and also has https://phabricator.wikimedia.org/T361593 with no response
10:29 AM
<moritzm> Moritz Mühlenhoff let me have a look
10:31 AM I haven't used it for ages and I think it was mostly used to stage/test the puppet 7. from my PoV it can be phased out unless Jesse or Taavi still use it
10:32 AM <andrewbogott> Andrew Bogott ok, thanks moritzm, let's see if anyone else has an opinion :)
10:34 AM
<jhathaway> Jesse Hathaway I agree with moritzm, I would like to keep the project around, but the instances can be removed
10:35 AM <andrewbogott> Andrew Bogott great, shall I delete things right now?
10:37 AM
<jhathaway> Jesse Hathaway fine by me, unless taavi objects
10:37 AM
<jbond> John Bond ftw also good with me
10:37 AM
<taavi> Taavi Väänänen no objections from me

In T362438#9718112, @Jhancock.wm wrote:

what are we doing with cloudbackup2001-array1 and cloudbackup2002-array1?

The latest puppetserver code is prone to gobbling RAM; I'd check for oom messages and see about using profile::puppetserver::java_max_mem

It's not really a buster thing -- the puppet code for geoip is entirely different in the puppetserver manifests vs. the old puppetmaster manifests.