I've built a new puppetserver in this project, wdqspuppetserver-1. Nothing was using the old one so probably this effort was in vain.

It is safe to reimage cloudbackup1003 on April 30.

Everything seems happy now. Thanks!

This is taavi messing with ovs

In T361804#9734320, @dcaro wrote:

In T361804#9727897, @Andrew wrote:

I'm late to this, but I also agree with B5. Do we need another rule regarding what to do with existing code when applying new checks or standards and would result in a reformat?

I would say no, specially as this decision does not force anyone to change anything.

A rule of the thumb though would be to separate those changes in it's own MR (if the changes are big), or in a first commit inside the MR (if they are small).

In T363125#9734178, @taavi wrote:

B: Fishbowl wiki hosted on wikikube, accounts in ldap. This option could be a final state OR a temporary state on the way to the SUL option.
Con (long-term): Allowing r/w ldap access from wikitech/wikikube may continue to introduce surprising edge cases for product maintenance.

I don't think Wikitech will require r/w access after T359544: Disable SSH key management on Wikitech is done.

...I just checked and Bobcat is still using greenlet 2.0.2 so this is likely not fixed in bobcat :(

I think this is the same issue (but different log message) as T352635

I've been seeing this crash periodically since we upgraded to A -- if this is the same failure then believe this is a bug in the python threading library that we're using and the full queue is a symptom of a stuck listener.

I'm late to this, but I also agree with B5. Do we need another rule regarding what to do with existing code when applying new checks or standards and would result in a reformat?

codfw1dev is now running bobcat. The only (minor) issue I'm aware of so far is T350807

Coincidentally, I just did a dist-upgrade that pulled in this new package. The 0.14 package installs its binary here:

These are now in service and working fine.

puppetserver is upgraded but everything in this project is Buster so puppet 7 will be unhappy until that's fixed.

This project was managed by jbond -- for now I will do this upgrade.

10:28 AM taavi, jhathaway, moritzm, is the puppet-dev project effectively defunct now that jbond has departed? It's unmarked on the purge page and also has https://phabricator.wikimedia.org/T361593 with no response
10:29 AM
<moritzm> Moritz Mühlenhoff let me have a look
10:31 AM I haven't used it for ages and I think it was mostly used to stage/test the puppet 7. from my PoV it can be phased out unless Jesse or Taavi still use it
10:32 AM <andrewbogott> Andrew Bogott ok, thanks moritzm, let's see if anyone else has an opinion :)
10:34 AM
<jhathaway> Jesse Hathaway I agree with moritzm, I would like to keep the project around, but the instances can be removed
10:35 AM <andrewbogott> Andrew Bogott great, shall I delete things right now?
10:37 AM
<jhathaway> Jesse Hathaway fine by me, unless taavi objects
10:37 AM
<jbond> John Bond ftw also good with me
10:37 AM
<taavi> Taavi Väänänen no objections from me

In T362438#9718112, @Jhancock.wm wrote:

what are we doing with cloudbackup2001-array1 and cloudbackup2002-array1?

The latest puppetserver code is prone to gobbling RAM; I'd check for oom messages and see about using profile::puppetserver::java_max_mem

It's not really a buster thing -- the puppet code for geoip is entirely different in the puppetserver manifests vs. the old puppetmaster manifests.

etcd nodes are now all bullseye. Moving them to bookworm will require more work; the bookworm nodes seem to cluster properly but they return 404s to normal api requests like 'member list'

Fair enough. Do you have any preference on the name? cloudidm2001.codfw.wmnet, e.g.?

In T347432#9706520, @taavi wrote:

I think this is moot at this point.

I'd prefer that it go on its own ganeti VM, as I'm trying to pare down on the total number if weird things that run on cloudservices.

Oops, 'puppetserver ca' is still not working on the puppet server (or perhaps it was working but broke again):

Yeah, the first thing to test here is reverting dbf47bd5a4100b96741fae320e7e0326e72880dc and see if that changes this.

That's correct yeah, I don't think there are security implications. What I'm after is the possibility to upload/config a keypair once and reuse that across instances launch, which at the time I couldn't find a way to do. Perhaps the underlying APIs do support it: once a named keypair has been used once then it can be reused for subsequent launches? That'd be enough for me

<long pause> Thanks for the feedback!

nope, thanks!

So puppet is removing eject, and since cloud-init requires it, puppet also removes it. that is not how I expect dependencies to work!

I started with with a VM from a raw debian image, and cloud-init is installed and configured:

I added catalyst-qte.wmcloud.org to the domain config by following Taavi's instructions. I have not tested creating/accessing the proxy but the option does appear in the UI.

I'm sure there's a perfectly reasonable, linear path to getting this fixed but it's going to have to wait until I get some sleep, or fresh eyes intervene.

This is my favorite kind of joke:

So I have at least two questions: