+1 to @tstarling's proposal.

Generated a list of all $hosts_allow arguments from rsync::server::module invocations across all of Puppet: P9544

Another thing that just came up: not all users of rsync::server::module are actually passing an array to the $hosts_allow argument: https://gerrit.wikimedia.org/r/c/operations/puppet/+/549142
Need to go through PuppetDB and look for this.

There's also some added logspam to be fixed:

After far too many things that were subtly wrong, it works!

In terms of identifying services that use keys issued by the puppet CA -- is it wrong to think that the following would be a complete list?

• keys created using cergen
• users of base::expose_puppet_certs
• the few users we have that are referencing either puppet_ssldir() or manually hardcoding the /var/lib/puppet/ssl directory

Okay, I think this is done and working!

A community member noticed this today; seen in #wikimedia-tech on IRC:

12:24:33	<Nemo_bis>	Since when a limit > 500 no longer works in Special:Contributions? https://it.wikipedia.org/w/index.php?title=Speciale:Contributi&limit=550&target=Nemo+bis

@Reedy that's the wrong place to look nowadays. The two good places to look for production database configs:
https://noc.wikimedia.org/db.php
https://noc.wikimedia.org/dbconfig/eqiad.json

In T236104#5595508, @Jdforrester-WMF wrote:

How reliable is the filemtime function in a scap world? Does scap / rsync always migrate the mtime from the host to the target machine?

@Reedy Can you elaborate? The only rate-limiting I'm aware of is the simultaneous edit limiting provided by PoolCounter, and the limited support for limiting simultaneous requests from the same IP address in Varnish.

The bug was turned into a private issue because of potential DoS concerns: a relatively low volume of these long-date-range limit=5000 Special:Contributions queries (a volume that I could trivially send from my laptop, for instance) create sufficient load to cause a great deal of hurt for our database servers.

This happened yesterday, during firefighting work on {T235949}

something to note/fix for future migrations: one option for how to push DNS changes when gerrit is down https://tools.wmflabs.org/sal/log/AW3wKZvNfYQT6VcDX2pS

Yes, sorry for the delay -- I was out a few days and then some other stuff
took priority. I'll try to get it done by next week.

How hard would it be to make the limit 500 instead of 5000?

Hi Andrew!

An update: as of now, 20.8% of the fleet (or 30% of hosts with software RAID enabled) have redundant bootloaders. This is just from fixing the partman configs and waiting for reimages to happen 'naturally'. That's about all the work I'm going to do on this for the time being; I figure improvements will continue as services need to move to Buster.

In T215183#5496348, @jcrespo wrote:

Buster failed to install on my md install for 2552d12fe15ec1 with "grub install sda sdb failed, cannot install on sda". I cannot be sure it is that change because buster, and because there is weird hybrid software raid + hw raid going on, and it could be a question of using the wrong recipe because of the extra disk, but FYI.

Is this still an issue?

In T234450#5545511, @Catrope wrote:

Turns out, we are correctly capping the limit already!
Note LIMIT 5001 at the end.

Was this discussed during the Monday meeting? What was the outcome?

We probably want to let the new recording rule accumulate some data -- a week's worth? -- and then start using it in the dashboard

@Krinkle @aaron -- could one of you please find some time soon to make sure I'm understanding things correctly as I state them below?

@Marostegui please give me some input on the UI -- what would you like dbctl to call the section that will represent "everything that refers to $wmgOldExtTemplate"? For now I'm thinking cluster1, does that sound good to you? (So you would write something like dbctl instance es1018 depool --section cluster1)

Sorry, I just realized that this won't work as-is; we don't allow outbound HTTP from production, except via a few proxy servers.

Sorry, I just realized that this won't work as-is; we don't allow outbound HTTP from production, except via a few proxy servers.

Yup, that's part of the datasource definition -- a simple access: proxy stanza

Adding datasources via the UI is disabled, as all datsource definitions are Puppetized in our installation.

In T226044#5524171, @greg wrote:

@mmodell @ema After a discussion with TechEng and SRE and the rest of tech-mgt this week in Portland, we are going to hold off on this work (phame as techblog) and instead wait for guidance from TechEng as they have a plan in place and people dedicated to the work. Basically this is going to be rolled into TechEng work on our communication and social plans from a wikimedia tech perspective.

checking the return value of $EDITOR is released but I think there's more work to do here

released

released