Done:

See T385529: Automatically export and publish a list of WMF deployed code repositories in Bitergia's JSON format for some related background.

AFAICT, just the 2 minor ones in the phabricator repo. It's up for testing in devtools.

Created as: https://gitlab.wikimedia.org/repos/projects/user-scripts-data-tooling

PHPUNIT_PARALLEL_GROUP_COUNT=cpus / (executers / 2) (gives us 8 for most instance, 16 for our bigger instance)

As I understand it: gitlab.api_key in the local.json created by each scap deployment is derived from the phabricator.local.gitlab_api_key set in puppet (from a lookup of profile::phabricator::main::gitlab_api_key) and written to /etc/phabricator/config.yaml.

Online and can take over train ops from here.

Going ahead and resolving this, please reopen if anything breaks with those jobs.

After updating analytics-refinery-maven-release, this one succeeded: https://integration.wikimedia.org/ci/view/All/job/analytics-refinery-maven-release/48/console

@brennen: I assume you don't know or remember anything about this, just like me. :)

If editing /etc/phabricator/config.yaml and restarting doesn't change things about the "Local Config" then what else needs to happen for that to sync to the Local Config? (I assume a proper deploy)

Currently stable on all wikis.

Haven't seen this in prod since rolling out backport.

Blocker resolved, logs clean-ish, rolling to all wikis.

Planning to deploy backports prior to rolling train; currently waiting on all-clear for deployment server DC switchover (T399891).

Raising to UBN as train blocker. Can handle backport shortly.

Planning a deploy in the upcoming window.

Planning a deploy in the upcoming window.

Marked this private, assuming it's for high-level data and nothing remotely at the level of PII, I think it fits the rubric. Let us know if you run into any issues.

Will need some investigation. May be Phorge upstream (Phabricator (Upstream)).

There's followup work to be done in improving the setup checks upstream and digging into repo caching, but the proximate problem seems resolved after tuning the APCu cache.

It still shows 128MB there. Let me do some restarts to change that.

Thanks for investigating! usage_ping change LGTM.

With the above mitigation in place, this seems less urgent in terms of web requests, but I'd still like to get to the bottom of it.

Question: Do we still have data if the number of duck-sound=quack requests was roughly the same level before and after our 2025-07-15 15:30UTC deployment in T370266? (If it was the same, that would rule out f2a01dca392.)

BUT, actually running a phpinfo() shows:

Puppet 7 servers never fail to surprise me with how much RAM they want. I'd definitely start by doubling the RAM on that VM before investigating anything else.

After reboot:

Unable to ssh to puppetserver. In horizon log tab for integration-puppetserver-01:

Stable on all wikis.