In T411516#11426742, @Catrope wrote:

Hmm now that I'm thinking about it a little more... do we run mergeMessageFileList for each wmf.N branch separately? If we run it only once and reuse the result across both branches, that would be a problem if an extension was missing in one branch but present in the other.

Changes deployed via scap 4.229.0

I'm currently blocked from updating my train-dev development environment because I can't clone https://phabricator.wikimedia.org/diffusion/NLSP/new-lexeme-special-page.git from offsite.

In T410018#11417894, @A_smart_kitten wrote:

In T410018#11374839, @dancy wrote:

Does anyone know where https://noc.wikimedia.org sources its data?

I may be wrong but judging by https://github.com/wikimedia/operations-mediawiki-config/blob/a0f366b8b706c963ed9badac8ebe7a074a292549/src/Noc/utils.php#L9-L18, I believe it might read the wikiversions.json file that's local to the server its running on.

In T371620#11420555, @MatthewVernon wrote:

@dancy it's been a while now, but I think we just moved trafficserver to specify tags: [wmcs] for all its jobs and that worked around the issue.

In T410049#11420530, @dduvall wrote:

@dancy Not quite. We still need the WMCS/trusted runner changes in puppet.

@dduvall Is this ticket effectively resolved?

@MatthewVernon Can you give me an update on the status of your gitlab builds? Have you managed to work around the original problems?

I didn't get an email notification this morning so I checked the host and it looks like puppet is working now.

Thanks @jnuche !

@jnuche Any ideas about this?

In T409855#11393299, @sbassett wrote:

I think they're fine. I think it's more about just changing the way someone has done something for 7 or 8 years.

In T409855#11371578, @sbassett wrote:

In T409855#11370703, @Lucas_Werkmeister_WMDE wrote:

So what is the officially sanctioned or maintained way to deploy security patches, then?

I don't think there is one, but most of us on the Security-Team that deploy security patches typically do so manually.

In T405921#11384564, @JVanderhoop-WMF wrote:

@dancy I don't believe this relates to Metrics Platform. Could I ask why you had thought it did?

In T396166#10952265, @CodeReviewBot wrote:

dancy merged https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/864

Drop php_fpm_restart_script support

Noting for the record that I had prior authorization from Mónica to delete the monica_wmde account.

I'll take this one.

<hashar> my guess is scap thinks that because it reads the local /srv/mediawiki-staging/wikiversions.json

https://versions.toolforge.org/ correctly showed still old 1.46.0-wmf.1 for group2 instead of 1.46.0-wmf.2.

Fix deployed in scap 4.227.0.

In T397931#11344804, @Lucas_Werkmeister_WMDE wrote:

Actually, that SpiderPig deployment is still confusing to me. The change it’s complaining about was merged last Monday, and made it into last week’s train (1.45.0-wmf.25), which by now is fully deployed (we’re now partway through 1.46.0-wmf.1). Why is it complaining about a change missing from wmf.24? Is it “deployable” because it hasn’t been removed from /srv/mediawiki-staging on deploy2002 yet?

Final rule 3b:

• Master branch MW code dependencies MUST be included in all live branches (error if not)
• Master branch MW code dependencies MUST be included in upcoming branches after newest live (error if not)
• When only one live branch exists, master dependencies SHOULD be in the previous deployable branch (warning if not)

In T362987#11349520, @jnuche wrote:

@dancy do you have a take on this?

In T362987#11345387, @jnuche wrote:

Then we can adapt the current logic to make a distinction for this particular case between:

• Production branches: If the change is missing from a version in wikiversions.json -> Error. Same behavior as now
• Non-production, deployable branch: If it's missing from such a branch -> Warning prompt. "Are you sure you want to go ahead? Version xyz is still deployable and if it gets deployed things will break"

Deployed via scap 4.223.0.

In T397931#11344804, @Lucas_Werkmeister_WMDE wrote:

Is it “deployable” because it hasn’t been removed from /srv/mediawiki-staging on deploy2002 yet?

In T397931#11344804, @Lucas_Werkmeister_WMDE wrote:

Actually, that SpiderPig deployment is still confusing to me. The change it’s complaining about was merged last Monday, and made it into last week’s train (1.45.0-wmf.25), which by now is fully deployed (we’re now partway through 1.46.0-wmf.1). Why is it complaining about a change missing from wmf.24? Is it “deployable” because it hasn’t been removed from /srv/mediawiki-staging on deploy2002 yet?

In T397931#11344804 (https://spiderpig.wikimedia.org/jobs/859) we have someone being blocked from backporting a change due to the "all deployable branches" logic.

Error for Change '1197613', project: 'operations/mediawiki-config', branch: 'master': Change '1197613' has dependency '1198084' targeting the master branch of MediaWiki code project 'mediawiki/extensions/Wikibase', but the commit is not present in deployable branch(es): wmf/1.45.0-wmf.24. Master dependencies must be deployed to all active branches.

Also related: T408868: Log deployment activities on auto-detected changes during deployment

In T100575#11335488, @LSobanski wrote:

Is this still relevant in the age of SpiderPig?

This should be fixed in scap 4.221.0.

This should be fixed in scap 4.221.0.

This should be fixed in scap 4.221.0.

This should be fixed in scap 4.221.0.

Implemented and deployed in scap 4.221.0.

Proposal implemented and deployed via scap 4.221.0.

Deployed via scap 4.220.0

Looks like xterm.js is doing this.

Scap 4.219.0 has been deployed with this change.

In T405037#11323834, @dancy wrote:

In T405037#11323817, @thcipriani wrote:

Alternatively, matching what we do with Jenkins, cleaning up jobs older than some date.

This is what I intend to do.

In T405037#11323817, @thcipriani wrote:

Alternatively, matching what we do with Jenkins, cleaning up jobs older than some date.

Resolved in scap 4.219.0

Thanks for the report @Tgr. This is fixed as of scap 4.218.0.

Noting T408525 as a new source of logspam.

In T393895#11313606, @Tgr wrote:

There is a link to the mediawiki-errors dashboard but not to the mwdebug dashboard. (Not objecting to declining it, just FWIW.)

Already exists. Also, I think the "Mediawiki error logs" tab was added to SpiderPig since this ticket was filed.

@jeena Can you make a mock-up of what you would like the Job History section to look like with respect to your suggestion?

The full exception message is:

PHP Warning: API call failed trying to get remote jsontransform result: error={"code":"jsonconfig-transform-missing-data","info":"Missing source data page Sample.tab.","*":"See https://commons.wikimedia.org/w/api.php for API usage. Subscribe to the mediawiki-api-announce mailing list at \u0026lt;https://lists.wikimedia.org/postorius/lists/mediawiki-api-announce.lists.wikimedia.org/\u0026gt; for notice of API deprecations and breaking changes."}, query= [Called from JsonConfig\JCUtils::warn in /srv/mediawiki/php-1.45.0-wmf.24/extensions/JsonConfig/includes/JCUtils.php at line 46]

A batch of these warnings started up again:

Adding a fresh stack trace here for searchability:

In T407996#11300064, @Kgraessle wrote:

@dancy once https://gerrit.wikimedia.org/r/1198138 is merged do we need to back port or will it deploy either way since it's a train blocker?

Thanks @ssingh !

I finished rolling the MediaWiki train to group1 before I noticed this task was added as an Unbreak Now. I will leave the train at group1 unless someone recommends rolling back (or error counts start increasing rapidly).

I asked for help in T404826

https://gerrit.wikimedia.org/r/c/operations/puppet/+/1197986 has caused puppet to break on deployment-cache-upload08.deployment-prep. Please help!

Probably introduced via https://gerrit.wikimedia.org/r/c/operations/puppet/+/1197986

In T402548#11296222, @ABreault-WMF wrote:

@dancy Ok, there hasn't been any occurrences of T407323 in the past half hour since the deploy

In T402548#11295809, @ABreault-WMF wrote:

@Lucas_Werkmeister_WMDE @dancy The stack trace from the logs on mnwwiktionary that I'm seeing look more like T407323 than this one. There's already a patch out for that and the logs were supposed to be suppressed. I can backport the patch in the late window in the next 10 mins or so if you'd like