I believe that the work done in T408740 has resolved this issue.

In T412951#11572476, @elukey wrote:

@dancy @Scott_French I think we are ready to move forward with https://gerrit.wikimedia.org/r/c/operations/puppet/+/1229145, what do you think?

I am ok to proceed with the sync-world and full_image_rebuild during a MediaWiki infrastructure window (could be good event tomorrow's), but it will be during your night so you'll not be able to join (but if needed I can loop in Matthew to check the apus/Ceph side). The alternative is to schedule the upgrade during a MediaWiki infra window happening during your workday, totally fine for me. Lemme know what you prefer!

@Reedy I deployed a change to the kubernetes deployment monitor code in scap. Now it will not give up monitoring when it gets an error from kubectl.

In T412941#11548781, @MLechvien-WMF wrote:

Release-Engineering-Team could you please provide inputs on the Scap specific works in the description?

In T401197#11559575, @jeena wrote:

I have archived the documentation page so now we can remove the chart.

Here's a working .gitlab-ci.yml config using wmcs runners: https://gitlab.wikimedia.org/repos/m3api/tmp-m3api-oauth2/-/merge_requests/1/diffs

@LucasWerkmeister Thanks for setting up https://gitlab.wikimedia.org/repos/m3api/tmp-m3api-oauth2 and adding me as a member. I tried to run the CI pipeline of that repo on the main branch (using the normal D.O. runners) but it fails: https://gitlab.wikimedia.org/repos/m3api/tmp-m3api-oauth2/-/jobs/727347. I need a working pipeline as a starting point for debugging.

In T414273#11551749, @A_smart_kitten wrote:

Thanks @dancy :) FWIW, the issue (for me at least) would be less creating/submitting the actual patch, and more getting together a local environment that I'd be able to test it in...

In T337570#11550214, @bd808 wrote:

In T337570#11549528, @dancy wrote:

If a merge request description body consists of a single Bug: TXXXXX line, the link to Phabricator does not get created. I'm wondering if this is a new problem since the recent Gitlab UI changes. See the job link referenced in the task description for an example (https://gitlab.wikimedia.org/repos/abstract-wiki/wikifunctions/function-orchestrator/-/merge_requests/18).

Phabricator linking seems to be working on https://gitlab.wikimedia.org/toolforge-repos/versions/-/merge_requests/8. I wonder if there is something interesting going on in that wikifunctions MR description? Does the regex that is in use allow for trailing whitespace? Is there a newline requirement?

Deployed via scap 4.235.0.

If a merge request description body consists of a single Bug: TXXXXX line, the link to Phabricator does not get created. I'm wondering if this is a new problem since the recent Gitlab UI changes. See the job link referenced in the task description for an example (https://gitlab.wikimedia.org/repos/abstract-wiki/wikifunctions/function-orchestrator/-/merge_requests/18).

Change deployed via scap 4.234.0

In T412951#11539048, @elukey wrote:

Thanks! When you have a moment let me know if what I wrote above is sound, namely if it is ok to just start clean or not (basically if it is ok not to be able to rollback when we build/deploy the first time after the switch). If it is fine, I think that we can probably choose when to do the test after the SRE summit (that will happen next week). Ideally we could just rebuild all images and try to push/pull them, and see how it goes.

@Clement_Goubert Please use scap lock --unlock-all --yes <unlock reason> (note: --bg is not passed here).

In T414864#11535504, @bd808 wrote:

The runners you would like unblocked are hosted on Digital Ocean. I do not think that it would be reasonable to open the Beta Cluster to the full DO IPv4 address space. If we have a fixed sub-space for egress we can unblock that. @dancy can you help me figure out if there is a restricted range that the runner egress through?

In T414864#11532038, @LucasWerkmeister wrote:

FWIW, I’ve tried to get m3api-oauth2 CI running on the WMCS runners instead (wmcs tag), but so far haven’t managed to get Chrome/Chromium running there yet (latest job).

Thanks for the report @elukey. This sounds very promising!

I confirmed that I did receive an email when I created a beta wiki account.

xref T412975

I ran sudo /usr/local/sbin/clean-stale-puppet-certs --clean on deployment-puppetserver-1.deployment-prep to take care of this.

Thanks for the work on this @thcipriani !

@jeena @thcipriani What's the status of the mediawiki-dev deployment-chart? I found that it is referenced in https://wikitech.wikimedia.org/wiki/Deployment_pipeline/Migration/Tutorial

@Blake I've installed a new release of scap on the deploy servers. You can now use scap lock --all --bg and scap lock --unlock-all to achieve your goal. I recommend testing things out on deployment.eqiad.wmnet well in advance of the datacenter switch. Let me know if you have any issues.

@A_smart_kitten patches are welcome! Let me know if you need help with getting something together.

Potential ideas:

• Drop a lock file on the deployment server that scap detects, remove it at a later step
• Add a switch to scap lock --all that puts the lock on and returns, add a scap unlock --all command

Added

profile::tlsproxy::envoy::upstream_sni: null
profile::tlsproxy::envoy::upstream_tls: false

to devtools project puppet according to @bd808's recommendation.

Looks similar to T414304

I turned off this instance as part of T412975

In T399348#11513384, @vaughnwalters wrote:

@dancy I tried changing the tags here and am getting the same error... maybe I'm still doing something wrong though?

In T412975#11498683, @Dzahn wrote:

Should I abandon or keep https://gerrit.wikimedia.org/r/c/operations/puppet/+/1219180 now?

In T399348#11508642, @vaughnwalters wrote:

In T399348#11507969, @dancy wrote:

After reading the history of this ticket, it's not clear to me if the memory-optimized runners have been tried yet. Can someone try and confirm?

Heya @dancy, this is done by tags: [wmcs] correct (looking at T399348#11007932)? That tag been added here for the orchestrator. So if that tag looks correct, then it has been tried.

After reading the history of this ticket, it's not clear to me if the memory-optimized runners have been tried yet. Can someone try and confirm?

@Andrew has since changed where he performing the build of the wikitech-static container image, so the main cause of this ticket has been resolved.

@Jdforrester-WMF FYI

@thcipriani Making sure you see this.

In T412975#11477379, @LSobanski wrote:

@dancy do you have any clues as to the need for this MX server?

@fnegri wrote in T373815:
For future reference, I suspect there was a failure in the bash command substitution that is used to determine the default branch. Below is the PCC output from my patch:

Presumably caused by https://gerrit.wikimedia.org/r/c/operations/puppet/+/1219137
@Muehlenhoff @Jelto

In T408277#11464912, @hashar wrote:

@dancy 1.46.0-wmf.7 has been rolled to testwikis.

The pretrain Jenkins job failed due to some npm connection issue. I reran it, the image took some efforts to get published and finally made it there. So I think you are all set to continue! 🚆

Here are pretrain image build and push times for the last several days:

In T411516#11426742, @Catrope wrote:

Hmm now that I'm thinking about it a little more... do we run mergeMessageFileList for each wmf.N branch separately? If we run it only once and reuse the result across both branches, that would be a problem if an extension was missing in one branch but present in the other.

Changes deployed via scap 4.229.0

I'm currently blocked from updating my train-dev development environment because I can't clone https://phabricator.wikimedia.org/diffusion/NLSP/new-lexeme-special-page.git from offsite.

In T410018#11417894, @A_smart_kitten wrote:

In T410018#11374839, @dancy wrote:

Does anyone know where https://noc.wikimedia.org sources its data?

I may be wrong but judging by https://github.com/wikimedia/operations-mediawiki-config/blob/a0f366b8b706c963ed9badac8ebe7a074a292549/src/Noc/utils.php#L9-L18, I believe it might read the wikiversions.json file that's local to the server its running on.

In T371620#11420555, @MatthewVernon wrote:

@dancy it's been a while now, but I think we just moved trafficserver to specify tags: [wmcs] for all its jobs and that worked around the issue.

In T410049#11420530, @dduvall wrote:

@dancy Not quite. We still need the WMCS/trusted runner changes in puppet.

@dduvall Is this ticket effectively resolved?

@MatthewVernon Can you give me an update on the status of your gitlab builds? Have you managed to work around the original problems?

I didn't get an email notification this morning so I checked the host and it looks like puppet is working now.

Thanks @jnuche !

@jnuche Any ideas about this?

In T409855#11393299, @sbassett wrote:

I think they're fine. I think it's more about just changing the way someone has done something for 7 or 8 years.

In T409855#11371578, @sbassett wrote:

In T409855#11370703, @Lucas_Werkmeister_WMDE wrote:

So what is the officially sanctioned or maintained way to deploy security patches, then?

I don't think there is one, but most of us on the Security-Team that deploy security patches typically do so manually.

In T405921#11384564, @JVanderhoop-WMF wrote:

@dancy I don't believe this relates to Metrics Platform. Could I ask why you had thought it did?

In T396166#10952265, @CodeReviewBot wrote:

dancy merged https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/864

Drop php_fpm_restart_script support

Noting for the record that I had prior authorization from Mónica to delete the monica_wmde account and I'm syncing with her via email on the changes.

I'll take this one.

<hashar> my guess is scap thinks that because it reads the local /srv/mediawiki-staging/wikiversions.json

https://versions.toolforge.org/ correctly showed still old 1.46.0-wmf.1 for group2 instead of 1.46.0-wmf.2.