I just noticed this while doing local catalyst-api development.

@brennen I heard you were going to be looking into this stuff.

Notice to train operator @jeena: I merged https://gitlab.wikimedia.org/repos/releng/release/-/merge_requests/236 today. This removes the 5 minute sleep that occurs after a full image build (T390251) now that T412951 has been implemented. If this causes a problem, merge a revert commit for https://gitlab.wikimedia.org/repos/releng/release/-/merge_requests/236 to recover.

bookworm-php-sury and its dependent images have been updated.

I'm running into this problem today so I'll take this ticket.

Thanks everyone!

A burst of these errors happened on Feb 16 between 13:35 and 13:40. Here's a fresh trace:

Thanks for the work on this. I'm holding the train until the fix is backported.

In T417820#11629590, @Tgr wrote:

Should probably do a quick fix rather than waiting for the client credentials issue to be fully worked out.

Thanks @jnuche !

@elukey Congrats on the experiment!

In T412951#11606818, @elukey wrote:

This includes two 5-mins wait time, so it could have potentially be completed in 27 mins.

elukey@deploy2002:~$ grep "300 seconds" /home/elukey/scap-image-build-and-push-log
11:25:42 [mediawiki-publish-83-next] Waiting 300 seconds for swift after full mediawiki image build (T390251)
11:32:11 [mediawiki-publish-83] Waiting 300 seconds for swift after full mediawiki image build (T390251)

In T417071#11604751, @Andrew wrote:

I have bodged up an a record and ptr record for this host, so you should be able to access it now. Because I created the record by hand it will likely live on forever after the VM is deleted so I'm going to keep this task open for a while.

If someone has a story about what happened and/or if that VM worked in the past, I'm still interested!

In T417071#11604700, @Andrew wrote:

ok! It would be useful to know if/when that VM was ever functional and reachable.

In T417071#11604654, @Andrew wrote:

@dancy have you been able to access this VM in the past?

In T361724#11604248, @Scott_French wrote:

Just to confirm, the terminal-multiplexer check feature works as expected, and all that remains is docs updates and comms to ops@lists before it's reenabled, correct? (i.e., the description remains accurate)

I believe that the work done in T408740 has resolved this issue.

In T412951#11572476, @elukey wrote:

@dancy @Scott_French I think we are ready to move forward with https://gerrit.wikimedia.org/r/c/operations/puppet/+/1229145, what do you think?

I am ok to proceed with the sync-world and full_image_rebuild during a MediaWiki infrastructure window (could be good event tomorrow's), but it will be during your night so you'll not be able to join (but if needed I can loop in Matthew to check the apus/Ceph side). The alternative is to schedule the upgrade during a MediaWiki infra window happening during your workday, totally fine for me. Lemme know what you prefer!

@Reedy I deployed a change to the kubernetes deployment monitor code in scap. Now it will not give up monitoring when it gets an error from kubectl.

In T412941#11548781, @MLechvien-WMF wrote:

Release-Engineering-Team could you please provide inputs on the Scap specific works in the description?

In T401197#11559575, @jeena wrote:

I have archived the documentation page so now we can remove the chart.

Here's a working .gitlab-ci.yml config using wmcs runners: https://gitlab.wikimedia.org/repos/m3api/tmp-m3api-oauth2/-/merge_requests/1/diffs

@LucasWerkmeister Thanks for setting up https://gitlab.wikimedia.org/repos/m3api/tmp-m3api-oauth2 and adding me as a member. I tried to run the CI pipeline of that repo on the main branch (using the normal D.O. runners) but it fails: https://gitlab.wikimedia.org/repos/m3api/tmp-m3api-oauth2/-/jobs/727347. I need a working pipeline as a starting point for debugging.

In T414273#11551749, @A_smart_kitten wrote:

Thanks @dancy :) FWIW, the issue (for me at least) would be less creating/submitting the actual patch, and more getting together a local environment that I'd be able to test it in...

In T337570#11550214, @bd808 wrote:

In T337570#11549528, @dancy wrote:

If a merge request description body consists of a single Bug: TXXXXX line, the link to Phabricator does not get created. I'm wondering if this is a new problem since the recent Gitlab UI changes. See the job link referenced in the task description for an example (https://gitlab.wikimedia.org/repos/abstract-wiki/wikifunctions/function-orchestrator/-/merge_requests/18).

Phabricator linking seems to be working on https://gitlab.wikimedia.org/toolforge-repos/versions/-/merge_requests/8. I wonder if there is something interesting going on in that wikifunctions MR description? Does the regex that is in use allow for trailing whitespace? Is there a newline requirement?

Deployed via scap 4.235.0.

If a merge request description body consists of a single Bug: TXXXXX line, the link to Phabricator does not get created. I'm wondering if this is a new problem since the recent Gitlab UI changes. See the job link referenced in the task description for an example (https://gitlab.wikimedia.org/repos/abstract-wiki/wikifunctions/function-orchestrator/-/merge_requests/18).

Change deployed via scap 4.234.0

In T412951#11539048, @elukey wrote:

Thanks! When you have a moment let me know if what I wrote above is sound, namely if it is ok to just start clean or not (basically if it is ok not to be able to rollback when we build/deploy the first time after the switch). If it is fine, I think that we can probably choose when to do the test after the SRE summit (that will happen next week). Ideally we could just rebuild all images and try to push/pull them, and see how it goes.

@Clement_Goubert Please use scap lock --unlock-all --yes <unlock reason> (note: --bg is not passed here).

In T414864#11535504, @bd808 wrote:

The runners you would like unblocked are hosted on Digital Ocean. I do not think that it would be reasonable to open the Beta Cluster to the full DO IPv4 address space. If we have a fixed sub-space for egress we can unblock that. @dancy can you help me figure out if there is a restricted range that the runner egress through?

In T414864#11532038, @LucasWerkmeister wrote:

FWIW, I’ve tried to get m3api-oauth2 CI running on the WMCS runners instead (wmcs tag), but so far haven’t managed to get Chrome/Chromium running there yet (latest job).

Thanks for the report @elukey. This sounds very promising!

I confirmed that I did receive an email when I created a beta wiki account.

xref T412975

I ran sudo /usr/local/sbin/clean-stale-puppet-certs --clean on deployment-puppetserver-1.deployment-prep to take care of this.

Thanks for the work on this @thcipriani !

@jeena @thcipriani What's the status of the mediawiki-dev deployment-chart? I found that it is referenced in https://wikitech.wikimedia.org/wiki/Deployment_pipeline/Migration/Tutorial

@Blake I've installed a new release of scap on the deploy servers. You can now use scap lock --all --bg and scap lock --unlock-all to achieve your goal. I recommend testing things out on deployment.eqiad.wmnet well in advance of the datacenter switch. Let me know if you have any issues.

@A_smart_kitten patches are welcome! Let me know if you need help with getting something together.

Potential ideas:

• Drop a lock file on the deployment server that scap detects, remove it at a later step
• Add a switch to scap lock --all that puts the lock on and returns, add a scap unlock --all command

Added

profile::tlsproxy::envoy::upstream_sni: null
profile::tlsproxy::envoy::upstream_tls: false

to devtools project puppet according to @bd808's recommendation.

Looks similar to T414304

I turned off this instance as part of T412975

In T399348#11513384, @vaughnwalters wrote:

@dancy I tried changing the tags here and am getting the same error... maybe I'm still doing something wrong though?

In T412975#11498683, @Dzahn wrote:

Should I abandon or keep https://gerrit.wikimedia.org/r/c/operations/puppet/+/1219180 now?

In T399348#11508642, @vaughnwalters wrote:

In T399348#11507969, @dancy wrote:

After reading the history of this ticket, it's not clear to me if the memory-optimized runners have been tried yet. Can someone try and confirm?

Heya @dancy, this is done by tags: [wmcs] correct (looking at T399348#11007932)? That tag been added here for the orchestrator. So if that tag looks correct, then it has been tried.

After reading the history of this ticket, it's not clear to me if the memory-optimized runners have been tried yet. Can someone try and confirm?

@Andrew has since changed where he performing the build of the wikitech-static container image, so the main cause of this ticket has been resolved.