Page MenuHomePhabricator
Create Task
Maniphest T405738

Debug scap partial deployment, 25 Sept 2025
Closed, ResolvedPublic
Actions

Assigned To
awight
Authored By
awight
Sep 26 2025, 1:45 PM
Project Tags
Referenced Files
F66701143: image.png
Sep 26 2025, 1:45 PM
F66701135: image.png
Sep 26 2025, 1:45 PM
F66701132: image.png
Sep 26 2025, 1:45 PM
F66701130: image.png
Sep 26 2025, 1:45 PM
F66701128: image.png
Sep 26 2025, 1:45 PM
F66701124: image.png
Sep 26 2025, 1:45 PM
Subscribers
Aklapper
awight
dancy
Lina_Farid_WMDE
Tgr
Tobi_WMDE_SW
WMDE-Fisch

Description

Somehow, it seems that deployment job https://spiderpig.wikimedia.org/jobs/633 resulted in an inconsistent state on the servers. I'm seeing that the JS component of patch 1191341 is live, but PHP or at least the Special:Version page is reporting an older revision of PHP. Evidence of my "fixed" feature still failing in production but working on the beta cluster corroborates the code version mismatch.

Fetching JS from testwiki I can see new code:

image.png (1,105×90 px, 15 KB)

it was only introduced in this fix patch:

image.png (1,882×159 px, 98 KB)

However, low-level VE debugging shows that the exported attribute is still missing on testwiki:

image.png (751×98 px, 18 KB)

Although healthy on the beta cluster:

image.png (751×98 px, 18 KB)

The SpiderPig deployment job looks correct

Fetching submodule extensions/Cite
From https://gerrit.wikimedia.org/r/mediawiki/extensions/Cite
   03bc65ea..e2445e96  master            -> origin/master
   0d5d8be2..e2aff282  wmf/1.45.0-wmf.20 -> origin/wmf/1.45.0-wmf.20

However, https://test.wikipedia.org/wiki/Special:Version reports an older PHP revision!

image.png (440×102 px, 8 KB)

This was one revision previous and doesn't match what is expected to be deployed:

image.png (1,111×623 px, 105 KB)

Details

Related Changes in GitLab:
TitleReferenceAuthorSource BranchDest Branch
Fix git info cache for master branchrepos/releng/scap!1013dancymaster-I6b69c1fd2d852047b36794c5e6bffb2af484b345master
git.py: Fix info() public commit selectionrepos/releng/scap!1004dancymaster-I378b7ff36ab2d84f145818e8e29265db2d641278master
Customize query in GitLab

Related Objects
Search...

Event Timeline

awight created this task.Sep 26 2025, 1:45 PM
awight triaged this task as High priority.Sep 26 2025, 1:53 PM
awight added projects: Scap, Release-Engineering-Team.

We're okay waiting for the next train to fix the immediate problem—but the underlying issue is very concerning, deployment should be transactional with no possibility of ending in an incomplete state like this, if that's what happened...

Tobi_WMDE_SW subscribed.Sep 26 2025, 1:56 PM
dancy subscribed.Sep 26 2025, 3:08 PM

The deploy server does have the desired commit in the staging directory.

dancy@deploy2002:/srv/mediawiki-staging/php-1.45.0-wmf.20/extensions/Cite$ git log -1
commit e2aff282c075d44ffebbce754d9bad1760f2abef (HEAD, origin/wmf/1.45.0-wmf.20)
Author: Adam Wight <adam.wight@wikimedia.de>
Date:   Thu Sep 18 14:58:32 2025 +0200

    Fix subref attribute order

    By preserving the `name` attribute, we're able to cleanly round-trip
    refs which have not been edited, and refs with name appearing before
    details.

    The new parser test options enable all parsoid testing variations
    including selser.

    Bug: T389363
    Change-Id: I61c4ae011be54cdce2b4e3d692e53e99dc7e33fb
    (cherry picked from commit 9fc882323beacaed3cf377218b8b519a90ea4d0b)
dancy updated the task description. (Show Details)Sep 26 2025, 3:10 PM
dancy added a comment.Sep 26 2025, 3:18 PM

I have also verified that the mediawiki container image used in production does contain the changes from rECITe2aff282c075.

dancy added a comment.Sep 26 2025, 3:32 PM

I think the commit reference on https://test.wikipedia.org/wiki/Special:Version is just wrong. I'll investigate how to improve that information.

CodeReviewBot added a project: Patch-For-Review.Sep 26 2025, 4:32 PM

dancy opened https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/1004

git.py: Fix info() public commit selection

CodeReviewBot added a comment.Sep 26 2025, 11:30 PM

thcipriani merged https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/1004

git.py: Fix info() public commit selection

Maintenance_bot removed a project: Patch-For-Review.Sep 26 2025, 11:31 PM
awight added a comment.Sep 29 2025, 8:45 AM

@dancy Thanks for doing this investigation!

I have a hard time following exactly which container is deployed, maybe that information was in /var/lib/spiderpig/scap-image-build-and-push-log and we should expose it in the SpiderPig Web UI? It would be very convenient if the container ID were exposed, at least.

Looking through debmonitor, I see docker-registry.discovery.wmnet/restricted/mediawiki-multiversion:2025-09-26-130659-publish-81 but this is already odd, since the timestamp seems to predate the scap deployment which began at 25 Sep 2025 13:25:51 GMT. I haven't figured out how to access the restricted images yet...

lilients_WMDE moved this task from Incoming to Dev-Work on the Cite (Sub-referencing) board.Sep 29 2025, 9:19 AM
awight closed this task as Resolved.Sep 29 2025, 9:50 AM
awight claimed this task.
awight moved this task from Sprint Backlog to Done on the WMDE-TechWish-Sprint-2025-09-16-Green-Tea-Icecream board.

I see that the number at the end is not expected to match the deployment time. I also see evidence that our change *is* deployed but that its caching behavior is a surprise to me, so this task can be closed now.

dancy added a comment.Sep 29 2025, 3:05 PM
In T405738#11223644, @awight wrote:

@dancy Thanks for doing this investigation!

I have a hard time following exactly which container is deployed, maybe that information was in /var/lib/spiderpig/scap-image-build-and-push-log and we should expose it in the SpiderPig Web UI? It would be very convenient if the container ID were exposed, at least.

Agreed. There are plans for building something like this eventually.

Stashbot added a comment.Sep 29 2025, 3:45 PM

Mentioned in SAL (#wikimedia-operations) [2025-09-29T15:45:21Z] <dancy@deploy2002> Started scap sync-world: Testing gitinfo fix (T405738)

Stashbot added a comment.Sep 29 2025, 3:56 PM

Mentioned in SAL (#wikimedia-operations) [2025-09-29T15:56:37Z] <dancy@deploy2002> Finished scap sync-world: Testing gitinfo fix (T405738) (duration: 11m 16s)

dancy added a comment.Sep 29 2025, 3:58 PM

The fix for the incorrect commit info in https://test.wikipedia.org/wiki/Special:Version has been deployed. You'll see good info there now.

CodeReviewBot added a project: Patch-For-Review.Oct 16 2025, 4:17 PM

dancy opened https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/1013

Fix git info cache for master branch

CodeReviewBot added a comment.Oct 16 2025, 4:57 PM

dancy merged https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/1013

Fix git info cache for master branch

Maintenance_bot removed a project: Patch-For-Review.Oct 16 2025, 5:31 PM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits