Page MenuHomePhabricator

Special:Version on Wikimedia wikis shows outdated commit hashes for submodules
Open, LowPublic

Description

Special:Version on Wikimedia wikis shows outdated commit hashes. For example, currently https://en.wikipedia.org/wiki/Special:Version shows 13847cc for the Cite extension, while it should be showing 7aad7f5. Bryan says he knows why.

Event Timeline

matmarex assigned this task to bd808.
matmarex raised the priority of this task from to Needs Triage.
matmarex updated the task description. (Show Details)
matmarex added a project: Scap.
matmarex added subscribers: matmarex, ori.
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptOct 22 2015, 11:41 PM

Scap generated this gitinfo file:

{
  "head": "b00d06a08fc62053e54f99bd234780f2ce0a07a0",
  "remoteURL": "https://gerrit.wikimedia.org/r/p/mediawiki/extensions/Cite.git",
  "branch": "b00d06a08fc62053e54f99bd234780f2ce0a07a0",
  "headCommitDate": "1444676619",
  "headSHA1": "13847ccede3cd7f7d2160a148b79f85435f6f466",
  "@directory": "/srv/mediawiki-staging/php-1.27.0-wmf.3/extensions/Cite"
}

The key bits here are "head" (sha1 of the HEAD of extensions/Cite on tin at the time) and "headSHA1" (the "disclosable head" of the branch). At the time that scap was last run (2015-10-20T23:20Z), the local git log looked like this:

* b00d06a Display 'cite_error_references_duplicate_key' next to the affected ref (2 weeks ago, Bartosz Dziewoński)
* f08984d Creating new wmf/1.27.0-wmf.3 branch (9 days ago, Mukunda Modell)
* 13847cc Localisation updates from https://translatewiki.net. (10 days ago, Translation updater bot) (master)
* a8fe5e5 Localisation updates from https://translatewiki.net. (11 days ago, Translation updater bot)
* a374453 Localisation updates from https://translatewiki.net. (13 days ago, Translation updater bot)

HEAD was b00d06a08fc62053e54f99bd234780f2ce0a07a0 which is the version we would expect to see on Special:Version. Scap instead decided that the "disclosable head" was 13847ccede3cd7f7d2160a148b79f85435f6f466 which happens to be the last commit to master before the wmf/1.27.0-wmf.3 branch was cut.

This "disclosable head" business is our attempt to keep security patches that might be deployed on the WMF production cluster from being displayed on Special:Version. The scap.utils.get_disclosable_head function tries to find the newest common commit between the clone on the deploy server and the origin repository by running git rev-list -1 @{upstream}.

@{upstream} here is a git magic shortcut for "the currently tracking remote branch for this local branch". It turns out that on our wmf.X branches right now submodule checkouts do not have a remote tracking branch. This causes the git rev-list -1 @{upstream} lookup to fail. When that happens, scap falls back to running git merge-base HEAD $(git remote). This returns the last commit before the submodule's release branch was cut and ignores any commits made on the release branch itself.

So... long story long... "most" of the time this would be close enough. Sync-dir and sync-file don't update the gitinfo cache files (T38271) we would typically expect the disclosed hash to be the branch point of the release branch. I think with some strong git-fu we can do better however and get the remote commits that came after the branch point as well. I'll look into that and report back.

bd808 renamed this task from Special:Version on Wikimedia wikis shows outdated commit hashes to Special:Version on Wikimedia wikis shows outdated commit hashes for submodules.Oct 23 2015, 12:15 AM
bd808 edited projects, added Deployments; removed Wikimedia-General-or-Unknown, Scap.
bd808 set Security to None.
bd808 triaged this task as Low priority.Oct 23 2015, 12:18 AM

I'm not sure why there aren't remote tracking branches, but it seems like that is the best way to tell what the upstream commit is.

I wonder why the remote branches didn't get set up by multiversion/checkoutMediaWiki?

bd808 added a comment.Oct 23 2015, 3:00 AM

I'm not sure why there aren't remote tracking branches, but it seems like that is the best way to tell what the upstream commit is.
I wonder why the remote branches didn't get set up by multiversion/checkoutMediaWiki?

I think the problem is that submodule tracking branches requires git 1.8.2+ and we have git 1.7.9.5 on tin. The .gitmodules file has the needed data for tracking branches to be used if we had a version of git that supported it:

$ git config -f /srv/mediawiki-staging/php-1.27.0-wmf.3/.gitmodules submodule.extensions/Cite.branch
wmf/1.27.0-wmf.3

So... this might all go away when tin is upgraded/replaced with a Ubuntu 14.04 host. We could try to add support for the git config trick above as a fallback when @{upstream} fails. It's a bit kludgy though because we need to know where the .gitmodules file is, which submodule we are looking at, and to prepend origin/ to the branch name we get back:

$ git rev-list -1 origin/$(git config -f /srv/mediawiki-staging/php-1.27.0-wmf.3/.gitmodules submodule.extensions/Cite.branch)
7aad7f5d0a449e5a0b562000bb3f448243edc8dd
bd808 added a comment.Oct 23 2015, 3:26 AM

The branches are all set in the top level .gitconfig as hoped for. I'm guessing this is because @mmodell is using a not-freaking-ancient version of git when he runs make-wmf-branch.

1$ git submodule foreach -q 'echo $name $(git config -f $toplevel/.gitmodules submodule.$name.branch)'
2extensions/AbuseFilter wmf/1.27.0-wmf.3
3extensions/AccountAudit wmf/1.27.0-wmf.3
4extensions/ActiveAbstract wmf/1.27.0-wmf.3
5extensions/AntiSpoof wmf/1.27.0-wmf.3
6extensions/ApiFeatureUsage wmf/1.27.0-wmf.3
7extensions/ApiSandbox wmf/1.27.0-wmf.3
8extensions/Babel wmf/1.27.0-wmf.3
9extensions/BetaFeatures wmf/1.27.0-wmf.3
10extensions/BounceHandler wmf/1.27.0-wmf.3
11extensions/Calendar wmf/1.27.0-wmf.3
12extensions/Campaigns wmf/1.27.0-wmf.3
13extensions/CategoryTree wmf/1.27.0-wmf.3
14extensions/CentralAuth wmf/1.27.0-wmf.3
15extensions/CentralNotice wmf_deploy
16extensions/CharInsert wmf/1.27.0-wmf.3
17extensions/CheckUser wmf/1.27.0-wmf.3
18extensions/CirrusSearch wmf/1.27.0-wmf.3
19extensions/Cite wmf/1.27.0-wmf.3
20extensions/CiteThisPage wmf/1.27.0-wmf.3
21extensions/Citoid wmf/1.27.0-wmf.3
22extensions/CleanChanges wmf/1.27.0-wmf.3
23extensions/CodeEditor wmf/1.27.0-wmf.3
24extensions/CodeReview wmf/1.27.0-wmf.3
25extensions/Collection wmf/1.27.0-wmf.3
26extensions/CommonsMetadata wmf/1.27.0-wmf.3
27extensions/ConfirmEdit wmf/1.27.0-wmf.3
28extensions/ContactPage wmf/1.27.0-wmf.3
29extensions/ContentTranslation wmf/1.27.0-wmf.3
30extensions/ContributionTracking wmf/1.27.0-wmf.3
31extensions/CreditsSource wmf/1.27.0-wmf.3
32extensions/CustomData wmf/1.27.0-wmf.3
33extensions/DisableAccount wmf/1.27.0-wmf.3
34extensions/Disambiguator wmf/1.27.0-wmf.3
35extensions/DismissableSiteNotice wmf/1.27.0-wmf.3
36extensions/DonationInterface wmf/1.27.0-wmf.3
37extensions/DoubleWiki wmf/1.27.0-wmf.3
38extensions/DynamicSidebar wmf/1.27.0-wmf.3
39extensions/Echo wmf/1.27.0-wmf.3
40extensions/EducationProgram wmf/1.27.0-wmf.3
41extensions/Elastica wmf/1.27.0-wmf.3
42extensions/EventLogging wmf/1.27.0-wmf.3
43extensions/ExtensionDistributor wmf/1.27.0-wmf.3
44extensions/FeaturedFeeds wmf/1.27.0-wmf.3
45extensions/FlaggedRevs wmf/1.27.0-wmf.3
46extensions/Flow wmf/1.27.0-wmf.3
47extensions/FundraiserLandingPage wmf/1.27.0-wmf.3
48extensions/FundraisingTranslateWorkflow wmf/1.27.0-wmf.3
49extensions/GWToolset wmf/1.27.0-wmf.3
50extensions/Gadgets wmf/1.27.0-wmf.3
51extensions/Gather wmf/1.27.0-wmf.3
52extensions/GeoCrumbs wmf/1.27.0-wmf.3
53extensions/GeoData wmf/1.27.0-wmf.3
54extensions/GettingStarted wmf/1.27.0-wmf.3
55extensions/GlobalBlocking wmf/1.27.0-wmf.3
56extensions/GlobalCssJs wmf/1.27.0-wmf.3
57extensions/GlobalUsage wmf/1.27.0-wmf.3
58extensions/GlobalUserPage wmf/1.27.0-wmf.3
59extensions/GoogleNewsSitemap wmf/1.27.0-wmf.3
60extensions/Graph wmf/1.27.0-wmf.3
61extensions/GuidedTour wmf/1.27.0-wmf.3
62extensions/ImageMap wmf/1.27.0-wmf.3
63extensions/ImageMetrics wmf/1.27.0-wmf.3
64extensions/InputBox wmf/1.27.0-wmf.3
65extensions/Insider wmf/1.27.0-wmf.3
66extensions/Interwiki wmf/1.27.0-wmf.3
67extensions/Josa wmf/1.27.0-wmf.3
68extensions/JsonConfig wmf/1.27.0-wmf.3
69extensions/LabeledSectionTransclusion wmf/1.27.0-wmf.3
70extensions/LandingCheck wmf/1.27.0-wmf.3
71extensions/LdapAuthentication wmf/1.27.0-wmf.3
72extensions/LiquidThreads wmf/1.27.0-wmf.3
73extensions/Listings wmf/1.27.0-wmf.3
74extensions/LocalisationUpdate wmf/1.27.0-wmf.3
75extensions/MapSources wmf/1.27.0-wmf.3
76extensions/MassMessage wmf/1.27.0-wmf.3
77extensions/Math wmf/1.27.0-wmf.3
78extensions/MobileApp wmf/1.27.0-wmf.3
79extensions/MobileFrontend wmf/1.27.0-wmf.3
80extensions/MoodBar wmf/1.27.0-wmf.3
81extensions/MultimediaViewer wmf/1.27.0-wmf.3
82extensions/MwEmbedSupport wmf/1.27.0-wmf.3
83extensions/NavigationTiming wmf/1.27.0-wmf.3
84extensions/NewUserMessage wmf/1.27.0-wmf.3
85extensions/Nuke wmf/1.27.0-wmf.3
86extensions/OAI wmf/1.27.0-wmf.3
87extensions/OATHAuth wmf/1.27.0-wmf.3
88extensions/OAuth wmf/1.27.0-wmf.3
89extensions/OpenStackManager wmf/1.27.0-wmf.3
90extensions/PageImages wmf/1.27.0-wmf.3
91extensions/PageTriage wmf/1.27.0-wmf.3
92extensions/PagedTiffHandler wmf/1.27.0-wmf.3
93extensions/ParserFunctions wmf/1.27.0-wmf.3
94extensions/ParsoidBatchAPI wmf/1.27.0-wmf.3
95extensions/PdfHandler wmf/1.27.0-wmf.3
96extensions/Petition wmf/1.27.0-wmf.3
97extensions/Poem wmf/1.27.0-wmf.3
98extensions/PoolCounter wmf/1.27.0-wmf.3
99extensions/Popups wmf/1.27.0-wmf.3
100extensions/ProofreadPage wmf/1.27.0-wmf.3
101extensions/Quiz wmf/1.27.0-wmf.3
102extensions/RSS wmf/1.27.0-wmf.3
103extensions/RandomRootPage wmf/1.27.0-wmf.3
104extensions/RelatedArticles wmf/1.27.0-wmf.3
105extensions/RelatedSites wmf/1.27.0-wmf.3
106extensions/Renameuser wmf/1.27.0-wmf.3
107extensions/RestBaseUpdateJobs wmf/1.27.0-wmf.3
108extensions/SandboxLink wmf/1.27.0-wmf.3
109extensions/Score wmf/1.27.0-wmf.3
110extensions/Scribunto wmf/1.27.0-wmf.3
111extensions/SemanticForms wmf/1.27.0-wmf.3
112extensions/SemanticMediaWiki 1.8.x
113extensions/SemanticResultFormats 1.8.x
114extensions/ShortUrl wmf/1.27.0-wmf.3
115extensions/SiteMatrix wmf/1.27.0-wmf.3
116extensions/SpamBlacklist wmf/1.27.0-wmf.3
117extensions/SubPageList3 wmf/1.27.0-wmf.3
118extensions/SubpageSortkey wmf/1.27.0-wmf.3
119extensions/SyntaxHighlight_GeSHi wmf/1.27.0-wmf.3
120extensions/TemplateData wmf/1.27.0-wmf.3
121extensions/TemplateSandbox wmf/1.27.0-wmf.3
122extensions/TextExtracts wmf/1.27.0-wmf.3
123extensions/Thanks wmf/1.27.0-wmf.3
124extensions/TimedMediaHandler wmf/1.27.0-wmf.3
125extensions/TitleBlacklist wmf/1.27.0-wmf.3
126extensions/TocTree wmf/1.27.0-wmf.3
127extensions/TorBlock wmf/1.27.0-wmf.3
128extensions/Translate wmf/1.27.0-wmf.3
129extensions/TranslationNotifications wmf/1.27.0-wmf.3
130extensions/TrustedXFF wmf/1.27.0-wmf.3
131extensions/UnicodeConverter wmf/1.27.0-wmf.3
132extensions/UniversalLanguageSelector wmf/1.27.0-wmf.3
133extensions/UploadWizard wmf/1.27.0-wmf.3
134extensions/UserDailyContribs wmf/1.27.0-wmf.3
135extensions/UserMerge wmf/1.27.0-wmf.3
136extensions/Validator 0.5.x
137extensions/VectorBeta wmf/1.27.0-wmf.3
138extensions/VipsScaler wmf/1.27.0-wmf.3
139extensions/VisualEditor wmf/1.27.0-wmf.3
140extensions/WikiEditor wmf/1.27.0-wmf.3
141extensions/WikiLove wmf/1.27.0-wmf.3
142extensions/Wikidata wmf/1.27.0-wmf.3
143extensions/WikidataPageBanner wmf/1.27.0-wmf.3
144extensions/WikimediaEvents wmf/1.27.0-wmf.3
145extensions/WikimediaIncubator wmf/1.27.0-wmf.3
146extensions/WikimediaMaintenance wmf/1.27.0-wmf.3
147extensions/WikimediaMessages wmf/1.27.0-wmf.3
148extensions/XAnalytics wmf/1.27.0-wmf.3
149extensions/ZeroBanner wmf/1.27.0-wmf.3
150extensions/ZeroPortal wmf/1.27.0-wmf.3
151extensions/cldr wmf/1.27.0-wmf.3
152extensions/intersection wmf/1.27.0-wmf.3
153extensions/timeline wmf/1.27.0-wmf.3
154extensions/wikihiero wmf/1.27.0-wmf.3
155skins/CologneBlue wmf/1.27.0-wmf.3
156skins/Modern wmf/1.27.0-wmf.3
157skins/MonoBook wmf/1.27.0-wmf.3
158skins/Nostalgia wmf/1.27.0-wmf.3
159skins/Vector wmf/1.27.0-wmf.3
160vendor wmf/1.27.0-wmf.3

@bd808: I run make-wmf-branch on my laptop for that reason, and a few others. Tin is ridiculously outdated and it manages to annoy me frequently.

greg edited projects, added scap2; removed Deployments.Feb 10 2016, 12:25 AM
bd808 removed bd808 as the assignee of this task.Feb 21 2016, 4:36 AM
bd808 added a subscriber: bd808.
bd808 added a comment.Feb 21 2016, 6:35 AM

Git on the new Ubuntu 14.04 based deployment servers (tin & mira) is version 1.9.1. I speculated in T116345#1747370 that this might fix the problem. I just checked tin:/srv/mediawiki-staging/php-1.27.0-wmf.13/extensions/Cite and the bug here is not fixed however.

tin:/srv/mediawiki-staging/php-1.27.0-wmf.13/extensions/Cite  (git (91eab16...))
bd808$ git log --graph --oneline --decorate --all | head -5
* 91eab16 (HEAD, origin/wmf/1.27.0-wmf.13) VE: Fix i18n names broken during migration
* 8276d9a Creating new wmf/1.27.0-wmf.13 branch
| * eb58f79 (origin/master, origin/HEAD) VE: Fix i18n names broken during migration
| * 483dbf8 Localisation updates from https://translatewiki.net.
|/
tin:/srv/mediawiki-staging/php-1.27.0-wmf.13/extensions/Cite  (git (91eab16...))
bd808$ git rev-list -1 @{upstream}
fatal: HEAD does not point to a branch
tin:/srv/mediawiki-staging/php-1.27.0-wmf.13/extensions/Cite  (git (91eab16...))
bd808$ git merge-base HEAD $(git remote)
c859818a705fc6ee9f3812a319e329ba8f517802

git rev-list -1 @{upstream} is still failing and thus scap will fall back to using git merge-base HEAD $(git remote) which still points to the pre-branch commit on master.

I tried doing the initial clones using git submodule update --init --recursive --remote (see stackoverflow). git remote show -n origin still shows that the tracking branch is master rather than the hoped for wmf/1.27.0-wmf.13. This probably has something to do with a submodule always being in a detached head state and the tracking branch information only being kept in .gitmodules rather than the cloned module itself.

I think I may have found another sort of way to find the disclosable hash using git log. git log --pretty=format:'%H %d' will print the commit hash and the "ref names" that correspond to the commit.

tin:/srv/mediawiki-staging/php-1.27.0-wmf.13/extensions/Cite  (git (91eab16...))
bd808$ git log --pretty=format:'%H %d' | head -5
91eab16282a47cedecf1aec0343da7f4917efb69  (HEAD, origin/wmf/1.27.0-wmf.13)
8276d9ae80632a377ad4c42b7357d7d16ab2ea23
c859818a705fc6ee9f3812a319e329ba8f517802  (master)
150f87b4985a6fe9f70790fdf0eb03505f673bb7
71889ff017bec4238a608f51c1fcd1c348232879

You can see here that the 91eab16 commit that we want to show on Special:Version has a ref naming a remote branch. I've tested in a submodule with a security patch applied and the results are something like:

abc123 (HEAD)
def456 (origin/wmf/1.27.0-wmf.13)
789abc
012def (master)

Using this output, the first line that includes either origin or master in the ref names should be safe to disclose. This is still a bit hacky feeling but not as much as baking in support for fishing around on the filesystem for the needed .gitmodules file and figuring out the submodule name that using the git config -f trick requires.

Here's the right shell magic to find our hash:

tin:/srv/mediawiki-staging/php-1.27.0-wmf.13/extensions/Cite  (git (91eab16...))
bd808$ git log --pretty=format:'%H %d' |
  grep -E 'origin|master' |
  awk '{print $1}' |
  head -1
91eab16282a47cedecf1aec0343da7f4917efb69

Using @{upstream} is still best when it works, but this fallback method could be used instead of git merge-base HEAD $(git remote) when no tracking branch is available.

mmodell edited projects, added Scap; removed scap2.Feb 10 2017, 6:22 PM