Page MenuHomePhabricator

Wikibase submodule is linked to github, causing scap prep to fail
Closed, ResolvedPublicBUG REPORT

Description

This change c1a1a3492d563d36f2304bfa9966b2e4d67abfde cause scap prep to fail because the submodule is linked to github instead of gerrit

List of steps to reproduce (step by step, including full links if applicable):

  • git clone https://github.com/wmde/new-lexeme-special-page /tmp/xxx (on the deployment server)

What happens?:
The clone fails

Cloning into '/srv/mediawiki-staging/php-1.38.0-wmf.21/extensions/Wikibase/lib/resources/wikibase-api'...
Cloning into '/srv/mediawiki-staging/php-1.38.0-wmf.21/extensions/Wikibase/view/lib/wikibase-data-model'...
Cloning into '/srv/mediawiki-staging/php-1.38.0-wmf.21/extensions/Wikibase/view/lib/wikibase-data-values-value-view'...
Cloning into '/srv/mediawiki-staging/php-1.38.0-wmf.21/extensions/Wikibase/view/lib/wikibase-data-values'...
Cloning into '/srv/mediawiki-staging/php-1.38.0-wmf.21/extensions/Wikibase/view/lib/wikibase-serialization'...
Cloning into '/srv/mediawiki-staging/php-1.38.0-wmf.21/extensions/Wikibase/view/lib/wikibase-termbox'...
Submodule 'resources/special/new-lexeme' (https://github.com/wmde/new-lexeme-special-page) registered for path 'extensions/WikibaseLexeme/reso
urces/special/new-lexeme'
Cloning into '/srv/mediawiki-staging/php-1.38.0-wmf.21/extensions/WikibaseLexeme/resources/special/new-lexeme'...
fatal: unable to access 'https://github.com/wmde/new-lexeme-special-page/': Failed to connect to github.com port 443: Connection timed out
fatal: clone of 'https://github.com/wmde/new-lexeme-special-page' into submodule path '/srv/mediawiki-staging/php-1.38.0-wmf.21/extensions/Wik
ibaseLexeme/resources/special/new-lexeme' failed
Failed to clone 'resources/special/new-lexeme'. Retry scheduled
Cloning into '/srv/mediawiki-staging/php-1.38.0-wmf.21/extensions/WikibaseLexeme/resources/special/new-lexeme'...
fatal: unable to access 'https://github.com/wmde/new-lexeme-special-page/': Failed to connect to github.com port 443: Connection timed out
fatal: clone of 'https://github.com/wmde/new-lexeme-special-page' into submodule path '/srv/mediawiki-staging/php-1.38.0-wmf.21/extensions/Wik
ibaseLexeme/resources/special/new-lexeme' failed
Failed to clone 'resources/special/new-lexeme' a second time, aborting
Failed to recurse into submodule path 'extensions/WikibaseLexeme'

What should have happened instead?:
The clone should succeed. The repository should be moved to gerrit

Event Timeline

WikibaseLexeme is a Wikimedia deployed extension, doesn't (SRE-imposed) policy require that it's canonically developed on WMF hosted platforms (Gerrit or in the future GitLab)?

Ladsgroup triaged this task as Unbreak Now! priority.Feb 8 2022, 7:09 PM
Ladsgroup subscribed.
In T301273#7694569, @Majavah wrote:

WikibaseLexeme is a Wikimedia deployed extension, doesn't (SRE-imposed) policy require that it's canonically developed on WMF hosted platforms (Gerrit or in the future GitLab)?

Yes. And Yes. It also must get a security review first.

wrt to canonical, it's not mandatory but encouraged. It must have a mirror though.

Reverted out of .21 in https://gerrit.wikimedia.org/r/c/mediawiki/extensions/WikibaseLexeme/+/760979 (as the code isn't used, this is the quickest way to cleanup)

I let the security team decide if this needs a security review or not (before deployment).

I let the security team decide if this needs a security review or not (before deployment).

Thank you! We will now look into mirroring the codebase. IIUC there are two actionables on our part:

  1. Mirror the codebase
  2. Start a security review

Is that correct? Is there any place in the documentation you can refer us to wrt starting a sec review process? Also, any docs about starting new repositories around production deployed extensions will be appreciated for future reference.

FWIW, our plan so far is that the submodule wouldn’t be directly used by WikibaseLexeme at all – the built artifacts would be checked into WikibaseLexeme.git, updated each time the submodule is updated (with a CI job to verify that they match). So the submodule is mainly there to record which commit the dist files belong to. (And also, in current WikibaseLexeme.git it’s not used at all and the whole feature is behind a feature flag too.)

Regarding the mirroring. We need to create a repo in diffusion that pulls from wmde's repo (and I assume this is fine because of arrangements and policies in place, including mandatory 2FA for admins, etc.). I can take care of that, I need to know the name and callsign (random phabricator stuff) but then it will be there.

Regarding two, Lydia has done it a couple of times before, if you want to see an example: T264822

I created https://phabricator.wikimedia.org/diffusion/NLSP/ but somehow the pulling from github is not working, ugh. I will work on it later.

Hey @Ladsgroup Thank you for creating the repo for us, we'd be happy to help in making this work, if you can point us to the right direction.

Regardless, we would like to also be able to create diffusion repositories on our own, what would be the process on adding myself, @karapayneWMDE and potentially other EMs and additional engineers in WMDE?

It now works, it was the mismatch between default branch of github repo (main) vs. default branch of diffusion (master), I changed the diffusion one and you should be able to access it now.

Thank you so much for your help @Ladsgroup it appears to be working

Tagging the Security-Team here to provide some context in preparation to a potential upcoming security review, as we would like to understand whether a security review is required or not.

What we are trying to achieve is inclusion of new code for a micro frontend in the Special:NewLexeme page. All code for this Javascript repsitory (See: rNLSP) is contributed and approved by WMDE employees, the production package dependencies are externalized to use formally approved modules included with the RL (Vue, VueX) except for the WiKit design system, which is also already included as a dependency in another of our projects that was formerly reviewed and consequently approved: https://gerrit.wikimedia.org/r/admin/repos/wikidata/query-builder

Therefore, our questions are:

  • Is a security review required for this repository to be included as a git submodule within WikibaseLexeme.git? (Similarly to the submodules in Wikibase.git)
  • Would it be advisable to request a review for WiKit separately, so that we could potentially use it in other projects?
Addshore lowered the priority of this task from Unbreak Now! to High.Feb 9 2022, 4:37 PM

Sounds like this is no longer an UBN!

Our current approach can now be found in this WikibaseLexeme change. We’re adding back the submodule, but from Diffusion instead of GitHub this time, so the clone should work in production; the dist files are copied into WikibaseLexeme (i.e. the submodule isn’t used at runtime), and the whole thing is also still behind a feature flag, which we won’t enable in production without further notice. (We may want to enable it on Beta sooner – as far as I’m aware, that would be less of an issue.) And we’ll also postpone merging that change until at least after the wmf.22 branch cut.

sbassett subscribed.

@ItamarWMDE - re: security reviews, please see the current SOP at https://www.mediawiki.org/wiki/Security/SOP/Application_Security_Reviews.

From what I'm seeing in the change set @Lucas_Werkmeister_WMDE mentions above, this doesn't appear to be a large volume of new or security-sensitive code, so it can likely just go through CR within gerrit. The Security-Team absolutely cannot review every new line of code added to Wikimedia projects and so we typically reserve the application security review process to major new codebases bound for production or major changes to core or other deployed extensions and skins. We also encourage folks to run various automated tools (SCA, SAST, etc.) against their own codebases, as tests or manually, for which we can help advise. The initial security concern here was the submodule update from github, which we don't allow in wikimedia production.

Lydia_Pintscher subscribed.

Thank you!
Closing this as the submodule is no longer linked to github now.