@Aklapper, please add any use cases that might be of interest to you. I'll do the same.

This is definitely not all, but one part of the puzzle is searching operations/puppet repo for "git::clone" to get this kind of list:

And then you'll have to add all the things that are deployed via scap.

https://phabricator.wikimedia.org/source/tool-ci/browse/master/build_table.py;3ed885277a38ddb5c7c8a6c3c9b666fb7b13ae10$49 is how my tools currently get a list of Wikimedia deployed extensions.

Some of the repositories listed in P6909 are only deployed in Cloud Services (e.g. extdist), not production.

I made a bunch of updates today on https://www.mediawiki.org/w/index.php?title=Developers/Maintainers&action=history and corresponding extension wiki home pages, however help with sorting out these items is welcome:

Deployed on WMF servers according to https://phabricator.wikimedia.org/diffusion/MREL/browse/master/make-wmf-branch/config.json and https://phabricator.wikimedia.org/source/mediawiki-config/browse/master/wmf-config/extension-list but the extension wiki page does not use {{TNT|OnWikimedia}} and it is not listed on https://www.mediawiki.org/wiki/Developers/Maintainers :

• extensions/CongressLookup
• extensions/FileExporter
• extensions/intersection
• extensions/JADE
• extensions/LdapAuthentication
• extensions/PropertySuggester
• extensions/WikimediaBadges

Extension wiki page uses {{TNT|OnWikimedia}} and it is listed as deployed on https://www.mediawiki.org/wiki/Developers/Maintainers but not in https://phabricator.wikimedia.org/diffusion/MREL/browse/master/make-wmf-branch/config.json and https://phabricator.wikimedia.org/source/mediawiki-config/browse/master/wmf-config/extension-list :

• https://www.mediawiki.org/wiki/Extension:DynamicPageList_(Wikimedia)

Extension wiki page uses {{TNT|OnWikimedia}} but not listed as deployed on https://www.mediawiki.org/wiki/Developers/Maintainers but not in https://phabricator.wikimedia.org/diffusion/MREL/browse/master/make-wmf-branch/config.json and https://phabricator.wikimedia.org/source/mediawiki-config/browse/master/wmf-config/extension-list :

• Extension:DataModel
• Extension:DataTypes
• Extension:DataValues
• Extension:DataValuesCommon
• Extension:DataValuesInterfaces
• Extension:Diff
• Extension:FundraisingEmailUnsubscribe
• Extension:ValueView

In T190891#4530610, @Aklapper wrote:

Extension wiki page uses {{TNT|OnWikimedia}} and it is listed as deployed on https://www.mediawiki.org/wiki/Developers/Maintainers but not in https://phabricator.wikimedia.org/diffusion/MREL/browse/master/make-wmf-branch/config.json and https://phabricator.wikimedia.org/source/mediawiki-config/browse/master/wmf-config/extension-list :

• https://www.mediawiki.org/wiki/Extension:DynamicPageList_(Wikimedia)

It's mediawiki/extensions/intersection.

Extension wiki page uses {{TNT|OnWikimedia}} but not listed as deployed on https://www.mediawiki.org/wiki/Developers/Maintainers but not in https://phabricator.wikimedia.org/diffusion/MREL/browse/master/make-wmf-branch/config.json and https://phabricator.wikimedia.org/source/mediawiki-config/browse/master/wmf-config/extension-list :

• Extension:DataModel
• Extension:DataTypes
• Extension:DataValues
• Extension:DataValuesCommon
• Extension:DataValuesInterfaces
• Extension:Diff
• Extension:ValueView

These are all PHP libraries that should be in [[Category:PHP libraries]], not listed as MediaWiki extensions afaik. Someone more familiar with Wikidata should confirm.

@Aklapper, I've been spending a bit of time on this and one of the directions I'm leaning is to have the ROO be an output of the deployment review process (Review Queue today). My thought being that the ROO could be used by deployers as an authoritative source to determine whether or not a new extensions, services, etc... should be deployed. I spoke a to Tyler a bit about this and it seems like it would be of value.

What are some other use cases for the ROO? One thing that's mentioned is it's need to be machine readable. What systems would you see consuming this list? FWIW, I too think it should be machine readable. I only ask to tease out some of the use cases that I may not be aware of.

On a related topic, I'm planning to spin up work around augmenting/revamping the review process.

What systems would you see consuming this list?

• Supersede manual {{OnWikimedia}} creating https://www.mediawiki.org/wiki/Category:Extensions_used_on_Wikimedia
• Supersede manual editing of https://www.mediawiki.org/wiki/Developers/Maintainers#MediaWiki_extensions_deployed_by_WMF
• Vague: Be able to differentiate on wikimedia.biterg.io between repos that are deployed vs repos that are not deployed to get more accurate and relevant statistics.
• Super vague: In the long run, somehow (maybe via T187661) have a code review performance dashboard for each team on wikimedia.biterg.io (see T179240)

All this still won't fix outdated information per se but I'd say it's less likely to have outdated information when there is one place to update instead of several places. My latest example was two days ago trying to find out who to fix https://phabricator.wikimedia.org/T203427 : I went to mw:Developers/Maintainers which listed "Discovery" for "Kartographer". On the Discovery IRC channel I was told that's not correct anymore (was corrected here).

Recently had a discussion with @mark and @faidon about some work that SRE is planning to do. Specifically the creation of a Service Catalog. On the surface this seems like it would potentially become the ROO mentioned in this task.

Posting this draft comment that I wrote at some point:

For MediaWiki extensions:

• For the next deployment, the reference of the repositories we deploy is the tool we use to cut the wmf branches . That was in Gerrit as mediawiki/tools/release.git in the file make-wmf-branch/config.json.
• For the currently deployed version, the tool above registers the extensions as submodule of the mediawiki/core wmf branch for the week. Hence we can get the list by looking at .gitmodule for each wmf branch.

The problem is that we can branch an extension weekly but not deploy it. Jade got branched constantly until it got archived and never saw production. There is a extension-list file in mw config which can be used instead I think.

I did create some hacky code that may help, it queries the API for each wiki to determine deployment status of skins/extensions. https://gitlab.wikimedia.org/acooper/extusage. There is a dump from the historical data in october here: https://docs.google.com/spreadsheets/d/1SBU6sPHSrkWmxLbMaUu1WoEVEPVQTmSHUgo_DVT7c4c/edit?usp=sharing

The main gap I am currently aware of is fundraising tech and any other wiki that does not allow non-authenticated users to access this data (which is rare I think).

The Security Team is also thinking about and prioritizing a public dashboard containing this information together with missing security controls per repo.

Beyond extensions/skins, we don't currently have a solution to tracking all deployed code like microservices/k8s. I am not sure if that is part of the requirement for this ticket.

Beyond extensions/skins, we don't currently have a solution to tracking all deployed code like microservices/k8s. I am not sure if that is part of the requirement for this ticket.

Thanks for sharing. There is some discussion spinning up in DX regarding developing a "Service Catalog". It's in very early discussions, but it might make sense to collaborate a bit on this.

Thank you both! Let me know if I can help on anything, eventually we would love to have a catalogue of python services to audit for security issues and do updates semi-automatically.

Something to consider, there is a lot of overlap between this ticket and SBOMs that are already better standardized and even there are tools to build them.
https://www.cisa.gov/sbom

http://federalregister.gov/documents/2021/05/17/2021-10460/improving-the-nations-cybersecurity

We probably could repurpose this to build some SBOMs and then see what's missing.

In T190891#10453803, @Ladsgroup wrote:

While that covers MediaWiki repos, we need a machine readable catalog for all deployed projects, e.g. in the example above repos deployed via puppet are exampled: T190891#4086712 This ties into using SBOMs and being able to quickly find if a CVE affects production plus easing maintenance of the codesearch indexing which currently is a mess. So that is covered for your usecase but I don't think this is fully addressed yet.

Noteworthy: Blubber is capable of producing SBOMs (blubber docs) for software running through the deployment pipeline to production (though I'm unsure if it's used by anything at this moment).

In T190891#10453904, @thcipriani wrote:

In T190891#10453803, @Ladsgroup wrote:

While that covers MediaWiki repos, we need a machine readable catalog for all deployed projects, e.g. in the example above repos deployed via puppet are exampled: T190891#4086712 This ties into using SBOMs and being able to quickly find if a CVE affects production plus easing maintenance of the codesearch indexing which currently is a mess. So that is covered for your usecase but I don't think this is fully addressed yet.

Noteworthy: Blubber is capable of producing SBOMs (blubber docs) for software running through the deployment pipeline to production (though I'm unsure if it's used by anything at this moment).

Thanks. That's great news, I want to eventually start using SBOMs more broadly (to compliment debmonitor and other toolings we have). Here is the attempt to use adopt them in MediaWiki: T359634: Adopt Software Bill of Materials (SBOM) for MediaWiki but that's one side of it. Codesearch also needs a catalog of repos to index as well.

@Aklapper Curious if we can move this ticket away from the Quality Engineering backlog - I don't see that there's action from my team on this and I'm trying to clean up that backlog in prep for incoming projects.

@SLong-WMF: Sure you can revert T190891#6665514. :) Would be very good though to have at least one team (or codebase) associated to an open task.
I can imagine that this is in the Release-Engineering-Team realm again.