Page MenuHomePhabricator
Create Task
Maniphest T190891

Develop canonical/single record of origin, machine readable list of all repos deployed to WMF sites
Open, MediumPublic
Actions

Description

From a discussion between Andre and I.

Would like to have a realtime list of repos/code that is deployed to production WMF sites.

In order to be of benefit, this list would likely need to be a "live" list that is reflective of what is currently deployed.

Existing [partial] "lists" of deployed repositories:

See Also:
T186290: codesearch should have search profiles for WMF production and for bundled/"tarball" extensions

Related Objects
Search...

Event Timeline

Jrbranaa created this task.Mar 27 2018, 10:57 PM
Restricted Application edited projects, added Release-Engineering-Team (Kanban); removed Release-Engineering-Team. · View Herald TranscriptMar 27 2018, 10:57 PM
Restricted Application added a subscriber: Aklapper. · View Herald Transcript
Jrbranaa added a comment.Mar 27 2018, 10:58 PM

@Aklapper, please add any use cases that might be of interest to you. I'll do the same.

Dzahn added a subscriber: Dzahn.EditedMar 27 2018, 11:02 PM

This is definitely not all, but one part of the puzzle is searching operations/puppet repo for "git::clone" to get this kind of list:

P6909 git::clones in puppet repo
1modules/jupyterhub_old/manifests/init.pp: git::clone { $wheels_repo:
2modules/labs_vagrant/manifests/init.pp: git::clone { 'vagrant':
3modules/wikimetrics/manifests/base.pp: git::clone { 'wikimetrics-deploy':
4modules/wikimetrics/manifests/base.pp: git::clone { 'wikimetrics':
5modules/releases/manifests/init.pp: git::clone { 'mediawiki/core':
6modules/releases/manifests/init.pp: git::clone { 'mediawiki/tools/release':
7modules/scap/manifests/l10nupdate.pp: git::clone { 'mediawiki/core':
8modules/scap/manifests/l10nupdate.pp: git::clone { 'mediawiki/extensions':
9modules/scap/manifests/l10nupdate.pp: git::clone { 'mediawiki/skins':
10modules/scap/manifests/master.pp: git::clone { 'operations/mediawiki-config':
11modules/beta/manifests/autoupdater.pp: git::clone { 'beta-mediawiki-core':
12modules/beta/manifests/autoupdater.pp: git::clone { 'beta-portal':
13modules/beta/manifests/autoupdater.pp: git::clone { 'beta-mediawiki-extensions':
14modules/beta/manifests/autoupdater.pp: git::clone { 'beta-mediawiki-skins':
15modules/beta/manifests/autoupdater.pp: git::clone { 'mediawiki/vendor':
16modules/puppet_compiler/manifests/init.pp: git::clone { 'operations/puppet':
17modules/puppet_compiler/manifests/init.pp: git::clone { 'labs/private':
18modules/git/spec/defines/clone_spec.rb:describe 'git::clone' do
19modules/git/manifests/install.pp: git::clone{$title:
20modules/git/manifests/clone.pp:# Definition: git::clone
21modules/git/manifests/clone.pp:# git::clone { 'my_clone_name':
22modules/git/manifests/clone.pp:# git::clone { 'analytics/wikimetrics':
23modules/git/manifests/clone.pp:define git::clone(
24modules/vagrant/manifests/mediawiki.pp: git::clone { 'mediawiki/vagrant':
25modules/eventschemas/manifests/init.pp:# [*ensure*] Passed directly to git::clone. Default: latest.
26modules/eventschemas/manifests/init.pp: git::clone { 'mediawiki/event-schemas':
27modules/snapshot/manifests/cron/wikidatadumps/common.pp: git::clone { 'DCAT-AP':
28modules/statistics/manifests/discovery.pp: git::clone { 'wikimedia/discovery/golden':
29modules/statistics/manifests/wmde/wdcm.pp: git::clone { 'analytics/wmde/WDCM':
30modules/statistics/manifests/wmde/graphite.pp: git::clone { 'wmde/scripts':
31modules/statistics/manifests/wmde/graphite.pp: git::clone { 'wmde/toolkit-analyzer-build':
32modules/statistics/manifests/aggregator/projectview.pp: git::clone { 'aggregator_projectview_data':
33modules/statistics/manifests/compute.pp: git::clone { 'statistics_mediawiki':
34modules/statistics/manifests/sites/analytics.pp: git::clone { 'analytics.wikimedia.org':
35modules/statistics/manifests/sites/stats.pp: git::clone { 'wikistats-v2':
36modules/statistics/manifests/aggregator.pp: git::clone { 'aggregator_code':
37modules/service/manifests/deploy/gitclone.pp: git::clone { $repository:
38modules/jupyterhub/manifests/init.pp: git::clone { $deploy_repository:
39modules/extdist/manifests/init.pp: git::clone {'labs/tools/extdist':
40modules/extdist/manifests/init.pp: git::clone { 'integration/composer':
41modules/visualdiff/manifests/init.pp: git::clone { 'integration/visualdiff':
42modules/contint/manifests/phpunit.pp: git::clone { 'jenkins CI phpunit':
43modules/contint/manifests/slave_scripts.pp: git::clone { 'jenkins CI slave scripts':
44modules/contint/manifests/composer.pp: git::clone { 'jenkins CI Composer':
45modules/authdns/spec/classes/authdns_spec.rb: 'define git::clone($directory, $origin, $branch,$owner,$group) {}',
46modules/authdns/manifests/init.pp: git::clone { $workingdir:
47modules/reportupdater/manifests/init.pp: git::clone { 'analytics/reportupdater':
48modules/reportupdater/manifests/job.pp: git::clone { $repository_name:
49modules/noc/manifests/init.pp: git::clone { 'operations/software/dbtree':
50modules/tendril/manifests/init.pp: git::clone { 'operations/software/tendril':
51modules/quarry/manifests/base.pp: git::clone { 'analytics/quarry/web':
52modules/sentry/manifests/init.pp: git::clone { 'operations/software/sentry':
53Binary file modules/role/manifests/grafana/.base.pp.swp matches
54modules/role/manifests/labs/ores/staging.pp: git::clone { 'ores-wm-config':
55modules/role/manifests/labs/db/common.pp: git::clone { 'operations/mediawiki-config':
56modules/role/manifests/xhgui/app.pp: git::clone { 'operations/software/xhprof':
57modules/role/manifests/xhgui/app.pp: git::clone { 'operations/software/xhgui':
58modules/mediawiki_singlenode/manifests/init.pp: git::clone { 'vendor':
59modules/mediawiki_singlenode/manifests/init.pp: git::clone { 'mediawiki':
60modules/mediawiki_singlenode/manifests/mwextension.pp: git::clone { $name:
61modules/profile/manifests/ci/gitcache.pp: git::clone { 'operations/puppet':
62modules/profile/manifests/ci/gitcache.pp: git::clone { 'mediawiki/core':
63modules/profile/manifests/ci/gitcache.pp: git::clone { 'mediawiki/vendor':
64modules/profile/manifests/zuul/server.pp: git::clone { 'integration/config':
65modules/profile/manifests/analytics/refinery/source.pp: git::clone { 'refinery_source':
66modules/profile/manifests/switchdc.pp: git::clone { 'operations-switchdc':
67modules/profile/manifests/discovery_dashboards/production.pp: git::clone { 'wikimedia/discovery/rainbow':
68modules/profile/manifests/discovery_dashboards/production.pp: git::clone { 'wikimedia/discovery/twilightsparql':
69modules/profile/manifests/discovery_dashboards/production.pp: git::clone { 'wikimedia/discovery/prince':
70modules/profile/manifests/discovery_dashboards/production.pp: git::clone { 'wikimedia/discovery/wetzel':
71modules/profile/manifests/discovery_dashboards/production.pp: git::clone { 'wikimedia/discovery/wonderbolt':
72modules/profile/manifests/discovery_dashboards/development.pp: git::clone { 'wikimedia/discovery/rainbow':
73modules/profile/manifests/discovery_dashboards/development.pp: git::clone { 'wikimedia/discovery/twilightsparql':
74modules/profile/manifests/discovery_dashboards/development.pp: git::clone { 'wikimedia/discovery/prince':
75modules/profile/manifests/discovery_dashboards/development.pp: git::clone { 'wikimedia/discovery/wetzel':
76modules/profile/manifests/discovery_dashboards/development.pp: git::clone { 'wikimedia/discovery/wonderbolt':
77modules/profile/manifests/microsites/annualreport.pp: git::clone { 'wikimedia/annualreport':
78modules/profile/manifests/microsites/design.pp: git::clone { 'design/landing-page':
79modules/profile/manifests/microsites/design.pp: git::clone { 'design/style-guide':
80modules/profile/manifests/microsites/wikibase.pp: git::clone { 'wikibase/wikiba.se-deploy':
81modules/profile/manifests/microsites/transparency.pp: git::clone { 'wikimedia/TransparencyReport':
82modules/profile/manifests/microsites/transparency.pp: git::clone { 'wikimedia/TransparencyReport-private':
83modules/profile/manifests/microsites/research.pp: git::clone { 'research/landing-page':
84modules/profile/manifests/calico/builder.pp: git::clone{ 'operations/calico-containers':
85modules/profile/manifests/calico/builder.pp: git::clone { 'operations/calico-cni':
86modules/profile/manifests/calico/builder.pp: git::clone { 'operations/calico-k8s-policy-controller':
87modules/profile/manifests/openstack/base/nodepool/service.pp: git::clone { 'integration/config':
88modules/profile/manifests/wmcs/tenants/libraryupgrader.pp: git::clone {'labs/libraryupgrader':
89modules/profile/manifests/performance/site.pp: git::clone { 'performance/docroot':
90modules/profile/manifests/kubernetes/deployment_server.pp: git::clone { 'operations/deployment-charts':
91modules/profile/manifests/docker/builder.pp: git::clone { 'operations/docker-images/production-images':
92modules/profile/manifests/grafana.pp: git::clone { 'operations/software/grafana/simple-json-datasource':
93modules/toollabs/manifests/images.pp: git::clone { 'operations/docker-images/toollabs-images':
94modules/toollabs/manifests/composer.pp: git::clone { 'composer':
95modules/puppetmaster/manifests/gitprivate.pp: git::clone { 'operations/private':
96modules/puppetmaster/manifests/gitclone.pp: git::clone { 'labs/private':
97modules/puppetmaster/manifests/gitclone.pp: git::clone {
98modules/puppetmaster/manifests/base_repo.pp: git::clone { 'operations/puppet':
99modules/testreduce/manifests/init.pp: git::clone { 'mediawiki/services/parsoid/testreduce':
100modules/wikilabels/manifests/web.pp: git::clone { 'wikilabels-wikimedia-config':
101modules/geowiki/manifests/init.pp: git::clone { 'geowiki-scripts':
102modules/geowiki/manifests/private_data.pp: git::clone { 'geowiki-data-private':
103modules/phragile/manifests/init.pp: git::clone { 'phragile':
104modules/phragile/manifests/init.pp: git::clone { 'composer':
105modules/wikistats/manifests/init.pp: git::clone { 'operations/debs/wikistats':

And then you'll have to add all the things that are deployed via scap.

Jrbranaa added a project: Code-Health.Mar 27 2018, 11:05 PM
Aklapper added a project: wikimedia.biterg.io.Mar 28 2018, 9:08 AM
Aklapper updated the task description. (Show Details)Apr 18 2018, 2:41 PM
Legoktm added a subscriber: Legoktm.Apr 18 2018, 4:00 PM

https://phabricator.wikimedia.org/source/tool-ci/browse/master/build_table.py;3ed885277a38ddb5c7c8a6c3c9b666fb7b13ae10$49 is how my tools currently get a list of Wikimedia deployed extensions.

Some of the repositories listed in P6909 are only deployed in Cloud Services (e.g. extdist), not production.

mmodell added a subscriber: mmodell.Apr 23 2018, 4:40 PM
greg triaged this task as Medium priority.Apr 23 2018, 4:41 PM
Aklapper updated the task description. (Show Details)Jun 20 2018, 7:01 PM
Jrbranaa added a parent task: T199253: TEC13:O1.1:Q1 Goal - Investigate and propose record of origin (ROO) for deployed code (currently Developers/Maintainers page).Jul 10 2018, 5:25 PM
zeljkofilipin awarded a token.Aug 8 2018, 2:49 PM
zeljkofilipin added a subscriber: zeljkofilipin.
Jdforrester-WMF added a subscriber: Jdforrester-WMF.Aug 24 2018, 12:42 AM
Aklapper updated the task description. (Show Details)Aug 24 2018, 2:48 PM
Aklapper added a comment.Aug 24 2018, 4:33 PM

I made a bunch of updates today on https://www.mediawiki.org/w/index.php?title=Developers/Maintainers&action=history and corresponding extension wiki home pages, however help with sorting out these items is welcome:

Deployed on WMF servers according to https://phabricator.wikimedia.org/diffusion/MREL/browse/master/make-wmf-branch/config.json and https://phabricator.wikimedia.org/source/mediawiki-config/browse/master/wmf-config/extension-list but the extension wiki page does not use {{TNT|OnWikimedia}} and it is not listed on https://www.mediawiki.org/wiki/Developers/Maintainers :

  • extensions/CongressLookup
  • extensions/FileExporter
  • extensions/intersection
  • extensions/JADE
  • extensions/LdapAuthentication
  • extensions/PropertySuggester
  • extensions/WikimediaBadges

Extension wiki page uses {{TNT|OnWikimedia}} and it is listed as deployed on https://www.mediawiki.org/wiki/Developers/Maintainers but not in https://phabricator.wikimedia.org/diffusion/MREL/browse/master/make-wmf-branch/config.json and https://phabricator.wikimedia.org/source/mediawiki-config/browse/master/wmf-config/extension-list :

Extension wiki page uses {{TNT|OnWikimedia}} but not listed as deployed on https://www.mediawiki.org/wiki/Developers/Maintainers but not in https://phabricator.wikimedia.org/diffusion/MREL/browse/master/make-wmf-branch/config.json and https://phabricator.wikimedia.org/source/mediawiki-config/browse/master/wmf-config/extension-list :

  • Extension:DataModel
  • Extension:DataTypes
  • Extension:DataValues
  • Extension:DataValuesCommon
  • Extension:DataValuesInterfaces
  • Extension:Diff
  • Extension:FundraisingEmailUnsubscribe
  • Extension:ValueView
Legoktm added a comment.Aug 26 2018, 4:18 AM
In T190891#4530610, @Aklapper wrote:

Extension wiki page uses {{TNT|OnWikimedia}} and it is listed as deployed on https://www.mediawiki.org/wiki/Developers/Maintainers but not in https://phabricator.wikimedia.org/diffusion/MREL/browse/master/make-wmf-branch/config.json and https://phabricator.wikimedia.org/source/mediawiki-config/browse/master/wmf-config/extension-list :

It's mediawiki/extensions/intersection.

Extension wiki page uses {{TNT|OnWikimedia}} but not listed as deployed on https://www.mediawiki.org/wiki/Developers/Maintainers but not in https://phabricator.wikimedia.org/diffusion/MREL/browse/master/make-wmf-branch/config.json and https://phabricator.wikimedia.org/source/mediawiki-config/browse/master/wmf-config/extension-list :

  • Extension:DataModel
  • Extension:DataTypes
  • Extension:DataValues
  • Extension:DataValuesCommon
  • Extension:DataValuesInterfaces
  • Extension:Diff
  • Extension:ValueView

These are all PHP libraries that should be in [[Category:PHP libraries]], not listed as MediaWiki extensions afaik. Someone more familiar with Wikidata should confirm.

Jrbranaa moved this task from Backlog to In-progress on the Release-Engineering-Team (Kanban) board.Aug 30 2018, 5:57 PM
Jrbranaa added a comment.Sep 6 2018, 6:09 PM

@Aklapper, I've been spending a bit of time on this and one of the directions I'm leaning is to have the ROO be an output of the deployment review process (Review Queue today). My thought being that the ROO could be used by deployers as an authoritative source to determine whether or not a new extensions, services, etc... should be deployed. I spoke a to Tyler a bit about this and it seems like it would be of value.

What are some other use cases for the ROO? One thing that's mentioned is it's need to be machine readable. What systems would you see consuming this list? FWIW, I too think it should be machine readable. I only ask to tease out some of the use cases that I may not be aware of.

On a related topic, I'm planning to spin up work around augmenting/revamping the review process.

Aklapper added a comment.EditedSep 7 2018, 11:54 AM

What systems would you see consuming this list?

All this still won't fix outdated information per se but I'd say it's less likely to have outdated information when there is one place to update instead of several places. My latest example was two days ago trying to find out who to fix https://phabricator.wikimedia.org/T203427 : I went to mw:Developers/Maintainers which listed "Discovery" for "Kartographer". On the Discovery IRC channel I was told that's not correct anymore (was corrected here).

hashar added a subscriber: hashar.Sep 7 2018, 12:01 PM
zeljkofilipin mentioned this in T199133: Find top 15 target projects that could use Selenium tests to prevent incidents.Sep 20 2018, 3:39 PM
Jrbranaa merged a task: T199253: TEC13:O1.1:Q1 Goal - Investigate and propose record of origin (ROO) for deployed code (currently Developers/Maintainers page).Oct 22 2018, 4:42 PM
Quiddity added a subscriber: Quiddity.Oct 25 2018, 7:00 AM
Quiddity mentioned this in Unknown Object (Task).Oct 25 2018, 7:04 AM
Quiddity updated the task description. (Show Details)Oct 25 2018, 7:08 AM
greg edited projects, added Release-Engineering-Team-TODO (201907); removed Release-Engineering-Team (Kanban).Jul 1 2019, 9:20 PM
greg moved this task from INBOX to Doing on the Release-Engineering-Team-TODO (201907) board.Jul 1 2019, 9:21 PM
greg added a project: Release-Engineering-Team (Code Health).Jul 6 2019, 5:59 AM
Jrbranaa edited projects, added Release-Engineering-Team-TODO (201908); removed Release-Engineering-Team-TODO (201907).Jul 29 2019, 4:30 PM
Jrbranaa edited projects, added Release-Engineering-Team-TODO; removed Release-Engineering-Team-TODO (201908).Jul 29 2019, 4:33 PM
Jrbranaa moved this task from Should be empty (use Release-Engineering-Team) to Epics on the Release-Engineering-Team-TODO board.
hashar removed a subscriber: hashar.Oct 7 2019, 12:33 PM
brennen added a subscriber: brennen.Dec 2 2019, 5:18 AM
Jrbranaa added subscribers: mark, faidon.Mar 6 2020, 12:13 AM

Recently had a discussion with @mark and @faidon about some work that SRE is planning to do. Specifically the creation of a Service Catalog. On the surface this seems like it would potentially become the ROO mentioned in this task.

Jrbranaa removed Jrbranaa as the assignee of this task.Dec 3 2020, 1:26 AM
Jrbranaa edited projects, added Quality-and-Test-Engineering-Team (QTE); removed Release-Engineering-Team-TODO, Release-Engineering-Team (Code Health).Dec 3 2020, 1:29 AM
Dzahn removed a subscriber: Dzahn.Dec 3 2020, 2:42 AM
Elitre mentioned this in T271700: Easier way to understand volunteers impact/interaction with WMF teams.Jan 11 2021, 10:42 AM
Aklapper added a parent task: T271700: Easier way to understand volunteers impact/interaction with WMF teams.Feb 23 2021, 11:51 AM
Jayprakash12345 mentioned this in T281977: Reconsidering of eligibility criteria for technical contributions in elections.May 5 2021, 12:48 PM
Jrbranaa mentioned this in T285058: Canonical Service Catalog.Aug 10 2021, 6:44 PM
• Addshore added a subscriber: • Addshore.Sep 21 2021, 2:44 PM
Jrbranaa moved this task from Inbox to Quality Engineering on the Quality-and-Test-Engineering-Team (QTE) board.Apr 29 2022, 4:36 PM
Aklapper removed a project: wikimedia.biterg.io.Jul 21 2022, 2:04 PM
zeljkofilipin mentioned this in T318599: Developers/Maintainers graph automation.Sep 26 2022, 6:14 PM
Ladsgroup added a subscriber: Ladsgroup.Feb 27 2023, 10:35 PM
TAdeleye_WMF edited projects, added Quality-and-Test-Engineering-Team (QTE) (Quality Engineering); removed Quality-and-Test-Engineering-Team (QTE).Mar 29 2023, 2:10 PM
Ladsgroup renamed this task from Develop canonical/single record of origin, machine readable list of all repos deployed to WMF sites. to Develop canonical/single record of origin, machine readable list of all repos deployed to WMF sites.Apr 3 2023, 7:29 PM
brennen mentioned this in T330347: A tool for displaying the canonical location of all Wikimedia repositories and any known mirrors.Apr 4 2023, 6:46 PM
Ladsgroup mentioned this in T334956: Consolidate various in-repo docs about getting started into a single standard for all repos.Thu, May 11, 1:26 PM
kchapman added a subscriber: kchapman.Fri, May 12, 6:15 PM
EpicPupper added a subscriber: EpicPupper.Sun, May 28, 3:16 AM
Content licensed under Creative Commons Attribution-ShareAlike 3.0 (CC-BY-SA) unless otherwise noted; code licensed under GNU General Public License (GPL) or other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL