Page MenuHomePhabricator

integration-agent-docker machines excessively pull some Wikibase related Git repos in Diffusion
Open, Stalled, LowPublic

Description

This might be actual job configurations but no idea how to debug further, thus tagging Continuous-Integration-Config for the time being.

According to https://phabricator.wikimedia.org/diffusion/pulllog/ , integration-agent-docker-*.integration.eqiad1.wikimedia.cloud machines pull certain repos from Diffusion about 90 times within 60 minutes which seems excessive.
From a quick look, this seems to only be the case for 3 and a half repos.

Data from the last 30 days, not filtered on IPs (ofc there's also "normal" pulling from 3rd party or internal codesearch.eqiad1.wikimedia.cloud):

mysql:phstats@m3-slave.eqiad.wmnet [phabricator_maniphest]> SELECT COUNT(r.name) AS pulls, r.name FROM phabricator_repository.repository_pullevent rp INNER JOIN phabricator_repository.repository r ON r.phid = rp.repositoryPHID GROUP BY r.name ORDER BY pulls DESC LIMIT 9;
+-------+-----------------------------------+
| pulls | name                              |
+-------+-----------------------------------+
| 47199 | DataValues Javascript             |
| 47196 | Wikibase Data Model JavaScript    |
| 47195 | Wikibase Serialization JavaScript |
| 12138 | new lexeme special page           |
|  3836 | netbox-exported-dns               |
|  1407 | wikibase-termbox                  |
|   322 | MediaWiki                         |
|   293 | skin-Vector                       |
|   122 | extension-HeaderTabs              |
+-------+-----------------------------------+
9 rows in set (0.249 sec)

Event Timeline

Some of those pulls come from mediawiki/extensions/Wikibase which has several git submodules pointing to Phabricator diffusion. https://gerrit.wikimedia.org/g/mediawiki/extensions/Wikibase/+/refs/heads/master/.gitmodules has:

.gitmodules
[submodule "view/lib/wikibase-data-values-value-view"]
	path = view/lib/wikibase-data-values-value-view
	url = https://gerrit.wikimedia.org/r/data-values/value-view
[submodule "view/lib/wikibase-serialization"]
	path = view/lib/wikibase-serialization
	url = https://phabricator.wikimedia.org/source/wikibase-serialization.git
[submodule "view/lib/wikibase-data-values"]
	path = view/lib/wikibase-data-values
	url = https://phabricator.wikimedia.org/source/datavalues-javascript.git
[submodule "view/lib/wikibase-data-model"]
	path = view/lib/wikibase-data-model
	url = https://phabricator.wikimedia.org/source/wikibase-data-model.git
[submodule "view/lib/wikibase-termbox"]
	path = view/lib/wikibase-termbox
	url = https://gerrit.wikimedia.org/r/wikibase/termbox

[submodule "lib/resources/wikibase-api"]
	path = lib/resources/wikibase-api
	url = https://gerrit.wikimedia.org/r/wikibase/javascript-api

They are JavaScript npm packages and adding them as submodules was done via T177087 .


The libraries canonical places are on the WMDE GitHub organization. Since we want the deployment to not have a dependency upon GitHub what was done is that Phabricator is mirroring the Github repositories. That was done by 2edbce09c29b6ca33b8f5c05f7e98613e5c647cf which is that Phabricator is pulling from Github.

https://phabricator.wikimedia.org/source/wikibase-serialization/manage/uris/ has:

Aklapper changed the task status from Open to Stalled.Nov 27 2023, 3:10 PM
Aklapper triaged this task as Low priority.

Setting "stalled" status as this depends on authority to declare where all our deployed code bases are supposed to be hosted.

Setting "stalled" status as this depends on authority to declare where all our deployed code bases are supposed to be hosted.

@thcipriani, @ttaylor: I'd appreciate a general policy at some point, regarding Gerrit, GitLab, Github, etc. (See also related non-public T213246.)

In combination with T362401 the setup seems to create issues for some folks: https://www.mediawiki.org/wiki/Topic:Yepfvppzskaq0x6r