When looking at #Gerrit issues during spring 2019 (T221026) we noticed a lot of git operations emanating from various tools on WMCS.
They are mostly reads, and there is no firm indication that the batch requests are overloading Gerrit. But given the load they impose, it might be wise to shift the load to a mirror. They notably do not need to be 100% up to date with the master and could suffer the slight delay incurred by replication.
Relevant extracts from T221026:
@mmodell wrote:
From looking at http requests per minute in javamelody, over 1 year, I see that traffic has increased a lot:
https://gerrit.wikimedia.org/r/monitoring?part=graph&graph=httpHitsRate (http hits per minutes):
{F28682710}
@thcipriani pointed out the mean stays identical, but the max has grown in March 2019 from roughly 4k/minutes to 6k/minutes.
@hashar proposed: Would it make sense to set a readonly replica such as git.wikimedia.org to offload Gerrit? The bots/scripts running on WMCS could be easily made to point to that mirror. And listed:
Out of 623k https requests in April 17th access logs:
| Requests | IP | DNS PTR
|--|--|--
| 84110 | 172.16.1.221 | codesearch4.codesearch.eqiad.wmflabs.
| 69921 | 2620:0:861:102:10:64:16:8 | phab1001.eqiad.wmnet.
| 51736 | 172.16.1.85 | extdist-02.extdist.eqiad.wmflabs.
| 51736 | 172.16.1.84 | extdist-01.extdist.eqiad.wmflabs.
| 51676 | 172.16.1.86 | extdist-03.extdist.eqiad.wmflabs.
| 16465 | 172.16.5.187 | integration-slave-docker-1051
| 16116 | 172.16.5.162 | integration-slave-docker-1048
| 14709 | 172.16.5.181 | integration-slave-docker-1050
| 13660 | 172.16.1.36 | integration-slave-docker-1041
| 13579 | 172.16.0.26 | integration-slave-docker-1054
| 12990 | 172.16.6.184 | integration-slave-docker-1043
| 12909 | 172.16.3.86 | integration-slave-docker-1040
| 11672 | 172.16.7.168 | integration-slave-docker-1034
| 10847 | 172.16.5.190 | integration-slave-docker-1052
| 9705 | xxxxx | some public internet IP
| 8786 | 172.16.3.87 | integration-slave-docker-1037
Probably codesearch ( https://codesearch.wmflabs.org/ ), Phabricator and extdist ( https://www.mediawiki.org/wiki/Extension:ExtensionDistributor ) could be moved to a use a mirror.
The CI slaves do hammer Gerrit :-/
Note that its for any HTTP request, not just git-upload-pack. But the result is similar when filtering for upload-pack.
Not taken in account, git fetch from the zuul-mergers which are done over ssh with the `jenkins-bot` user.