Page MenuHomePhabricator

Improve git integration with the cloud-vps puppet/hiera enc
Closed, ResolvedPublic

Description

We've been hitting rate limits when the enc tries to update its archival git repo. There are also likely some race conditions in there.

Hashar's thoughts are:

you might need to add a global lock shared between the process and wait for it to avoid the race condition
if that scripts always run on the same machine, that is probably easy to add (a file based lock would do)
else push over https instead of ssh
it is not subject to a per user limit as far as I am aware, or if there is a limit it is way higher
a one off password can be generated for the user from https://gerrit.wikimedia.org/r/settings/#HTTPCredentials (there is surely a REST API entry point to generate the token)
which could then be hold in the puppet secret and passed to the script somehow
then you git push https://xxx
there is still the fetch/rebase/push short lived race condition, but it apparently hasn't been much of an issue until now
so push over https might be the easiest :]
while a global lock would be more correct,but I have no idea what kind of issue that could lead to :]

Event Timeline

Restricted Application added a subscriber: Aklapper. · View Herald Transcript

Change 871298 had a related patch set uploaded (by Majavah; author: Majavah):

[operations/puppet@production] openstack: encapi: perform git updates server-side

https://gerrit.wikimedia.org/r/871298

Change 871298 merged by Andrew Bogott:

[operations/puppet@production] openstack: encapi: perform git updates server-side

https://gerrit.wikimedia.org/r/871298

The current single worker based system should fix the concurrency issues.