Page MenuHomePhabricator

[lima-kilo] Improve convergence
Closed, ResolvedPublicFeature

Description

Feature summary (what you would like to be able to do and where):

Setup a lima-kilo instance on a low bandwidth/flakey internet connectivity that requires multiple convergence attempts.

Use case(s) (list the steps that you performed to discover that problem, and describe the actual underlying problem which you want to solve. Do not describe only a solution):

When making changes to toolforge infrastructure e.g. jobs-api, components-api it is useful to be able to test against a toolforge like environment, the current tooling for this is lima-kilo.

Sometimes work is being done in environments that have intermittent or slow network connectivity, while attempting to setup lima-kilo in this environment (4g/5g mobile data in the Azores) it was found to be very painful.

The main challenge is the binaries are re-downloaded on every execution, so if a run is interrupted the previously installed binaries need to be re-downloaded (and thrown away).

While it might be possible to converge using the existing tags, it is not clear if this is complete or which order things need to be executed in.

Benefits (why should this be implemented?):

Improve the developer experience when trying to contribute to and improve toolforge infrastructure.

Event Timeline

Restricted Application added a subscriber: Aklapper. · View Herald Transcript

Initial changes:

More contentious:

With the above applied, after the initial ansible run, you get a clean run except for foxtrot-ldap which T400167 is a good solution for (same pattern as lima_kilo_toolforge_deploy_components).

Creating sub-tasks for review, let me know if you want to discuss any of this in more detail.

Ansible has many shortcomings when trying to make it re-entrant, essentially you have to implement most if not all the logic yourself. We had some of that code in lima-kilo in the past and proved to be quite complex, error prone, and hard to maintain, enough for us to drop that approach and simplify drastically the ansible manifests, relying instead on the tags to run parts of the manifests again when needed.

I think that the best course of action here is:

  • use https://gitlab.wikimedia.org/repos/cloud/toolforge/lima-kilo/-/merge_requests/251, to run all the tags from the failed one to the end again (ex. ./start_devenv.sh --tags k9s: will rerun everything starting from the k9s tag)
  • split roles more fine grained, so they can be addressed with tags (currently the toolforge components are all in one single role/tag for example)
  • improve documentation to reflect all this
dcaro triaged this task as Medium priority.Aug 27 2025, 12:47 PM

I started creating a draft of decision request that would potentially address this too.

dcaro added a project: tools-platform-team.

I'll close this once the subtask is done, I think that we can reopen a new one once we finish deciding the decision request.