This might mean using a structure for the config, maybe something like:
source_repo: url: http://gitlab..... ref: main ...
This has to be tested to make sure it bypasses the caches of github, for example by setting up a github pipeline that does a deployment right after a merge, and making sure it's getting the latest commit, and not the previous "cached" one.
| Status | Subtype | Assigned | Task | ||
|---|---|---|---|---|---|
| Resolved | LucasWerkmeister | T320140 Migrate wd-shex-infer from Toolforge GridEngine to Toolforge Kubernetes | |||
| Resolved | matmarex | T319707 Migrate dtcheck from Toolforge GridEngine to Toolforge Kubernetes | |||
| Resolved | Legoktm | T320062 Migrate steve-adder from Toolforge GridEngine to Toolforge Kubernetes | |||
| Resolved | Legoktm | T320011 Migrate rfa-voting-history from Toolforge GridEngine to Toolforge Kubernetes | |||
| In Progress | dcaro | T194332 [builds-api,components-api,webservice,jobs-api] Make Toolforge a proper platform as a service with push-to-deploy and build packs | |||
| Resolved | dcaro | T393564 [Hypothesis] WE6.3.10 start a beta for the push-to-deploy features | |||
| Open | BUG REPORT | None | T401868 [components-api,beta] Config not updated from remote source | ||
| Open | dcaro | T402764 [components-api] allow specifying `source_repo`+`ref` for the config |
It would be a breaking change, but perhaps:
source: url:
source: repo_url: branch: main
That would follow the same structure as the other elements, where 1 key has multiple object types with it's own keys, and avoids needing to deal with e.g. source_url and source_repo both being specified.
The URL should probably be validated to either https:// or git:// with branch being optional (default to the remote).
I have a significant preference on also being able to specify path, which allows keeping all the files in 1 branch (e.g. https://github.com/cluebotng/component-configs) rather than needing to create a branch per tool (this is mostly so it's easy for a human to see what changed).
It might be worthwhile to perform the operation as an archive (apparently not supported by GitHub) or sparse checkout, to avoid very large repos from needing to be cloned.
I made https://phabricator.wikimedia.org/T402790 as it's not directly related to this, but implementation of this is perhaps a good time for consideration.
It's something that has niggled at me in multiple angles during trying to get this working. Feel free to un-link if you want to keep this issue clean.
+1 for both, though being non-backwards compatible we will have to support both syntaxes for a while
First stab at this:
https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/121 (validate repo url - not strictly required but generally makes sense)
https://gitlab.wikimedia.org/damian/components-api/-/merge_requests/1 (actual logic - on-top of 121, using new validated type)
https://gitlab.wikimedia.org/damian/components-api/-/merge_requests/2 (if happy using new lib, use it in the existing places - ontop of 1)
Will do some testing when lima-kilo finishes building.
Key question for now is if you are happy to bring in GitPython or want to stick to subprocess (can make a class around subprocess if needed)
I had to make some changes due to no writeable tmp directory, but the above appears to work (at least locally).
Some testing output added to https://gitlab.wikimedia.org/damian/components-api/-/merge_requests/1.
Marking all the above and not draft and leaving for review (assume the MRs will get re-created for the pipeline to run).
damian updated https://gitlab.wikimedia.org/damian/components-api/-/merge_requests/1
Add source support to ToolConfig
damian opened https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/127
Add validated type for git urls
damian closed https://gitlab.wikimedia.org/damian/components-api/-/merge_requests/1
Add source support to ToolConfig
damian updated https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/127
Add source support to ToolConfig
Rebased for the conflicts from reuse_from.
Moved all MRs to the target repo - will need re-creating for the pipeline anyway;
https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/121
1 -> https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/127
2 -> https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/128
damian opened https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/132
Expand source support in ToolConfig
damian closed https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/127
Add source support to ToolConfig
To close this out only https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/132 is pending, the rest of the links referenced above are either merged or superseded.
I struggle to see why we need an handle configurations saved in a git repo as a different from ordinary url, given that the files in a particular branch in both gitlab and github (and most git servers) can be defind as a simple url.
What am I missing?
typical example is https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/raw/replace_destination_image_comparison_with_image_name/LICENSE?ref_type=heads
the above link can be read by anything, we do not need to know it's in a git repo in branch replace_destination_image_comparison_with_image_name
@Raymond_Ndibe please read the history on the related tickets, specifically https://phabricator.wikimedia.org/T401868
Non-git http interfaces have full rights to cache, which they do, which combined with the inability to specify a specific ref, makes the http interface useless.
Honestly the code was written more than 2 weeks ago, this ticket has been open more than a month... so the time for a discussion on if it should be done seems a little time wasting at this point, especially given this ticket was for the literal implementation.