Hey @brennen -

This seems like as good a task as any to inquire as to whether Release Engineering expects the current blubber/pipelinelib config data stored within specific repos to look about the same for projects migrating to Gitlab. i.e the blubber.yaml and config.yaml files that currently exist within the .pipeline directory for most repos. Thanks.

This seems like as good a task as any to inquire as to whether Release Engineering expects the current blubber/pipelinelib config data stored within specific repos to look about the same for projects migrating to Gitlab. i.e the blubber.yaml and config.yaml files that currently exist within the .pipeline directory for most repos. Thanks.

I'd say that's an open question. We hope to have a better, if imperfect, idea of the answer at the end of the current sprint.

Ok, sounds fair. Thanks for the response.

For the archival of Gerrit repository it is done manually via Projects-Cleanup which has a form listing all the steps required. We had a long standing task to automatize that process which is T175499

I have looked at how SRE implements runbook. It is based on Spicerack and Cookbooks. The idea is to write the steps in Python reusing modules that gives a bunch of functionalities (such as !log), but that requires access to the Cumin orchestration hosts as far as I can tell and non root don't have access to it (T244840).

I am wondering whether we can implement the runbook as a Cookbook and have run those from the deployment server. Given the runbook would do REST API queries to Gerrit/Phabricator/Gitlab, it does not need to run on any specific hosts and thus has no need for Cumin. But maybe that is overkill.

An Ansible playbook running from the deployment server would be a good fit but I would rather not introduce it to our stack. Cookbook/Spicerack might well be overkill for issuing rest queries. So I am looking for an alternative.

SRE foundations pointed me at @Volans who wrote and knows everything about runbooks implementations and could surely recommend an existing tool or guide us toward using cookbook if it is at all possible :]

In T307538#7926644, @hashar wrote:

SRE foundations pointed me at @Volans who wrote and knows everything about runbooks implementations and could surely recommend an existing tool or guide us toward using cookbook if it is at all possible :]

@hashar There are current plans and partial implementation of allowing to run Cookbooks with our own accounts (via Kerberos) on a dedicated unprivileged host.
This is tracked at T244840 and T284302.
A basic implementation for Cumin is currently being tested, see https://wikitech.wikimedia.org/wiki/Cumin#Rootless_production_infrastructure, but that's just one small piece of the puzzle.

The available options that I see depends on the timeline that you have in mind:

• Add your use case to the above task(s), get in touch with Moritz and me to understand requirements and limitations and then try to push management for getting that project prioritized in the planning for next quarter.
• Write a normal cookbook that for now will require an SRE to be run, but will be easily be included in the above effort once that's done. Once the cookbook is stable enough and well tested we could also consider a way to allow RelEng engineers to write somewhere the repos for which it should be run and then have some automatic way of running it. [we have another separate long-term proposal to allow to schedule cookbook runs that would be related to this].
• If time wise you need something now, then I would probably implement it as a normal Python script in the puppet repo but using the wmflib package, in particular those features:
  • actions: to store the steps made by the script during execution
  • phabricator: to update a given task with the above results, that will become something like T307997#7933764
  • requests: to get a requests session with timeout, autoretry capabilities and User-Agent
  • irc: to write to IRC and/or !log to SAL via IRC
  • interactive: a bunch of helpers to interact with the user running the script
  • and much more

Re: The user-facing manual part of this, I've moved GitLab/Migrating_a_Project to GitLab/Hosting a project on GitLab and starting filling out some details there.

See also T327332