Page MenuHomePhabricator

Formalize and share the spicerack/cumin release process
Open, LowPublic

Description

Currently, spicerack and cumin are both released by a single person, using custom personal tailored scripts.
It would be very beneficial if that task was able to be done by at least someone else, and better formalized and documented.

This task is meant to discuss and agree on how to achieve the above.

It's expected for the managers to agree/guide on how to share responsibilities.

Snippet of the irc chat that triggered this:

121027    dcaro | volans: I would like to do a cumin + spicerack release, now that a bunch of the fixes and new functionality for wmcs toolforge/vps cookbooks is merged,
                | What's the process?
121108   volans | dcaro: I can take care of the spicerack one, for the cumin one I need to get something else finished before releasing it
121139    dcaro | volans: can I help anyhow? would be nice to learn how to do it
121331   volans | I run a script, it's a bit tailored to my setup, it requires an account on pypi and also I'm signing all the releases with my gpg
121405    dcaro | my pypi account user is dcaro
121458    dcaro | my gpg key fingerprint is '718083A2AC8B314FB4CE11714071C7E1D26269C3'
121503    dcaro | ^ is that enough?
122057   volans | for what? the release script is totally experimental and tailored to my local env
122141   kormat | volans has made his own ball, and it only works at his home
122221   volans | it's more that I didn't had time to make it generic enough and uses a config file per repo that are not committd
122227   volans | because experimental
122304   volans | it also requires a script in the build host and one in the apt host to build the deb package and then reprepro it
122331   volans | but at least I use the same script for all the python packages I manage, because they share the same setup
...
122837    dcaro | volans: I'll add a note in the spicerack roadmap to add/generalize that script, ack?
123021    dcaro | I think it would be good if more than one person can release ;)
123344   volans | I agree in general, but is not just fixing the script. It's also a matter of who's responsabile for any follout of its deploy to production. What/how to
                | test it when releasing it, coordinate with long-running cookbooks, etc.
123503    dcaro | that's ok, let's write that down
123732    dcaro | the only way of having more than one person being able to release, is well, sharing that responsibility and knowledge
123941   volans | yes and AFAIK SRE I/F is responsible for that area, but I'm not a manager
124132    dcaro | let's get one then :), I'll open a phab task to start a chat about it

Spicerack roadmap doc: https://docs.google.com/document/d/10FIxIw5yTmFUdDC0M9ETTi5zDFWiqPJ2665sLJBoDTs/edit

Event Timeline

Restricted Application added a subscriber: Aklapper. · View Herald Transcript
LSobanski renamed this task from Formalize and share the spicerack/cumin realease process to Formalize and share the spicerack/cumin release process.Mar 4 2021, 11:52 AM
LSobanski updated the task description. (Show Details)
jbond triaged this task as Medium priority.Mar 4 2021, 11:52 AM
dcaro updated the task description. (Show Details)
faidon lowered the priority of this task from Medium to Low.Mar 4 2021, 12:47 PM

Judging from the last two lines of that transcript, I've been summoned :)

Documenting the release process for posterity and to increase the bus factor is definitely a very good goal to have -- we should spend some time on that. This is a very valid task, thanks for opening it :)

In terms of the other question, around responsibility: Cumin and the Spicerack framework ownership lies in the SRE Infrastructure Foundations team, as a foundational piece of Tooling & Automation, one of the team's core responsibilities. These are open source projects, and everyone is more than welcome to contribute and help shape their future, and work together collaboratively (as you have, @dcaro!). My expectation, however, is that release management for both the projects themselves, as well as their deployment to our fleet are to be coordinated with the tooling and automation lead (@Volans). Hope that helps!

In terms of the other question, around responsibility: Cumin and the Spicerack framework ownership lies in the SRE Infrastructure Foundations team, as a foundational piece of Tooling & Automation, one of the team's core responsibilities. These are open source projects, and everyone is more than welcome to contribute and help shape their future, and work together collaboratively (as you have, @dcaro!). My expectation, however, is that release management for both the projects themselves, as well as their deployment to our fleet are to be coordinated with the tooling and automation lead (@Volans). Hope that helps!

Thanks for the explanation, but there's several points here that still need discussion. There's three different processes that seem to be bundled up here:

  1. Releasing a new Pypi package
  2. Releasing a new debian package (this could be discussed also on what repo that package is released to)
  3. Upgrading the wiki production Cumin hosts

Now, the last point is clear that should be taken over by someone that knows the ins and outs of that deployment, and the SRE I/F team is the right one. Same way that if WMCS ends up having a VM dedicated for this, I would expect WMCS to be the ones deploying it there.

For the first two (pypi + debian), it's not so clear, specially taking into account that these tools are not only being used in wiki prod but also being used in cloud prod (and laptops), so being able to share that release management duties with someone from WMCS (for example), allows both teams to collaborate in the tooling, and unblock themselves and move both environments at different speeds (given the different needs).

Of course, coordination and collaboration are part of the release process, and @Volans being a huge part of the project is going to be key coordinating those, but there's a very subtle and important difference between coordinating with someone and needing someone's action.

Even getting to be able to release on Pypi would help us considerably to be able to unblock ourselves when needed (btw. this might tie up with the goal of being able to restrict what versions of spicerack are allowed to run on certain hosts, that I forgot to add to the roadmap xd).

Thanks for the explanation, but there's several points here that still need discussion.

I recognize that gatekeeping at its extremes results in unnecessary blocking and waiting, reduces autonomy, and creates complexity in planning in order to align the work. At the same time, the other extreme is lack of ownership, accountability, design-by-commitee and analysis-paralysis. These trade-offs are hard to get right and need constant realignment. They have been incorporated into our strategy from the beginning, with a layered design, both in terms of technical architecture, and team structure, both across SRE and within the I/F team itself.

Underlying foundational pieces (choice of SRE-wide tooling itself, as well as various software components, themselves layered, that have been written for this purpose etc.) have clear ownership in SRE in the Foundations team, and clear ownership and leadership within the team, while other SRE teams and service teams self-service/self-maintain workflow-specific cookbooks, in a separate repository.

I don't have enough evidence so far that the current release management and maintenance model doesn't work. I'd be willing to be convinced otherwise, but for this particular collaboration with the WMCS team, I feel it's too early in the process (~1-2 months in?) to have enough data, especially in a quarter where we did not even plan this work together (so some amount of blocking is to be expected).

Appreciate your feedback here, but I'm afraid this is not something I'm going to revisit at this point in time. I welcome feedback from the team and team's leadership (as well from every other team reading this) over time of cases were one team blocked another, and data points and/or proposals across the technical/people spectrum -- we're in this together, and we're here to help one another be effective, and this is something I care deeply about. Hope this makes sense.

Thank you for weighing in Faidon. I thought I would also add my perspective.

I'd like us collectively to think about how we can effectively support each other as we seek to expand the usage and use cases for spicerack. Releasing / distributing a binary is but one piece of this.

Looking forward, this is an example of WMCS needing a different version of spicerack than is generally deployed across the foundation. How can we best support this in the future?

The second is how to ensure cookbooks can support new features without breaking older versions of spicerack that might be deployed. How can we best support a mixed environment of spicerack + cumin + cookbooks with differing "versions"?

Creating a cadence and ensuring we can all move as fast as the most urgent stakeholder wants to move is one solution. I'm inclined to allow for more flexibility if possible to avoid forcing synchronicity. I would propose this as a shared objective to explore in Q4.

In the interim, WMCS appreciates the timeliness of releasing a new pypi package as we develop this quarter. Thanks for pushing a new version and keeping us unblocked!