My votes: 4 -> {3,2} -> 1

In T325382#8484046, @aborrero wrote:

My votes: 4 -> {3,2} -> 1

Thanks for your vote!
I would appreciate if you have any comments on any of the options or some extra rationale on why you vote this way.
I'm trying to gather opinions/discuss on pros-cons of the current options or find new more suitable ones.

I do think moving to an API based design makes sense, but I don't have a preference whether it should be written in Go or Python.

Can you share more details on which library you might be targeting? Looking at https://kubernetes.io/docs/reference/using-api/client-libraries/, I see a list of official libs, along with many that are unofficial. This would help to look at example implementations and get a feel for each library before weighing an opinion. Thanks!

In T325382#8487254, @nskaggs wrote:

Can you share more details on which library you might be targeting? Looking at https://kubernetes.io/docs/reference/using-api/client-libraries/, I see a list of official libs, along with many that are unofficial. This would help to look at example implementations and get a feel for each library before weighing an opinion. Thanks!

Sure, so I was just checking the official ones, but let's give a look.

Python

• current libs - We are currently using a very thin, snippet of code to do requests directly:
  • one copy on each codebase that uses it
  • no types
  • no tests
  • any changes we have to do it ourselves
  • no async support
  • example

• https://github.com/kubernetes-client/python/ -- official one
  • has models but no types (https://github.com/kubernetes-client/python/issues/225)
  • no testing helpers
  • ok maintained (a bit behind k8s, 2 releases)
  • special async support (not using the bultin async/await)

• github.com/fiaas/k8s -- norway feed company behind (FINN.no), main contributor seems to have moved companies
  • not much activity
  • no compatibility matrix
  • does not support many resources (PSPs for example)
  • no async
  • no testing helpers

• github.com/gtsystem/lightkube -- HERE technologies (1 person maintaining it+ some contribs from canonical)
  • few maintainers
  • very active
  • not mature (they say not ready for prod)
  • has types
  • wide support on the same library
  • async support
  • no testing helpers

• github.com/mnubo/kubernetes-py - AspenTech
  • some maintainers
  • no activity in a while (the main contributor changed companies it seems)
  • limited resources support
  • no types
  • no async
  • no testing helpers

• github.com/tomplus/kubernetes_asyncio - personal project
  • one main maintainer, many small contributions
  • quite active
  • similar versioning as the official library
  • partial support for streaming (read-only)
  • no test helpers
  • no types

• github.com/Frankkkkk/pykorm - personal/sokube cloud solutions
  • few maintainers
  • little activity
  • claims not to be very stable
  • ORM pattern oriented
  • focusing on easy CRD support
  • no async support
  • no testing helpers
  • no types

Golang

• https://github.com/kubernetes/client-go/ - official libs
  • very well maintained
  • wider compatibility
  • wider support
  • testing helpers (see https://pkg.go.dev/k8s.io/client-go/kubernetes/fake)
  • typed
  • golang concurrency model

• github.com/ericchiang/k8s - archived in november (read-only)

I might have missed/misinterpreted something, if so please let me know (I'm not familiar with most of these libraries). If there's any others you want to consider, please add too.

If the main reason for this proposal was "we just want to have fun playing with & learning a new technology" then that's OK. I want to play & learn too. But that's not a technical reason :-) I don't see strong technical reasons, see below.

As you pointed already, so far we have been using 2 options to interact with k8s from python:

• using our tiny embedded custom library k8sclient.py, examples jobs-framework-api and tools-webservice --- about ~100 LOC
• using the official lib https://github.com/kubernetes-client/python/, example codebase using it: jobs-framework-emailer

In my experience running any of the two is more than enough. They are robust. They work well. They are easy to use. They work well with python-flask, which is mostly what we use here for building REST APIs. In particular, the statement The python k8s libs are not as well maintained, harder to test and develop with I think is quite the opposite, actually.

We also have a few Toolforge k8s custom components written in golang. They are tiny admission webhooks that do simple things, examples: volume-admission-controller, ingress-admission-controller, registry-admission-webhook, etc.
On the other hand, unlike the previous tiny webhooks, the proposed new codebase can't be comparable in scale. It will be at least 2 major projects (cli, api) that is already complex enough (build service for buildpack, etc), that could otherwise be made with either of the 2 python options. More so, if you consider that we already have similar codebases implementing the same CLI <-> API architecture (jobs-framework-api and jobs-framework-cli, both in python). I understand that some folks dislike the ~100 LOC embedded custom k8s library. But the solution being rewriting everything in a different lang feels a bit overkill.

For me, the reasons stated in the proposal aren't enough to justify the adoption of golang at such scale. That's why vote goes first for option 4, then for any of 3 or 2.

Moreover, the auth dancing that this new API would require is going to be very similar to what jobs-framework-api uses. This is already implemented in python. I would love to see the code refactored and reused in this new buildservice API.
In the future, if we decide so, it can also be adopted by tools-webservice (if we ever move that to an API architecture).

In T325382#8488430, @aborrero wrote:

Moreover, the auth dancing that this new API would require is going to be very similar to what jobs-framework-api uses. This is already implemented in python. I would love to see the code refactored and reused in this new buildservice API.
In the future, if we decide so, it can also be adopted by tools-webservice (if we ever move that to an API architecture).

Thanks a lot for your input Arturo!

btw. the auth dance is similar, and that part of the api was already sorted out in golang too:

In the future, if we decide so, it can also be adopted by tools-webservice (if we ever move that to an API architecture).

Yep, that's another reason why exploring another approach (golang) will help make a more informed decision. I would want to avoid making a decision mainly based on lack of knowledge/experience with the alternative or inertia from current practices.

I'd like to see this decisions request split into several smaller ones:

• Should we reimplement Toolforge Buildservices as a REST API? This question is independent of the Python vs Go discussion, as the architecture in itself is language-agnostic.
• Would it make sense to (re)implement some new and existing backend services in Go? If so, which ones? This should probably be discussed on a case-by-case basis, although I do see several overarching higher-level incentives for the team to develop some Go skills.
• Should we migrate toolforge-cli to Go? If we adopt an API architecture, toolforge-cli would no longer have any direct interaction with k8s. Unless the intention is to use the toolforge-cli project as a playground to learn Go, I see no real incentive to migrate it.

In T325382#8494657, @Slst2020 wrote:

I'd like to see this decisions request split into several smaller ones:

• Should we reimplement Toolforge Buildservices as a REST API? This question is independent of the Python vs Go discussion, as the architecture in itself is language-agnostic.
• Would it make sense to (re)implement some new and existing backend services in Go? If so, which ones? This should probably be discussed on a case-by-case basis, although I do see several overarching higher-level incentives for the team to develop some Go skills.

For the build service case, this one depends on the previous one, if we don't move to an API architecture, there's no rewrite needed.

For a wider scope, I agree we should check case-by-case, so probably out of scope for this decision, if we decide to write an API, and for it to be in golang, that will help deciding for the other cases.

• Should we migrate toolforge-cli to Go? If we adopt an API architecture, toolforge-cli would no longer have any direct interaction with k8s. Unless the intention is to use the toolforge-cli project as a playground to learn Go, I see no real incentive to migrate it.

And as you hint, yes, the rewrite of the toolforge-cli is purely to learn Go deployment, packaging and best practices, no other incentive. So again, this depends on the previous point, that depends on the first one xd

I wanted to avoid having three different consecutive decisions by summarizing this one as:

• 1) API no (if no API, there's no rewrite of anything, so no more choices)
• 2) API yes but in golang without toolforge-cli rewrite
• 3) API yes but in golang with toolforge-cli rewrite
• 4) API yes but in python (if in python, rewrite of toolforge-cli is moot)

So essentially you can map the three decisions to 4 options:

• If you want no api -> 1
• If you want api -> 2, 3 or 4
  • if you want python -> 4
  • if you want golang -> 2 or 3
    • if you want rewrite of toolforge-cli -> 2
    • if you don't want to rewrite the toloforge cli -> 3

Does this help or do you still think we need 3 different tasks?

In T325382#8494714, @dcaro wrote:

So essentially you can map the three decisions to 4 options:

• If you want no api -> 1
• If you want api -> 2, 3 or 4
  • if you want python -> 4
  • if you want golang -> 2 or 3
    • if you want rewrite of toolforge-cli -> 2
    • if you don't want to rewrite the toloforge cli -> 3

Does this help or do you still think we need 3 different tasks?

I think the decision about whether to move TBS to an API architecture warrants its own discussion, independent of the actual implementation. I understand that choosing a programming language would be an immediate follow-up decision, together with other implementation details. But as you said yourself, that discussion would be moot if we decided against the API, so this seems to me a separate question better left for later so that we all can focus on one concept at a time.

My preference would be to only discuss API vs no API to start with, then go from there.

Moved to T326136: Decision request - Toolforge build service to move to an API design, please add comments there, this discussion will be halted until that one is resolved, thanks!

A decision was made to continue with the API design on T326136: Decision request - Toolforge build service to move to an API design, this discussion can now continue, I have adapted the task description, please review, thanks

I find it hard to decide personally, so I will not cast a vote for one of the options.

I think there is value in using Python because that's the main language used in the team, everyone is already familiar with it, and I'm sure it can handle the requirements of this project pretty well. But I also see value in using Go, mainly because of the typing support, but also as a way to increase the skills of the team (myself included) in a language that is becoming more and more relevant in the SRE space.

I think there is value in using Python because that's the main language used in the team, everyone is already familiar with it, and I'm sure it can handle the requirements of this project pretty well. But I also see value in using Go, mainly because of the typing support, but also as a way to increase the skills of the team (myself included) in a language that is becoming more and more relevant in the SRE space.

Agree, I lean towards the golang option as it allows us to become more future proof, in the sense of having a better view of both options, and expanding our knowledge and skillset to more than one stack (that we are already using, so not really introducing much new, just expanding on the best practices).

Of course, if we develop it as we have been so far, more than one person at a time, otherwise the knowledge does not get shared.

I lean toward option 1. I think the pros outweigh both the cons of this option, and the pros of option 2.

From my personal experience with Go so far, it's a relatively easy language to get started with, e.g. compared to Rust. The syntax should be familiar to anyone with some experience with any C-type language, and core Go is by design minimal and devoid of any magic and syntactic sugar. The Gophers Slack community is also very active and helpful – anytime I've asked a question there, someone has helped me out within an hour, or sometimes even minutes.

I'd probably keep the toolforge-cli in Python, though. As k8s interactions become intermediated by the API, I see no obvious reasons to spend time porting it to Go.

my opinion on this is in two parts:

1. If the plan to work on this and maybe get it done in a month or less, we should go with option 4.
2. If we are ready to spend more time on this (seems like we are), then we can decide to build this with golang, option 2