Taavi says:
** I keep the Toolforge Kubernetes cluster up-to-date. The Kubernetes project releases three new versions a year, and we can't skip a release, so these tend to backlog rather quickly if they're neglected. Example: https://phabricator.wikimedia.org/T298005 <-- arturo, Raymond, slavina
There should be time for someone to ride along on an upgrade, if that hasn't happened recently.
| Status | Subtype | Assigned | Task | ||
|---|---|---|---|---|---|
| Restricted Task | |||||
| Resolved | Andrew | T305828 upgrade cloud-vps openstack to Openstack version 'Yoga' | |||
| Resolved | Andrew | T296561 upgrade cloud-vps openstack to Openstack version 'Xena' | |||
| Resolved | rook | T281275 upgrade cloud-vps openstack to Openstack version 'Wallaby' | |||
| Resolved | Andrew | T281276 Upgrade cloud-vps openstack hosts to Debian 'Bullseye' | |||
| Resolved | aborrero | T302050 prometheus-openstack-exporter in Bullseye | |||
| Resolved | dcaro | T302178 prometheus-openstack-exporter No module named 'urlparse' | |||
| In Progress | None | T333975 Cloud VPS: refresh openstack resources grafana dashboard | |||
| Resolved | Andrew | T335943 prometheus-openstack-exporter: collected data shows regular null intervals | |||
| Resolved | Andrew | T335978 openstack: consider removing references to old hardware from the database | |||
| Open | Andrew | T362443 Learn how to do what Taavi does | |||
| Open | None | T362445 Taavi knowledge transfer: Toolforge k8s upgrades |
Notes:
Apr 23, 2024 | Toolforge - Kubernetes
Attendees: Andrew Bogott Arturo Borrero David Caro Raymond Olisaemeka Taavi Väänänen Slavina Stefanova
Notes
* K8s upgrades
* We are a few years behind
* Upstream does 3 releases/year
* We want to catch up -> this means making extra effort at the beginning to upgrade **more often** than upstream
* Then stabilize at the same pace at upstream
* Loki
* Looks promising, have to do a re-POC as it has been a while since https://wikitech.wikimedia.org/wiki/User:Majavah/Loki_notes
* How does quotas work? -> find out
* Stored in S3
* Maintain-kubeusers
* Probably keep the current VCS tests to be able to run on CI without having to setup a full toolforge setup
* Try to improve the usage (recording/updating/etc.)
* Maybe use it in other places too
* Currently running on lima-kilo (arturo added the component not long ago)
*
Action items
* Create task to bootstrap upgrade working group (with the people in the meeting), to see how we want to organize ourselves
* Discuss striker tomorrow, if there's not enough time then schedule a meeting