Currently, setting a limit range in webservice manually using the mechanism described here, will set a higher limit with the same low "request". This results in a situation where the scheduler for Kubernetes will schedule the service as though it is expected to consume the "request" amount and oversubscribe the rest. To use the "Burstable" qosClass best for containers, we should set the request value to one that is proportional to the limit. This will prevent the wild oversubscription rates seen in the old cluster (often over 600%).
Description
Description
Details
Details
| Status | Subtype | Assigned | Task | ||
|---|---|---|---|---|---|
| Restricted Task | |||||
| Resolved | • Bstorm | T246122 Upgrade the Toolforge Kubernetes cluster to v1.16 | |||
| Restricted Task | |||||
| Resolved | bd808 | T232536 Toolforge Kubernetes internal API down, causing `webservice` and other tooling to fail | |||
| Resolved | • Bstorm | T236565 "tools" Cloud VPS project jessie deprecation | |||
| Resolved | aborrero | T101651 Set up toolsbeta more fully to help make testing easier | |||
| Resolved | • Bstorm | T166949 Homedir/UID info breaks after a while in Tools Kubernetes (can't read replica.my.cnf) | |||
| Resolved | • Bstorm | T246059 Add admin account creation to maintain-kubeusers | |||
| Resolved | • Bstorm | T154504 Make webservice backend default to kubernetes | |||
| Declined | None | T245230 Investigate cpu/ram requests and limits for DaemonSets pods | |||
| Resolved | • Bstorm | T214513 Deploy and migrate tools to a Kubernetes v1.15 or newer cluster | |||
| Resolved | • Bstorm | T244289 Improve limit range management in webservice for Kubernetes |
Event Timeline
Comment Actions
Change 570734 had a related patch set uploaded (by Bstorm; owner: Bstorm):
[operations/software/tools-webservice@master] kubernetes: resource requests should be proportional to limits
Comment Actions
Change 570734 merged by jenkins-bot:
[operations/software/tools-webservice@master] kubernetes: resource requests should be proportional to limits
Comment Actions
Mentioned in SAL (#wikimedia-cloud) [2020-02-07T21:09:19Z] <bstorm_> upgraded toollabs-webservice package for stretch toolsbeta to 0.59 T244293 T244289 T234617 T156626
Comment Actions
Change 570974 had a related patch set uploaded (by Bstorm; owner: Bstorm):
[operations/software/tools-webservice@master] k8s-resources: resources must be converted to str from Decimal
Comment Actions
Change 570974 merged by jenkins-bot:
[operations/software/tools-webservice@master] k8s-resources: resources must be converted to str from Decimal
Comment Actions
Mentioned in SAL (#wikimedia-cloud) [2020-02-10T21:18:57Z] <bstorm_> upgraded toollabs-webservice package for stretch toolsbeta to 0.62 T244293 T244289 T234617 T156626
Comment Actions
Mentioned in SAL (#wikimedia-cloud) [2020-02-10T21:25:10Z] <bstorm_> upgraded toollabs-webservice package for tools to 0.62 T244293 T244289 T234617 T156626