Q1 FY2025-26 Goal: Operational Excellence - LiftWing Platform Updates & Improvements
Open, Needs TriagePublic
Actions

Assigned To

None

Authored By

	isarantopoulos
	Jul 8 2025, 11:53 AM

Project Tags

Referenced Files

None

Subscribers

Description

As an engineer on the Machine Learning team,
I want to upgrade existing infrastructure, improve build tools, respond to alerts, and fix bugs,
so that we can efficiently deliver value by serving models in a reliable and scalable way.

During this quarter we want to achieve the following things:

Update kserve + knative T380723 + T380722
Update debian docker images based on bullseye to bookworm 🎉
~~Decide on replacements of existing servers that are reaching EOL support~~ [postponed for next quarter]
Decide and define a standard access path for LiftWing for canonical data via Eventstreams together with Data Platform
Migrate the following SLO dashboards to pyrra

Related Objects
Search...

Status	Assigned	Task
Open	None	T398948 Q1 FY2025-26 Goal: Operational Excellence - LiftWing Platform Updates & Improvements
Declined	klausman	T380723 Update knative-serving+net-istio to v1.12.x on ML clusters
Resolved	kevinbazira	T385173 Use rocm/vllm image on Lift Wing
Resolved	kevinbazira	T415627 Update WMF Debian vLLM image to support latest upstream software stack
Resolved	kevinbazira	T398533 Update knative's queue proxy image and the Swift/S3 accounts used on ml-serve clusters
Resolved	klausman	T380722 Update kserve to v0.15.2* on ML clusters
Open	isarantopoulos	T393475 ML Services causing log spam
Resolved	kevinbazira	T399683 Fix AlertLintProblem for ORESFetchScoreJobKafkaLag
Resolved	isarantopoulos	T399733 Investigate reference-need-predictor alert triggered by BrokenProcessPool error
Resolved	isarantopoulos	T399936 Investigate reference-need-predictor alert triggered by reverse proxying request error
Resolved	klausman	T398600 Upgrade the AMD GPU plugin for k8s to support MI300 GPUs
Open	None	T393566 Add the ML team to the POSIX group `docker` on `ml-build1001`
Resolved	isarantopoulos	T400144 Upgrade remaining model servers from debian bullseye to bookworm
Resolved	gkyziridis	T400266 Upgrade revertrisk model server from the debian bullseye base image to bookworm.
Resolved	gkyziridis	T400347 Upgrade langid model server from debian bullseye to bookworm
Resolved	gkyziridis	T400348 Upgrade ores-legacy from debian bullseye to bookworm
Resolved	gkyziridis	T400349 Upgrade articletopic-outlink model servers from debian bullseye to bookworm
Resolved	kevinbazira	T400350 Upgrade revscoring model servers from debian bullseye to bookworm
Resolved	BWojtowicz-WMF	T400351 Upgrade article-descriptions model servers from debian bullseye to bookworm
Resolved	achou	T400352 Upgrade readability model server from debian bullseye to bookworm
Open	BWojtowicz-WMF	T414431 Optimize resource utilization for InferenceServices on LiftWing cluster
Resolved	DPogorzelski-WMF	T419235 Fix revertrisk Pyrra SLO

Event Timeline

isarantopoulos created this task.Jul 8 2025, 11:53 AM

Restricted Application added a subscriber: Aklapper. · View Herald TranscriptJul 8 2025, 11:53 AM

isarantopoulos added a subtask: T380723: Update knative-serving+net-istio to v1.12.x on ML clusters.Jul 8 2025, 12:02 PM

isarantopoulos renamed this task from Q1 25-26 Goal: Operational Excellence - LiftWing Platform Updates & Improvements to Q1 FY2025-26 Goal: Operational Excellence - LiftWing Platform Updates & Improvements.Jul 8 2025, 1:56 PM

isarantopoulos added subtasks: T385173: Use rocm/vllm image on Lift Wing, T398533: Update knative's queue proxy image and the Swift/S3 accounts used on ml-serve clusters, T380722: Update kserve to v0.15.2* on ML clusters, T393475: ML Services causing log spam.Jul 11 2025, 9:51 AM

isarantopoulos closed subtask T398533: Update knative's queue proxy image and the Swift/S3 accounts used on ml-serve clusters as Resolved.Jul 15 2025, 2:51 PM

kevinbazira closed subtask T399683: Fix AlertLintProblem for ORESFetchScoreJobKafkaLag as Resolved.Jul 18 2025, 8:40 AM

kevinbazira subscribed.Jul 18 2025, 10:30 AM

isarantopoulos added a subtask: T398600: Upgrade the AMD GPU plugin for k8s to support MI300 GPUs.Jul 22 2025, 1:27 PM

isarantopoulos added a subtask: T393566: Add the ML team to the POSIX group `docker` on `ml-build1001`.

isarantopoulos added subtasks: T400350: Upgrade revscoring model servers from debian bullseye to bookworm, T400144: Upgrade remaining model servers from debian bullseye to bookworm.Aug 12 2025, 2:38 PM

isarantopoulos updated the task description. (Show Details)Sep 2 2025, 2:36 PM

isarantopoulos closed subtask T400350: Upgrade revscoring model servers from debian bullseye to bookworm as Resolved.

isarantopoulos closed subtask T400144: Upgrade remaining model servers from debian bullseye to bookworm as Resolved.Sep 8 2025, 7:02 AM

klausman closed subtask T398600: Upgrade the AMD GPU plugin for k8s to support MI300 GPUs as Resolved.Sep 30 2025, 3:12 PM

BWojtowicz-WMF created subtask T414431: Optimize resource utilization for InferenceServices on LiftWing cluster.Jan 13 2026, 11:20 AM