Page MenuHomePhabricator
Create Task
Maniphest T386889

MinT: Deployment timeouts for eqiad
Closed, ResolvedPublic4 Estimated Story PointsBUG REPORT
Actions

Description

MinT deployment timeouts when deploying in eqiad, most likely due to several times it has to download models for each worker. Deployment in codfw seems good so far.

This can be probably fixed by a similar fix (afdd1cf985842cf13d2eaaf86453fc618a03ab79) we did for recommendation-api-ng.

Details

Related Changes in Gerrit:
SubjectRepoBranchLines +/-
WIP: services/machinetranslation: adjust startup probe delaysoperations/deployment-chartsmaster+26 -9
MinT: Increase liveness probeoperations/deployment-chartsmaster+3 -0
MinT: staging: Increase liveness probeoperations/deployment-chartsmaster+7 -0
Customize query in gerrit

Related Objects
Search...

Event Timeline

KartikMistry created this task.Feb 20 2025, 5:11 AM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptFeb 20 2025, 5:11 AM
KartikMistry claimed this task.Feb 20 2025, 5:11 AM
Nikerabbit triaged this task as High priority.Feb 20 2025, 8:44 AM
Nikerabbit set the point value for this task to 4.
Nikerabbit moved this task from Backlog to Infrastructure on the MinT board.
KartikMistry mentioned this in T385552: MinT: Add support for Obolo, Central Dusun, Iban and, South Ndebele.Feb 20 2025, 11:40 AM
KartikMistry moved this task from Backlog to In Progress on the LPL Essential (LPL Essential 2025 Feb-Mar) board.
KartikMistry mentioned this in T383750: MinT: Fails to download models/files from peopleweb.discovery.wmnet.Feb 28 2025, 4:50 AM
Nikerabbit added a parent task: T335491: Provide better long-term storage for translation models.Mar 3 2025, 9:13 AM
gerritbot added a comment.Mar 6 2025, 7:55 AM

Change #1125093 had a related patch set uploaded (by KartikMistry; author: KartikMistry):

[operations/deployment-charts@master] MinT: Increase rediness probe

https://gerrit.wikimedia.org/r/1125093

gerritbot added a project: Patch-For-Review.Mar 6 2025, 7:55 AM
gerritbot added a comment.Mar 17 2025, 5:11 AM

Change #1128067 had a related patch set uploaded (by KartikMistry; author: KartikMistry):

[operations/deployment-charts@master] MinT: staging: Increase rediness probe

https://gerrit.wikimedia.org/r/1128067

gerritbot added a comment.Mar 25 2025, 6:28 AM

Change #1128067 merged by jenkins-bot:

[operations/deployment-charts@master] MinT: staging: Increase liveness probe

https://gerrit.wikimedia.org/r/1128067

KartikMistry added a comment.Apr 14 2025, 7:40 AM

I'm check with https://gerrit.wikimedia.org/r/c/operations/deployment-charts/+/1136148 - if bumping the chart helps to take the config (https://gerrit.wikimedia.org/r/1128067) we did into effect. Currently, diff isn't listing the config change in the staging.

Nikerabbit changed the task status from Open to In Progress.Apr 14 2025, 12:28 PM
Nikerabbit edited projects, added LPL Essential (LPL Essential 2025 Apr-Jun: CX); removed LPL Essential (LPL Essential 2025 Feb-Mar).
Nikerabbit moved this task from Backlog to In Progress on the LPL Essential (LPL Essential 2025 Apr-Jun: CX) board.
KartikMistry added a comment.Apr 15 2025, 6:55 AM

Now, even staging is timing out when downloading any models: https://logstash.wikimedia.org/app/discover#/doc/logstash-*/logstash-k8s-1-7.0.0-1-2025.04.15?id=0To1OJYB-jfKnJisF2Oj

Nikerabbit changed the task status from In Progress to Stalled.May 5 2025, 8:35 AM
Nikerabbit moved this task from In Progress to Backlog on the LPL Essential (LPL Essential 2025 Apr-Jun: CX) board.
Nikerabbit subscribed.

Focusing on T335491: Provide better long-term storage for translation models as way to resolve this.

Nikerabbit removed a parent task: T335491: Provide better long-term storage for translation models.May 6 2025, 7:10 AM
Nikerabbit added a subtask: T335491: Provide better long-term storage for translation models.
gerritbot added a comment.Jun 23 2025, 6:24 AM

Change #1125093 abandoned by KartikMistry:

[operations/deployment-charts@master] MinT: Increase liveness probe

Reason:

Not useful with S3, but can be restore if needed.

https://gerrit.wikimedia.org/r/1125093

Maintenance_bot removed a project: Patch-For-Review.Jun 23 2025, 6:30 AM
gerritbot added a comment.Jun 23 2025, 5:36 PM

Change #1162985 had a related patch set uploaded (by Klausman; author: Klausman):

[operations/deployment-charts@master] WIP: services/machinetranslation: adjust startup probe delays

https://gerrit.wikimedia.org/r/1162985

gerritbot added a project: Patch-For-Review.Jun 23 2025, 5:36 PM
KartikMistry changed the task status from Stalled to In Progress.Jun 24 2025, 3:13 AM
KartikMistry moved this task from Backlog to In Progress on the LPL Essential (LPL Essential 2025 Apr-Jun: CX) board.
Nikerabbit edited projects, added LPL Essential (2025 Jul-Oct); removed LPL Essential (LPL Essential 2025 Apr-Jun: CX).Jul 14 2025, 8:21 AM
Nikerabbit moved this task from Backlog to In Progress on the LPL Essential (2025 Jul-Oct) board.
Nikerabbit added a project: LPL Projects (MinT for Wikireaders – FY26 WE 3.1.5).Jul 14 2025, 8:31 AM
Nikerabbit changed the task status from In Progress to Stalled.Jul 14 2025, 8:36 AM
Nikerabbit removed KartikMistry as the assignee of this task.
Nikerabbit moved this task from In Progress to Backlog on the LPL Essential (2025 Jul-Oct) board.
KartikMistry added a subscriber: klausman.Jul 15 2025, 4:30 AM

@klausman While deploying T335491, I didn't see any timeout for eqiad. Should we close this task?

Nikerabbit closed subtask T335491: Provide better long-term storage for translation models as Resolved.Jul 16 2025, 7:17 AM
Nikerabbit changed the task status from Stalled to In Progress.Jul 16 2025, 7:20 AM
Nikerabbit assigned this task to KartikMistry.
Nikerabbit moved this task from Backlog to In Progress on the LPL Essential (2025 Jul-Oct) board.

Marking as not stalled now that s3 migration is done.

KartikMistry added a comment.Jul 21 2025, 4:36 AM
In T386889#11003047, @KartikMistry wrote:

@klausman While deploying T335491, I didn't see any timeout for eqiad. Should we close this task?

Should we go ahead?

KartikMistry moved this task from In Progress to Backlog on the LPL Essential (2025 Jul-Oct) board.Aug 11 2025, 7:56 AM
klausman added a comment.Aug 11 2025, 8:05 AM
In T386889#11019056, @KartikMistry wrote:
In T386889#11003047, @KartikMistry wrote:

@klausman While deploying T335491, I didn't see any timeout for eqiad. Should we close this task?

Should we go ahead?

Yes, please do! Sorry for the late response, I was out on PTO until today.

Nikerabbit closed this task as Resolved.Aug 11 2025, 11:16 AM
Nikerabbit moved this task from Backlog to Done on the LPL Essential (2025 Jul-Oct) board.
Nikerabbit removed a project: Patch-For-Review.
gerritbot added a comment.Aug 11 2025, 11:17 AM

Change #1162985 abandoned by Nikerabbit:

[operations/deployment-charts@master] WIP: services/machinetranslation: adjust startup probe delays

Reason:

S3 migration makes this less/unnecessary.

https://gerrit.wikimedia.org/r/1162985

• MusikAnimal mentioned this in T408160: Make the MinT translation preference on by default if possible.Oct 29 2025, 3:02 AM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits