Page MenuHomePhabricator
Create Task
Maniphest T386371

Request capacity increase in preparation for MinT for wiki Readers experiment
Closed, ResolvedPublic4 Estimated Story Points
Actions

Description

As part of the preparations (T381406) to continue with the MinT for Wiki Readers experiment, we want to identify if there are quick ways to improve the response time for machine translation requests. Slow response times can add some noise to the experiment results.

This ticket proposes to identify which server resources could have the most impact in translation speed from those that could be easily increased. In this way we can make a request, and have the service in a better condition for the experiment.

Details

Related Changes in Gerrit:
SubjectRepoBranchLines +/-
machinetranslation: Increase replicasoperations/deployment-chartsmaster+1 -1
Customize query in gerrit

Related Objects
Search...

Event Timeline

Pginer-WMF created this task.Feb 13 2025, 3:46 PM
Pginer-WMF added a project: LPL Essential.
Pginer-WMF edited projects, added LPL Essential (LPL Essential 2025 Feb-Mar); removed LPL Essential.
Pginer-WMF mentioned this in T385784: Explore options reduce the impact of potentially slow MinT response time.Feb 14 2025, 2:12 PM
Pginer-WMF added a parent task: T385784: Explore options reduce the impact of potentially slow MinT response time.
Nikerabbit triaged this task as Medium priority.Feb 20 2025, 8:46 AM
Nikerabbit set the point value for this task to 4.
Nikerabbit moved this task from Backlog to Infrastructure on the MinT board.
Nikerabbit removed KartikMistry as the assignee of this task.Mar 10 2025, 7:50 AM
Nikerabbit renamed this task from Request server capcity increase in preparation for MinT for wiki Readers experient to Request server capacity increase in preparation for MinT for wiki Readers experiment.Mar 11 2025, 7:58 AM
Nikerabbit renamed this task from Request server capacity increase in preparation for MinT for wiki Readers experiment to Request capacity increase in preparation for MinT for wiki Readers experiment.
Nikerabbit moved this task from Backlog to Ready for dev on the LPL Essential (LPL Essential 2025 Feb-Mar) board.
abi_ mentioned this in T388402: MinT for Wiki Readers MVP: Enable 13 wikis to MinT for Wiki Readers MVP (revised 2025 A/B list) .Mar 18 2025, 2:10 PM
• PWaigi-WMF mentioned this in T390023: MinT for Wiki Readers MVP: Pre-Pilot enablement on 4 wikis.Mar 26 2025, 7:02 AM
Pginer-WMF mentioned this in T391365: Analyze the Impact of MinT for Wiki Readers on Pre-pilot wikis .Apr 11 2025, 11:24 AM
KartikMistry added a comment.Apr 14 2025, 7:07 AM

After deploying 'MinT for Wiki Readers' in 4 Wikipedias (T390023), traffic (and other metrics: Memory, CPUs) seems normal, we will observe it this week before going forward to deploy in more Wikis.

Nikerabbit edited projects, added LPL Essential (LPL Essential 2025 Apr-Jun: CX); removed LPL Essential (LPL Essential 2025 Feb-Mar).Apr 14 2025, 12:32 PM
Nikerabbit changed the task status from Open to Stalled.Jun 3 2025, 10:36 AM
Pginer-WMF added a comment.Jun 24 2025, 1:59 PM

The recent report for the initial MinT for Wikipedia pre-pilot wikis (T391365) shows high loading times for machine translation views:

  • Over 3s for 68% of the views
  • Over 2s for 90% of the views.

Loading times over 2-3 seconds can affect the abandonment rate, introducing noise to the experiment results.
[This old Google post(https://blog.google/products/admanager/the-need-for-mobile-speed/) summarizes some of the research in this space:

  • 53% of visits are likely to be abandoned if pages take longer than 3 seconds to load.
  • One out of two people expect a page to load in less than 2 seconds.

(These numbers are from 2016, but I see no reason for today expectations being for slower response times)

I think it may be worth trying to make a request for additional server capacity that could result in faster loading times. Especially considering that the experiment is targeting more wikis with more traffic than the 4 in the pre-pilot.

@KartikMistry is there a simple request we can make to increase resources that could help with loading times?

Pginer-WMF mentioned this in T381406: MinT for Wiki Readers MVP: Complete key issues before continuing with experimentation.Jun 25 2025, 9:57 AM
Nikerabbit moved this task from LPL Essential 2025 Apr-Jun: CX to 2025 Jul-Oct on the LPL Essential board.Jul 8 2025, 6:54 AM
Nikerabbit edited projects, added LPL Essential (2025 Jul-Oct); removed LPL Essential (LPL Essential 2025 Apr-Jun: CX).
Nikerabbit changed the task status from Stalled to Open.Jul 16 2025, 7:21 AM
Nikerabbit moved this task from Backlog to Ready for dev on the LPL Essential (2025 Jul-Oct) board.Jul 17 2025, 11:11 AM
Nikerabbit added a project: LPL Projects (MinT for Wikireaders – FY26 WE 3.1.5).Jul 23 2025, 8:05 AM
Nikerabbit edited projects, added LPL Essential (FY2025-26 Q2); removed LPL Projects (MinT for Wikireaders – FY26 WE 3.1.5), LPL Essential (2025 Jul-Oct).Oct 23 2025, 11:20 AM
Nikerabbit mentioned this in T408160: Make the MinT translation preference on by default if possible.Oct 29 2025, 8:26 AM
Nikerabbit removed a project: LPL Essential (FY2025-26 Q2).Nov 6 2025, 7:37 AM
KartikMistry changed the task status from Open to In Progress.Nov 6 2025, 8:25 AM
KartikMistry claimed this task.
KartikMistry added a project: LPL Essential (FY2025-26 Q2).
KartikMistry moved this task from Backlog to In Progress on the LPL Essential (FY2025-26 Q2) board.
KartikMistry removed a subscriber: • PWaigi-WMF.
gerritbot added a comment.Nov 6 2025, 8:39 AM

Change #1202642 had a related patch set uploaded (by KartikMistry; author: KartikMistry):

[operations/deployment-charts@master] machinetranslation: Increase replica and memory

https://gerrit.wikimedia.org/r/1202642

gerritbot added a project: Patch-For-Review.Nov 6 2025, 8:39 AM
Nikerabbit added a project: LPL Projects (Other).Nov 11 2025, 8:07 AM
jijiki subscribed.Nov 11 2025, 1:00 PM

Looking at the memory usage

https://grafana.wikimedia.org/goto/OxM6IrkvR?orgId=1

It seems like only one of the pods goes from 32G to 64G, and it is the same one (it is visible if you toggle between the two pods and the two DCs (eqiad/codfw)

Is there anything else that might help explain this behaviour? Unless, of course, it is expected. Thank you!

gerritbot added a comment.Nov 13 2025, 6:04 AM

Change #1202642 merged by jenkins-bot:

[operations/deployment-charts@master] machinetranslation: Increase replicas

https://gerrit.wikimedia.org/r/1202642

Stashbot added a comment.Nov 13 2025, 6:18 AM

Mentioned in SAL (#wikimedia-operations) [2025-11-13T06:18:33Z] <kart_> machinetranslation: Increase replicas (T386371)

Maintenance_bot removed a project: Patch-For-Review.Nov 13 2025, 6:30 AM
Pginer-WMF closed this task as Resolved.Nov 26 2025, 8:43 AM

Capacity has been increased, without apparent issues. When the experiment is launches, we'll hopefully have higher chances of faster request responses and lower chances of servers going out of capacity.

abi_ moved this task from In Progress to Done on the LPL Essential (FY2025-26 Q2) board.Nov 26 2025, 11:01 AM
Nikerabbit mentioned this in T411058: Can't deploy machinetranslation due to exceeding resource quotas.Dec 16 2025, 1:06 PM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits