As part of the exploration for self-hosted translation service (T331505) there was the question about where to store the models. Currently machine learning models for translation are stored at: https://people.wikimedia.org/~santhosh/nllb/ and https://people.wikimedia.org/~santhosh/opusmt
However, since the ML cluster uses Swift, that may be more resilient long term.
| Status | Subtype | Assigned | Task | ||
|---|---|---|---|---|---|
| Resolved | santhosh | T331505 Self hosted machine translation service | |||
| Resolved | BUG REPORT | KartikMistry | T386889 MinT: Deployment timeouts for eqiad | ||
| Resolved | KartikMistry | T335491 Provide better long-term storage for translation models | |||
| Resolved | BUG REPORT | KartikMistry | T383750 MinT: Fails to download models/files from peopleweb.discovery.wmnet | ||
| Resolved | kevinbazira | T391958 Create a new S3 bucket for MinT |
@KartikMistry definitely yes, I added most of my thoughts earlier on but feel free to ping me anytime if you need help!
With T391958, we've migrated from people.wikimedia.org to https://analytics.wikimedia.org/published/wmf-ml-models/mint/ for external storage. Internally, models will use s3 storage.
Change #1147812 had a related patch set uploaded (by KartikMistry; author: KartikMistry):
[mediawiki/services/machinetranslation@master] entrypoint: Update models download URL
Change #1147812 merged by jenkins-bot:
[mediawiki/services/machinetranslation@master] entrypoint: Update model storage download
Change #1149391 had a related patch set uploaded (by Klausman; author: Klausman):
[operations/deployment-charts@master] charts: Add secrets template for Machine Translation
Change #1149391 merged by jenkins-bot:
[operations/deployment-charts@master] charts: Add secrets template for Machine Translation
With the above patch (and the private repo stuff) merged, we can diff on the deployment server (I elided some unrelated changes):
# cd /srv/deployment-charts/helmfile.d/services/machinetranslation/ # helmfile -e eqiad -i diff --context=3 skipping missing values file matching "values-eqiad.yaml" skipping missing values file matching "values-production.yaml" Comparing release=production, chart=wmf-stable/machinetranslation, namespace=machinetranslation machinetranslation, machinetranslation-production-secret-config, Secret (v1) has been added: + # Source: machinetranslation/templates/secret.yaml + apiVersion: v1 + kind: Secret + metadata: + labels: + app: machinetranslation + chart: machinetranslation-0.0.23 + heritage: Helm + release: production + name: machinetranslation-production-secret-config + data: + AWS_ACCESS_KEY_ID: '++++++++ # (18 bytes)' + AWS_SECRET_ACCESS_KEY: '++++++++ # (16 bytes)' + type: Opaque
Change #1159696 had a related patch set uploaded (by KartikMistry; author: KartikMistry):
[operations/deployment-charts@master] WIP: machinetranslation: Use s3 storage for production
@klausman is there any reason why we can't see following in the diff in staging?
+ data: + AWS_ACCESS_KEY_ID: '++++++++ # (18 bytes)' + AWS_SECRET_ACCESS_KEY: '++++++++ # (16 bytes)'
I am not sure. As far as I can see, the secrets are wired up independently of clusters (e.g. both eqiad and codfw show a correct diff, and the diff for staging has everything except the actual secrets), so I suspect it's something in how the chart works. I will investigate.
Found it: the secrets were not wired up for staging because I had a brain fart when setting that up. It's been fixed in the private repo with commit 7bc13c5d2 (https://gerrit.wikimedia.org/r/c/labs/private/+/1160032 on the pseudo-private one), and staging now shows the correct diff:
machinetranslation, machinetranslation-staging-secret-config, Secret (v1) has been added: + # Source: machinetranslation/templates/secret.yaml + apiVersion: v1 + kind: Secret + metadata: + labels: + app: machinetranslation + chart: machinetranslation-0.0.23 + heritage: Helm + release: staging + name: machinetranslation-staging-secret-config + data: + AWS_ACCESS_KEY_ID: '++++++++ # (18 bytes)' + AWS_SECRET_ACCESS_KEY: '++++++++ # (16 bytes)' + type: Opaque
Change #1159696 merged by jenkins-bot:
[operations/deployment-charts@master] machinetranslation: Use S3 storage for production
Change #1162952 had a related patch set uploaded (by Jelto; author: Jelto):
[operations/puppet@production] peopleweb: add KUBEPOD ranges to firewall
Change #1162952 merged by Jelto:
[operations/puppet@production] peopleweb: add KUBEPOD ranges to firewall
Change #1163291 had a related patch set uploaded (by KartikMistry; author: KartikMistry):
[mediawiki/services/machinetranslation@master] entrypoint: Update model storage download
Change #1163739 had a related patch set uploaded (by Klausman; author: Klausman):
[labs/private@master] hiera/k8s: Add missing :prod suffix to machinetranslation S3 credentials
Change #1163739 merged by Klausman:
[labs/private@master] hiera/k8s: Add missing :prod suffix to machinetranslation S3 credentials
Change #1164235 had a related patch set uploaded (by Klausman; author: Klausman):
[operations/puppet@production] hiera/thanos-swift: Fix MinT user
Change #1165901 had a related patch set uploaded (by Elukey; author: Elukey):
[operations/puppet@production] profile::thanos::swift: rework machinetranslation account
Change #1165901 abandoned by Elukey:
[operations/puppet@production] profile::thanos::swift: rework machinetranslation account
Change #1166543 had a related patch set uploaded (by KartikMistry; author: KartikMistry):
[operations/deployment-charts@master] WIP: machinetranslation: Use s3 for model download in staging
Change #1166754 had a related patch set uploaded (by Klausman; author: Klausman):
[labs/private@master] hiera/deployment-server: change name of MT AWS user
Change #1166754 merged by Klausman:
[labs/private@master] hiera/deployment-server: change name of MT AWS user
Change #1164235 merged by Klausman:
[operations/puppet@production] hiera/thanos-swift: Fix MinT user
Mentioned in SAL (#wikimedia-operations) [2025-07-07T10:24:08Z] <Emperor> remove swift-account-stats_machinetranslation:prod time & service from thanos-fe1004 T335491
Change #1163291 merged by jenkins-bot:
[mediawiki/services/machinetranslation@master] entrypoint: Update model storage download
Change #1166543 merged by jenkins-bot:
[operations/deployment-charts@master] machinetranslation: Use s3 for model download in staging
Change #1167328 had a related patch set uploaded (by KartikMistry; author: KartikMistry):
[operations/deployment-charts@master] machinetranslation: Remove extra / from s3 URL
Change #1167328 merged by jenkins-bot:
[operations/deployment-charts@master] machinetranslation: Remove extra / from s3 URL
Change #1167448 had a related patch set uploaded (by KartikMistry; author: KartikMistry):
[mediawiki/services/machinetranslation@master] entrypoint.sh: Configure config path properly for s3cmd
Change #1167448 merged by jenkins-bot:
[mediawiki/services/machinetranslation@master] entrypoint.sh: Configure config path properly for s3cmd
Change #1167608 had a related patch set uploaded (by KartikMistry; author: KartikMistry):
[operations/deployment-charts@master] machinetranslation: staging: Update MinT to 2025-07-09-124154-production
Change #1167608 merged by jenkins-bot:
[operations/deployment-charts@master] machinetranslation: staging: Update MinT to 2025-07-09-124154-production
Status update:
We're testing the entrypoint.sh in the staging (using values-staging.yaml). Currently, s3cmd is still failing to find or parse the config file.
[2025-07-10 05:39:27] s3cfg written to models/.s3cfg ERROR: Configuration file not available. ERROR: models/.s3cfg: None
Using --debug flag locally shows that the file is parsed, but the error is still shown.
$ s3cmd --debug --config .s3cfg ls DEBUG: s3cmd version 2.4.0 DEBUG: ConfigParser: Reading file '.s3cfg' DEBUG: ConfigParser: access_key->${...17_chars...} DEBUG: ConfigParser: secret_key->${...21_chars...} DEBUG: ConfigParser: host_base->https://thanos-swift.discovery.wmnet DEBUG: ConfigParser: host_bucket->https://thanos-swift.discovery.wmnet DEBUG: ConfigParser: use_https->True DEBUG: ConfigParser: signature_v2->False ERROR: .s3cfg: None ERROR: Configuration file not available. ERROR: Consider using --configure parameter to create one.
Change #1167742 had a related patch set uploaded (by Elukey; author: Elukey):
[operations/deployment-charts@master] machinetranslation: add snippet to fetch private env variables
Change #1167742 merged by Elukey:
[operations/deployment-charts@master] machinetranslation: add snippet to fetch private env variables
Change #1167854 had a related patch set uploaded (by KartikMistry; author: KartikMistry):
[operations/deployment-charts@master] machinetranslationt: Use s3 model storage for production
Update: We've now staging server running using S3 model storage and observing logs, startup time. The patch for production is ready for deploy.
Change #1167854 merged by jenkins-bot:
[operations/deployment-charts@master] machinetranslationt: Use s3 model storage for production
Mentioned in SAL (#wikimedia-operations) [2025-07-14T12:34:34Z] <kart_> machinetranslationt: Use s3 model storage for production (T335491)
Since logs are fine, we don't have anything specific to QA for this task. Thanks to all for the help!
Hello,
I see this ticket is resolved now. I have been watching it to see if you don't need to use people hosts anymore to store large files once this is used instead.
Is that the case now and would it be ok to delete those large files from the home dirs (of Kartik and Santhosh)?
Thanks,
Daniel
@Dzahn Yes. We can remove MinT models from our home directories at the people.wikimedia.org.
Yay, thank you @KartikMistry ! :) (this means I should not get warning emails anymore and we can move forward with a low prio but stalled task, appreciate it).