Page MenuHomePhabricator

Investigate high usage of Apertium and V2 endpoint
Closed, ResolvedPublic

Description

Since today (ie https://grafana.wikimedia.org/dashboard/db/service-cxserver?orgId=1&from=1543257000000&to=1543312725861), Apertium MT API requests with V2 endpoint was too high. Investigate and find the root cause.

Event Timeline

KartikMistry updated the task description. (Show Details)

I 've helped with the debugging. Starting from apertium it was clear something automated was POSTing a lot of requests to it. It turned out they were mostly for the rus|bel langpair but that was a red herring as it was just the snapshot in time I looked at. Moving from apertium to cxserver it became clear something was POSTing to /v2/translate endpoint. The things I noted were mostly about another language pair ca|oc but again that was a snapshot in time. Then a VM IP caught my eye, one that was of wcdo.wcdo.eqiad.wmflabs. I 've jumped into said VM and stopped a process that was clearly heavily hitting the cxserver API

@marcmiquel as I understand it this was your process. I am really sorry I had to stop your stats generation, I hope I did not cause much damage, but it was causing stress to the API (~700 requests per second). This was 2 orders of magnitude more than the standard traffic and was deemed excessive. Also there was no clear User-Agent header in the requests, which would have helped more in identifying the requester and notifying you accordingly. Please have a look at https://www.mediawiki.org/wiki/API:Etiquette if you haven't already.

Thanks for the message akosiaris. I'm sorry the posts were heavy. This process runs only once a month.

I will concentrate text in longer queries. What is the text length limit for a single query?

Thanks for the message akosiaris. I'm sorry the posts were heavy. This process runs only once a month.

I will concentrate text in longer queries. What is the text length limit for a single query?

Hi,

It is more about number of queries per second (ie lots of requests) Also, please inform Language-Team before you run queries.

Let me know if you've any questions.

I'm moving this to done as https://gerrit.wikimedia.org/r/476233 is merged and will deploy today (most probably).

@marcmiquel Hi, are you running the query again by any chance? Please let us know. Thanks!

Nope. I haven't for days. Since your last email.

Nope. I haven't for days. Since your last email.

Thanks!

Pginer-WMF claimed this task.