Page MenuHomePhabricator
Create Task
Maniphest T182663

labtestwiki memcache errors, lots of requests from parsoid to labtestweb2001
Closed, ResolvedPublic
Actions

Description

Today the mediawiki memcached alerts error triggered, turns out the errors were from labtestwiki, in this form:

Memcached error for key "{memcached-key}" on server "{memcached-server}": SERVER HAS FAILED AND IS DISABLED UNTIL TIMED RETRY  	143,765
Memcached error for key "{memcached-key}" on server "{memcached-server}": CONNECTION FAILURE	13,441

labtestweb2001 was overwhelmed and its cpu at 100%

looking at apache access log on labtestweb2001 there are a few entries from ltx71.com and then a whole lot of entries from parsoid

2017-12-12T10:38:52     204526  CENSORED   application/x-httpd-php/200     6508    GET     http://labtestwikitech.wikimedia.org/wiki/Nova_Resource:Ragefixing      -       text/html       -       -       ltx71 - (http://ltx71.com/)
     -       -       -       CENSORED
2017-12-12T10:39:16     225272  CENSORED   application/x-httpd-php/200     5760    GET     http://labtestwikitech.wikimedia.org/wiki/Category:Bn-N -       text/html       -       -       ltx71 - (http://ltx71.com/)     -       -
       -       CENSORED
2017-12-12T10:40:23     195031  CENSORED   application/x-httpd-php/200     6374    GET     http://labtestwikitech.wikimedia.org/wiki/User_talk:Arjunaraoc  -       text/html       -       -       ltx71 - (http://ltx71.com/)     -
       -       -       CENSORED
2017-12-12T10:40:26     293628  2620:0:861:103:10:64:32:13      application/x-httpd-php/200     43679   GET     http://labtestwikitech.wikimedia.org/w/api.php?action=query&format=json&formatversion=2&siprop=extensions&meta=siteinfo -
       application/json        -       -       parsoid-reparse/0.1 python3/3.4.2       -       -       -       2620:0:861:103:10:64:32:13
2017-12-12T10:40:27     237032  2620:0:861:103:10:64:32:13      application/x-httpd-php/200     4703    GET     http://labtestwikitech.wikimedia.org/w/api.php?action=query&format=json&formatversion=2&siprop=namespaces&meta=siteinfo -
       application/json        -       -       parsoid-reparse/0.1 python3/3.4.2       -       -       -       2620:0:861:103:10:64:32:13
2017-12-12T10:40:27     317621  2620:0:861:103:10:64:32:13      application/x-httpd-php/200     60688   GET     http://labtestwikitech.wikimedia.org/w/api.php?action=query&formatversion=2&gapfilterredir=nonredirects&prop=info&generator=allpages&gapnamespace=10&format=json&gaplimit=max   -       application/json        -       -       parsoid-reparse/0.1 python3/3.4.2       -       -       -       2620:0:861:103:10:64:32:13
2017-12-12T10:40:28     248414  2620:0:861:103:10:64:32:13      application/x-httpd-php/200     979     GET     http://labtestwikitech.wikimedia.org/w/api.php?action=query&formatversion=2&gapfilterredir=nonredirects&prop=info&generator=allpages&gapnamespace=11&format=json&gaplimit=max   -       application/json        -       -       parsoid-reparse/0.1 python3/3.4.2       -       -       -       2620:0:861:103:10:64:32:13
2017-12-12T10:40:28     276708  2620:0:861:103:10:64:32:13      application/x-httpd-php/200     32105   GET     http://labtestwikitech.wikimedia.org/w/api.php?action=query&formatversion=2&gapfilterredir=nonredirects&prop=info&generator=allpages&gapnamespace=110&format=json&gaplimit=max  -       application/json        -       -       parsoid-reparse/0.1 python3/3.4.2       -       -       -       2620:0:861:103:10:64:32:13
2017-12-12T10:40:28     254918  2620:0:861:103:10:64:32:13      application/x-httpd-php/200     805     GET     http://labtestwikitech.wikimedia.org/w/api.php?action=query&formatversion=2&gapfilterredir=nonredirects&prop=info&generator=allpages&gapnamespace=111&format=json&gaplimit=max  -       application/json        -       -       parsoid-reparse/0.1 python3/3.4.2       -       -       -       2620:0:861:103:10:64:32:13
2017-12-12T10:40:28     445148  10.64.32.232    application/x-httpd-php/200     122672  GET     http://labtestwikitech.wikimedia.org/w/api.php?format=json&action=query&meta=siteinfo&siprop=namespaces%7Cnamespacealiases%7Cmagicwords%7Cfunctionhooks%7Cextensiontags%7Cgeneral%7Cinterwikimap%7Clanguages%7Clanguagevariants%7Cprotocols%7Cspecialpagealiases%7Cdefaultoptions%7Cvariables&rawcontinue=1     -       application/json        -       -       Parsoid/0.8.0+git       -
       -       -       10.64.32.232
2017-12-12T10:40:28     439944  10.64.32.229    application/x-httpd-php/200     122672  GET     http://labtestwikitech.wikimedia.org/w/api.php?format=json&action=query&meta=siteinfo&siprop=namespaces%7Cnamespacealiases%7Cmagicwords%7Cfunctionhooks%7Cextensiontags%7Cgeneral%7Cinterwikimap%7Clanguages%7Clanguagevariants%7Cprotocols%7Cspecialpagealiases%7Cdefaultoptions%7Cvariables&rawcontinue=1     -       application/json        -       -       Parsoid/0.8.0+git       -
       -       -       10.64.32.229
2017-12-12T10:40:28     431428  10.64.16.90     application/x-httpd-php/200     122672  GET     http://labtestwikitech.wikimedia.org/w/api.php?format=json&action=query&meta=siteinfo&siprop=namespaces%7Cnamespacealiases%7Cmagicwords%7Cfunctionhooks%7Cextensiontags%7Cgeneral%7Cinterwikimap%7Clanguages%7Clanguagevariants%7Cprotocols%7Cspecialpagealiases%7Cdefaultoptions%7Cvariables&rawcontinue=1     -       application/json        -       -       Parsoid/0.8.0+git       -
       -       -       10.64.16.90
2017-12-12T10:40:28     452604  10.64.32.230    application/x-httpd-php/200     122672  GET     http://labtestwikitech.wikimedia.org/w/api.php?format=json&action=query&meta=siteinfo&siprop=namespaces%7Cnamespacealiases%7Cmagicwords%7Cfunctionhooks%7Cextensiontags%7Cgeneral%7Cinterwikimap%7Clanguages%7Clanguagevariants%7Cprotocols%7Cspecialpagealiases%7Cdefaultoptions%7Cvariables&rawcontinue=1     -       application/json        -       -       Parsoid/0.8.0+git       -
       -       -       10.64.32.230

Event Timeline

fgiunchedi created this task.Dec 12 2017, 11:06 AM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptDec 12 2017, 11:06 AM
fgiunchedi renamed this task from labtestwiki memcache errors to labtestwiki memcache errors, lots of requests from parsoid to labtestweb2001.Dec 12 2017, 11:06 AM
• chasemp assigned this task to Andrew.Dec 12 2017, 12:41 PM
• chasemp triaged this task as High priority.
ssastry subscribed.Dec 12 2017, 3:07 PM

I am running a script on terbium that is touching all pages on all wikis in non-Main and non-Talk namespaces ( I processed the main and talk namespaces earlier ) to initialize linter errors for all pages on all wikis so we have a baseline. The script asks parsoid to parse those pages which is how you are setting Parsoid's requests to labtestwiki. The script is now done passed along beyond labtestwiki.

But, I want to know if you are alarmed by Parsoid's requests, if those requests caused an error, of if the request rate from Parsoid to the wiki was too high. There is a possibility I might have to run that script again in the future and good to know if there is anything I should do differently.

Andrew added a comment.Dec 12 2017, 3:14 PM

@ssastry, labtestweb and wikitech (basically anything with 'lab' in its db name) are baby little servers that are unlikely to handle automated traffic very well. Can you just exclude them from future tests?

ssastry added a comment.Dec 12 2017, 3:15 PM
In T182663#3831243, @Andrew wrote:

@ssastry, labtestweb and wikitech (basically anything with 'lab' in its db name) are baby little servers that are unlikely to handle automated traffic very well. Can you just exclude them from future tests?

Ok, will do.

• chasemp closed this task as Resolved.Dec 13 2017, 2:47 PM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL