My bot was using /w/index.php?action=raw to fetch the content of each page/redirect at zhwiki, then it will do some simple search/replace/template addition work.

1、 I has never seen anything go wrong under such a request rate (about 4000-7000 req/s ). When my bot was running, the total CPU usage of app servers just increased by 5-10 percent. Neither database lag nor http error rate increased. As this is the only way to check the entire zhwiki in a few hours, I see no point in throttling the rate.

2、My bot was sending requests to the 3128 port of cp1008.wikimedia.org. Firstly, this is the only way for me to bypass the TLS redirect, which is nothing but nonsense. We use TLS to protect our private data or to prevent man-in-the-middle attack. Since tool labs was inside the data center, again, I see no point in forcing internal http requests to be secure. Secondly, none of my requests (api.php?action=query or index.php?action=raw) can be cached. Sending them to the front cache will only add unnecessary load to the caching layer. Thirdly, as has pointed above, cp1008 is a test machine. So,there's actually no production impact.

3、Using API doesn't help, as page content api querying and proceeding involves xml encoding and decoding, only to cause unnecessary performance decrease.

@Antigng_ you might not have seen anything go wrong, but your bot was accounting for 50% of the uncached requests to our backends or more. It's a clearly unacceptable amount of traffic from a single tool.

See https://ganglia.wikimedia.org/latest/stacked.php?m=ap_rps&c=Application%20servers%20eqiad&r=day&st=1465831227&host_regex=

As for the "5-10% increase in cpu" on the appservers, that means that, given we have 101 servers doing that job, 5-10 servers were fully committed to serving your bot.

It's clearly too much and you should explore alternative means of fetching all that information (I can't really help there, though I guess dumps or labs replicas might do the job).

Labs replicas can't do that job, as revision tables are removed on such databases. Dumps are not updated such often.

If you don't give me a good reason why cp1008.wikimedia.org:3128 / index.php?action=raw shouldn't be used, I will start some of my jobs that don't involve mass page content fetching, such as projectstat.

@Antigng_ just to understand, what is your bot doing? If dumps are not refreshed fast enough for you, maybe you should make your bot follow one of the RecentChanges streams we provide.

As for good reasons not to do what you are currently doing:

1. cp1008 is a test host that is not intended for bot use, rather for human testing of new features. Actually having other traffic there can make debug sessions a pain
2. Connecting to 3128 directly is possible because of a bug and is both wrong and hurtful, as it skips some request mangling we do in the frontend varnish
3. The rate at which you are polling is way too high to be sustainable in general

Most of my tasks don't generate such " unacceptable amount of traffic". They usually send a few hundred to thousand requests before exit. But they still need a way to bypass the TLS redirect.

In T137707#2376258, @Antigng_ wrote:

My bot was using /w/index.php?action=raw to fetch the content of each page/redirect at zhwiki, then it will do some simple search/replace/template addition work.

1、 I has never seen anything go wrong under such a request rate (about 4000-7000 req/s ). When my bot was running, the total CPU usage of app servers just increased by 5-10 percent. Neither database lag nor http error rate increased. As this is the only way to check the entire zhwiki in a few hours, I see no point in throttling the rate.

2、My bot was sending requests to the 3128 port of cp1008.wikimedia.org. Firstly, this is the only way for me to bypass the TLS redirect, which is nothing but nonsense. We use TLS to protect our private data or to prevent man-in-the-middle attack. Since tool labs was inside the data center, again, I see no point in forcing internal http requests to be secure. Secondly, none of my requests (api.php?action=query or index.php?action=raw) can be cached. Sending them to the front cache will only add unnecessary load to the caching layer. Thirdly, as has pointed above, cp1008 is a test machine. So,there's actually no production impact.

3、Using API doesn't help, as page content api querying and proceeding involves xml encoding and decoding, only to cause unnecessary performance decrease.

In T137707#2379894, @Antigng_ wrote:

If you don't give me a good reason why cp1008.wikimedia.org:3128 / index.php?action=raw shouldn't be used, I will start some of my jobs that don't involve mass page content fetching, such as projectstat.

There are so many things wrong here it's hard to know where to begin, but I'll try:

1. 4-7K reqs/sec is not normal traffic. One user's script taking 5-10% of our entire application server pool's CPU capacity is not normal traffic, either. @Joe has already gone into this above, though. As a reference point, we've been debating certain API request limits and one of the exception cases is Google's bots that pull all our data. Even for them, we've been talking about ballpark request ratelimits of ~200/s.
2. On the subject of using cp1008:3128:
   1. cp1008.wikimedia.org shouldn't be used at all. It's a test entrypoint for a very specific kind of manual public-facing testing by operations, and it's never been advertised as a valid place for users to direct traffic at.
   2. Regardless of the above, port 3128 should also never be used at all. It bypasses the front edge of our infrastructure, which does many other things aside from just caching. The correct entry points for HTTP(S) traffic are the standard ports 80 and 443. There are technical measures in place to prevent outside access to port 3128 normally in a couple of different ways:
      1. Most of our edge cp machines are inside our private networks anyways, so nothing in the public or labs worlds can connect directly to them in the first place.
      2. cp1008 is an exception, but even then there's a VCL rule design to reject port 3128 traffic unless it originates from inside WMF production networks (because that port is used for inter-cache traffic internally). This rule accidentally included the labs networks, which it shouldn't have, which is one of the reasons you were able to reach port 3128 there. That's been fixed since. You won't be able to make successful requests to port 3128 of cp1008 from labs anymore.

3. Bypassing TLS redirects is not an option for our public entrypoints, and labs is not an exception. It would've worked on port 80 yesterday if you faked the X-Forwarded-Proto header, again because labs networks were accidentally included in the ACL for production networks, but that's been fixed since. This ticket isn't the place to debate our TLS policy. I'm sorry you think it's unnecessary, but it's not your call.

4. That you don't want the performance hit of decoding XML is also not a good reason to avoid using the API. You should talk to the API developers about supporting your needs better if you think it's deficient.

Also, there doesn't exist a clear request rate limit for mediawiki api, as[[T135240| the rest api]] does. If you want to set one, you should document it.

For the API part, I would like to add that API infrastructure (application servers and databases) is specifically prepared to be separated from non-api traffic and better ready for mass requests than regular browser queries, so that both cannot interfere each other. It produces information in nice JSON format, that you can parse with any json decoder, (or even a regex!), with little to no performance loss.

If you think that the API is non performant (both the action API or the restbase one), please send a bug and we will look at it.

We can discuss more or less usage of the API, but not using the API for API-like requests is definitely not OK. From https://www.mediawiki.org/wiki/API:Etiquette :

There is no hard and fast limit on read requests, but we ask that you be considerate and try not to take a site down. Most sysadmins reserve the right to unceremoniously block you if you do endanger the stability of their site.

I don't think api.php?action=query&prop=revisions&rvprop=content can be the same performant as index.php?action=raw, and the latter is the easiest way to get the source code of a page. I would appreciate it if there was a way to perform api.php?action=raw.

I would appreciate it if there was a way to perform api.php?action=raw

Please file a separate bug report for that.

BTW, the API is definitely faster, one just need to use it efficiently:

$ time curl 'https://en.wikipedia.org/w/api.php?action=query&prop=revisions&rvprop=content&format=json&titles=January|February|March|April|May|June|July|August|September' > /dev/null
real	0m0.717s
user	0m0.004s
sys	0m0.004s

$ time (curl 'https://en.wikipedia.org/w/index.php?action=raw&title=January' && curl 'https://en.wikipedia.org/w/index.php?action=raw&title=February' && curl 'https://en.wikipedia.org/w/index.php?action=raw&title=March'  && curl 'https://en.wikipedia.org/w/index.php?action=raw&title=April'  && curl 'https://en.wikipedia.org/w/index.php?action=raw&title=May'  && curl 'https://en.wikipedia.org/w/index.php?action=raw&title=June' && curl 'https://en.wikipedia.org/w/index.php?action=raw&title=July' && curl 'https://en.wikipedia.org/w/index.php?action=raw&title=September' ) > /dev/null

real	0m3.654s
user	0m0.024s
sys	0m0.008s

In T137707#2380275, @jcrespo wrote:

BTW, the API is definitely faster, one just need to use it efficiently:

$ time curl 'https://en.wikipedia.org/w/api.php?action=query&prop=revisions&rvprop=content&format=json&titles=January|February|March|April|May|June|July|August|September' > /dev/null
real	0m0.717s
user	0m0.004s
sys	0m0.004s

$ time (curl 'https://en.wikipedia.org/w/index.php?action=raw&title=January' && curl 'https://en.wikipedia.org/w/index.php?action=raw&title=February' && curl 'https://en.wikipedia.org/w/index.php?action=raw&title=March'  && curl 'https://en.wikipedia.org/w/index.php?action=raw&title=April'  && curl 'https://en.wikipedia.org/w/index.php?action=raw&title=May'  && curl 'https://en.wikipedia.org/w/index.php?action=raw&title=June' && curl 'https://en.wikipedia.org/w/index.php?action=raw&title=July' && curl 'https://en.wikipedia.org/w/index.php?action=raw&title=September' ) > /dev/null

real	0m3.654s
user	0m0.024s
sys	0m0.008s

OK, I may have a try.

In T137707#2379997, @jcrespo wrote:

For the API part, I would like to add that API infrastructure (application servers and databases) is specifically prepared to be separated from non-api traffic and better ready for mass requests than regular browser queries, so that both cannot interfere each other. It produces information in nice JSON format, that you can parse with any json decoder, (or even a regex!), with little to no performance loss.

If you think that the API is non performant (both the action API or the restbase one), please send a bug and we will look at it.

We can discuss more or less usage of the API, but not using the API for API-like requests is definitely not OK. From https://www.mediawiki.org/wiki/API:Etiquette :

There is no hard and fast limit on read requests, but we ask that you be considerate and try not to take a site down. Most sysadmins reserve the right to unceremoniously block you if you do endanger the stability of their site.

Lack of hard and fast limit on read requests can be a problem, since your definition of request limit can be different from mine. You are obligated to give me a clear cutoff.

In T137707#2382924, @Antigng_ wrote:

Lack of hard and fast limit on read requests can be a problem, since your definition of request limit can be different from mine. You are obligated to give me a clear cutoff.

Just make your requests serially, and not in parallel. From https://www.mediawiki.org/wiki/API:Etiquette#Request_limit:

If you make your requests in series rather than in parallel (i.e. wait for the one request to finish before sending a new request, such that you're never making more than one request at the same time), then you should definitely be fine.