Page MenuHomePhabricator

Investigation: why did parse-time geoshape expansion cause an increase in service requests?
Open, Needs TriagePublic

Description

Our theory was that each page save mostly corresponds to one parse and one pageview, so moving the geoshape expansion to parse time could only result in fewer service requests. There would be one request made while parsing, and then the pageview after save would be served from ParserCache. What we found in practice is that the geoshape requests increased by roughly 3x:

image.png (833×1 px, 185 KB)

Experimentally resolve various potential explanations.

Possible explanations:

  • The backend is making many parse requests for each page save, even though these could have been deduplicated and served from the cache.
    • Diagnostic: Check the geoshape webrequest log for repeated titles.
    • Diagnostic: Add page_id, revision_id, and request_id to the logged geoshape expansion event.
  • Maplinks would not have been rendered on Wikipedias with static mapframe thumbnails, so these would be new requests.
    • Diagnostic: We surveyed maplinks as being much more rare, with 1k maplinks vs. 650k mapframes. This can't explain such a large jump in requests.
    • Diagnostic: Add maplink vs mapframe to our logged event. This will help us monitor whether maplinks ever become more prominent in the future.
  • The slight decay over the enablement period suggests that some cache is filling.
    • However, no theory behind this idea yet.
    • Diagnostic: Leave enabled until the graph levels out.
  • Could the broken static thumbnails have caused repeated attempts to render from kartotherian, and somehow repeated expansions of the geoshapes?
    • Theory is incomplete.

Event Timeline

awight updated the task description. (Show Details)

It looks like the additional requests did cause problems for our service. These are ETIMEDOUT errors and correspond to a long window after the new feature was deployed,
https://logstash.wikimedia.org/goto/6d04422e5f1cf09a4e870fc83a63fabc

image.png (330×947 px, 26 KB)