Page MenuHomePhabricator
Create Task
Maniphest T341634

MediaWiki frequently receives HTTP 500 from AQS (via PageViewInfo extension)
Open, Needs TriagePublic
Actions

Description

Impact

The "Page information" panel on Wikipedia will (wrongly) claim that the article has had 0 pageviews during the past 30 days:

https://en.wikipedia.org/w/index.php?title=Scream_(2022_film)&action=info

Screenshot 2023-07-11 at 17.15.44.png (1,662×1,282 px, 233 KB)

Similarly, the MediaWiki API for accessing this data returns all nulls (with a warning):

https://en.wikipedia.org/w/api.php?action=query&titles=Scream%20(2022%20film)&prop=pageviews&format=json

{
  "warnings": {
    "pageviews": {
      "*": "An earlier attempt to fetch page \"Scream_(2022_film)\" failed. To limit server load, retries have been blocked for 30 minutes."
    }
  },
  "query": {
    "pages": {
      "31716869": {
        "title": "Scream (2022 film)",
        "pageviews": {
          …
          "2023-07-01": null,
          "2023-07-02": null,
          "2023-07-03": null,
          "2023-07-04": null,
          "2023-07-05": null,
          "2023-07-06": null,
          "2023-07-07": null,
          "2023-07-08": null,
          "2023-07-09": null,
          "2023-07-10": null
        }
      }
    }
  }
}

The choice to cache errors for 30 minutes is also discussed at T181012, but that's a MW extension bug, wheras this task is about the analytics service returning an error so often in the first place.

Error

Around 70,000 errors in the past 7 days. There's a continous trickle with several failures every minute.

I don't have stats on succesfull requests from this specific feature, but according to Turnilo (https://w.wiki/6$yL) there's about ~12,000 requests to the "Page information" action on Wikipedia per minute. That makes half a dozen failures per minute, amplied by caching, fairly visible. In spot-checking random articles, I got several after checking 10-15 articles.

Screenshot 2023-07-11 at 17.24.42.png (1,948×768 px, 110 KB)

https://logstash.wikimedia.org/goto/4c6640971ac029b5cff2f9787e540cb5

Jul 11, 2023 @ 16:05:26.964Failed fetching http://localhost:6011/wikimedia.org/v1/metrics/pageviews/per-article/en.wikipedia.org/all-access/user/Camelot_%28film%29/daily/20230512/20230710: There was a problem during the HTTP request: 503 Service Unavailable
Jul 11, 2023 @ 16:05:45.685Failed fetching http://localhost:6011/wikimedia.org/v1/metrics/pageviews/per-article/en.wikipedia.org/all-access/user/Hell_Baby/daily/20230512/20230710: There was a problem during the HTTP request: 503 Service Unavailable
Jul 11, 2023 @ 16:05:56.553Failed fetching http://localhost:6011/wikimedia.org/v1/metrics/pageviews/per-article/en.wikipedia.org/all-access/user/La_scuola/daily/20230512/20230710: There was a problem during the HTTP request: 503 Service Unavailable
Jul 11, 2023 @ 16:05:59.389Failed fetching http://localhost:6011/wikimedia.org/v1/metrics/pageviews/per-article/en.wikipedia.org/all-access/user/Bling_Bling_%28video%29/daily/20230512/20230710: There was a problem during the HTTP request: 503 Service Unavailable
Jul 11, 2023 @ 16:06:14.344Failed fetching http://localhost:6011/wikimedia.org/v1/metrics/pageviews/per-article/www.wikidata.org/all-access/user/Q1174509/daily/20230512/20230710: There was a problem during the HTTP request: 503 Service Unavailable
Jul 11, 2023 @ 16:06:40.686Failed fetching http://localhost:6011/wikimedia.org/v1/metrics/pageviews/per-article/en.wikipedia.org/all-access/user/The_Little_Vampire_%28film%29/daily/20230512/20230710: There was a problem during the HTTP request: 503 Service Unavailable

I helped analyze a similar issue with the Shellbox service, you can fine the way I did that at T292663#8867819.

Related Objects

Event Timeline

Krinkle created this task.Jul 11 2023, 10:01 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptJul 11 2023, 10:01 PM
BTullis subscribed.Jul 12 2023, 11:42 AM

This is interesting and I'd like to help track down the source of the errors if I can. However, other folks on the Data-Engineering team will know more about the nuances of the API endpoints themselves than I do.

These queries go through RESTBase and then they go through the LVS service at: aqs.svc.eqiad.wmnet:7232
See: https://gerrit.wikimedia.org/g/operations/puppet/+/production/hieradata/common/profile/restbase.yaml for there that is configured.

That endpoint is backed by the 12 physical servers aqs10[10-21] - These servers each run:

  • Two instances of cassandra
  • One nodejs based service called aqs, which I understand was originally based on RESTBase.

This nodejs based daemon is the 'legacy' AQS 1.0 service and is what receives traffic on port 7232.

You can see the config file template for the aqs service here: https://gerrit.wikimedia.org/r/plugins/gitiles/operations/puppet/+/production/modules/aqs/templates/config.yaml.erb
It's configured to send metrics to statsd and that's what I believe is powering the graphs here: https://grafana-rw.wikimedia.org/d/000000526/aqs?from=now-24h&orgId=1&to=now&viewPanel=27
There are a few 500s there, but I see no 503s. I'm not sure if the number of 500s would equate to the number of 503s that you're seeing from the appservers, but I suspect it's probably a bit low.

Checking the rsyslog configuration it's not configured to forward the systemd unit's logs to kafka, so logs from that service just end up in /srv/log/aqs/syslog.log on each host.
I had a quick scan of one of those logs and it doesn't seem to be logging individual 503s or 500s but I didn't do an exhaustive check.

The endpoint itself (https://wikimedia.org/api/rest_v1/#/Pageviews%20data/get_metrics_pageviews_per_article__project___access___agent___article___granularity___start___end_) is served by the local Cassandra cluster on the same hosts.
Some of the other AQS 1.0 end-points, the ones related to mediawiki_history, are served by a Druid cluster on different hosts, but all of the paths shown are to the same endpoint.

I hope that gives you some clues to help investigate. Let me know if I can help any further.
Maybe it is another timeout issue from envoy, so a retry might help.

Do you happen to know if the error occurs in the same manner when requests are served by mw-on-k8s?

Dominicbm subscribed.Aug 23 2023, 7:11 PM
• VirginiaPoundstone moved this task from Incoming to To be discussed/To be estimated on the AQS2.0 board.Aug 25 2023, 5:17 PM
Krinkle mentioned this in T342201: MediaWiki\Extension\Notifications\Api\ApiEchoUnreadNotificationPages::getUnreadNotificationPagesFromForeign: Unexpected API response from {wiki}.Sep 7 2023, 8:59 PM
Krinkle mentioned this in T346971: Uncaught ConfigException: Failed to load configuration from etcd.Sep 20 2023, 9:01 PM
• VirginiaPoundstone edited projects, added: Test Kitchen; removed: API Platform (AQS 2.0 Roadmap).EditedJan 24 2024, 9:15 PM
• VirginiaPoundstone subscribed.

@BTullis is this issue still present? This task was filed prior to the switchover to AQS 2, and Dan and Sam worked on resolving the issue. If they were successful, please mark the ticket resolved.

• VirginiaPoundstone moved this task from Incoming to NEEDS ESTIMATION on the Test Kitchen board.Jan 24 2024, 10:01 PM
Krinkle added a comment.EditedFeb 7 2024, 5:33 PM

logstash.wikimedia.org -> mediawiki -> channel:PageViewInfo AND message:"Failed fetching"

In the last day, it seems AQS failed over 700 times for these kinds of queries (not counting expected "404" errors), and about 3,500 in the past week.

Failed fetching http://localhost:6033/wikimedia.org/v1/metrics/pageviews/per-article/fr.wikipedia.org/all-access/user/Dichiarazioni_d%27amore/daily/20231209/20240206: There was a problem during the HTTP request: 503 Service Unavailable

• lbowmaker moved this task from Incoming (new tickets) to Tag with Radar on the Data-Engineering board.Feb 12 2024, 2:25 PM
• VirginiaPoundstone moved this task from NEEDS ESTIMATION to Wikistats Backlog on the Test Kitchen board.Mar 18 2024, 7:51 PM
• VirginiaPoundstone moved this task from Wikistats Backlog to AQS Backlog on the Test Kitchen board.Mar 29 2024, 5:13 PM
Krinkle mentioned this in T371633: Burst of GuzzleHttp Exception for http://localhost:6025/call/constraint-regex-checker.Aug 20 2024, 6:21 PM
• VirginiaPoundstone removed a project: Test Kitchen.Jan 6 2025, 8:54 PM
• VirginiaPoundstone added a project: Data-Engineering-Radar.Jan 6 2025, 9:12 PM
Krinkle mentioned this in T350717: Math image replaced with "Math extension cannot connect to Restbase." due to "upstream connect error or disconnect/reset before headers".Feb 24 2026, 3:35 AM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits