@TheDJ is this issue blocking for you? We will investigate next week.

@VirginiaPoundstone it's not blocking me (i just did the investigation), it's blocking Theklan

@Theklan we will try to investigate this end of next week. Please share any relevant details about what this blocks and urgency. It will help us prioritize appropriately.

I don't know the details on how much this blocks the deploy of the "Explore" section in the parent task. We are planning to deliver the new Grid Main Page in a couple of weeks, when we reach the 33rd Wikipedia with most articles and the 17th in the List of articles every Wikipedia should have. I guess we can deploy it without the "Explore" section, but this was a highlight of the new design.

@VirginiaPoundstone @TheDJ interesting.. I am getting a 200 status code and proper JSON response ( its large) body

import requests

prod_url = 'https://wikimedia.org/api/rest_v1/metrics/pageviews/top/eu.wikipedia.org/all-access/2024/05/03'

header = {"accept": "application/json",
          "user-agent": 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/113.0.0.0 Safari/537.36'}

response = requests.get(prod_url, headers=header)

print(response.status_code)
print(response.json())

Response

status code: 200

thanks @EChukwukere-WMF, @Theklan are you still able to reproduce the issue on your side?

In T364253#9844970, @EChukwukere-WMF wrote:

@VirginiaPoundstone @TheDJ interesting.. I am getting a 200 status code and proper JSON response ( its large) body

I note that I was using http2, not sure if python uses http2 by default. Also it's been almost 30 days, the chance that the url survived for 30 days in the cache (of all routes that potentially ingress) is of course pretty low. We already knew that from the hackathon we had two different ingress routes in play

I very much suspect that this is/was a corner case in the proxy edge or https layers, either with a specific host, or a very specific situation (host restart?). I'm mostly wondering what generated that 301.

It went from /api/rest_v1/metrics/pageviews/top/eu.wikipedia.org/all-access/2024/05/03
to /wikimedia.org/v1/metrics/pageviews/top/eu.wikipedia.org/all-access/2024/05/03

so what hosts would modify that first part of the url ? What has the knowledge wikimedia.org/v1 ? That seems like a rewrite to an internal path, that accidentally got exposed to external ?

@Sfaci @mforns can you help answer the question from @TheDJ ?

I have never seen 301 when requesting anything to any AQS services. I have no idea about why that happened. In my case, all request responses include envoy as the server value even when a there was a hit on the cache.
If the data were not available yet we should see a 404 (that could be cached as well). It seems you were able to skip envoy somehow and reached the wrong destination. Is mw-web.eqiad.main-55b8c76fd7-k745s a mediawiki pod?

According to https://wikitech.wikimedia.org/wiki/Analytics/AQS/Pageviews#Updates_and_backfilling data is loaded at the end of the specific timespan (next hour or day for example). But in any case case data could be still unavailable because it seems loading time could tale 24 hours. Anyway, In that case, you should receive a 404 response.

There are different cache hosts depending on were we live, right? Maybe it's something we cannot reproduce from where we are (in my case at least). Anyway the caching period for AQS should be 14400 (4 hours).

I'm sorry. For now I have no idea about this issue or why it could have happened.

@hnowlan Could you take a look at this? Any ideas?

In T364253#9849840, @Sfaci wrote:

I have never seen 301 when requesting anything to any AQS services. I have no idea about why that happened. In my case, all request responses include envoy as the server value even when a there was a hit on the cache.

That makes sense. Envoy is the building block behind both the service mesh/proxy, the api/rest gateway and the k8s ingress gateway. You should be seeing it almost anywhere in our infra these days.

If the data were not available yet we should see a 404 (that could be cached as well). It seems you were able to skip envoy somehow and reached the wrong destination. Is mw-web.eqiad.main-55b8c76fd7-k745s a mediawiki pod?

It is. But this hasn't anything to do with envoy but probably some routing logic, either in the CDN layer or the api-gateway layer.

According to https://wikitech.wikimedia.org/wiki/Analytics/AQS/Pageviews#Updates_and_backfilling data is loaded at the end of the specific timespan (next hour or day for example). But in any case case data could be still unavailable because it seems loading time could tale 24 hours. Anyway, In that case, you should receive a 404 response.

There are different cache hosts depending on were we live, right? Maybe it's something we cannot reproduce from where we are (in my case at least). Anyway the caching period for AQS should be 14400 (4 hours).

The following should allow you to iterate across all CDN and control for "where I live" factor.

for i in eqsin eqiad codfw esams ulsfo drmrs ; do curl -v --resolve wikimedia.org:443:$(host text-lb.${i}.wikimedia.org | grep -Eo '([0-9]{1,3}\.){3}[0-9]{1,3}') -X GET 'https://wikimedia.org/api/rest_v1/metrics/pageviews/top/eu.wikipedia.org/all-access/2024/05/03' ; done

when the Brazil PoP is added, add magru after drmrs to include that one too.

I just ran the above and there are no differences between all the PoPs for that URL.

I'm sorry. For now I have no idea about this issue or why it could have happened.

If we can have a reproduction it would be actionable. Otherwise, I 'd suggest to resolve this and reopen if something shows up again.

For what is worth, the CDN's max TTL is 24 hours.

yes this can be marked as resolved. Thanks all