@JoeWalsh
I use a REST client tool to keep sending the request multiple times to the https://zh.wikipedia.org/api/rest_v1/page/mobile-html/2019冠状病毒病疫情 with Accept-Language: zh-hant and Cache-Control: no-cache, the issue can be reproduced.

You can notice the article title changed one time:
嚴重特殊傳染性肺炎疫情 -> 2019冠状病毒病疫情

It looks like the page loses its Accept-Language and shows the default content, which is a mixed-variants content of the article.

@cooltey good catch, I can also reproduce that behavior locally with curl. I'll update the description and tag this task into Core Platform Team Clinic Duty for investigation.

@Pchelolo thanks for investigating and thanks for the update @cooltey can you verify that the android app isn't percent-encoding the titles and if you percent-encode the title the issue is resolved? I'm unable to repro locally with the title percent-encoded using:

curl -sv -H "Accept-Language: zh-hant" https://zh.wikipedia.org/api/rest_v1/page/mobile-html/2019%E5%86%A0%E7%8A%B6%E7%97%85%E6%AF%92%E7%97%85%E7%96%AB%E6%83%85 | egrep -ohi '(?:<h1 data-id="0" class="pcs-edit-section-title">)[^<]+'

Thanks @Pchelolo and @JoeWalsh.

In the Android app, it does percent-encoding the titles and the following screenshot is the Logcat from the Android Studio.

and here is the code:
https://github.com/wikimedia/apps-android-wikipedia/blob/c6939be39bbdea26ca9dba629429369d9e5c2fd1/app/src/main/java/org/wikipedia/bridge/CommunicationBridge.java#L76-L78

@Pchelolo It turns out I am able to repro locally with the title percent encoded when passing "Cache-Control: no-cache". I've updated the task description with the repro steps and example headers for correct and incorrect results. Here is the curl command that shows the problem locally:

curl -sv -H "Accept-Language: zh-hant" -H "Cache-Control: no-cache" https://zh.wikipedia.org/api/rest_v1/page/mobile-html/2019%E5%86%A0%E7%8A%B6%E7%97%85%E6%AF%92%E7%97%85%E7%96%AB%E6%83%85 | egrep -ohi '(?:<h1 data-id="0" class="pcs-edit-section-title">)[^<]+'

Why are you passing Cache-Control: no-cache exactly? These should are not forwarded to RESTBase but stripped on Varnish/ATS level. Theoretically, it shouldn't do anything, but maybe it messes up how CDN cache works.

I'll try to see why are no-cache headers make any difference while they should be ignored, but my immediate suggestion would be to stop adding this header.

It looks like we are indeed asking for no-cache when the user explicitly refreshes the article in the app. Not sure how it got there, but it probably dates back to when we were seeing change-propagation issues (edits not being visible immediately) and were trying (mistakenly?) to see if adding no-cache would help. We can probably remove it safely, but note that this has been part of the app for a long time, with other endpoints such as summary and page-lead.

...or rather, when the user explicitly refreshes the article, we want to force our network stack to read from the network instead of from local cache, and setting no-cache is the recommended guidance to do that.

Tagging Traffic - seems like Varnish/ATS layer no longer unsets 'cache-control: no-cache' header on requests to RESTBase. It used to be the case before. RESTBase stack heavily depends on the caching layer not passing through the cache-control header. It's possible for us to unset those in RESTBase depending on the client IP, but that doesn't seem like a preferred solution. Why has Varnish/ATS stopped unsetting the cache-control header for incoming requests?

In T249284#6091252, @Pchelolo wrote:

Tagging Traffic - seems like Varnish/ATS layer no longer unsets 'cache-control: no-cache' header on requests to RESTBase. It used to be the case before. RESTBase stack heavily depends on the caching layer not passing through the cache-control header. It's possible for us to unset those in RESTBase depending on the client IP, but that doesn't seem like a preferred solution. Why has Varnish/ATS stopped unsetting the cache-control header for incoming requests?

There's no mention of Cache-Control as a request header in our VCL, and the last somehow related change I see was made in 2016 (and it's about parsoid). Unless I'm missing something I'd be thus inclined to say that we haven't been unsetting Cache-Control from requests at the Varnish/ATS layer for a while, possibly ever. We can do that though if needed.

You've just challenged my core belief - I was 100% sure Varnish layer was not passing through the 'cache-control' request header for us, and a lot of restbase updates design was based on that "fact". Do we have some whitelist of request headers that are allowed through? Was there maybe some other software that used to chop off the header?

In any case, there are 2 options to mitigate that:

1. We can unset client cache-control in varnish layer
2. We can unset cache-control for external IPs in RESTBase.

I am inclined to say, that maybe unsetting it in RESTBase actually makes more sense - after all RESTBase depends on it and needs to do it's own cleanup. Depending on the specifics of the frontend caching behavior is too fragile, as we are witnessing here. I will do that.

I am inclined to say, that maybe unsetting it in RESTBase actually makes more sense - after all RESTBase depends on it

It sounds reasonable to me that this assumption be enforced by the entity making it, in this case RESTBase.

In T249284#6034265, @Pchelolo wrote:

RESTBase stack only supports requests with titles encoded with percent-encoding. Supplying unencoded titles in most cases don't break things, but in this particular case it makes nodejs go nuts - RESTBase receives some crazy gibberish instead of a title, and everything goes sideways from there.

This is a bit surprising to me that Android app is not encoding the titles - if that is the case, none of the pages with / in them will work in the app. Please check if that's the case.

Nothing we can really do here from RESTBase standpoint except updating the documentation which I am doing here: https://github.com/wikimedia/restbase/pull/1254

@Pchelolo thanks for updating this documentation - are there specifics about which characters are allowed and which characters need to be encoded?

@Pchelolo thanks for updating this documentation - are there specifics about which characters are allowed and which characters need to be encoded?

In JS clients using encodeURIComponent on the title component should work. In other languages - use any built-in percent-encoding methods. Ideally, need to be compliant with https://tools.ietf.org/html/rfc3986

Is this related to T256491?