@alexhollender, I would like to collect your thoughts on the 3 options. Personally, option 2 seems the most straightforward but would need input from you on what that threshold would look like to give a good UX while minimising wasted related articles requests for those that never get to the bottom of the page.

@Jdlrobson I agree that Option 2 seems like the best UX. In terms of optimizing the threshold so that we minimize requests, I'm not quite sure how to go about figuring that out. Trial+error: trying, looking at data, and revising. Or some kind of engineering solution that predicts the chances of someone reaching the bottom?

Option 3 also seems fine. Is there any way to send a cheaper request just to find out how many related articles we're going to display, without fetching the actual article preview content?

@Jdlrobson Could you summarise the original reasoning for why related article metadata is lazy-loaded, and evaluate whether that still applies?

I did some preliminary comparisons and it seems there is an opportunity to gain a net benefit to the user experience and perf metrics by removing this indirection. I'll let you look it further first, though. Would like to understand this better before I look further.

In T208180#4806271, @alexhollender wrote:

Is there any way to send a cheaper request just to find out how many related articles we're going to display, without fetching the actual article preview content?

The "related articles" API request made by MobileFrontend (aka gsrsearch=morelike&prop=pageimages+description) has a response smaller than 1KB (typically 0.3K-0.5KB it seems).

Reducing the response to just numerical information would not result in significant savings. In particular because the standard cost involved with making any server-request (request headers, response headers, JavaScript handling for that) would be larger than the bytes saved. On top of that, even if the JS code and request overhead was free, spitting the response in two would always produce a larger total number of bytes transferred due to response headers and JSON wrapping. (Keeping in mind that images are not part of the payload, only the url to where the image is located.)

Could you summarise the original reasoning for why related article metadata is lazy-loaded, and evaluate whether that still applies?

The original thinking was a concern around wasted HTTP requests. We use the MediaWiki API to source related articles. If we didn't lazy load the content, we would hit the API for every request to an article page (and mediawiki api doesn't cache). On projects like Wikivoyage where related articles show on desktop as well as mobile this becomes more significant. This would be wasteful given the majority of people do not scroll to the bottom of the article and thus do not see it and the additional (but maybe not significant) bytes shipped in HTML. It also allows users to inject related articles on wikis where they are not enabled via user script (https://www.mediawiki.org/wiki/Topic:Uq5kapm403lfj2i3).

This also allowed us during development to run an A/B test for anonymous users as well as the ability to disable related articles during roll out.

At this point, we could of course investigate doing this all server side but this would require combining data from CirrrusSearch, PageImages and Wikidata descriptions which would amount to a considerable but not impossible rewrite.

I assume the existing backend query can be performed as-is in MediaWiki PHP.

In terms of backend performance, that would presumably improve not regress, because its result would be cached by Varnish. Thus leaving only cache-fill and logged-in user requests. If we go a step further and compute it in ParserOutput rather than OutputPage, we'd also cache it for logged-in users. Thus effectively reducing computation to only occur in post-send time and job-queue time; after edits and after automatic purging from bank-link references. (E.g. when changing the summary of a related article).

In terms of user-perceived performance, for users not triggering the lazy-load I would guesstimate it as on-par with today. Because in return they'd save bandwidth and CPU time for the startup module, and bandwidth and CPU time for the lazy-loading code. I don't know how large that is, but the API data is only 0.3KB after gzip. If the JS code is even smaller than that, users not triggering the lazy-load would trade only a handful of bytes and CPU milliseconds from the time-to-interactive path, in favour of bytes mostly outside the critical rendering path (the end of the HTML stream). That seems like a worthy trade-off in principle, but in practice too small to reason about.

For users that do trigger the lazy-load, its saves additional CPU time on the main thread and bandwidth for the extra JS code. That bandwidth currently exceeds the size of the generated HTML. It'd also save additional network activity. This can be significant because after the page is finished loading, mobile radios eventually power down, and this might be one of only a few reasons that currently power it back up in a way that isn't deferrable to a point where it can be combined with other network activity (given that XHR has high fetch priority).