So you want to try this thing out on the beta cluster, and then if it's successful, enable it on the beta cluster?

In T207508#4681903, @Krenair wrote:

So you want to try this thing out on the beta cluster, and then if it's successful, enable it on the beta cluster?

More, if it seems to be useful we can enable it more broadly across beta cluster, making it a feature that beta cluster provides developers (a run around the "real" content request).

I, of course, feel a bit weird with this being in MFE and understand it's just the most expedient way that it was created.

What are the potential downsides/limitations of this model?

Idea would be to expose people to testing with it on https://es.wikipedia.beta.wmflabs.org/ and if successful enable on https://en.wikipedia.beta.wmflabs.org/ and other beta cluster instances.

My bet is that many skin regressions will be captured before hitting production by doing this.

While the content provider lives in MobileFrontend right now, there's no reason not to break it out into its own extension if there's a need to. Although built initially to test MobileFormatting, we actually used it heavily for testing Popups (page previews).

What are the potential downsides/limitations of this model?

I am not aware of any limitations, other than the fact that the pages pulled in are not real pages so will show as missing in watchlist/special pages/locally created pages. The page will also render as a 404 rather than a 200. We purposely still send a 404 on these pages to avoid them getting cached by search engines.

In T207508#4682061, @Jdlrobson wrote:

What are the potential downsides/limitations of this model?

I am not aware of any limitations, other than the fact that the pages pulled in are not real pages so will show as missing in watchlist/special pages/locally created pages. The page will also render as a 404 rather than a 200. We purposely still send a 404 on these pages to avoid them getting cached by search engines.

Interesting. @zeljkofilipin do you have any thoughts on whether this would be an issue with any of the popular browser test patterns?

Browser tests will presumably be using local content, so they'd still render as 200s and behave identically as before.
https://en.wikipedia.beta.wmflabs.org/wiki/Barcelona right now 404s for example but with the change would look like http://reading-web-staging.wmflabs.org/wiki/Barcelona

If a local version of Barcelona is created, nothing will change.

Note: MediaWiki:Common.css styles are not copied across, but I believe we mirror these anyway.

While this might make it easier to detect some kinds of regressions, I think you also lose out on a certain level of integration testing. MobileFrontend does mess with the parser (at least it did last time I looked), and I think it's important that we continue to test that.

I also do think beta cluster should be entirely independent of production, mostly for independence of bug reproducibility.

If the problem is that content isn't available or has diverged the last time someone did an import, maybe we need to do another mass import or automate regular imports?

If the problem is that content isn't available or has diverged the last time someone did an import, maybe we need to do another mass import or automate regular imports?

For most articles these days import is completely broken due to a heavy use of wikidata and complex template transclusion. Our content changes all the time so imports are always inferior when it comes to backs. The parser on the beta cluster also cannot handle mature articles e.g. https://en.m.wikipedia.beta.wmflabs.org/wiki/Barack_Obama

For these reasons for any kind of collaborative frontend development the beta cluster is an inferior environment for testing. While it may be great for backend changes frontend bugs are heavily dependent and coupled to content. The number of regressions on Minerva has fallen dramatically ever since we started testing with production quality articles (can provide data if necessary). I strongly believe page previews would still be in development without it.

As well as that having such an environment has been great for newcomers as we can get them replicating frontend bugs in no time with a little bit of config.

As you can see I strongly believe this is something we should be making more heavy use of and I'm frustrated every time I see a UI regression in Vector which could have been avoided by having this kind of testing available.

While this might make it easier to detect some kinds of regressions, I think you also lose out on a certain level of integration testing.

The parser issues you point to are no longer problems. As I've stated locally created content will always be honored. MobileFrontend does not mess with the parser for pages that exist. It's been specifically built so we can run integration tests as well. Mobilefrontend/Minerva are one of our most heavy consumers of browser tests!

I realise there may be unknown unknowns but this is why i am suggesting testing it on one of the lesser used beta clusters e.g. https://es.wikipedia.beta.wmflabs.org/ or https://en.wikivoyage.beta.wmflabs.org/wiki/Salzburg and gathering more feedback before using it more widely.

Change 479256 had a related patch set uploaded (by Jdlrobson; owner: Jdlrobson):
[operations/mediawiki-config@master] Beta cluster shows production content on mobile only for non-existent pages

https://gerrit.wikimedia.org/r/479256

The above patch limits this to the mobile domain (labs). This should only impact reading-web products and open up discussions for expanding to desktop. If there's no complaints I'd like to swat this later today.

Change 479256 merged by jenkins-bot:
[operations/mediawiki-config@master] Beta cluster shows production content on mobile only for non-existent pages

https://gerrit.wikimedia.org/r/479256

For what it's worth, this is typical "interwiki transclusion" territory.

(Adding this task to Developer Productivity to keep track of it in that context; we're still working through our current goals for devprod (localdev bootstraped) so this isn't "now" but "later/someday" territory for now... until we know the next (fiscal) year looks like more clearly)

@ovasileva just a note as we push into desktop refresh I suspect testing real world content is going to become super important. We might want to check in with RelEng about a strategy (or dedicated test wiki) here given the importance of the project.