Page MenuHomePhabricator

RFC: Remove .m. subdomain, serve mobile and desktop variants through the same URL
Open, MediumPublic

Assigned To
None
Authored By
tstarling
Jan 31 2019, 12:19 AM
Tokens
"Love" token, awarded by MusikAnimal."Love" token, awarded by Ltrlg."Like" token, awarded by Majavah."Like" token, awarded by Mobiledesktop."Barnstar" token, awarded by Gloin_Oin."Like" token, awarded by Njk."Like" token, awarded by Darylgolden."Like" token, awarded by Benjaminikuta."Like" token, awarded by Darxus."Like" token, awarded by Jdforrester-WMF."Love" token, awarded by dbarratt."Dislike" token, awarded by TerraCodes."Like" token, awarded by Platonides."Love" token, awarded by Kaartic."Orange Medal" token, awarded by Addshore."Love" token, awarded by Legoktm."Orange Medal" token, awarded by Krinkle."Mountain of Wealth" token, awarded by Krenair."Love" token, awarded by Harej.

Description

  • Affected components: Wikimedia site configuration.
  • Engineer for implementing the change: Core Platform Team (WMF) and SRE ServiceOps (WMF).
  • Code steward: (idem).

Motivation

The m. subdomain, as in en.m.wikipedia.org, is annoying. Links shared on social media are randomly mobile or non-mobile, and so desktop users often accidentally visit the mobile site. Having separate domains complicates analytics and search engine optimisation.

The m. subdomain does not appear to reflect best current practice. Many websites are able to detect mobile clients and deliver the correct variant through an unmodified URL. For example, WikiTravel does this, using CloudFront and MediaWiki/MobileFrontend.

Requirements
  1. When browsing Wikipedia from mobile web or desktop, and sharing the browser address with someone, the address should produce the default Wikipedia experience for the receiver's end.

Exploration

The first requirement means we either have to redirect in both directions, or vary the server response without a redirect. The latter seems simple and also more performant for the end-user by not having a significant portion of page views delayed by an additional blocking/synchronous client-server roundtrip over the network.

I propose removing the m. subdomain. The frontend would detect the desired device variant based on UA and request cookies, and send it to MediaWiki in a single header, and would vary the resulting cache based on this request header. MediaWiki (MobileFrontend, or core after T100402 is fixed) would use this request header to select the skin.

@BBlack suggests starting with a soft launch. We would initially remove the device detection redirect from en.wikipedia.org to en.m.wikipedia.org. We would redirect away from the old domains only when the traffic on them has declined. Obviously, to avoid breaking links, the redirect will have to be retained forever.

We could look at how device detection works in third-party CDNs for ideas on what to call the request header or any other unresolved details.


Known issues

Related Objects

Mentioned In
T270603: Module site.styles generates different output depending on mobile cookie, if $wgMFSiteStylesRenderBlocking = true;
T264678: UrlShortener should use desktop link has primary
T225814: m.wikidata: login status not detected
T171398: On mobile domain, interwiki links for WMF wikis should be resolved as mobile rather than desktop
T206497: Enable $wgMFNoindexPages for: Italian, Dutch, Korean, Arabic, Chinese, and Hindi Wikipedias
T52399: {{fullurl:}} and {{SERVER}} when used on mobile do not resolve to commons.m.wikimedia.org but to commons.wikimedia.org
T211559: Notification links redirects to desktop site
T195494: Handle mobile domains in core
T198969: Ensure links on the mobile version of pages are not to the desktop version
T60425: Mobile site does not automatically redirect to desktop version (and not possible to use browser "use desktop view")
Mentioned Here
T225814: m.wikidata: login status not detected
T74186: Varnish: Mobile site redirect interferes with OAuth authorization process
T171398: On mobile domain, interwiki links for WMF wikis should be resolved as mobile rather than desktop
T172581: [EPIC] Set up mechanism for archiving Google Search Console data
T215071: Merge Wikipedia subdomains into one, to discourage censorship
T32618: Desktop version fine on mobile UNTIL clicking "disable mobile view"
T44480: Disabling desktop view is broken
T48473: desktop view on mobile domain
T60425: Mobile site does not automatically redirect to desktop version (and not possible to use browser "use desktop view")
T100402: Core should have support for mobile detection and customization

Event Timeline

There are a very large number of changes, so older changes are hidden. Show Older Changes

Re: SEO, Google's Mobile SEO Overview document says "Google does not favor any particular URL format as long as the page(s) and all page assets are accessible to all Googlebot user-agents" about separate mobile site vs. dynamic serving. If we opt for dynamic serving, we should probably test it first on a smaller wiki (and hope for the best - SEO is notoriously hard to test).

It complicates SEO in the sense that, when I wrote this task, I was looking at Google Search Console for a few of our domains with an eye towards SEO for sister projects, and found that mobile traffic was split 50/50 between the dashboards for the m and non-m subdomains. So it was hard to draw any conclusions without manually aggregating the data.

Joe triaged this task as Medium priority.Feb 7 2019, 12:37 PM

(People interested in merging subdomains may also be interested in T215071: Merge Wikipedia subdomains into one, to discourage censorship which is about merging languages)

[..] This feels like an RFC to me. [..]

I've put it in our inbox to discuss this (or next) week, to figure out if it needs an RFC, and if not, we'll suggest an alternate facilitator to help solve the use cases and problems described in this task.

[..] This feels like an RFC to me. [..]

I've put it in our inbox to discuss this (or next) week, to figure out if it needs an RFC, and if not, we'll suggest an alternate facilitator to help solve the use cases and problems described in this task.

Without going into the question of how to best structure this discussion:

As @Jdlrobson noted above (T214998#4929700), this is a decision with major product impact. And it seems that many of the rationales brought in favor of the change (e.g. the first two sentences of the current task description, or T214998#4919905 ) are about changing the user experience, i.e. product aspects.

Are product decisions now considered to be in the remit of TechCom? In that case we should update the corresponding sentence at https://www.mediawiki.org/wiki/Wikimedia_Technical_Committee ("has standing on questions around ...").

It complicates SEO in the sense that, when I wrote this task, I was looking at Google Search Console for a few of our domains with an eye towards SEO for sister projects, and found that mobile traffic was split 50/50 between the dashboards for the m and non-m subdomains. So it was hard to draw any conclusions without manually aggregating the data.

We're now ingesting and aggregating Search Console data on our own analytics infrastructure (T172581) which allows more flexible aggregation. So this data analysis inconvenience when using Google's interface may not be a very strong argument for such a far-reaching change.

Are there other ways in which the current setup complicates SEO?
Conversely, as @Tgr already mentioned above, we should make sure the SEO risks of the proposed new setup (serving different content depending on user agent) are well understood before making such a decision.

Just wanted to chime in with a product perspective on this. This change is not currently a priority for us, and introducing it would alter the workflows for current readers and editors. This would require non-trivial amounts of work that would include additional research (user testing as well as thorough analysis on potential impact on SEO), as well as communication with both readers and communities. If we were to consider the change in the future, I think we need a clear problem statement, as well as stronger motivations than the ones currently mentioned in this task (as others have shown .m. does not go against current practice (T214998#4922797) and is not significantly detrimental to overall analytics (T214998#4929700) or the analysis of SEO (T214998#5007592)).

That said, I’d like to highlight that we are in the midst of medium-term planning during which our product direction might shift for the next few years. It seems likely that we will be exploring sharing entire articles or portions of articles to across devices. Perhaps we can revisit this proposal after the planning period and assess whether it aligns better with our future priorities.

Great framing, nice job! One question, though, what's this part about and how does that tie into the conversation?

"It seems likely that we will be exploring sharing entire articles or portions of articles to across devices."

I figured better to ask than to try to re-parse the conversation and guess.

^ Well, I intended for that to be on email. But it stands: I think Olga put this in terms that I could understand - and as I've said in other places, I think the implementation is non-trivial even if the consequences can be studied sufficiently to be well understood. That said, what is this "exploring sharing entire articles or portions of articles" part about?

I've put it in our inbox to discuss this (or next) week, to figure out if it needs an RFC, and if not, we'll suggest an alternate facilitator to help solve the use cases and problems described in this task.

[..]

Are product decisions now considered to be in the remit of TechCom?

No. The questions of whether a user-facing problem is considered valid, whether it is something that should be solved, and when it would be solved (e.g. roadmap, planning, deployment); those are questions for a product owner.

If solving a problem requires significant changes from technical perspective, then the question that TechCom ultimately has to answer is about how the desired outcome may be achieved. If and when an idea reaches that stage, the RFC process becomes mandatory. Before and unless that stage is reached, neither TechCom nor the RFC process have to be involved.

But people are welcome to (and often do) start the RFC process before that stage. In most cases this simply means that the RFC will reside in our Backlog until participants on the task have figured out what specific problem they want to focus on, and until there is product approval and commitment for solving that problem.

Exposure through the RFC workboard or TechCom newsletter can sometimes help mature or grow an idea, others may come and offer feedback, help identify who else may be interested, who the product owner might be, etc.

Back to this task. This task is not yet an RFC. Jon's comment led to the open question: Should this be an RFC? Or, in other words, would the current version of this proposal likely need an RFC if it were to be accepted by product? I've volunteered to answer Jon's question, by asking the other committee members in the next internal meeting. To make sure I don't forget, I've tagged our team workboard (not the RFC workboard).

Could we sort of default to the last site domain the user has visited ? I mean even if the link points to a mobile site, he should ideally be getting the desktop site if he browses on Wikipedia on desktop most of the time (regardless of whether the device is a phone or laptop).

^ Well, I intended for that to be on email. But it stands: I think Olga put this in terms that I could understand - and as I've said in other places, I think the implementation is non-trivial even if the consequences can be studied sufficiently to be well understood. That said, what is this "exploring sharing entire articles or portions of articles" part about?

@dr0ptp4kt - I was referring to the following statement in the description: "Links shared on social media are randomly mobile or non-mobile, and so desktop users often accidentally visit the mobile site", which I believe that this is a valid issue. However, I think it's also a part of the the overall experience and functionality of sharing to social media, link sharing, etc which is a topic that has been brought up numerous times in conversations around medium term planning and annual planning, although no decisions have been made so far. I think it would be wise to wait until planning is complete, decide whether this area is something that we want to invest time in, and if so, reassess whether this change aligns with the overall product direction we settle on. Hope that makes more sense!

Do we know how much of a burden it will be to vary the cache based on the user-agent? That seems like it would be a whole lot more cache objects and cache MISS, but this may not be a problem.

Do we know how much of a burden it will be to vary the cache based on the user-agent?

Yes, impact would be none as we already do this.

Desktop and mobile domains both have their page views cached in Varnish (with different output each), and the canonical desktop url varies by user agent to power the redirect we see today (cache/response variance). Note that MediaWiki never sees the m-dot domain. The caching and internal traffic always operates on the canonical domain, using the internal header to distinguish mobile mode.

Our Varnish logic today (simplified)

  • Internally rewrite the Host name to strip m. from domain name.
  • If original Host "en.m.wikipedia.org"
    • Set IsMobile to true for MobileFrontend.
    • Fetch from Apache backend (cached under "en.wikipedia.org" + IsMobile + Path).
  • Request for Host "en.wikipedia.org"
    • If the user-agent matches a mobile user agent, then redirect to the same path+query on "(sub).m.(project.suffix)". This is an internal redirect from the Varnish VCL code at the edge.
    • Else, fetch from Apache backend (cached under "en.wikipedia.org" + IsMobile + Path).

What our Varnish logic could be (simplified):

  • Internally rewrite the Host name to strip any m. from domain name.
  • Request for Host "en.wikipedia.org"
    • If the user-agent matches a mobile user agent, then set IsMobile to true for MobileFrontend.
    • Fetch from Apache backend (cached under "en.wikipedia.org" + IsMobile + Path).

Impact

  • Simpler VCL code.
  • Same amount of cache objects.
  • Fewer Varnish responses (fewer redirects, right now a canonical url opened on mobile is always 2 requests and 2 responses instead of 1).
  • Better performance for end-users on mobile, by no longer requiring 2 full network roundtrips for them when they open a "wrong" url.

@tstarling, this problem:

The m. subdomain, as in en.m.wikipedia.org, is annoying. Links shared on social media are randomly mobile or non-mobile, and so desktop users often accidentally visit the mobile site.

can be easily resolved by detecting desktop/mobile client by User-Agent and 302 redirecting by default desktop clients from mobile version of site to desktop version of site. Now exists redirection only from desktop version to mobile version of site for mobile clients (detected by User-Agent).

I have standalone wiki based on MediaWiki engine and already create such logic using nginx web server and njs script. If you or someone else interested - I can share my njs code and nginx wiki config at github.

main idea of my code looks like this:

// ...
var arg_mobileaction = r.args.mobileaction;
var redirect_host;
if (arg_mobileaction === 'toggle_view_desktop') {
    redirect_host = desktop_host;
} else if( arg_mobileaction === 'toggle_view_mobile' ) {
    redirect_host = mobile_host;
} else {
    redirect_host = host;
}

var mobile_browser = is_mobile_browser(r.variables.http_user_agent);
var desktop_browser = !mobile_browser;
var cookie_stopMobileRedirect = r.variables.cookie_stopMobileRedirect;
var cookie_stopDesktopRedirect = r.variables.cookie_stopDesktopRedirect;

if (mobile_browser && arg_mobileaction !== 'toggle_view_desktop' && cookie_stopMobileRedirect !== 'true') {
    redirect_host = mobile_host;
}
if (desktop_browser && arg_mobileaction !== 'toggle_view_mobile' && cookie_stopDesktopRedirect !== 'true') {
    redirect_host = desktop_host;
}

// set $stop_redirect_cookie

if (mobile_browser && arg_mobileaction === 'toggle_view_desktop') {
    r.variables.stop_redirect_cookie = ['stopMobileRedirect=true', 'domain=' + wildcard_host(r), 'path=/', 'Max-Age=86400', 'HttpOnly', ''].join(';')
} else if (desktop_browser && arg_mobileaction === 'toggle_view_mobile') {
    r.variables.stop_redirect_cookie = ['stopDesktopRedirect=true', 'domain=' + wildcard_host(r), 'path=/', 'Max-Age=86400', 'HttpOnly', ''].join(';')
} else {
    r.variables.stop_redirect_cookie = '';
}
// ...

this allow by default redirect users to proper version of site, bit also this allow users to manually switch between desktop and mobile versions in any time.

I suppose what such simple logic can be also implemented using varnish.

Having separate domains complicates analytics and search engine optimisation.

Search engines already know about mobile and desktop versions of wikipedia and understand difference.

See https://developers.google.com/search/mobile-sites/mobile-seo/separate-urls for details.

The m. subdomain does not appear to reflect best current practice. Many websites are able to detect mobile clients and deliver the correct variant through an unmodified URL. For example, WikiTravel does this, using CloudFront and MediaWiki/MobileFrontend.

You are talking about https://wikitravel.org/en/Main_Page site?

Looks like this site is broken. I set mobile user agent, but this site still display me desktop version.

Also I don't see any ability to change from desktop to mobile site or from mobile site to desktop site manually.

I propose removing the m. subdomain. The frontend would detect the desired device variant based on UA and request cookies, and send it to MediaWiki in a single header, and would vary the resulting cache based on this request header. MediaWiki (MobileFrontend, or core after T100402 is fixed) would use this request header to select the skin.

If I use desktop browser - how I can manually switch to mobile version of site, is it possible after implementing your proposal?

Also, If I use mobile browser - how I can manually switch to desktop version of site, is it possible after implementing your proposal?

If answer is "No" - removing m. subdomain is bad idea, from my point of view.

@Koavf, ok I see, it work. But WikiTravel don't allow me to browse mobile version of site if I have desktop browser and WikiTravel don't allow me to browse desktop version of site if I use mobile browser.

It permanently show mobile version for mobile users and desktop version for desktop users, and don't remember user choice. This is bug or feature?

For example, even if I manually switch to mobile version of page - https://wikitravel.org/wiki/en/index.php?title=Western_Sahara&mobileaction=toggle_view_mobile - all links on this page, for example https://wikitravel.org/en/North_Africa show me desktop version.

If I use desktop browser - how I can manually switch to mobile version of site, is it possible after implementing your proposal?

Also, If I use mobile browser - how I can manually switch to desktop version of site, is it possible after implementing your proposal?

Yes, you will still be able to do this, and it will work the same way as today. Nothing is changing about this.

Links shared on social media are randomly mobile or non-mobile, and so desktop users often accidentally visit the mobile site.

can be easily resolved by detecting desktop/mobile client by User-Agent and 302 redirecting by default desktop clients from mobile version of site to desktop version of site. [..]

Serving 50% of our social-traffic (or more) with redirects is not an acceptable resolution from a performance perspective.

Current cache proxy logic
handle (cookies and mode-switch logic) {
  
}

if (domain is normal) {
  if (isMobileUserAgent) {
    exit (redirect to mobile domain);
    # user gets redirected...
    # browser makes new URL request over network ...
    # browser URL request arrives in our data centre, and we start another instance of this cache proxy logic ...
  }
}

if (domain is mobile) {
  exit (route to MediaWiki with MobileFrontend=on);
} else {
  exit (route to MediaWiki with MobileFrontend=off);
}
Proposed cache proxy logic
handle (cookies and mode-switch logic) {
  
}

if (isMobileUserAgent) {
  exit (route to MediaWiki with MobileFrontend=on);
} else {
  exit (route to MediaWiki with MobileFrontend=off);
}

See also T214998#5038999. Basically:

  • Better performance for our mobile users, by no longer forcing redirects for many of their page views.
  • Better UX for our desktop users, by no longer serving the mobile site when they do not expect this (random switches based on sharer's url).
  • Nothing else changes for users.

If I use desktop browser - how I can manually switch to mobile version of site, is it possible after implementing your proposal? Also, If I use mobile browser - how I can manually switch to desktop version of site, is it possible after implementing your proposal?

Yes, you will still be able to do this, and it will work the same way as today. Nothing is changing about this.

This proposal already implemented on WikiTravel site, but browsing mobile version of site is not possible using browser with desktop User-Agent, and browsing desktop version of site is not possible using browser with mobile User-Agent. Manually user can switch mobile/desktop view only for each page, but globally, for entire site - can't:

For example, even if I manually switch to mobile version of page - https://wikitravel.org/wiki/en/index.php?title=Western_Sahara&mobileaction=toggle_view_mobile - all links on this page, for example https://wikitravel.org/en/North_Africa show me desktop version.

If this is bug in WikiTravel configuration - probably this bug should be fixed before removing m. subdomain from all Wikipedia sites?

See also T214998#5038999.

As I understand, T214998#5038999 proposal forces mobile users to browse only mobile version of site and forces desktop users to browse only desktop version of site without ability to select which version of site user want to see and browse.

If mobile and desktop versions of site has the same uri - remembering user choice is only possible via cookies, something like this:

userPreferredVersion=mobile or userPreferredVersion=desktop and logic should be something like this:

Proposed cache proxy logic - version 2.0
# user choice has higher priority
if (cookie userPreferredVersion exists) {
  if (cookie userPreferredVersion === 'mobile') {
    exit (route to MediaWiki with MobileFrontend=on);
  }
  if (cookie userPreferredVersion === 'desktop') {
    exit (route to MediaWiki with MobileFrontend=off);
  }
}
# default fallback, if no userPreferredVersion cookie exists
# or if it value is not 'desktop' or 'mobile'
if (isMobileUserAgent) {
  exit (route to MediaWiki with MobileFrontend=on);
} else {
  exit (route to MediaWiki with MobileFrontend=off);
}

[..] it will work the same way as today. Nothing is changing about this.

This proposal already implemented on WikiTravel site, but [..]. If this is bug in WikiTravel configuration - probably this bug should be fixed before removing m. subdomain from all Wikipedia sites?

WikiTravel have done something similar, but their architecture is quite different. It is not comparable. We do not have this bug currently on Wikipedia, and we will not introduce this bug as part of this change. You can rest assured it will keep working. The code for it already exists, it works fine today, and will continue to work. It will not change with this proposal.

Regarding your code suggestion, note that this Phabricator task is not a code review. We are not yet at the implementation stage. I simplified the code in my comment so that it can explain the impact to a wider audience. The actual Wikipedia configuration is in a different syntax and more complex. If and when the plan is accepted, the team implementing it will apply the changes as necessary. Then, you can review the code in the Gerrit system.

Krinkle updated the task description. (Show Details)
Krinkle removed a subscriber: wikibugs-l-list.

What does this mean for me as for a user who edits Wikimedia projects from mobile using desktop view for almost 9 years? Does it mean I would not be able to continue doing so and would have to stick to limited in functionality mobile view instead? (Well, in Firefox for Android there is also a "require desktop view" flag, but I am not sure if I can use this preference per web-site and thus it is an undesirable way).

It could be a good idea as long as there is a preference to opt out of mobile view for good for registered users and a long living cookie remembering the view used for anonymous users. Otherwise it sounds like a terrible idea.

What does this mean for me as for a user who edits Wikimedia projects from mobile using desktop view for almost 9 years?

The proposal here is to provide the current mobile experience by default to visitors with detected mobile devices, and the current desktop experience to visitors with detected desktop devices, with the ability to switch experiences back and forth. Logged in users would continue to get the skin of their preference. However, it's too early to make commitments about exactly how this would work and what the experience would be, if we did this.

@Jdlrobson If I understand correctly, you're saying that it is considered a requirement that a user can still find an easy way into the mobile/minerva experience from a desktop browser and stay there when browsing around. In other words, what they can do today by manually going to en.m.wikipedia.org, which they presumably have stumbled upon by accident or figured out by reproducing the url they see on mobile.

Preserving this seems fair indeed. However, to what extent is this not already satisfied by the skin preference and the "Mobile view" cookie? The latter is a session-bound preference for logged-out users. And the former is a sticky preference that works across devices for logged-in users.

This change would affect MobileFrontend and therefore make you an important stakeholder. However in terms of resourcing, this seems like it would mainly involve CDN and site configuration changes which CPT, SRE and Perf could handle I think. Are there changes required within MobileFrontend to make this change possible? If this is indeed not a prioroity for Readers Web currently, it'd be good to have a rough sense of what areas might need changing, so that if other participants are able/willing to help out that they can volunteer :)

Krinkle renamed this task from Remove .m. subdomain, serve mobile and desktop variants through the same URL to RFC: Remove .m. subdomain, serve mobile and desktop variants through the same URL.Apr 4 2020, 2:34 AM

Have been somewhat expecting this to be fixed for the better part of 7+ years now. Hopefully this can be implemented soon!