Page MenuHomePhabricator

Remove .m. subdomain, serve mobile and desktop variants through the same URL
Open, NormalPublic

Subscribers
Tokens
"Love" token, awarded by dbarratt."Dislike" token, awarded by TerraCodes."Like" token, awarded by Platonides."Love" token, awarded by Kaartic."Orange Medal" token, awarded by Addshore."Love" token, awarded by Legoktm."Orange Medal" token, awarded by Krinkle."Mountain of Wealth" token, awarded by Krenair."Love" token, awarded by Harej.
Assigned To
None
Authored By
tstarling, Jan 31 2019

Description

The m. subdomain, as in en.m.wikipedia.org, is annoying. Links shared on social media are randomly mobile or non-mobile, and so desktop users often accidentally visit the mobile site. Having separate domains complicates analytics and search engine optimisation.

The m. subdomain does not appear to reflect best current practice. Many websites are able to detect mobile clients and deliver the correct variant through an unmodified URL. For example, WikiTravel does this, using CloudFront and MediaWiki/MobileFrontend.

I propose removing the m. subdomain. The frontend would detect the desired device variant based on UA and request cookies, and send it to MediaWiki in a single header, and would vary the resulting cache based on this request header. MediaWiki (MobileFrontend, or core after T100402 is fixed) would use this request header to select the skin.

@BBlack suggests starting with a soft launch. We would initially remove the device detection redirect from en.wikipedia.org to en.m.wikipedia.org. We would redirect away from the old domains only when the traffic on them has declined. Obviously, to avoid breaking links, the redirect will have to be retained forever.

We could look at how device detection works in third-party CDNs for ideas on what to call the request header or any other unresolved details.

Event Timeline

There are a very large number of changes, so older changes are hidden. Show Older Changes
Restricted Application added a project: Operations. · View Herald TranscriptJan 31 2019, 12:19 AM
Restricted Application added a subscriber: Aklapper. · View Herald Transcript
Koavf added a subscriber: Koavf.Jan 31 2019, 12:31 AM

Agreed. There is zero need for a subdomain: this is why CSS exists and has been supported since at least 1998.

Krinkle added a subscriber: Krinkle.

The m. subdomain, as in en.m.wikipedia.org, is annoying. Links shared on social media are randomly mobile or non-mobile, and so desktop users often accidentally visit the mobile site.

This is partly why stuff like https://addons.mozilla.org/en-US/firefox/addon/skip-mobile-wikipedia/ exists. Love the plan on removing it.

Koavf added a comment.Jan 31 2019, 6:46 AM

Alternate proposal: create

  • [langcode].braille.[project].org
  • [langcode].embossed.[project].org
  • [langcode].handheld.[project].org
  • [langcode].print.[project].org
  • [langcode].projection.[project].org
  • [langcode].screen.[project].org
  • [langcode].speech.[project].org
  • [langcode].tty.[project].org
  • [langcode].tv .[project].org

Or maybe buy up every germane .mobi domain name and just redirect all users there!

This comment was removed by tstarling.
Koavf triaged this task as Unbreak Now! priority.Jan 31 2019, 7:45 AM
Restricted Application added subscribers: Liuxinyu970226, TerraCodes. · View Herald TranscriptJan 31 2019, 7:45 AM
Koavf added a comment.Jan 31 2019, 7:45 AM

Escalated to unbreak now to bring attention to the vandal: https://phabricator.wikimedia.org/p/Funnyjokes2019/

I've never seen vandalism on Phab before. So weird.

Ammarpad lowered the priority of this task from Unbreak Now! to Needs Triage.EditedJan 31 2019, 8:09 AM

Escalated to unbreak now to bring attention to the vandal: https://phabricator.wikimedia.org/p/Funnyjokes2019/

The account has been disabled.

Alternate proposal: create

  • [langcode].braille.[project].org

I assume this sarcastic comment is intended merely to offend whoever made this decision. I don't think this is helpful. Please keep your comments respectful.

Koavf added a comment.Jan 31 2019, 8:24 AM

Alternate proposal: create

  • [langcode].braille.[project].org

I assume this sarcastic comment is intended merely to offend whoever made this decision. I don't think this is helpful. Please keep your comments respectful.

What? Which decision? Including braille as a possibility for media...? Of course not--I have always argued that accessibility is non-negotiable. I don't know who is supposed to be offended at this.

Yeah, very funny. In case it needs to be said, your comment was an attempt to support my proposal by reductio ad absurdum, via a reference to the CSS 2.1 media types, following on from your previous comment that CSS has existed since 1998. But @media is off topic, since the mobile website differs significantly from the desktop website in ways that cannot efficiently be dealt with in CSS. See for example MobileFormatter. I'm not proposing to remove MobileFormatter or to somehow merge the mobile and desktop skins so that they differ only by CSS. I'm an advocate for responsive design, but that's a big project and a very long way off topic from what I'm proposing here.

For the record, MobileFrontend was deployed in 2011, when Squid was the frontend. It's impossible to do what I'm suggesting with an unpatched Squid, so 2014 is the earliest date this change could have happened.

My problem is that you are making fun of MobileFrontend, and by implication, the engineers who designed MobileFrontend, who may well be reading your comments. Please note that Phabricator comments are subject to a Code of Conduct — discussion here is held to a higher standard of civility than on a Wikipedia talk page.

TheDragonFire added a comment.EditedJan 31 2019, 1:47 PM

I don't think there is any significant animosity toward any engineers involved in building what was at the time a pragmatic solution (media queries only being introduced in 2012 and device detection still being erratic), but perhaps there is a cultural difference between the wiki peanut gallery and people trying to do their jobs on Phabricator. So without further ado, I'll invite everyone back over to the afterparty at en:WP:VPT as necessary and we'll let these fine people get on with making *.m.* history!

Koavf added a comment.Jan 31 2019, 3:54 PM

My problem is that you are making fun of MobileFrontend, and by implication, the engineers who designed MobileFrontend, who may well be reading your comments. Please note that Phabricator comments are subject to a Code of Conduct — discussion here is held to a higher standard of civility than on a Wikipedia talk page.

I was not, as I had never heard of that and had no idea it existed. This is also why I wasn't clear as to who would even be offended. To any engineer who applied his skill to making this feature that I cannot even understand, I definitely have respect for that person's hard work and talent; I never had a go at any of you for working to make the sites function better on mobile. For that matter, I have no clue what (if anything) anyone has done to improve the experience for TTY or Braille, so I'm happy that anyone has put forth any effort on those fronts. There was never a joke at their expense.

stjn added a subscriber: stjn.Jan 31 2019, 6:59 PM

If you add some redirection, add a way to circumvent it please (via some specific query?). Right now testing for mobile version involves just changing a domain, with this proposal it would involve changing a user agent string which is much harder to explain to people (and impossible to link to, in fact).

If you add some redirection, add a way to circumvent it please (via some specific query?). Right now testing for mobile version involves just changing a domain, with this proposal it would involve changing a user agent string which is much harder to explain to people (and impossible to link to, in fact).

There is already useformat e.g. https://en.wikipedia.org/wiki/Ipswich?useformat=mobile . And you would still have the "mobile view" link in the footer, although I'm not sure whether it would be better to link to the page with useformat or do a preference switch. In any case, we would stop delivering the site via the m. domain, and remove the relevant code, so you couldn't just override the redirect.

stjn added a comment.Jan 31 2019, 8:29 PM

Wasn’t aware of this link, thanks for addressing my concern. (‘Mobile view’ link sets cookies that make mobile version permanent for a visitor, though.)

Cirdan added a subscriber: Cirdan.Jan 31 2019, 10:55 PM
Pcoombe added a subscriber: Pcoombe.Feb 2 2019, 7:25 AM
Tgr added a subscriber: Tgr.Feb 2 2019, 9:52 PM

The m. subdomain does not appear to reflect best current practice.

Going by Wikipedia's list of top websites:

  1. Google: a big grab bag of standalone apps on various subdomains / subdirectories. As far as I can tell none of them use a mobile domain. Many of them don't seem available on mobile at all, though, Google just pushes you to use the app instead.
  2. Youtube: uses m.youtube.com for mobile
  3. Facebook: uses m.facebook.com for mobile
  4. Baidu: like Google it's many different things, but mostly does not seem to use mobile domains, with the exception of m.help.baidu.com
  5. (Wikipedia)
  6. Tencent QQ: I have a hard time figuring out what the various subdomains are for (forum boards?). Just from a quick Google search most seem not to use a mobile subdomain.
  7. Taobao: some of its shops use a *.m.taobao.com scheme. The others don't seem to be mobile-friendly at all.
  8. Tmall: same situation as Taobao (it's more or less the same company, too)
  9. Yahoo: does not use mobile domains
  10. Amazon: does not use mobile domains
  11. Twitter: uses mobile.twitter.com
  12. Sohu: uses m.sohu.com
  13. Jingdong Mall: mostly uses m.jd.com
  14. Windows Live: also many things. No mobile domains as far as I can tell.
  15. Instagram: does not use mobile domains.

So about half of the top 15 websites use mobile subdomains; not sure what that says about best current practice.

Thanks for raising this. This is clearly not something we ever got satisfyingly right and clearly it leaves many unhappy [0] but I think solving this is not so clear cut and there are quite a few things to think about before doing this.

The m. subdomain, as in en.m.wikipedia.org, is annoying. Links shared on social media are randomly mobile or non-mobile, and so desktop users often accidentally visit the mobile site.

This statement really needs data to back it up. Luckily we should be able to obtain that with a few Hive queries, but we should definitely fact check this. There is similar anecdotal evidence (people tweeting [3]) that people use wikipedia desktop on mobile as a reader mode. I see this as a "feature" in a similar way to skin preferences and I think it's important we ensure that we preserve those users preferred choice of reading . As @stjn points out we also want to make it easier for editors to test pages on mobile. I've seen much more attention from editors on addressing display of content mobile recently and I'd hate for us to lose that momentum by making things harder to test.

Having separate domains complicates analytics...

Not sure about the analytics side - I believe all our analytics consider mobile edits based on mobile domain NOT browser user agent (which is plain wrong). While I'd not argue it complicates things having a separate domain, I think the domain is the least of our worries. How are we going to detect "mobile" in future? Based on user agent alone? How will existing EventLogging schemas distinguish between mobile and desktop? Right now most of our EventLogging schemas are relying on domain rather than user agent and/or skin (to my frustration) to distinguish mobile and desktop and we'll need to update all of those before any change here.

... and search engine optimisation.

I'd also recommend we talk to google and friends about how they crawl our site before implementing this. We've had issues in the past with mobile pages being indexed despite canonical URIs and google in particular value mobile experiences in search rankings so I'd be worried about impacting external search referrers. The web purist in me feels these are 2 different resources with different URIs, content (formatted vs non formatted) and interfaces. How will we clarify the canonical one if they share the same URI to crawlers? The canonical meta tag will no longer work.

The m. subdomain does not appear to reflect best current practice.

@Tgr covered this one in https://phabricator.wikimedia.org/T214998#4922797 so I won't add to that. Sites which do not use mobile domains tend to also serve the same HTML for both mobile and desktop. We don't do that.

General suggestions
This feels like an RFC to me. The technical details look sane but I don't think the work here is trivial and I'm not 100% sure it's the right decision. I expect a lot of vigor in making this happen as this has a lot of potential product impact. There may also be iterative steps we can take (e.g. first making Varnish redirect desktop user agents on mobile domain to the desktop domain). Rather than jump to solution mode, I'd encourage thinking about it from a problem statement point of view. My interpretation of the problem is that it could be solved in 3 ways:

  1. Remove m domain (as per proposal)
  2. User agent sniff on mobile URIs in Varnish and bounce to desktop (T60425)
  3. An intervention screen - Intermediate screen/overlay where users can explicitly say "give me the desktop site" / "I want to read!" and we remember that preference.

The biggest problem I want to flag other than working through the analytics changes that would be neeed here (which is helpful if this is treated as an RFC) is that with any of these 3 solutions we need to fix the opt out button. We have a long history of problems with people who are not getting the desktop experience on mobile for some reason despite switching to desktop and I'm a little worried this problem will escalate if we are sharing the same URI for these 2 entities. Right now we use a cookie strategy (set via JS) with a long history of problems there (T60425, T48473, T32618, T44480... ). With this new setup we'd need to revise that to work both ways allowing viewers to force on mobile mode on their desktop mode (see below) and in particular make that work for legacy browsers where we don't run JS. I'd ask we take some time to get this working correctly this time round.

This is partly why stuff like https://addons.mozilla.org/en-US/firefox/addon/skip-mobile-wikipedia/ exists.

Thanks for raising this! There are also add-ons that do the exact opposite (redirecting desktop to mobile) [1,2] with quite a few users. It should go without saying that this change should be done in such a way that these extensions continue to function (or are no longer needed). From the look of it, most of the desktop to mobile redirects are happening using the "m" domain so these would break with this change. Would be good to give their developers a heads up.

My recommendations:

  • The task needs an RFC that evaluates the three possible options here so we go into this informed and clearly define the problem statement and the specification for how things like "desktop view"/"mobile view" links work and how they remain "sticky".
    • An output of the RFC should be informing browser extension developers and their users of how they will be impacted and how they can continue to function
    • Another output should be a spec for how the "sticky" desktop link behaves.
  • As part of the RFC we should run some queries so we can go into this informed and clear of what problem we are solving rather than what we think is correct
  • We must talk to google and other search vendors and understand potential impacts on how we are crawled to de-risk potential impacts on SEO. We might want to consider/dismiss running A/B tests for SEO to help give us this confidence.

Thanks for starting this conversation!

[0] https://twitter.com/_Hypron/status/1022312984058703879
[1] https://chrome.google.com/webstore/detail/m-wiki/ibnmikddaopgfbbngcgcfmanjfgbcopf
[2] https://addons.mozilla.org/en-US/firefox/search/?platform=linux&q=Wikipedia%20mobile
[3] https://twitter.com/amihailovskis/status/1091338577823318017, https://twitter.com/vkalpgupta/status/1025749762174201856, https://twitter.com/tejasmanohar/status/972621494814519296, https://twitter.com/cwhite_92/status/825024789257482240, https://twitter.com/pczajkowski/status/797030909367218177, https://twitter.com/LiquidLightUK/status/701784161271545857, https://twitter.com/ShaineHatch/status/696517847099396096 (and many more)

Tgr added a comment.Feb 5 2019, 10:49 PM

Re: SEO, Google's Mobile SEO Overview document says "Google does not favor any particular URL format as long as the page(s) and all page assets are accessible to all Googlebot user-agents" about separate mobile site vs. dynamic serving. If we opt for dynamic serving, we should probably test it first on a smaller wiki (and hope for the best - SEO is notoriously hard to test).

It complicates SEO in the sense that, when I wrote this task, I was looking at Google Search Console for a few of our domains with an eye towards SEO for sister projects, and found that mobile traffic was split 50/50 between the dashboards for the m and non-m subdomains. So it was hard to draw any conclusions without manually aggregating the data.

Joe triaged this task as Normal priority.Feb 7 2019, 12:37 PM

(People interested in merging subdomains may also be interested in T215071: Merge Wikipedia subdomains into one, to discourage censorship which is about merging languages)

Tbayer added a subscriber: Tbayer.Feb 12 2019, 8:01 PM
ema moved this task from Triage to Caching on the Traffic board.Wed, Mar 6, 9:42 AM

[..] This feels like an RFC to me. [..]

I've put it in our inbox to discuss this (or next) week, to figure out if it needs an RFC, and if not, we'll suggest an alternate facilitator to help solve the use cases and problems described in this task.

Tbayer added a comment.Thu, Mar 7, 8:40 AM

[..] This feels like an RFC to me. [..]

I've put it in our inbox to discuss this (or next) week, to figure out if it needs an RFC, and if not, we'll suggest an alternate facilitator to help solve the use cases and problems described in this task.

Without going into the question of how to best structure this discussion:

As @Jdlrobson noted above (T214998#4929700), this is a decision with major product impact. And it seems that many of the rationales brought in favor of the change (e.g. the first two sentences of the current task description, or T214998#4919905 ) are about changing the user experience, i.e. product aspects.

Are product decisions now considered to be in the remit of TechCom? In that case we should update the corresponding sentence at https://www.mediawiki.org/wiki/Wikimedia_Technical_Committee ("has standing on questions around ...").

Tbayer added a comment.Thu, Mar 7, 9:03 AM

It complicates SEO in the sense that, when I wrote this task, I was looking at Google Search Console for a few of our domains with an eye towards SEO for sister projects, and found that mobile traffic was split 50/50 between the dashboards for the m and non-m subdomains. So it was hard to draw any conclusions without manually aggregating the data.

We're now ingesting and aggregating Search Console data on our own analytics infrastructure (T172581) which allows more flexible aggregation. So this data analysis inconvenience when using Google's interface may not be a very strong argument for such a far-reaching change.

Are there other ways in which the current setup complicates SEO?
Conversely, as @Tgr already mentioned above, we should make sure the SEO risks of the proposed new setup (serving different content depending on user agent) are well understood before making such a decision.

Just wanted to chime in with a product perspective on this. This change is not currently a priority for us, and introducing it would alter the workflows for current readers and editors. This would require non-trivial amounts of work that would include additional research (user testing as well as thorough analysis on potential impact on SEO), as well as communication with both readers and communities. If we were to consider the change in the future, I think we need a clear problem statement, as well as stronger motivations than the ones currently mentioned in this task (as others have shown .m. does not go against current practice (T214998#4922797) and is not significantly detrimental to overall analytics (T214998#4929700) or the analysis of SEO (T214998#5007592)).

That said, I’d like to highlight that we are in the midst of medium-term planning during which our product direction might shift for the next few years. It seems likely that we will be exploring sharing entire articles or portions of articles to across devices. Perhaps we can revisit this proposal after the planning period and assess whether it aligns better with our future priorities.

dr0ptp4kt added a comment.EditedFri, Mar 8, 11:10 AM

Great framing, nice job! One question, though, what's this part about and how does that tie into the conversation?

"It seems likely that we will be exploring sharing entire articles or portions of articles to across devices."

I figured better to ask than to try to re-parse the conversation and guess.

^ Well, I intended for that to be on email. But it stands: I think Olga put this in terms that I could understand - and as I've said in other places, I think the implementation is non-trivial even if the consequences can be studied sufficiently to be well understood. That said, what is this "exploring sharing entire articles or portions of articles" part about?

I've put it in our inbox to discuss this (or next) week, to figure out if it needs an RFC, and if not, we'll suggest an alternate facilitator to help solve the use cases and problems described in this task.

[..]

Are product decisions now considered to be in the remit of TechCom?

No. The questions of whether a user-facing problem is considered valid, whether it is something that should be solved, and when it would be solved (e.g. roadmap, planning, deployment); those are questions for a product owner.

If solving a problem requires significant changes from technical perspective, then the question that TechCom ultimately has to answer is about how the desired outcome may be achieved. If and when an idea reaches that stage, the RFC process becomes mandatory. Before and unless that stage is reached, neither TechCom nor the RFC process have to be involved.

But people are welcome to (and often do) start the RFC process before that stage. In most cases this simply means that the RFC will reside in our Backlog until participants on the task have figured out what specific problem they want to focus on, and until there is product approval and commitment for solving that problem.

Exposure through the RFC workboard or TechCom newsletter can sometimes help mature or grow an idea, others may come and offer feedback, help identify who else may be interested, who the product owner might be, etc.

Back to this task. This task is not yet an RFC. Jon's comment led to the open question: Should this be an RFC? Or, in other words, would the current version of this proposal likely need an RFC if it were to be accepted by product? I've volunteered to answer Jon's question, by asking the other committee members in the next internal meeting. To make sure I don't forget, I've tagged our team workboard (not the RFC workboard).

stjn removed a subscriber: stjn.Sat, Mar 9, 9:21 PM

Could we sort of default to the last site domain the user has visited ? I mean even if the link points to a mobile site, he should ideally be getting the desktop site if he browses on Wikipedia on desktop most of the time (regardless of whether the device is a phone or laptop).

^ Well, I intended for that to be on email. But it stands: I think Olga put this in terms that I could understand - and as I've said in other places, I think the implementation is non-trivial even if the consequences can be studied sufficiently to be well understood. That said, what is this "exploring sharing entire articles or portions of articles" part about?

@dr0ptp4kt - I was referring to the following statement in the description: "Links shared on social media are randomly mobile or non-mobile, and so desktop users often accidentally visit the mobile site", which I believe that this is a valid issue. However, I think it's also a part of the the overall experience and functionality of sharing to social media, link sharing, etc which is a topic that has been brought up numerous times in conversations around medium term planning and annual planning, although no decisions have been made so far. I think it would be wise to wait until planning is complete, decide whether this area is something that we want to invest time in, and if so, reassess whether this change aligns with the overall product direction we settle on. Hope that makes more sense!

daniel edited projects, added TechCom-RFC; removed TechCom.Thu, Mar 14, 6:24 AM

Accepted as an RFC

daniel moved this task from Inbox to Under discussion on the TechCom-RFC board.Thu, Mar 14, 6:25 AM

Do we know how much of a burden it will be to vary the cache based on the user-agent? That seems like it would be a whole lot more cache objects and cache MISS, but this may not be a problem.

CDanis added a subscriber: CDanis.Mon, Mar 18, 1:14 PM
Krinkle added a comment.EditedWed, Mar 20, 1:40 AM

Do we know how much of a burden it will be to vary the cache based on the user-agent?

Yes, impact would be none as we already do this.

Desktop and mobile domains both have their page views cached in Varnish (with different output each), and the canonical desktop url varies by user agent to power the redirect we see today (cache/response variance). Note that MediaWiki never sees the m-dot domain. The caching and internal traffic always operates on the canonical domain, using the internal header to distinguish mobile mode.

Our Varnish logic today (simplified)

  • Internally rewrite the Host name to strip m. from domain name.
  • If original Host "en.m.wikipedia.org"
    • Set IsMobile to true for MobileFrontend.
    • Fetch from Apache backend (cached under "en.wikipedia.org" + IsMobile + Path).
  • Request for Host "en.wikipedia.org"
    • If the user-agent matches a mobile user agent, then redirect to the same path+query on "(sub).m.(project.suffix)". This is an internal redirect from the Varnish VCL code at the edge.
    • Else, fetch from Apache backend (cached under "en.wikipedia.org" + IsMobile + Path).

What our Varnish logic could be (simplified):

  • Internally rewrite the Host name to strip any m. from domain name.
  • Request for Host "en.wikipedia.org"
    • If the user-agent matches a mobile user agent, then set IsMobile to true for MobileFrontend.
    • Fetch from Apache backend (cached under "en.wikipedia.org" + IsMobile + Path).

Impact

  • Simpler VCL code.
  • Same amount of cache objects.
  • Fewer Varnish responses (fewer redirects, right now a canonical url opened on mobile is always 2 requests and 2 responses instead of 1).
  • Better performance for end-users on mobile, by no longer requiring 2 full network roundtrips for them when they open a "wrong" url.
Peter added a subscriber: Peter.Wed, Mar 20, 3:46 PM